GUI Says Starting Backup Failed

Server
1

Hello,
I’m running two Urbackup servers (2.4.13) one each in jails on two TrueNAS servers (12.0-U8.1) named Plum and Shed.
Recently the Shed Urbackup instance went totally pink and will not do backups any more to any client.
The Plum server is working just fine.

Here is the Shed Urbackup GUI:

StartingBackupFails
StartingBackupFails1352×1047 122 KB

Notice that all clients except Puff show online.
Puff is ON but it disappeared into “Online = No” status on both servers at the same time.
I assume that this is an issue with the Puff client which is not germain to this discussion.
I will deal with Puff’s issues in another post after I get the rest of the clients on the Plum server to work.

Here is the end of the Muff client’s debug log:
2022-06-22 21:42:34: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#SIGNATURE#pubkey=MIIBtjCCASsGByqGSM44BAEwggEeAoGBAKssAlbiS1Domzi30g7UWpoTOd88VE4BAFYHaTY1gJAkETnvLEX5B3oJL3RMwar1qXVf/cFQWJKaN5MlSeTEWY1umjV6HwF0O1HELXllk0t1Q3B8Gmq4hn4TMWKSyyxUfcAYWrbm/JhKXRu5xf6qlMlvXd9ehP2/buoy9QyIB8SNAhUAzj57EGw46Pd25zOXG0XHWcxOQS0CgYAUNmrZwYkco6d1/7O837BA64A+eOsZYpYTQfW9V/V6sfBvTsJlqkj+/2pRWxQ77/SPOFS0Xofqgnsz9NuCtoz1TlOGsvMMMD68BKWZJS6DlFnePk9cPlVAoHzqCUiGuRmgLE3kxO0V8yxfqnUy9d9tw/fyRYS0ygg6v684T17YwAOBhAACgYAY9ojgjb+n+TBTqERFUrDnDKSwemZhalNaK8FWKZ38+/0BdMPZRBEODQIvtzUKNtl/9GxH1U0CDemNC9CkvM0dW1GoSfe1IxL7sGTYRHIQo3j/r7bjafGJFo2SUFyOtK2z8mNI+dTaqDH1ZIUGvZWEHS63AXbBpNg9w8chXMQKYg–&pubkey_ecdsa409k1=MH4wEAYHKoZIzj0CAQYFK4EEACQDagAEAVaS+IWC08s9Q3+QrBkPpOnp6Dt/JKTMdka5KJjnE1P28qwtqybqZLum4UGciSKvFl3h4gD7qxrjWAp5kFsibkWU0JOLKd436R+Vd/jz/thanSbzWuSwli0kvJaRn10XK58YT66i/FU-&signature=qP7FHGF0y2LqudfwMJbwhr9ifrCQxd4qL54dW6dBEW/dt6VwQAf3Ug–&signature_ecdsa409k1=G9CnYkUNRibjXGccukuWgVwMo7kghGQy2ds1gGRgBjKy5d5s/W6iJXhTsAOsJ5det0whPpRdTjNK7qyw0h8NSpSV2aPmURVRATHoZR4YSTK6FSjFyRNXcVP+xa8VWNtqcHWBlhgI&session_identity=roBDyMNhjifzrL5jilWB
2022-06-22 21:42:35: rc=0 hasError=true state=0
2022-06-22 21:42:39: ClientService cmd: PONG
2022-06-22 21:43:19: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:43:39: ClientService cmd: PONG
2022-06-22 21:44:11: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:44:39: ClientService cmd: PONG
2022-06-22 21:45:03: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:45:39: ClientService cmd: PONG
2022-06-22 21:45:55: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:46:39: ClientService cmd: PONG
2022-06-22 21:46:47: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:47:39: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:47:39: ClientService cmd: PONG
2022-06-22 21:48:31: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:48:39: ClientService cmd: PONG
2022-06-22 21:49:23: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:49:39: ClientService cmd: PONG
2022-06-22 21:50:15: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:50:39: ClientService cmd: PONG
2022-06-22 21:51:07: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:51:39: ClientService cmd: PONG
2022-06-22 21:51:59: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:52:39: ClientService cmd: PONG
2022-06-22 21:52:51: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:53:39: ClientService cmd: PONG
2022-06-22 21:53:43: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:54:34: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:54:39: ClientService cmd: PONG
2022-06-22 21:55:26: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:55:39: ClientService cmd: PONG
2022-06-22 21:56:18: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:56:39: ClientService cmd: PONG
2022-06-22 21:56:52: Looking for old Sessions… 0 sessions
2022-06-22 21:57:09: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:57:39: ClientService cmd: PONG
2022-06-22 21:58:01: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:58:31: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#ADD IDENTITY
2022-06-22 21:58:31: rc=0 hasError=true state=0
2022-06-22 21:58:31: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#GET CHALLENGE
2022-06-22 21:58:31: rc=0 hasError=true state=0
2022-06-22 21:58:31: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#SIGNATURE#pubkey=MIIBtjCCASsGByqGSM44BAEwggEeAoGBAKssAlbiS1Domzi30g7UWpoTOd88VE4BAFYHaTY1gJAkETnvLEX5B3oJL3RMwar1qXVf/cFQWJKaN5MlSeTEWY1umjV6HwF0O1HELXllk0t1Q3B8Gmq4hn4TMWKSyyxUfcAYWrbm/JhKXRu5xf6qlMlvXd9ehP2/buoy9QyIB8SNAhUAzj57EGw46Pd25zOXG0XHWcxOQS0CgYAUNmrZwYkco6d1/7O837BA64A+eOsZYpYTQfW9V/V6sfBvTsJlqkj+/2pRWxQ77/SPOFS0Xofqgnsz9NuCtoz1TlOGsvMMMD68BKWZJS6DlFnePk9cPlVAoHzqCUiGuRmgLE3kxO0V8yxfqnUy9d9tw/fyRYS0ygg6v684T17YwAOBhAACgYAY9ojgjb+n+TBTqERFUrDnDKSwemZhalNaK8FWKZ38+/0BdMPZRBEODQIvtzUKNtl/9GxH1U0CDemNC9CkvM0dW1GoSfe1IxL7sGTYRHIQo3j/r7bjafGJFo2SUFyOtK2z8mNI+dTaqDH1ZIUGvZWEHS63AXbBpNg9w8chXMQKYg–&pubkey_ecdsa409k1=MH4wEAYHKoZIzj0CAQYFK4EEACQDagAEAVaS+IWC08s9Q3+QrBkPpOnp6Dt/JKTMdka5KJjnE1P28qwtqybqZLum4UGciSKvFl3h4gD7qxrjWAp5kFsibkWU0JOLKd436R+Vd/jz/thanSbzWuSwli0kvJaRn10XK58YT66i/FU-&signature=GGcAvbyf6gGIO5Aync5bSyXNtiweK41Zk/GK+aFqhLadrh3+sayYFQ–&signature_ecdsa409k1=QnaZQLrwPb8YkA5nMgpD2luAYpah7oh2qpDYurDjab09Dqo4b9e+wl5w3Jc1XDPlMeafdc+sP4VtMs9jgcGjsst8Rkiw5OQWdYK0GU7crLbggbEIx6uXXSH/aRazOBaonSNd8kra&session_identity=BQEhsutPKVuCz1inWnIU
2022-06-22 21:58:31: rc=0 hasError=true state=0
2022-06-22 21:58:39: ClientService cmd: PONG
2022-06-22 21:58:53: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 21:59:39: ClientService cmd: PONG
2022-06-22 21:59:45: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:00:37: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:00:39: ClientService cmd: PONG
2022-06-22 22:01:29: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:01:39: ClientService cmd: PONG
2022-06-22 22:02:21: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:02:39: ClientService cmd: PONG
2022-06-22 22:03:12: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:03:39: ClientService cmd: PONG
2022-06-22 22:04:04: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:04:39: ClientService cmd: PONG
2022-06-22 22:04:55: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:05:39: ClientService cmd: PONG
2022-06-22 22:05:47: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:06:38: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:06:39: ClientService cmd: PONG
2022-06-22 22:07:29: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:07:39: ClientService cmd: PONG
2022-06-22 22:08:21: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:08:39: ClientService cmd: PONG
2022-06-22 22:09:13: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:09:39: ClientService cmd: PONG
2022-06-22 22:10:05: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:10:39: ClientService cmd: PONG
2022-06-22 22:10:57: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:11:39: ClientService cmd: PONG
2022-06-22 22:11:49: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:12:39: ClientService cmd: PONG
2022-06-22 22:12:40: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:13:32: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:13:39: ClientService cmd: PONG
2022-06-22 22:14:24: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:14:30: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#ADD IDENTITY
2022-06-22 22:14:30: rc=0 hasError=true state=0
2022-06-22 22:14:30: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#GET CHALLENGE
2022-06-22 22:14:30: rc=0 hasError=true state=0
2022-06-22 22:14:30: ClientService cmd: #I7ICPqSsdo9c1l0vlEtUu#SIGNATURE#pubkey=MIIBtjCCASsGByqGSM44BAEwggEeAoGBAKssAlbiS1Domzi30g7UWpoTOd88VE4BAFYHaTY1gJAkETnvLEX5B3oJL3RMwar1qXVf/cFQWJKaN5MlSeTEWY1umjV6HwF0O1HELXllk0t1Q3B8Gmq4hn4TMWKSyyxUfcAYWrbm/JhKXRu5xf6qlMlvXd9ehP2/buoy9QyIB8SNAhUAzj57EGw46Pd25zOXG0XHWcxOQS0CgYAUNmrZwYkco6d1/7O837BA64A+eOsZYpYTQfW9V/V6sfBvTsJlqkj+/2pRWxQ77/SPOFS0Xofqgnsz9NuCtoz1TlOGsvMMMD68BKWZJS6DlFnePk9cPlVAoHzqCUiGuRmgLE3kxO0V8yxfqnUy9d9tw/fyRYS0ygg6v684T17YwAOBhAACgYAY9ojgjb+n+TBTqERFUrDnDKSwemZhalNaK8FWKZ38+/0BdMPZRBEODQIvtzUKNtl/9GxH1U0CDemNC9CkvM0dW1GoSfe1IxL7sGTYRHIQo3j/r7bjafGJFo2SUFyOtK2z8mNI+dTaqDH1ZIUGvZWEHS63AXbBpNg9w8chXMQKYg–&pubkey_ecdsa409k1=MH4wEAYHKoZIzj0CAQYFK4EEACQDagAEAVaS+IWC08s9Q3+QrBkPpOnp6Dt/JKTMdka5KJjnE1P28qwtqybqZLum4UGciSKvFl3h4gD7qxrjWAp5kFsibkWU0JOLKd436R+Vd/jz/thanSbzWuSwli0kvJaRn10XK58YT66i/FU-&signature=V5481QWkFTmigrSmel9FiY3/EqQcygBgNZAESc4SszHeJPnYzigYWw–&signature_ecdsa409k1=eNJXVTdnMF3FerP73ruOdCvBXibbFAOjyeKlEfvYLojW9CSWW18euAqjfOCYHf1hdp4xUV9XzmYXJwPq8AnWmaE6AFpS6ODAqAni7rqCePghzz/VVe7X0Jz2P5psMIPpHuLOTuxX&session_identity=K8KLGFTJwWpcU3r0SHck
2022-06-22 22:14:30: rc=0 hasError=true state=0
2022-06-22 22:14:39: ClientService cmd: PONG
2022-06-22 22:15:16: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR
2022-06-22 22:15:39: ClientService cmd: PONG
2022-06-22 22:16:08: ClientService cmd: STATUS#pw=dkPbYlw6mw8aBoOQwtVLZBi0udLFWR

At this point I have to assume that all clients cannot see the Shed Urbackup server for some as yet undetermined reason.
Each server runs in a Truenas jail with it’s own IP address.
Plum is 192.168.2.7 and Shed is 192.168.2.17.
All devices are on the same 192.168.2 subnet.
All devices do DHCP from the router at 192.168.2.1 and all devices have DHCP reservations on the router. Essentially creating a static address IP network subnet. This ensures that all devices have fixed IP addresses on the network but, easily permits changing a clients address by diddling with the router only.
Muff’s client status looks like this:
MuffStatus

All clients show only the Plum 192.168.2.7 server.
Please, some guidance on how to troubleshoot this issue would be greatly appreciated.

Thank You

2

I wish to thank the moderators for promoting my issue to the Discourse mailing.
I have restarted the urbackup server jail multiple times to no avail.
Always, the clients can’t see the server but, the server can see the clients.
After exhausting tests to solve the probable connectivity issues, I started looking to the improbable.
I saw the last date that the urbackup server worked was 05-29-22.
I checked and discovered that the Shed truenas server was last rebooted on 05-29-22.
This discovery caused me to reboot the truenas server.
I restarted the urbackup server jail and urbackup started to work again.
The truenas server had performed without any other type of issue for the some 40 days from 05-29-22 to today.
The truenas server must have crashed some critical service at the time of that last boot up 40 days ago.
I had not considered that it could be caused by the truenas server itself.
Thanks to those who may have spent time chasing this rabbit down the rabbit hole.