Driving me crazy why it stopped working!

[Andy_Newby]

Argh, this is doing my head in. It was all working, and now it won’t listen on 55415:

netstat -tulpn | grep 55415

This is the debug from tail -n20 -f /var/log/urbackup.log

20-12-14 09:04:44: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2020-12-14 09:04:44: SQLite: recovered 263 frames from WAL file /var/urbackup/backup_server.db-wal code: 283
2020-12-14 09:04:44: SQLite: recovered 3 frames from WAL file /var/urbackup/backup_server_settings.db-wal code: 283
2020-12-14 09:04:44: SQLite: recovered 24 frames from WAL file /var/urbackup/backup_server_link_journal.db-wal code: 283
2020-12-14 09:04:44: Started UrBackup…
2020-12-14 09:04:44: Removing temporary files…
2020-12-14 09:04:44: Recreating temporary folder…
2020-12-14 09:04:45: Testing if backup destination can handle subvolumes and snapshots…
2020-12-14 09:04:45: Backup destination cannot handle subvolumes and snapshots. Snapshots disabled.
2020-12-14 09:04:45: Testing if backup destination can handle filesystem transactions…
2020-12-14 09:04:45: Testing for hardlinks in backup destination…
2020-12-14 09:04:45: Could create hardlink at backup destination. Hardlinks enabled.
2020-12-14 09:04:45: Testing for reflinks in backup destination…
2020-12-14 09:04:45: Reflink ioctl failed. errno=95
2020-12-14 09:04:45: Could not create reflink at backup destination. Reflinks disabled. Operation not supported (code: 95)
2020-12-14 09:04:45: Binding to interface eth0 (ipv4) for broadcasting…
2020-12-14 09:04:45: Broadcasting on ipv4 interface eth0 addr 194.29.101.139
2020-12-14 09:04:45: Binding to interface eth1 (ipv4) for broadcasting…
2020-12-14 09:04:45: Broadcasting on ipv4 interface eth1 addr 10.184.1.74
2020-12-14 09:04:45: Binding to interface eth0 (ipv6) for broadcasting…
2020-12-14 09:04:45: Broadcasting on ipv6 interface eth0 addr fe80::216:3cff:fe96:7da5
2020-12-14 09:04:45: Binding to interface eth1 (ipv6) for broadcasting…
2020-12-14 09:04:45: Broadcasting on ipv6 interface eth1 addr fe80::217:3eff:fee7:726a
2020-12-14 09:04:45: UrBackup Server start up complete.
2020-12-14 09:04:45: Server started up successfully!
2020-12-14 09:04:45: Looking for old Sessions… 0 sessions
2020-12-14 09:04:46: Downloading version file…
2020-12-14 09:04:46: Downloading version file…
2020-12-14 09:04:46: Downloading server version info…
2020-12-14 09:04:46: Downloading dataplan database…

What am I missing?

[Andy_Newby]

OMG this is really doing my head in. My firewall settings:

ACCEPT udp – 0.0.0.0/0 0.0.0.0/0 udp dpt:35623
ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 multiport dports 55413,55415
ACCEPT tcp – 81.174.134.133 0.0.0.0/0 tcp dpt:55414 # this is me - as only I want access to admin panel!
ACCEPT tcp – 0.0.0.0/0 0.0.0.0/0 multiport dports 55413,55415

Debug when restarting is:

2020-12-15 16:37:08: Reflink ioctl failed. errno=95
2020-12-15 16:37:08: Could not create reflink at backup destination. Reflinks disabled. Operation not supported (code: 95)
2020-12-15 16:37:08: Binding to interface eth0 (ipv4) for broadcasting…
2020-12-15 16:37:08: Broadcasting on ipv4 interface eth0 addr 194.29.101.139
2020-12-15 16:37:08: Binding to interface eth1 (ipv4) for broadcasting…
2020-12-15 16:37:08: Broadcasting on ipv4 interface eth1 addr 10.184.1.74
2020-12-15 16:37:08: Binding to interface eth0 (ipv6) for broadcasting…
2020-12-15 16:37:08: Broadcasting on ipv6 interface eth0 addr fe80::216:3cff:fe96:7da5
2020-12-15 16:37:08: Binding to interface eth1 (ipv6) for broadcasting…
2020-12-15 16:37:08: Broadcasting on ipv6 interface eth1 addr fe80::217:3eff:fee7:726a

So AFAIK it should be running? This is what I see in the admin area:

…and yet it still doesn’t seem to work:

root@backup:/backup/servers# curl 194.29.101.xx:55415
curl: (7) Failed to connect to 194.29.101.xx port 55415: Connection refused
root@backup:/backup/servers# curl 127.0.0.1:55415
curl: (7) Failed to connect to 127.0.0.1 port 55415: Connection refused
root@backup:/backup/servers# curl localhost:55415
curl: (7) Failed to connect to localhost port 55415: Connection refused

PLEASE PLEASE someone help me debug this. It was working perfectly last night and this morning. We had 6 or 7 backups of each server done. I’ve done bugger all to the server, apart from reboot it, and now it just doesn’t want to work

Thanks in advance!

Andy

[Andy_Newby]

Some more updates

 Server:

 nc -z -v -u 194.29.101.139 35623
 Connection to 194.29.101.139 35623 port [udp/*] succeeded!

 Client:

 nc -z -v -u 80.85.84.52 35623
 Connection to 80.85.84.52 35623 port [udp/*] succeeded!

Yet its refusing to see the server any more. PLEASE does anyone have any ideas on how to debug this? I’m loosing my hair over it! The only way I get it to work again, is to wipe the server, re-install, and then it works for a bit. Then eventually it stops working. So something is causing it to break, but I can’t see what

Cheers

Andy

[Bearded_Blunder]

In my experience strange connectivity issues and also some apparent hangs have been down to IPv6, that’s a complete stab in the dark though.

If you enable the correct columns it’ll show if it’s using / trying to use that, sorry I can’t give you exactly which without looking & I have my server stopped to run CHKDSK on the storage, so I can’t look right now, CHKDSK is likely to take many hours to complete.

However strange issues all went away on my setup when I disabled IPv6 at the OS level so UrBackup could no longer attempt to use it.

Can’t really help you troubleshoot Linux networking, it’s evident you’re more familiar with it than me anyway.

[Andy_Newby]

Thanks for the reply So going based on:

How to Disable IPv6 in Ubuntu Server 18.04/16.4 LTS

I edited /etc/sysctl.conf, so it has this at the bottom:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth1.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

and then I reloaded with:

sysctl -p

Then, run the following command to check the IPv6 status:

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

which prints out “1” (which I believe is correct!). So I can’t see the ipv6 trying to be connected to any more. Yet it still doesn’t work Do you have to maybe set the clients to use ipv4 as well, instead of trying ipv6?

[Bearded_Blunder]

Possibly, I only had to set the server, but I only have Windows clients.

[Andy_Newby]

Its like the server cant see them:

My backup server on HostHatch doesn’t come with both ipv4 and v6 configured on building. I tried to work out how to add ipv6, but seemed to cause more problems then it solved lol

[Bearded_Blunder]

Well if your clients are set to internet mode, & configured to talk to the correct server ip/hostame, then I’m getting out of my depth, ipv6 was really a stab in the dark having solved similar awkwardness here.

[Andy_Newby]

I think I’m going to try and remove them, and re-add them via the NAT option. Not as pretty as the “hint” method, but if it works - who knows. Will update once I’ve tested it out on one client

[Andy_Newby]

Ok cool. So I’ve moved to another storage host ( https://www.mythic-beasts.com/ ), and they include ipv6 as a standard setup for their server. So I setup a 1 core server, with a few hundred Gb of storage, installed Ubuntu, HestiaCP, and UrBackup. Its been running for a couple of days now without incident So I guess the issue must have been around the ipv6 stuff on the HostHatch Storage server