Backup is corrupt (cannot restore some files)

Hi

I’ve been testing urbackup for the past couple of months and had the need to use it in anger this morning - a directory I was working in, which hasn’t been changed for weeks, was deleted and I needed it back.

I can download a .zip file of the whole directory via the webui, but some of the files are corrupt - two immediate examples are .JSON text files, they are a reasonable size (1200 bytes and 1963 bytes respectively) but they full of zero byte data. These files were full of actual data this morning (they were actually loaded in my IDE when the directory was deleted), and I’ve tried going back to backups from a couple of days ago and downloading them from the web interface, but still they are corrupt.

My workstation is OSX 10.13 and the server is CentOS Linux release 7.4.1708 (Core) 3.10.0-693.2.2.el7.x86_64.

Any ideas?

Regards
John

Can you check if the files are also corrupted on disk (i.e. it is a ZIP problem) on the server?
Also can you run e.g. urbackupsrv verify-hashes -v '*' on the server?

Mac OS doesn’t have snapshots for backups, so that might be the problem if files get modified during backup.

It’s still running but I’m getting lots of errors like:

2017-10-23 13:38:35: ERROR: Error opening file "/home/backup/Johns-MacBook-Pro/170914-1221/dev/sparlm/Source/google-my-business-api-sample-2/tmp/google-oauth-java-client/dependencies/images/icon_info_sml.gif"

2017-10-23 13:38:35: ERROR: Error opening file "/home/backup/Johns-MacBook-Pro/170914-1221/dev/sparlm/Source/google-my-business-api-sample-2/tmp/google-oauth-java-client/dependencies/images/icon_success_sml.gif"

(By “a lot” I mean that the list of files has scrolled off the top of my terminal window buffer)

These files exist on my workstation and should be included in the backup schedule; the error opening on the server is because they do not exist at all (ie it’s not a permissions problem).

I’ve tried to download versions of the specific files direct from webui dating back a few days and they are equally corrupt; these files have not been touched for weeks, and at least one of them I did not edit today either so I don’t think it’s a snapshot issue. I have found the files on the server, and they are corrupt too - if I got back as far as the 171020-0657 directory on the server, I can find the file without following symlinks and it is corrupt, yet this morning I had that very text file in my IDE.

So, the backup is not storing some files and storing other files as junk (although some files are stored OK)

The urbackup on my client is v 2.1.16

tested verify-hashes -v ‘*’ at home
for me it complains when there are broken symlinks for .directory_pool/ folders

Does it fail often and how does this happen? Can it be corrected on the server? It sounds like these errors by verify-hashes are something different, although missing files is pretty serious and has to be fixed, my original issue is that the files are corrupt.

It seems to me that a backup solution has to be 100% reliable otherwise it is 100% useless and this is a pretty new installation on the server and the client, so I am surprised that the devs aren’t all over this problem. Is there a “real” support channel I should be raising this with?

Hello

That forum is the best support channel you’ll find. @Uroni is the dev, so he knows his stuff pretty well, and usually he is quite quick to produce fixes.

Only if I get enough information to narrow down the problem :wink: Which is why I asked for the verify. If that fails for the file something propably corrupted the e.g. the file system.
If there is a problem with directory links, I’ll need more information as well. No reports/complains about that before.

The verify completed late yesterday but the verification file was not created. I’m happy to re-run it if you think it will help. Is there any other information you need?

This server was built especially to run URBackup, and only a short while ago - it’s a VM hosted on a SAN and there no errors on the hosts or the disks, so the corruption would appear to be at the application level rather than the OS or hardware.

As far as I am aware, the urbackup client on my Mac is running just fine - it’s in the menu bar and if I open the status window it says that the last backup was 7 minutes ago and it’s connected to the public address of my backup server just fine.

BTW, although I have a direct route to the backup server, it has two NICs, one public internet and one private, the server is configured for access via the internet NIC. My workstation is the other side of a VPN but ignores the VPN for the purposes of backing up.

Hi

I am at the office, i’ll check at home the .directory_pool/ folders links. If they can be properly restored and whatnot

Can you confirm, when you restore :
The file is about 1-2kb in size, but when you open it in a text editor or with less , it contains only 000000… ?

Can you do on the folder you need to restore on your osx worsktation
ls -l “the path to the folder”
Same on the server (maybe one recent , one old date) ?
Can you also post a a screenshot of the gui where you see theses files?

I am curious to know if they are stored in a .directory_pool/ folders, and if the links to the pool are fine.

To find the file you want to restore on the server you can try something like:
find “/path to urbackup storage” -iname “filename” -exec ls -ld {} ;

It still unclear for me how this works exactly.

thats a broken symlink :
ls: cannot access ‘/data/urbackup2/home/170706-2345/home/orogor/.thunderbird/bpy1ky4i.default/saved-telemetry-pings’: No such file or directory

if i browse to the file of the broken symlink in the web ui i can restore it

so i searched for that file on the drive, found a few instances of it

that s a strange file
ls -lh /data/urbackup2/home/.directory_pool/82/821pPcrwDk150184295025964574062/00befc13-b5e4-42e3-8715-1660eb4d6bde
-rwxr-x— 2 urbackup urbackup 306 juil. 7 00:04 /data/urbackup2/home/.directory_pool/82/821pPcrwDk150184295025964574062/00befc13-b5e4-42e3-8715-1660eb4d6bde

strange file content (if needed, tomorow i ll use find -L /dir/to/start -samefile /tmp/orig.file to find what links to it and check if i can open it)
F�<Fm�K7���1�3�$�1�rT���AN��@F�<Fm�K7���1�3�$�1�rT��e/home/orogor/.thunderbird/bpy1ky4i.default/saved-telemetry-pings/00befc13-b5e4-42e3-8715-1660eb4d6bde����
����
H�GvF�xC�,����h�h����
5�����
5�����
5�r

thats a good file (if i open it , its valid json data and the same that the original file on the client)
ls -l /data/urbackup2/[home]/.directory_pool/M7/M776M2X9Vx150106347625185100462/00befc13-b5e4-42e3-8715-1660eb4d6bde
-rwxr-x— 12 urbackup urbackup 8134 oct. 11 2016 /data/urbackup2/home/.directory_pool/M7/M776M2X9Vx150106347625185100462/00befc13-b5e4-42e3-8715-1660eb4d6bde

Hi @orogor

Sure, here’s the listing from that morning’s archive directory:

# pwd
/home/backup/Johns-MacBook-Pro/171023-0846/dev/Qooxdoo/demoapp
# ls -la config.json
-rwxr-x---. 64 urbackup urbackup 1963 Aug 30 17:25 config.json
# cat config.json
# hexdump config.json
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
00007a0 0000 0000 0000 0000 0000 0000
00007ab

That config.json was all text, and if I go back further in time it’s still the same contents. I think I’ve found the directory_pool file and its the same:

# pwd
/home/backup/Johns-MacBook-Pro/.directory_pool/7w/7weXOoWJHw15087647382439673027/demoapp
# cat config.json
# hexdump config.json
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
00007a0 0000 0000 0000 0000 0000 0000
00007ab

On my workstation, I have replaced the file from an old backup on another drive, and the current backup seems fine:

# pwd
/home/backup/Johns-MacBook-Pro/current/dev/Qooxdoo/demoapp
# head config.json
{
	"name": "demoapp",
	"include": [
		{
			"path": "${QOOXDOO_PATH}/tool/data/config/application.json"
		}
	],
	"export": [
		"api",
		"api-data",

This is probably just the file in .hashes (metadata information since it includes the original file path).

Can you go through the hierarchy and find out where it is broken? The web interface resolves the symlinks manually, maybe that’s why it works there.

In the advanced settings there is an option (marked debugging) to check the files after backup. This will fail if the files change during backups because it wasn’t snapshotted, like on macs unfortunately.

@uroni thanks, I’ve enabled the debugging option. Is there a way to make the backup client re-check all of the files? There are still some files missing from the archive. I don’t want to delete and restart the entire backup because it is 70GB.

Well, I’m giving up with this. Urbackup shows a lot of promise but it is clearly not suitable for production use, there is very little support in the community and apparently no commercial support either. That it can fail to backup a text file in it’s first month of evaluation is very disappointing and worrying, but the poor support in tracking it down and fixing the problem is just unacceptable.

A backup just has to be completely reliable or it’s worthless, and UrBackup is worthless.

Hi, all.

Maybe also interesting for @uroni then, I have spent quite some time with a fresh installation - and I also run into this. It is consistent behaviour and vers reproducable.

Server: raspberry pi running Jessie with btrfs-tools as only extra, latent urbackup server (2.1.19)
Client: Mac client 2.1.16 on high sierra with APFS

The probleem is that during backup, THE FILES ARE NOT ACTUALLY TRANSFERRED. The server shows that everything is fine, but it seems that only the metadata of the files is transferred (Size, name, hash, …?).

ACTUALLY, this is quite easy to spot: a multi-GB backup is done in a only 1 or 2 minutes…and watching it in the ‘live log view’ on the server shows this insane speed. Furthermore, the backup directory on the server hardly grows, just some 10’s of MBs in a multi-GB download - so it is pretty obvious that NOT FILES ARE TRANSFERRED.

Trying to restore files exactly produces the problem that the original TS sees: a file seems to download/restore OK, but the file is completely corrupted, as if UrBackup just fills empty space until the correct file size is reached…

I have tried many things: change the filesystem (btrfs, ext4 - all the same results), complete reinstall (both of server and client), etc. But the problem persists. I am about to give up. Maybe this is a Mac OS High Sierra (10.13) / APFS problem? I can’t tell as I only have Mac OS High Sierra clients here…

Anyone?

Grtz,
R.

I have also seen my urbackup become corrupt and useless.
Working on building backup solution with other software now. :cry:
I really like urbackup client / server software, but I can’t use backup software not working.

That was also on mac?
There s a guy who submited system traces of the problem and as i remember uroni spotted something with sparse files, so there are chances it maybe fixed.

No debian stretch server, with backup from linux servers and win clients.
but due to missing client encryption I need to look for another solution.