Is there a best practise to move a huge backupdir to a virtual client?

1

Hi, basically the title, a little context: I have experienced some intermittent issues with backing up qcow files and need to investigate that later to see what the pattern is exactly, but that kinda forced me to finally set the client up with virtual clients so i could split the different types of data under different virtual clients. That way a failed backup of the qcows will not fail the backup for other types of data. I have experienced issues configuring virtual clients, more on that later.

Now that i have defined a virtual client “btrfs” for the bulk data for that client (several folders under /btrfs on that client), i did a urbackupclientctl remove-backupdir and add-backupdir -v btrfs to move the paths that contains the big amount under that virtual client for every single folder path under /btrfs. Share hashes has always been on, no snapshoting mechanism has ever been used. However after not only waiting for (i estimate) 1,5day to have the client finish hashing:

Screenshot_20260718_172157
Screenshot_20260718_1721571529×333 30 KB

It seems to re-download all that data again and actually take up new storage, not what i expected.

What i expected to see was that the client would already have known the hashes for those paths and find the files could be linked instead of re-downloaded. I am now waiting for the job to cancel and have moved the biggest offender (/btrfs/media ~9TB) back from the virtual client to the regular client. I have to admit that during my adventures setting up the virtual client i had issues: i was able to define new virtual client names on the server end and those would reflect on the client output of list-backupdirs, but i when i removed a virtual client name from the server, it would not disappear on the client. Later i found that i could fix that by downloading and re-installing the client software (the one from the web gui for that specific client). I think that might have reset the filelists it had stored under /usr/local/var/urbackup/data and caused the client to need to rehash everything. But the data is mostly static, so the hashes should have matched with data that was backed up before, right? It seems i was mistaken. Also later i found that removing a virtual client just takes way longer at the client side; about an hour and it would finally disappear, lesson learned.

Anyway, im not sure what to do next. Eventually i wish to have all the backup folders that are under /btrfs on that client to be “under” the virtual client “serverddlin3[btrfs]”. What should i do once the backup cancellation is done?

I think i should run a urbackupsrv remove-unkown after this failed attempt, because the backupstorage is now completely full.

Should i remove all the previous backups for that client and just accept that i will have a timewindow of 1,5day (for hashing) and another 13TB/1Gbps (estimated time for data transfer: 36 hours) = 3 days total where there is no backup for that data or is there a way to efficiently move a backupfolder that was previously backed up under the virtual client?

If anybody knows, i would love to hear it.

2

Is that qcow unchanged in-between the file backups?

3

Hi Martin, yes the qcow is in use. I thought it would be best to create a seperate thread for that. But i can already share with you that the error is different from the (expected) warning. For now, the issue started on one host directly after the last server upgrade, then on another. The expected (example) warning was:
image
Now the (fatal) error was something i had not seen before:

Screenshot_20260713_170700
Screenshot_20260713_1707001280×565 208 KB

Also, the first screenshot shows a qcow that is on a glusterfs, and i have 3 small vms running like that. Those are still fine today. But the ones that fail are stored locally on ext4. So that was one of the weird things. After seperating the backup for that last one into a virtual client serverddlin7[opnsense_vm] it worked again but it came back later, then i ran a full backup and it worked again. If i shut down the vm (not the opnsense one ofcourse hehe) the incremental backup seems to work, but that is exactly one of the things i wanted to test: when it fails, does it work ok again if i shut it down? Things like that, a pattern. I have another VM that i can shutdown without bringing down my gateway.
I have some more to share and more to test but i have to go somewhere, i will check later today/this weekend and share what i found.
Let me know what you think about that 13TB that i need to separate from the rest, wdyt?

4

It does only do file-level dedup. So you might as well turn off the hashing in the Internet client setting for those virtual clients.

W.r.t. the error message. Could you look into/attach the server log? Maybe put it into debug mode.
(Also the versions involved would be interesting)

5

Hi again, back home again i was greeted with this:

Screenshot_20260718_212655
Screenshot_20260718_212655603×342 34.8 KB

I suspect the server had too much IO pressure stall, i have ran into performance issues before and need to revise the setup. After stopping urbackupsrv i attempted a urbackupsrv repair-database, which failed, there i expect there wasnt enough free space (i monitor that externally, i had a warning) after enlarging the underlying lvs and expaning the btrfs (raid1) filesystem (a subvolume mounted on /var/urbackup) i attempted the repair again, which succeeded but i think the damage was already done. The .db files in that directory were all ~50MB and urbackupsrv would not start, complaining about missing tables in the db. So today i setup snapper on that subvolume, just as an extra precaution for the future. I created a snapshot that holds the (corrupt) data, removed it and restored the data from a backup that was taken around 1pm yesterday so i dont think i lost too much. Now urbackupsrv starts again and is performing a “nightly cleanup” but the data in the database isnt consistent with the actual data in the backup storage, so im getting lots of errors with the cleanup and backups that are immediately started. I have stopped urbackupsrv and i am restoring the db data again, after that a remove-unknown should get that fixed (i assume).
Update while writing this: actually with that it is still complaining about database corruption and is now removing backups…argh…it feels like its removing backups because it cant find them in the (still corrupt) database and im pressing ctrl-c but its not quitting, ctrl-z puts it to sleep. sidenote; when you run a remove-unknown it seems to wait for 20 seconds to allow the user to ctrl-c and cancel the operation, but instead when you do that it just cancels the delay. The only way to cancel the remove-unknown is do a ctrl-z and then kill the stopped process, but additional damage seems to be done regardless:
image
image1617×615 196 KB

Especially those “backup(self)” backups are very important right now, i have no way to stop this, you can imagine how i feel about this. If only the remove-unknown would quit when i hit ctrl-c. I am trying to kill those subvolume deletions, but it doesnt look good. Im shutting the machine, thats the only thing at this moment i can do. I will have to check filesystems for issues first, this is getting out of hand.

Back on the subject, i might need to read through the admin manual again, because i think i need to have a better understanding of the transfer mechanisms and hashing methods, what you just said is not what i thought was going on. What i thought was that a client keeps a db of hashes of datablocks (or files?), compare them to what the server “knows” about and if needed “patch” and reflink the new files in the new backup. I have backups of the logs, i might be able to produce more information and i understand you wish to know. If you are able to see them i have a couple of drafts in my profile with possibly more information and context. But because things happen so fast at this end and the virtual client creation/removal issues i had resolved themselves while writing down the draft so i did not feel like they are relevant. Right now i am ofcourse prioritizing getting to a consistent and working state, get better knowledge of the hashing/deduping because it seems in my case i have not applied optimal settings. Also that might lead to me to better explain the situation to you. Thanks for now.

6

ab

7

Sorry, i dont know what “ab” means. I could just post the draft(s) i have on the subject if that is what you mean?

8

I was repeating my question in the hope you had an answer

9

I think i have that in the draft. Right now i am prioritizing getting things operational at my end. In the meantime i have recovered a database from the 17th, it repaired without issues (that was basically to see if it was corrupt) and now its running a remove-unknown. Looking at its output it seems to be removing the backups taken in the meantime and removing entries from the DB for which there are no backups in the storage, this could mean i am loosing some logs at this moment. It seems to me you are more interested in that specific error related to the qcows, ill update the draft where necessary, and once things are ok again im sure i can re-create the error so we can investigate in detail.