I have been using urbackup for a while without issue. Recently, my urbackup server (running on a raspberry pi), reached the SD limit and went into read-only mode. Since then, I haven’t been able to get a backup to complete. It remains at 100% with 0 files left with no progress.
The steps I’ve taken to resolve the issue
- Upgrade server with faster hardware (both processor and hard drive the database is on, currently on an SSD)
- Run urbackupsrv remove-unknown
- Run sqlite3 backup_server_settings.db “PRAGMA integrity_check”
- At this point, the multiple retries caused the backup to try a full backup instead of incremental, but the same issue remains
I turned on debug logging and am including the tail end of the log (as it’s quite large). If it would help to have the full log let me know. Any guidance is greatly appreciated!
urbackup.log (11.0 KB)
2 things I notice that might or might not be connected.
- Do you store your backups on an sd-card? Probably not what you want to do if that is the case.
- The log never shows “backup done”, pretty sure all my backup logs end with that.
You overfilled your sd-card, there is noway of telling what happened to your system and the risk of it being corrupted beyond repair is not unlikely.
I have never experienced a complete failure of the database in urbackup, but I 4 protections in place.
- I don’t have the database storage on my sd-card on my rpi.
- I make extra backups of the database and do not only rely on the single backup urbackup MIGHT do if you select it in the settings.
- I have a systemd service that is relying on stuff like working mounts and such, so if a filesystem fails, the urbackups systemd SHOULD stop (not sure in this instance to be honest, I will look into how to protect myself if this were to happen to me)
- I make constant img backups of / that I can restore if disaster were to strike, like this. I will probably loose the absolute last few backups, but at least I can restore the urbackup server to a working condition and redo the missing backups. Not perfect but better than nothing.
This might be considered a little bm, but I made this script to backup my SBC:s. Works on at least 4 different os:es (raspberry pi os amoung them). The size of the unpacked img file for my rpi with urback/plex/deluge etc. is 3.5GB, I restore that in like 5mins. If I compress the file it’s about 800mb big, pretty easy to manage.
On top of that, I exclude all *.img files so urbackup will backup its own compressed version but ignoring the img file. 
I’m sorry I can’t help more. Overfilling your storage on linux usually means “time to reinstall”. 
Hey Bedna, thank you very much for the information. I didn’t have the backups themselves stored on the SD card, just the database. I was hoping to be able to retain any and all information, but your comment on backing up the database reminded me I did keep bi-weekly snapshots of everything. I was able to pull the database from the last snapshot and was able to get the full backup to complete (I also used the tmp folder on the SSD which I believe helped)! Unfortunately, with timing, I lost about 4 weeks of backups (since it doesn’t seem like you can add missing backups back into the db). Definitely acceptable though not having to lose all my old backups.
Appreciate the help and guidance! @uroni If it’s something you’re interested in I could provide my backup database that wasn’t able to complete a backup in case you want to investigate.
John
Awesome!!! It always makes me happy to read when ppl find a solution!
Probably, since that directory is held in memory.
What you can do is symlink the database directory (/var/urbackup on my rpi) to somewhere not on the SD-card, sd-cards are NOT a good media to rewrite things over and over (like the urbackup database)
Glad you figured it out! 
So unfortunately the issue popped up again. After restoring the database and getting a full backup to complete, everything has been running fine taking incremental backups. However, the next full backup that was scheduled to occur had the same issue (backed everything up to 100% then got stuck on INFO All metadata was present and never finished). I’m going to attempt the same thing, but fear that there’s some systemic issue with my setup preventing the full backups from completing. Any thoughts or guidance on how to track down the issue? Much appreciated!
John
Did you move the database away from the sd-card?
If you keep getting a corrupted database (if that is the case here), there is something fishy going on, that should not “just happen”.
Hey bedna,
I ditched the PI completely and have it on a proper x86 system with the last time this happened. OS (including urbackup db) is now on an SSD. I agree, at this point I’m going to assume there’s something fishy with the db and am going to try and start fresh. I’ll still keep my old backups as reference in case I really need to get to them, but considering this is one of many types of backups I have I’m not going to spend too much time fixing the current backup chain.
1 Like