Thoughts and questions after install on Raspberry Pi 4

general discussion
1

Had a thread about choosing filesystem a while ago and you guys helped me out, thanks again for that.

Setup:
Rpi 4 8GB
6TB WD red
4TB WD red

NAME        FSTYPE FSVER LABEL      FSAVAIL  FSUSE% MOUNTPOINT
sda
├─sda1      ext4   1.0   diverse     1.8T    43%    /media/diverse
└─sda2      btrfs        usr        18.1G    20%    /media/usr
sdb
└─sdb1      btrfs        backup        3T    45%    /media/backup
mmcblk0
├─mmcblk0p1 vfat   FAT32 boot      221.8M    12%    /boot
└─mmcblk0p2 ext4   1.0   rootfs     24.8G     8%    /

Server on Rpi and client on both the rpi and Win10 on local network.

I ended up resizing both sda and sdb, made the “usr” directory (where the urbackup db is kept) on the 6TB first and then realized my stupidity of having both the working dir AND the backup on the same physical hd. This was after I had already ran backups for a week and finally got everything going so the disks were all used. But still lost no data (except for the usr data that I had to backup in advance, no biggie, only a few GB). At least got to deep learn parted, btrf, e2fsck and resize2fs. :slight_smile:

For both clients I had a full file backup and an incremental backup on the server. I accidentally pressed “full file backup” on the Rpi client, realized my mistake, stopped it. Stopped urbackupsrv.service and ran urbackupsrv cleanup --amount 0% and urbackupsrv remove-unknown. Rebooted and tried to make an incremental backup, but it still continues full file backup.

I checked in the backup folder under the rpi client and yeah, it didn’t remove the last backup.
Stopped the service again, and tried to redo the commands but this time keeping an eye on the backup directory and nope, doesn’t remove the “unwanted” full file backup dir.

Tried to remove it with sudo, but nope, did not let me. Only urbackup has any kind of privileges. Tried to chown them, but wouldn’t let me. So I restarted the backup again and had to wait another 72 hrs for the rpi to backup 1,3TB (or something like that).

Now with only 2 days to go, I realized I could have sudo su the user urbackup and removed the directory and then retry, but would that actually not break the urbackup server? I’m suspecting the database would still point to the directory somehow?
Guess I’ll just let it finish for another 2 days but it SUX, I already have a full file backup. Could it have been because I resized the partition somehow? The files never moved, but still? Maybe I HAVE to do a full backup?

About the speeds. From my win computer it registers speed about 200-250Mbit, witch is very strange, since I only have 100Mbit router bottlenecking in between (waiting for new router), how is that even possible? And even with that speed, it doesn’t seem accurate, since the backup from my computer (about 1TB) took about 13hrs. Should have gone quicker with that speed?

With the backup on the rpi itself, i see in top, and totally understand that its the cpu bottlenecking, not much to do about that, speeds are registering between 30-75Mbit, I’m ok with that but faster would be nicer. How does the transfer on a local machine with both client and server work? Is it still somehow utilizing the network, and that slows things down?
Running in background disabled, no compression and everything in raw.
Would/Should turning off encryption on the rpi client make it faster?

When I installed I made a symlink in /var/urbackup pointing to the usr mount and moved the database here since id rather not keep the database on the sd card. Seems to be working brilliantly except for one time, when running the database cleanup script it gave me an error about that exact directory but everything seems to work, should I be worried? (can’t run the script now to reproduce the error, still waiting 2 more days to finish backup xD)
Is this the correct way to move the database?
Tried searching for info on it but cant find anything about it.

Thanks for reading and hopefully you have some helpful input for me.

2

Updating this with a few solutions to problems I had and a small request to update the documentation about snapshots.

As you can se above in my previous post I have two partitions in btrfs and one in ext4. The reason I want the /media/diverse in ext4 is because I read somewhere that downloading torrent files to a btrfs partition could be hurtful for btrfs so I opted for ext4 (not backing up that directory).

Finding information about how to get this work was NOT easy. For setting the backupfolder mode to btrfs is described in the documentation, easy peasy.

But when it comes to snapshotting all info (if any at all) had to be found on this forum.
Quite a few times I found forum posts with questions about the exact things I needed answers to but those posts were all unanswered. :frowning:
Since I installed the client with “no snapshot” I had to find out how to change that.

I figured out I need to remove/rename /usr/local/etc/urbackup/no_filesystem_snapshot and create a file called snapshot.cfg.
I read about this file in forum posts and figured out that all the snapshot scripts are in /usr/local/share/urbackup/

$ ls /usr/local/share/urbackup/ -l
total 48
-rwxr-xr-x 1 root root  732 Oct 18 14:41 btrfs_create_filesystem_snapshot
-rwxr-xr-x 1 root root  662 Oct 18 14:41 btrfs_remove_filesystem_snapshot
-rwxr-xr-x 1 root root 3447 Oct 18 14:41 dattobd_create_snapshot
-rwxr-xr-x 1 root root 2332 Oct 18 14:41 dattobd_remove_snapshot
-rwxr-xr-x 1 root root 5145 Oct 18 14:41 dm_create_snapshot
-rwxr-xr-x 1 root root 2191 Oct 18 14:41 dm_remove_snapshot
-rwxr-xr-x 1 root root  378 Oct 18 14:41 filesystem_snapshot_common
-rwxr-xr-x 1 root root 1784 Oct 18 14:41 lvm_create_filesystem_snapshot
-rwxr-xr-x 1 root root 1869 Oct 18 14:41 lvm_remove_filesystem_snapshot
drwx------ 2 root root 4096 Oct 18 14:41 scripts
-rw-r--r-- 1 root root  436 Oct 18 14:41 urbackup_ecdsa409k1.pub

I created the file /usr/local/etc/urbackup/snapshot.cfg and tried all of the different modes but they all gave different error messages because of the different filesystems.
I used nano to try to understand the scripts and try to figure out how to set different modes to different disks but could not figure it out. Searching the internet about this subject rendered absolutely nothing.

#This is a key=value config file for determining the scripts/programs to create snapshots
create_filesystem_snapshot=/usr/local/share/urbackup/btrfs_create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/share/urbackup/btrfs_remove_filesystem_snapshot

I spent quite a few days on trying around with different scripts in snapshot.cfg but could not get it to work.
I did not create an lvm on the ext4 (because I didn’t know about these mechanics at all before starting this project) and wasn’t too keen on shrinking the filesystem, spending space just for snapshots on that drive since it only contains media files, they most probably wont change when the backup is run and if any does, it’s not really a big deal if that file fails on one backup (backup wont fail), a new incremental backup is run in 6hrs.
So what made sense to me was to use the btrfs_create filesystem_snapshot and btrfs_remove_filesystem_snapshot on my /media/users directory because those files WILL change during backup.

But this produced this log entry for every backup

Level	Time	                Message
Errors  10/28/22 12:44      	Creating snapshot of "/media/diverse" failed
Errors  10/28/22 12:44      	Cannot create snapshot of file system with type ext4
Errors  10/28/22 12:44      	Creating snapshot of "diverse" failed.

So I simply want to exclude the drive from snapshotting.
When for the hundread time I read the contents of /usr/local/share/urbackup/ I suddenly realized there is a reference to /etc/urbackup/exclude_snapshot_mountpoints in the file filesystem_snapshot_common. It’s the same directory used to set the backupfolder to btrfs but the file was not there.

After searching the forum for exclude_snapshot_mountpoints I finally found my answer. Simply create /etc/urbackup/exclude_snapshot_mountpoints and put /media/diverse, ie the mountpoint you want to exclude, in that file.

$ sudo echo "/media/diverse" > /etc/urbackup/exclude_snapshot_mountpoints

This should really be mentioned in the documentation, I can’t be the only one with that problem and it really was an easy fix, if you just know how to do it.

Question:
When searching for info on /usr/local/etc/urbackup/snapshot.cfg I read in ONE post that it included “volumes_mounted_locally=0”. I have the client and the server on the same machine, should I add this to the file?

#This is a key=value config file for determining the scripts/programs to create snapshots
create_filesystem_snapshot=/usr/local/share/urbackup/btrfs_create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/share/urbackup/btrfs_remove_filesystem_snapshot
volumes_mounted_locally=1

Or maybe even volumes_mounted_locally=2 or even 3?
Both drives I backup are locally mounted but only one use snapshots. And the sdcard the rpi is running on is also obv mounted locally but I don’t back anything up from that drive.
As of now, volumes_mounted_locally is not in my file. Should it?

Maybe this post turns up in other peoples search having the same problems as me.
Still hoping for some input/answers on my first post. :pray:

3

Hi, man.
lets try it: GitHub - uroni/urbackup_backend: UrBackup - Client/Server Open Source Network Backup for Windows, MacOS and Linux

install_client_linux.sh