ERROR: No permission to access "/mypath/urbackup_tmp_files"

Server
#1

Hello,
I set up a machine with urbackupsrv : ubuntu 20.04 + LAMP with apache "ProxyPass “/x” “fcgi://127.0.0.1:55413” and SSL access.
I connected to my server and install a client. Everything seemed ok !
When client starts to connect, very quickly urbackupsrv bugs (process still running but impossible to connect via web interface & client lost internet connection)
Log gives that -> ERROR: No permission to access "/backupdata/urbackup_tmp_files"
I check chown and changed chmod -R 777 /backupdata
I rerun : service urbackupsrv restart
Bug still exits !
After that, ‘ls -lrtd /backupdata’ gives :
drwxrwxrwx 1 urbackup urbackup 0 Feb 12 17:08 /backupdata/clients
drwxr-x— 1 urbackup urbackup 0 Feb 12 22:49 /backupdata/urbackup_tmp_files
It seems that process apply chmod on /backupdata/urbackup_tmp_files

My conf
df -kh
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 392M 1.3M 391M 1% /run
/dev/sda1 15G 6.3G 7.4G 46% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mapper/vg-backupdata 1.9T 3.8M 1.8T 1% /backupdata
/dev/loop0 56M 56M 0 100% /snap/core18/1988
/dev/loop1 70M 70M 0 100% /snap/lxd/19188
/dev/loop2 32M 32M 0 100% /snap/snapd/10707
/dev/loop3 32M 32M 0 100% /snap/snapd/11036
tmpfs 392M 0 392M 0% /run/user/0

mount | grep backupdata
/dev/mapper/vg-backupdata on /backupdata type btrfs (rw,noatime,compress=zlib:3,space_cache,autodefrag,subvolid=5,subvol=/)

What’s wrong ?
Could you help me understand where to look for ?

#2

From documentation: 6.1 File backup (Bullet 6)

There is 2 options:

tmp_options
tmp_options918×180 7.95 KB

If you checked both, this will resolve your problem. But you will got another one.

tmp_issue

This error error occurs due low disk space on system partition (if you store /tmp on root filesystem).
Temporary directory filled with this cps.* files. And this files filled whole disk space because no limitation.

tmp_space
tmp_space531×832 16.5 KB

Suggest you to put /tmp on another partition with limited size for example 8-64Gb. I really don’t know how many it’s required… May be @uroni can suggest for us something? :slight_smile:

For me as you see 64Gb enough for 4 simultaneous image backup jobs. May be less is still will be fine.
I will test this strategy next week. Better to have server option that will limit this buffer, but I still didn’t find something.

  • As I see you use BTRFS over underlying LVM. My experience shows that this is bad idea. I got significant less iowait on ZFS against BTRFS (all due issues in 5.0+ kernels related to BTRFS Count of fixes make me crazy).
  • Looks like this is not a permissions mismatch, but high iowait try to track this.
  • You don’t need compression and autodefrag options except you doing uncompressed image/file backups I think.
#3

Thanks a lot for that very detailled explaination :grinning:

If I well understood, the root cause of my issue is btrfs on LVM creating high iowait.
You propose me a workaround using advanced options + another partition.
As I understand, going to ZFS would solve root cause and avoid advanced options and new partition.

My issue is that I rent a ‘kimsufi’ server. The configuration tool allows me to create a LVM partition, and, I used this to set up btrfs. I don’t see how I could set up ZFS in that context…

So, I see 2 options
a. The solution you propose with an additionnal partition (ext4) mount on /tmp
b. Simplify and go back without LVM partition, one filesystem ext4… mount on /

What do you thing ?
Is there an interest to have LVM btrfs ?

#4

Ok. This is not final solution. But.

  1. If you checked permissions on tmp folder and found it right then problem with high iowait (iostat usefull utility to diag this correlating error message with read/write metrics, also try to do read/write benchmarks on your backup storage).
  2. Any solution that you can apply to reduce iowait will be helpful in your situation, but for me I only achieved after moved to ZFS (At this point monitoring system no more alerting me on high iowait but high cpu load warnings still exist). In latest kernels btrfs got many fixes that I didn’t tested, you can try google something like “btrfs optimization”.
  3. There is my urbackup + ubuntu 20.04 server setup. This is for Hyper-V VM, all options based on my experience, be carefully and adapt for your environment before usage if you wishes. I still working on it.

What I can’t resolve: shortly still found no way to manage memory, cache, buffer for urbackup server.
Of course it’s working fine with small count of clients. But if we talking about tens then issues occurs.

In /backupdata/urbackup_tmp_files directory filled with this cps.* files. When advanced options to store it on system default tmp chosen, then system drive fully filled as result system crashes.Now I planning to configure urbackup user quota and test this settings. Better of course to have something like MAX_BUFFER_SIZE in urbackup server options but I didn’t found this option.

  • But putting buffer storage on different location (faster than same backup storage) that I think can be done very simply can improve backup jobs performance. And I wondered why this is not configurable.
#5

Hey Artyom, how can I thank you for all that ! :blush:

I see it becoming too complex !
I decide to move to bare dedicated server… not virtual :wink:

There is something that alert me : “Of course it’s working fine with small count of clients. But if we talking about tens then issues occurs.”

Do you means this, in general, or, for VM only ? Should I plan a machine for 8 clients at max… ?

#6

Finally, I solve my issue.
It was that bug : Urbackupsrv crashes on Ubuntu 20.04
I don’t know why it starts with “No permission to access” ?
I wonder if a compatibility resolution is in the pipe ?
Anyway, many thanks to all contributors :blush: