High memory utilization

Server
1

Hello! Server hardware HP Proliant, Ubuntu 18.04, urBackup Server 2.3.8, Number of clients 57, RAM 64Gb, backup storage HP hardware raid 5, partition btrfs. Server settings: Max simultaneous backups = 16, Max recently active clients = 32.

Memory
Memory1907Ă—361 85.7 KB

How to determine reason?
Also found process running for 342h.
Htop
Htop1000Ă—583 179 KB

2

You could enable Show custom thread names in htop display options, then it should show what the thread does that uses that much CPU time.

Best tool for Linux recently that finds memory leaks/usage problems is heaptrack.

  • Compile the urbackup server (so you have debug symbols)
  • Compile or install heaptrack https://github.com/KDE/heaptrack
  • Then run urbackupsrv with e.g. heaptrack urbackupsrv run -c /etc/default/urbackupsrv
  • Send me the heaptrack output or analyze it yourself
1 Like
4

Well, looking at heaptrack:

grafik
grafik1129Ă—795 130 KB

UrBackup never uses more than ~150 MB memory…

1 Like
5

Ok just a short video about how it’s going…

If you think this is not urBackup, then can you suggest how can I tune OS/disk system?

Htop
Htop1442Ă—1005 243 KB

Full record: Htop_urBackup.mp4

6

Besides returning to my first post I wonder that 50Gbs of reserved memory by urbackup process is not strange for you…

As I said this behavoiur on multiple servers: memory continues swallow then swapping and finally urbackup service going to exited status. All linux servers.

7
  • Fails in Heaptrack output
heaptrack output will be written to "/root/scripts/heaptrack.urbackupsrv.393001.gz"
starting application, this might take some time...
TEST FAILED: guestmount is missing (libguestfs-tools)
Testing for btrfs...
ERROR: not a btrfs filesystem: /mnt/Storage/Backups/testA54hj5luZtlorr494
TEST FAILED: Creating test btrfs subvolume failed
Testing for zfs...
TEST FAILED: Dataset is not set via /etc/urbackup/dataset
2021-02-10 08:34:59: WARNING: Shutting down (Signal 2)
free(): invalid pointer
heaptrack stats:
        allocations:            6748728
        leaked allocations:     49175
        temporary allocations:  4718224

How explain this errors and fails?
I can understand btrfs error because I use ZFS.
Also libguestfs-tools. Bacause this package not installed.

9

You probably have Temporary files as file/image backup buffer enabled? See the manual for that…

11

You’re using ZFS with only 8GB RAM. This is insufficent in your case. Additionally, you’re using raidz for your setup which make things worse. ZFS needs (a lot of) RAM and assings 50% of RAM by default and won’t free up while running. Try at least 64GB RAM and use striped mirrors of disks (aka RAID10) for better io performance.
My setup just for comparison: dedicated server with AMD EPYC 7232P 128GB RAM 8 disks striped mirrors with dedupliucation + compression enabled, RAM ist always around 80% full (I’ve tuned some ZFS parameters) while running, only 25 clients (Win + Linux) mixed, ~30TB logical, ~4TB allocated

12

Additionally, don’t (!) use never ever crappy RAID-Adapters with ZFS!

13

Ok. I understand.
But this is not ZFS related.

  1. I have same issue on server (just a standard PC no RAID) with single SATA HDD 10Tb with 8Gb of RAM.
  2. Same issue with EXT4 and BTRFS.
14

Ok, that’s strange behavior. What I’ve forgotten to write: I’m using Ubuntu Server 20.04 LTS. Even on 10 years old hardware for testing with only 32GB RAM, I’ve never had such issues like you. I’m sorry, but I can’t help.

15

Ok. I have 2 run modes:

  1. Default:
  • Temporary files as file backup buffer: DISABLED
  • Temporary files as image backup buffer: DISABLED
  • DAEMON_TMPDIR="/tmp" in /etc/default/urbackupsrv

Results: >3Gb memory utilized by urbackupsrv, following free command I think it’s buff/cached.

Default_TMP
Default_TMP1127Ă—463 108 KB

  1. Custom:
  • Temporary files as file backup buffer: ENABLED
  • Temporary files as image backup buffer: ENABLED
  • DAEMON_TMPDIR="/mnt/Tmp" in /etc/default/urbackupsrv

Result: same result + files on disk. Wondered about impact of this options.

image
image1150Ă—509 18.9 KB

Final summary.

  • urBackup issue related to memory / buffer / cache management and this issue have no dependency on file system, storage, OS (except Windows, have no experience with it) etc.
  • sync && echo 3 > /proc/sys/vm/drop_caches commands allow to control server stability, but not at all
  • After a while all memory filled and this cache starts fill swap
  • When memory and swap fulfilled urbackupsrv failed with “exited” status

And I still have no response from developers…

My mistake: in posts before i used free command from ubuntu it have differences with debian. On ubuntu “free” column includes buff/cache related to process in memory on debian not (StackOverflow)

In this post i show tests from Debian 10.8 and 2.4.13 server. same server for both tests with only single client. Doing Full image backups to VHDz.

16

Ok, maybe /tmp is using tmpfs (RAM)?

17

No just checked. And no ramfs mounts available.
image

18

One of my servers with 10 clients.

image
image1040Ă—384 16.3 KB

This is Ubuntu 20.04 with ZFS, 2.4.13, default settings on server’s Advanced tab.

All filled memory, swap, but still working!!! urbackup 4.6 gb in memory while doing nothing.
Have no idea what is going on.
Most other my servers failed in this resources utilization state.

19

You are using tmpfs:

/dev/shm

check https://wiki.archlinux.org/index.php/Tmpfs

20

Yes 0% and 0 bytes utilization while backup jobs running fr all mounts. I already checked all tmpfs mounts. There is nothing related to urbackup. And others too.

21

Just about your server with 10 clients:

As you can see 4,6GB for urbackupsrv (my System shows up 3GB) and if you remember ZFS needs 50% of RAM…
…there’s nothing left.

22

Ok.:slight_smile: What is in swap? :slight_smile: While server not doing anything.

23

I don’t know what’s inside swap…
…but did you changed the swapping behavior?

E.g. like adding “vm.swappiness=10” to /etc/sysctl.conf