See the last discussion here: High memory utilization
Some issues might already be fixed in 2.5.x (specifically with lmdb, see https://github.com/uroni/urbackup_backend/commits/dev/urbackupserver/lmdb ).
The ZSTD compression also uses a lot of memory for idle connections, it would be interesting to see if that is what causes your problem (you’d see that in a heaptrack trace). Nowadays heaptrack can be installed via apt install heaptrack.