UrBackup Server 2.0.32 100% CPU usage

SBTech · August 30, 2016, 6:04pm

When my workstation is backing up to the server (Image Backup) the CPU on the server goes to 100%, creating a blocking problem with other services.

My server is an Proliant with a single Intel® Xeon® CPU L5420 @ 2.50GHz and 8GB ram running FreeNAS (UR Backup is running in a jail).

I have seen this same issue on two other systems I have setup in a similar way. All clients are on the local lan, not connected via the internet.

Any thoughts on how to de-prioritize the urbackupsrv process? anyone else seeing this issue?

ttrammell · August 30, 2016, 7:46pm

~~On the server, under the settings -> advanced tab, do you have the box checked to be a background priority or not?~~

That is probably for workstations, not servers. My bad.

SBTech · August 30, 2016, 7:55pm

Yes, but that settings is for the clients? Full Text is “Run backups with background priority on the clients:”

trying to use nice in a rc.d script, but its not working and server is now really slow. Getting 71.15 KBit/s was getting 100+ MBit/s on the activity page.

#!/bin/sh
. /etc/rc.subr
name=urbackupsrv
rcvar=urbackup_srv_enable
command="/usr/local/sbin/${name}"
command_args=“run -d -g 104857600 -u root”
start_precmd=“cd /usr/local/var”
load_rc_config $name
run_rc_command “$1”
pidfile="/var/run/${name}.pid"

start_cmd="${name}_start"

urbackupsrv_start() {

    /usr/bin/nice -n 10 ${command} ${command_args}

}

ssemone · August 30, 2016, 8:55pm

I wonder if the jail is incurring some sort of performance penalty. Maybe try installing the server outside of a FreeBSD jail and see what the results are?

SBTech · August 30, 2016, 9:35pm

When I comment out the last few lines and eliminate the nice, I can you 100% of the server CPU to the point that other services start preforming poorly.

#!/bin/sh
. /etc/rc.subr
name=urbackupsrv
rcvar=urbackup_srv_enable
command="/usr/local/sbin/${name}"
command_args="run -d -g 104857600 -u root"
start_precmd="cd /usr/local/var"
load_rc_config $name
run_rc_command "$1"
pidfile="/var/run/${name}.pid"

#start_cmd="${name}_start"

#urbackupsrv_start() {

#       /usr/bin/nice -20 ${command} ${command_args}
#}

This is a backup running

But it looks like my issue with my first test was client side, after starting the sever with the rc.d script in my first post (with nice) I did not restart the client on my computer. Its now running at full speed, just need to test if its still blocking other process when hogging the CPU.

ssemone · August 30, 2016, 9:42pm

Great find!

SBTech · August 30, 2016, 10:57pm

no joy back to the drawing board, none of the above scripts where doing anything as they were not correct. once I got the correct order, and added an echo so I would know what was happening did i realize I had issues.

#!/bin/sh
. /etc/rc.subr
name=urbackupsrv
rcvar=urbackup_srv_enable

#start_cmd="${name}_start"
start_postcmd="${name}_poststart"
start_precmd="cd /usr/local/var"

command="/usr/local/sbin/${name}"
command_args="run -d -g 104857600 -u root"
pidfile="/var/run/${name}.pid"

urbackupsrv_start() {
        echo 'Starting "niced"'
        /usr/bin/nice --20 ${command} ${command_args}
}

urbackupsrv_poststart() {
        sleep 1
        echo 'Renicing'
        /usr/bin/renice -20 $(cat ${pidfile})
}

load_rc_config $name
run_rc_command "$1"

Problems

the rc.d init system does not have permission to nice or renice
even when I renice urbackupsrv process to -20, it still blocks other process with its 100% CPU usage during image backups.

ssemone · August 31, 2016, 3:52pm

Bummer. Maybe try outside of the FreeBSD Jail?

SBTech · August 31, 2016, 5:51pm

Installed on FreeNAS so outside the jails really sounds like a bad idea, plus I’m start to consider this a bug. Should backing up 1 computer send the backup server to 100% CPU?

I have found a “work around” that pretty much sucks but is working. Under settings -> Server I have set “Total max backup speed for local network:”, this seems to limit the CPU leaving some idle cycles and my servers still working.

I can confirm that this issues is happening on two separate FreeNAS servers running 2.0.32 in a jail and that image backups send the CPU thru the roof.

ssemone · August 31, 2016, 11:57pm

Hehe, probably right.

I wonder if there is something else in the FreeNAS setup that is trying to also read/write disk changes at the same time that the image is being written (e.g. file system snapshots?).

SBTech · September 1, 2016, 3:06pm

Just to be clear, its NOT an IO issue its a CPU issue.

The server that I’m testing in my office is a Quad Core Xeon, with URBackup server running and idle the CPU is at 98% idle, when I initiate a image backup from a single 100% usage and its starves enough resources from the iscsi that it breaks my connections.

This is with a backup running from a single computer.

This is right after I stopped the backup.

hippy1970 · September 1, 2016, 3:09pm

I’m running urbackup 2.0.33 server in a jail on freenas 9.10.1.
I’m idling while backing up 30+ clients simultaneously. Granted they’re file backups.

I am NOT using ISCSI. So maybe that’s where the problem lies…

SBTech · September 1, 2016, 9:41pm

Can you try to run an image backup :), see what your CPU does? I have no issues with with file backup.

I use the FreeNAS as an ISCSI target, not involved in the URBackups setup. URBackups is running in a jail on the FreeNAS so it has direct access to my storage rather then a VM over NFS or ISCSI.

I only have IO issues with the VM’s using FreeNAS over ISCSI (which is most of my VM’S) when urbackupsrv is running a t 100% of the CPU. local IO on the freenas server is fine.

hippy1970 · September 2, 2016, 8:05am

I’m running a full image backup of a W10 system right now.
The data is coming in at about 160Mbps, it’s on the LAN, and the CPU is averaging almost 97% idle. It also happens to be backing up 8 internet clients at the same time (file backup).

Are you using deduplication on the ZFS pool or encryption?
I only have compression running and rely on the urbackup deduplication.

The FreeNAS box here is running RAIDZ2 on 2 x 6 core AMD opterons with 48GB RAM and 15 x 3TB drives in groups of 5. Almost forgot, I have 2 x SSDs running as ZIL and L2ARC. Maybe that’s what helps…?

SBTech · September 2, 2016, 7:37pm

hippy1970 thanks! you pointed me in the right direction.

I setup both FreeNAS systems the same (vanila config). The one no-standard thing that I did do because it sounded good in my head was to use gzip level 9 compression on the URBackup datastore thinking that it would save me some space.

Well I switched back to LZ4 default and my CPU issues are gone.

hippy1970 · September 2, 2016, 7:54pm

That’s great news.
Glad I could be of help.