Out of memory urbackup-server-2.1.14beta - Gentoo Linux

1

Feb 06 15:00:05 mimir systemd[1]: Started Timeline of Snapper Snapshots.
Feb 06 15:00:05 mimir dbus[291]: [system] Activating service name=‘org.opensuse.Snapper’ (using servicehelper)
Feb 06 15:00:05 mimir dbus[291]: [system] Successfully activated service ‘org.opensuse.Snapper’
Feb 06 15:16:29 mimir fail2ban.actions[346]: NOTICE [sshd] Unban 59.63.188.3
Feb 06 15:17:35 mimir smartd[294]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 118 to 117
Feb 06 15:17:36 mimir smartd[294]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 171 to 166
Feb 06 15:17:37 mimir smartd[294]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 117 to 116
Feb 06 15:25:48 mimir fail2ban.actions[346]: NOTICE [sshd] Unban 175.144.150.77
Feb 06 15:47:35 mimir smartd[294]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 117 to 118
Feb 06 15:47:35 mimir smartd[294]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 166 to 171
Feb 06 15:47:37 mimir smartd[294]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 116 to 117
Feb 06 15:57:13 mimir systemd[1]: Started OpenSSH per-connection server daemon (113.108.21.16:44240).
Feb 06 15:57:13 mimir sshd[31885]: Connection from 113.108.21.16 port 44240 on 10.1.0.1 port 22
Feb 06 15:57:13 mimir sshd[31885]: Did not receive identification string from 113.108.21.16 port 44240
Feb 06 16:00:25 mimir systemd[1]: Started Timeline of Snapper Snapshots.
Feb 06 16:00:25 mimir dbus[291]: [system] Activating service name=‘org.opensuse.Snapper’ (using servicehelper)
Feb 06 16:00:25 mimir dbus[291]: [system] Successfully activated service ‘org.opensuse.Snapper’
Feb 06 16:01:47 mimir systemd[1]: Started OpenSSH per-connection server daemon (60.191.38.77:54692).
Feb 06 16:01:47 mimir sshd[31905]: Connection from 60.191.38.77 port 54692 on 10.1.0.1 port 22
Feb 06 16:01:47 mimir sshd[31905]: Did not receive identification string from 60.191.38.77 port 54692
Feb 06 16:02:43 mimir kernel: kthreadd invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), order=2, oom_score_adj=0
Feb 06 16:02:43 mimir kernel: kthreadd cpuset=/ mems_allowed=0
Feb 06 16:02:43 mimir kernel: CPU: 0 PID: 2 Comm: kthreadd Not tainted 4.7.10-hardened #8
Feb 06 16:02:43 mimir kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E350M1, BIOS P2.10 03/06/2014
Feb 06 16:02:43 mimir kernel: 0000000000000006 ffffffff8132dd0b 0000000000000006 ffff88042c9b7d20
Feb 06 16:02:43 mimir kernel: ffff8803ee32f500 ffffffff8111d265 0000000000000003 ffff880400000015
Feb 06 16:02:43 mimir kernel: ffff88043ec148d0 ffff88043effac00 ffffffff81c78c40 ffff8803ee32f500
Feb 06 16:02:43 mimir kernel: Call Trace:
Feb 06 16:02:43 mimir kernel: [] ? dump_stack+0x47/0x63
Feb 06 16:02:43 mimir kernel: [] ? dump_header+0x5b/0x1da
Feb 06 16:02:43 mimir kernel: [] ? oom_kill_process+0x93/0x30b
Feb 06 16:02:43 mimir kernel: [] ? has_ns_capability_noaudit+0x2a/0x39
Feb 06 16:02:43 mimir kernel: [] ? out_of_memory+0x367/0x397
Feb 06 16:02:43 mimir kernel: [] ? __alloc_pages_nodemask+0x9c3/0xa97
Feb 06 16:02:43 mimir kernel: [] ? alloc_loc_track+0x4f/0x5a
Feb 06 16:02:43 mimir kernel: [] ? alloc_kmem_pages_node+0x1f/0x5e
Feb 06 16:02:43 mimir kernel: [] ? copy_process.part.44+0x11a/0x1653
Feb 06 16:02:43 mimir kernel: [] ? update_load_avg+0x231/0x279
Feb 06 16:02:43 mimir kernel: [] ? init_completion+0x1d/0x1d
Feb 06 16:02:43 mimir kernel: [] ? set_next_entity+0x3f/0x62
Feb 06 16:02:43 mimir kernel: [] ? __switch_to+0x27b/0x46f
Feb 06 16:02:43 mimir kernel: [] ? _do_fork+0xb5/0x257
Feb 06 16:02:43 mimir kernel: [] ? __schedule+0x317/0x4c3
Feb 06 16:02:43 mimir kernel: [] ? kernel_thread+0x20/0x21
Feb 06 16:02:43 mimir kernel: [] ? kthreadd+0x13a/0x17e
Feb 06 16:02:43 mimir kernel: [] ? ret_from_fork+0x1e/0x40
Feb 06 16:02:43 mimir kernel: [] ? kthread_stop+0xd3/0xd3
Feb 06 16:02:43 mimir kernel: Mem-Info:
Feb 06 16:02:43 mimir kernel: active_anon:43216 inactive_anon:52449 isolated_anon:0
active_file:2807600 inactive_file:543290 isolated_file:32
unevictable:0 dirty:334466 writeback:3250 unstable:0
slab_reclaimable:487611 slab_unreclaimable:42568
mapped:39681 shmem:83 pagetables:1533 bounce:0
free:38107 free_pcp:274 free_cma:0
Feb 06 16:02:43 mimir kernel: Node 0 DMA free:15896kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15896kB mlo
Feb 06 16:02:43 mimir kernel: lowmem_reserve[]: 0 2703 15746 15746
Feb 06 16:02:43 mimir kernel: Node 0 DMA32 free:73496kB min:11588kB low:14484kB high:17380kB active_anon:29164kB inactive_anon:31100kB active_file:1283264kB inactive_file:620812kB unevictable:0kB isolated(anon):0kB isolated(file):0kB prese
Feb 06 16:02:43 mimir kernel: lowmem_reserve[]: 0 0 13043 13043
Feb 06 16:02:43 mimir kernel: Node 0 Normal free:61596kB min:55924kB low:69904kB high:83884kB active_anon:143700kB inactive_anon:178696kB active_file:9947880kB inactive_file:1552428kB unevictable:0kB isolated(anon):0kB isolated(file):128kB
Feb 06 16:02:43 mimir kernel: lowmem_reserve[]: 0 0 0 0
Feb 06 16:02:43 mimir kernel: Node 0 DMA: 24kB (U) 28kB (U) 216kB (U) 132kB (U) 364kB (U) 2128kB (U) 0256kB 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15896kB
Feb 06 16:02:43 mimir kernel: Node 0 DMA32: 154754kB (UE) 5278kB (UH) 516kB (H) 432kB (H) 564kB (H) 3128kB (H) 2256kB (H) 1512kB (H) 01024kB 02048kB 04096kB = 68052kB
Feb 06 16:02:43 mimir kernel: Node 0 Normal: 148114kB (UME) 2978kB (UMH) 4916kB (UH) 432kB (UH) 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 62532kB
Feb 06 16:02:43 mimir kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 06 16:02:43 mimir kernel: 3353010 total pagecache pages
Feb 06 16:02:43 mimir kernel: 530 pages in swap cache
Feb 06 16:02:43 mimir kernel: Swap cache stats: add 3755808, delete 3755278, find 1157666/1664200
Feb 06 16:02:43 mimir kernel: Free swap = 1022888kB
Feb 06 16:02:43 mimir kernel: Total swap = 1048572kB
Feb 06 16:02:43 mimir kernel: 4119994 pages RAM
Feb 06 16:02:43 mimir kernel: 0 pages HighMem/MovableOnly
Feb 06 16:02:43 mimir kernel: 84252 pages reserved
Feb 06 16:02:43 mimir kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 06 16:02:43 mimir kernel: [ 185] 0 185 8245 233 18 3 123 -1000 systemd-udevd
Feb 06 16:02:43 mimir kernel: [ 286] 109 286 25556 0 20 3 70 0 systemd-timesyn
Feb 06 16:02:43 mimir kernel: [ 291] 101 291 9346 381 23 3 43 -900 dbus-daemon
Feb 06 16:02:43 mimir kernel: [ 293] 107 293 9569 54 16 3 27 0 systemd-network
Feb 06 16:02:43 mimir kernel: [ 294] 0 294 3603 194 12 4 80 0 smartd
Feb 06 16:02:43 mimir kernel: [ 296] 0 296 5968 59 17 3 28 0 systemd-logind
Feb 06 16:02:43 mimir kernel: [ 311] 108 311 6177 35 17 3 28 0 systemd-resolve
Feb 06 16:02:43 mimir kernel: [ 346] 0 346 158182 1776 132 3 1093 0 fail2ban-server
Feb 06 16:02:43 mimir kernel: [28519] 0 28519 31200 58 14 3 61 0 agetty
Feb 06 16:02:43 mimir kernel: [ 6631] 0 6631 31812 5449 65 3 31 0 systemd-journal
Feb 06 16:02:43 mimir kernel: [ 7103] 0 7103 15193 401 35 3 219 0 apache2
Feb 06 16:02:43 mimir kernel: [ 7116] 81 7116 185739 1001 70 4 237 0 apache2
Feb 06 16:02:43 mimir kernel: [ 7117] 81 7117 218507 959 72 4 262 0 apache2
Feb 06 16:02:43 mimir kernel: [ 7179] 0 7179 72561 502 88 3 296 0 nmbd
Feb 06 16:02:43 mimir kernel: [ 7181] 0 7181 85125 573 115 3 403 0 smbd
Feb 06 16:02:43 mimir kernel: [ 7189] 118 7189 391163 121710 559 5 3463 0 urbackupsrv
Feb 06 16:02:43 mimir kernel: [24614] 0 24614 86730 882 118 3 352 0 smbd
Feb 06 16:02:43 mimir kernel: [31103] 0 31103 88415 3139 121 3 178 0 smbd
Feb 06 16:02:43 mimir kernel: Out of memory: Kill process 7189 (urbackupsrv) score 29 or sacrifice child
Feb 06 16:02:43 mimir kernel: Killed process 7189 (urbackupsrv) total-vm:1564652kB, anon-rss:363928kB, file-rss:122912kB, shmem-rss:0kB
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Main process exited, code=killed, status=9/KILL
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Unit entered failed state.
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Failed with result ‘signal’.

Last few messages from urbackup-server on the system log (these overlap with the above. Different filter)

Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/Local Settings” to "/media/raid/backups/urbackup/Lesl
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/My Documents” to "/media/raid/backups/urbackup/Leslie
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/NetHood” to "/media/raid/backups/urbackup/Leslies-Lap
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/PrintHood” to "/media/raid/backups/urbackup/Leslies-L
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/Recent” to "/media/raid/backups/urbackup/Leslies-Lapt
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/SendTo” to "/media/raid/backups/urbackup/Leslies-Lapt
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/Start Menu” to "/media/raid/backups/urbackup/Leslies-
Feb 06 14:54:25 mimir urbackupsrv[7189]: 2017-02-06 14:54:25: ERROR: Cannot copy directory metadata from “/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Users/jonesmz/Templates” to "/media/raid/backups/urbackup/Leslies-L
Feb 06 14:55:42 mimir urbackupsrv[7189]: 2017-02-06 14:55:42: ERROR: Cannot copy directory metadata from "/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Windows/SysWOW64/config/systemprofile/AppData/Local/Microsoft/Windo
Feb 06 14:55:51 mimir urbackupsrv[7189]: 2017-02-06 14:55:51: ERROR: Cannot copy directory metadata from "/media/raid/backups/urbackup/Leslies-Laptop/170206-0846/.hashes/C/Windows/System32/config/systemprofile/AppData/Local/Microsoft/Windo
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Main process exited, code=killed, status=9/KILL
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Unit entered failed state.
Feb 06 16:02:43 mimir systemd[1]: urbackup-server.service: Failed with result ‘signal’.
Feb 06 18:45:06 mimir systemd[1]: Stopped UrBackup backup service.

System info:
mimir ~ # emerge --info urbackup-server
Portage 2.3.3 (python 3.4.5-final-0, hardened/linux/amd64/no-multilib, gcc-4.9.4, glibc-2.23-r3, 4.7.10-hardened x86_64)

                     System Settings

=================================================================
System uname: Linux-4.7.10-hardened-x86_64-AMD_E-350D_APU_with_Radeon-tm-_HD_Graphics-with-gentoo-2.3
KiB Mem: 16142968 total, 174652 free
KiB Swap: 1048572 total, 1036960 free
Timestamp of repository gentoo: Fri, 03 Feb 2017 18:42:50 +0000
sh bash 4.3_p48-r1
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
app-shells/bash: 4.3_p48-r1::gentoo
dev-lang/perl: 5.22.3_rc4::gentoo
dev-lang/python: 2.7.12::gentoo, 3.4.5::gentoo
dev-util/cmake: 3.7.2::gentoo
dev-util/pkgconfig: 0.28-r2::gentoo
sys-apps/baselayout: 2.3::gentoo
sys-apps/openrc: 0.22.4::gentoo
sys-apps/sandbox: 2.10-r1::gentoo
sys-devel/autoconf: 2.69::gentoo
sys-devel/automake: 1.11.6-r1::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils: 2.25.1-r1::gentoo
sys-devel/gcc: 4.9.4::gentoo
sys-devel/gcc-config: 1.7.3::gentoo
sys-devel/libtool: 2.4.6-r2::gentoo
sys-devel/make: 4.2.1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc: 2.23-r3::gentoo
Repositories:

gentoo
location: /usr/portage
sync-type: git
sync-uri: https://github.com/gentoo-mirror/gentoo
priority: -1000

jonesmz-public-overlay
location: /usr/portage-overlays/jonesmz-public-overlay
sync-type: git
sync-uri: https://github.com/jonesmz/gentoo-overlay.git
masters: gentoo

Installed sets: @pc-base-system, @portage
ACCEPT_KEYWORDS=“amd64”
ACCEPT_LICENSE="* -@EULA"
CBUILD=“x86_64-pc-linux-gnu”
CFLAGS="-O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
CHOST=“x86_64-pc-linux-gnu”
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -O2 -pipe -march=x86-64 -mtune=generic -O2 -pipe"
DISTDIR="/usr/portage-distfiles"
EMERGE_DEFAULT_OPTS=" --jobs --keep-going --newuse --deep --backtrack=3000 --complete-graph --with-bdeps=y"
FCFLAGS="-O2 -pipe"
FEATURES=“assume-digests binpkg-logs buildpkg clean-logs compress-build-logs compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles installsources merge-sync news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms split-elog split-log splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr”
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS=“http://distfiles.gentoo.org”
LANG=“en_US.utf8”
LDFLAGS="-Wl,-O1 -Wl,–as-needed"
MAKEOPTS="-j1"
PKGDIR="/usr/portage-packages"
PORTAGE_COMPRESS=“xz”
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="–recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE=“acl amd64 avahi bzip2 clang cli cracklib crypt cxx dbus dri gd gdbm gudev hardened iconv ipv6 justify lm_sensors modules ncurses nls nptl openmp pam pax_kernel pcre pie python readline samba seccomp session ssl ssp systemd tcpd threads udisks unicode urandom xattr xtpax zeroconf zlib” ABI_X86=“64” ALSA_CARDS=“ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci” APACHE2_MODULES=“authn_core authz_core authz_host dir mime unixd socache_shmcb info log_config” CALLIGRA_FEATURES=“kexi words flow plan sheets stage tables krita karbon braindump author” COLLECTD_PLUGINS=“df interface irq load memory rrdtool swap syslog” CPU_FLAGS_X86=“mmx sse sse2 mmxext” ELIBC=“glibc” GPSD_PROTOCOLS=“ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx” GRUB_PLATFORMS=“coreboot efi-64 emu qemu pc” INPUT_DEVICES=“libinput” KERNEL=“linux” L10N=“en en-US” LCD_DEVICES=“bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text” LIBREOFFICE_EXTENSIONS=“presenter-console presenter-minimizer” LINGUAS=“en en_US” OFFICE_IMPLEMENTATION=“libreoffice” PHP_TARGETS=“php5-6” PYTHON_SINGLE_TARGET=“python2_7” PYTHON_TARGETS=“python2_7 python3_4” QEMU_SOFTMMU_TARGETS=“arm aarch64 x86_64” QEMU_USER_TARGETS=“arm aarch64 x86_64” RUBY_TARGETS=“ruby21 ruby21” USERLAND=“GNU” VIDEO_CARDS=“r600 radeon radeonsi amdgpu vesa modesetting fbdev qxl” XTABLES_ADDONS=“quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account”
Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
Package Settings

app-backup/urbackup-server-2.1.14::jonesmz-public-overlay was built with the following:
USE=“crypt zlib -fuse -hardened -mail”

This version of urbackup server was built with

Which is a modified / updated copy of Gentoo ebuild (WIP)

mimir ~ # cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 20
model : 2
model name : AMD E-350D APU with Radeon™ HD Graphics
stepping : 0
microcode : 0x500010d
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt hw_pstate vmmcall arat npt lbrv svm_lock nrip_save pausefilter
bugs : fxsave_leak sysret_ss_attrs null_seg
bogomips : 3200.09
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 1
vendor_id : AuthenticAMD
cpu family : 20
model : 2
model name : AMD E-350D APU with Radeon™ HD Graphics
stepping : 0
microcode : 0x500010d
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 6
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt hw_pstate vmmcall arat npt lbrv svm_lock nrip_save pausefilter
bugs : fxsave_leak sysret_ss_attrs null_seg
bogomips : 3200.02
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

mimir ~ # btrfs fi usage /media/raid
Overall:
Device size: 29.10TiB
Device allocated: 20.62TiB
Device unallocated: 8.48TiB
Device missing: 0.00B
Used: 17.92TiB
Free (estimated): 5.59TiB (min: 5.59TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)

Data,RAID1: Size:10.28TiB, Used:8.94TiB
/dev/sda2 1.65TiB
/dev/sdb2 1.65TiB
/dev/sdc2 2.56TiB
/dev/sdd2 2.56TiB
/dev/sde2 6.00TiB
/dev/sdf2 1.66TiB
/dev/sdg2 2.81TiB
/dev/sdh2 1.65TiB

Metadata,RAID1: Size:29.00GiB, Used:25.98GiB
/dev/sda2 3.00GiB
/dev/sdb2 4.00GiB
/dev/sdc2 2.00GiB
/dev/sdd2 3.00GiB
/dev/sde2 29.00GiB
/dev/sdf2 2.00GiB
/dev/sdg2 12.00GiB
/dev/sdh2 3.00GiB

System,RAID1: Size:32.00MiB, Used:1.53MiB
/dev/sde2 32.00MiB
/dev/sdf2 32.00MiB

Unallocated:
/dev/sda2 1.07TiB
/dev/sdb2 1.07TiB
/dev/sdc2 1.07TiB
/dev/sdd2 1.07TiB
/dev/sde2 1.25TiB
/dev/sdf2 1.07TiB
/dev/sdg2 831.02GiB
/dev/sdh2 1.07TiB

(Taken when making this report, not when urbackup-server ran out of memory)
mimir ~ # cat /proc/meminfo
MemTotal: 16142968 kB
MemFree: 178968 kB
MemAvailable: 15457436 kB
Buffers: 104 kB
Cached: 12477948 kB
SwapCached: 2076 kB
Active: 10878528 kB
Inactive: 1645624 kB
Active(anon): 21884 kB
Inactive(anon): 25004 kB
Active(file): 10856644 kB
Inactive(file): 1620620 kB
Unevictable: 0 kB
Mlocked: 44 kB
SwapTotal: 1048572 kB
SwapFree: 1036912 kB
Dirty: 304 kB
Writeback: 0 kB
AnonPages: 45132 kB
Mapped: 59764 kB
Shmem: 544 kB
Slab: 3372812 kB
SReclaimable: 3139472 kB
SUnreclaim: 233340 kB
KernelStack: 3792 kB
PageTables: 4524 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 9120056 kB
Committed_AS: 757652 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
AnonHugePages: 4096 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 16492 kB
DirectMap2M: 4929536 kB
DirectMap1G: 11534336 kB

I don’t see any options to export a debug log file from the server’s WebUI.

Are there relevant data or log files to track the leak down?

For reference, I have 7 clients.

Image backups are disabled

None of the clients have more than a few TB of harddrive space.

2

Memory leak?
Can’t tell more without debug info.

3

I’m no expert in debugging crashes, but i think this might have some value:

Feb 06 16:02:43 mimir kernel: lowmem_reserve: 0 0 0 0

I think this has something to do with the kernel running out of memory. I had a similar crash yesterday with server 2.0.38 and I think my issue was solved with:

vm.zone_reclaim_mode=1

and/or

vm.min_free_kbytes=1048576

I’m not sure which one helped since I implemented both at the same time… So far no crashes…

4

UrBackup is only using 363MB of your 16GB memory so it’s not UrBackup’s fault.

I’d use the newest kernel (4.9.x) as there were some memory reclaim issues since 4.4. There were some mailing list threads about this. This seems to be a work-around: https://lkml.org/lkml/2016/11/29/667

Could also be low-mem issues like Greg said.

If it occurs with the latest release it may help to post it to the mm@ and btrfs@ kernel lists.

5

I had this too, I fixed it by limiting the dirty-ratio (the amount of memory the kernel can use for ‘should write to disk’ cache). The default is 20% and 20% of 16GB == 3.2GB of data which isn’t freed fast enough when it’s needed. I did:

sysctl -w vm.dirty_background_ratio = 1
sysctl -w vm.dirty_ratio = 2

Haven’t had crashes so far.

6

Try to cron a drop caches.

Because as uroni said, since this summer there are heavy issue in linux memory management, where the memory get fragmented, then it won’t get freed or compacted when memory is requested. Issue is amplified on cow filesystemes (btrfs, zfs).
Don’t trust anything you see about this bug being solved, it has been solved/re-opened at least 3 times sine june/july.

Typically you’d get an error with an order 2 memory request (your logs shows ones)

I have this issue on all my urbackup servers , i run an hourly cron job to drop caches this solves the issue.

7

@orogor:
Can you please publish your cron file - the commands you have used?