Server cpu stuck at 100%

server cpu is at 100%, backups don t seems to run

it didn’t go up to our alerting system because it was 100% cpu of a single core (the server is actually 36 cores ( x2 with hyperthreading). So it register as 3% usage in the monitoring.
I tried a service restart first. it zombified the service

unusual in the log files, i had a lof of :
2017-06-11 07:03:33: SQLITE_BUSY in CQuery::Execute Stmt: [BEGIN IMMEDIATE;]
2017-06-11 07:03:33: Active query(0): PRAGMA wal_checkpoint(PASSIVE)
2017-06-11 07:03:33: Active query(1): END;
2017-06-11 07:03:33: Active query(2): BEGIN IMMEDIATE;
2017-06-11 07:03:33: Active query(3): BEGIN IMMEDIATE;
2017-06-11 07:03:33: Active query(4): UPDATE settings_db.settings SET value=? WHERE key=? AND clientid=?
2017-06-11 07:03:33: Active query(5): BEGIN IMMEDIATE;
2017-06-11 07:03:33: Active query(6): BEGIN IMMEDIATE;
2017-06-11 07:03:33: Active query(7): PRAGMA wal_checkpoint(PASSIVE)

then a lot of
2017-06-11 07:18:07: SQLITE_BUSY in CQuery::Execute Stmt: [UPDATE settings_db.automatic_archival SET next_archival=? WHERE id=?]
2017-06-11 07:18:07: Active query(0): PRAGMA wal_checkpoint(PASSIVE)
2017-06-11 07:18:07: Active query(1): END;
2017-06-11 07:18:07: Active query(2): BEGIN IMMEDIATE;
2017-06-11 07:18:07: Active query(3): BEGIN IMMEDIATE;
2017-06-11 07:18:07: Active query(4): UPDATE settings_db.settings SET value=? WHERE key=? AND clientid=?
2017-06-11 07:18:07: Active query(5): BEGIN IMMEDIATE;
2017-06-11 07:18:07: Active query(6): BEGIN IMMEDIATE;
2017-06-11 07:18:07: Active query(7): PRAGMA wal_checkpoint(PASSIVE)
2017-06-11 07:18:07: Active query(8): UPDATE settings_db.automatic_archival SET next_archival=? WHERE id=?
2017-06-11 07:18:17: SQLITE_BUSY in CQuery::Execute Stmt: [UPDATE settings_db.automatic_archival SET next_archival=? WHERE id=?]

then a lof of (only the same 3 inernet clients)
2017-06-12 14:22:54: Authed+capa for client ‘serverA’ (encrypted-v2, compressed-v2, token auth) - 1 spare connections
2017-06-12 14:23:27: Authed+capa for client ‘serverB’ (encrypted-v2, compressed-v2, token auth) - 1 spare connections
2017-06-12 14:23:54: Authed+capa for client ‘serverA’ (encrypted-v2, compressed-v2, token auth) - 1 spare connections
2017-06-12 14:24:27: Authed+capa for client ‘serverC’ (encrypted-v2, compressed-v2, token auth) - 1 spare connections
2017-06-12 14:24:54: Authed+capa for client ‘severA’ (encrypted-v2, compressed-v2, token auth) - 1 spare connections

Backups don t seems to start, as it s in debug , usually ther e san entry for each file operation

i spoted some messages in dmesgs kernel logs

[dim. juin 11 06:52:14 2017] INFO: task files checkpoin:43945 blocked for more than 120 seconds.
[dim. juin 11 06:52:14 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:52:14 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:52:14 2017] files checkpoin D 0 43945 19389 0x00000100
[dim. juin 11 06:52:14 2017] Call Trace:
[dim. juin 11 06:52:14 2017] __schedule+0x22f/0x700
[dim. juin 11 06:52:14 2017] schedule+0x3d/0x90
[dim. juin 11 06:52:14 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:52:14 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:52:14 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:52:14 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:52:14 2017] ? tsd_set+0x31a/0x4f0 [spl]
[dim. juin 11 06:52:14 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:52:14 2017] zfs_fsync+0x77/0xf0 [zfs]
[dim. juin 11 06:52:14 2017] zpl_fsync+0x68/0xa0 [zfs]
[dim. juin 11 06:52:14 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:52:14 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:52:14 2017] SyS_fsync+0x10/0x20
[dim. juin 11 06:52:14 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:52:14 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:52:14 2017] RIP: 0033:0x7f03694364cd
[dim. juin 11 06:52:14 2017] RSP: 002b:00007f034cff8130 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[dim. juin 11 06:52:14 2017] RAX: ffffffffffffffda RBX: 00007f0330013c98 RCX: 00007f03694364cd
[dim. juin 11 06:52:14 2017] RDX: 000000000278b968 RSI: 0000000000000002 RDI: 00000000000007c1
[dim. juin 11 06:52:14 2017] RBP: 0000000000000005 R08: 0000000000000000 R09: 00007f03300fedf8
[dim. juin 11 06:52:14 2017] R10: 000000000000002d R11: 0000000000000293 R12: 00000000ffffffff
[dim. juin 11 06:52:14 2017] R13: 00007f036aefe010 R14: 0000000000000004 R15: 000000000003f15f
[dim. juin 11 06:52:14 2017] INFO: task fbackup write:37055 blocked for more than 120 seconds.
[dim. juin 11 06:52:14 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:52:14 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:52:14 2017] fbackup write D 0 37055 19389 0x00000100
[dim. juin 11 06:52:14 2017] Call Trace:
[dim. juin 11 06:52:14 2017] __schedule+0x22f/0x700
[dim. juin 11 06:52:14 2017] ? zfs_remove+0x49b/0x900 [zfs]
[dim. juin 11 06:52:14 2017] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[dim. juin 11 06:52:14 2017] schedule+0x3d/0x90
[dim. juin 11 06:52:14 2017] bit_wait+0x11/0x60
[dim. juin 11 06:52:14 2017] __wait_on_bit+0x58/0x90
[dim. juin 11 06:52:14 2017] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[dim. juin 11 06:52:14 2017] __inode_wait_for_writeback+0xad/0xf0
[dim. juin 11 06:52:14 2017] ? autoremove_wake_function+0x40/0x40
[dim. juin 11 06:52:14 2017] inode_wait_for_writeback+0x26/0x40
[dim. juin 11 06:52:14 2017] evict+0xb3/0x190
[dim. juin 11 06:52:14 2017] iput+0x1c6/0x250
[dim. juin 11 06:52:14 2017] do_unlinkat+0x187/0x300
[dim. juin 11 06:52:14 2017] SyS_unlink+0x16/0x20
[dim. juin 11 06:52:14 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:52:14 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:52:14 2017] RIP: 0033:0x7f0369160dc7
[dim. juin 11 06:52:14 2017] RSP: 002b:00007efb26ffc1b8 EFLAGS: 00000206 ORIG_RAX: 0000000000000057
[dim. juin 11 06:52:14 2017] RAX: ffffffffffffffda RBX: 00007efb26ffc1d0 RCX: 00007f0369160dc7
[dim. juin 11 06:52:14 2017] RDX: 00007efb6c1c5aa0 RSI: 00007efb6c1784c0 RDI: 00007efb6c1c5aa0
[dim. juin 11 06:52:14 2017] RBP: 00007efb26ffc850 R08: 00007efb6c102f80 R09: 0000000000000080
[dim. juin 11 06:52:14 2017] R10: 00030f333a6f219f R11: 0000000000000206 R12: 00007efb26ffc7f0
[dim. juin 11 06:52:14 2017] R13: 0000000000449bc0 R14: 00007efb26ffc2b0 R15: 00007efb26ffcbe0
[dim. juin 11 06:52:14 2017] INFO: task fbackup main:31812 blocked for more than 120 seconds.
[dim. juin 11 06:52:14 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:52:14 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:52:14 2017] fbackup main D 0 31812 19389 0x00000100
[dim. juin 11 06:52:14 2017] Call Trace:
[dim. juin 11 06:52:14 2017] __schedule+0x22f/0x700
[dim. juin 11 06:52:14 2017] ? wb_queue_work+0x88/0xf0
[dim. juin 11 06:52:14 2017] schedule+0x3d/0x90
[dim. juin 11 06:52:14 2017] wb_wait_for_completion+0x5f/0x90
[dim. juin 11 06:52:14 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:52:14 2017] sync_inodes_sb+0xa9/0x2a0
[dim. juin 11 06:52:14 2017] ? __writeback_inodes_sb_nr+0x96/0xe0
[dim. juin 11 06:52:14 2017] sync_filesystem+0x5c/0xa0
[dim. juin 11 06:52:14 2017] SyS_syncfs+0x3e/0x70
[dim. juin 11 06:52:14 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:52:14 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:52:14 2017] RIP: 0033:0x7f0369165a87
[dim. juin 11 06:52:14 2017] RSP: 002b:00007efb25febc48 EFLAGS: 00000213 ORIG_RAX: 0000000000000132
[dim. juin 11 06:52:14 2017] RAX: ffffffffffffffda RBX: 00000000000007fd RCX: 00007f0369165a87
[dim. juin 11 06:52:14 2017] RDX: 00000000000007fd RSI: 0000000000080000 RDI: 00000000000007fd
[dim. juin 11 06:52:14 2017] RBP: 00007efb25ffab50 R08: 00007efaf01bf090 R09: 0000000000000000
[dim. juin 11 06:52:14 2017] R10: 002a6606490f54c8 R11: 0000000000000213 R12: 00007efb25ffa840
[dim. juin 11 06:52:14 2017] R13: 000000000001600d R14: 0000000000000001 R15: 00007f027010d890
[dim. juin 11 06:52:14 2017] INFO: task fbackup main:49433 blocked for more than 120 seconds.
[dim. juin 11 06:52:14 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:52:14 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:52:14 2017] fbackup main D 0 49433 19389 0x00000100
[dim. juin 11 06:52:14 2017] Call Trace:
[dim. juin 11 06:52:14 2017] __schedule+0x22f/0x700
[dim. juin 11 06:52:14 2017] schedule+0x3d/0x90
[dim. juin 11 06:52:14 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:52:14 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:52:14 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:52:14 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:52:14 2017] ? tsd_set+0x31a/0x4f0 [spl]
[dim. juin 11 06:52:14 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:52:14 2017] zfs_fsync+0x77/0xf0 [zfs]
[dim. juin 11 06:52:14 2017] zpl_fsync+0x68/0xa0 [zfs]
[dim. juin 11 06:52:14 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:52:14 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:52:14 2017] SyS_fsync+0x10/0x20
[dim. juin 11 06:52:14 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:52:14 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:52:14 2017] RIP: 0033:0x7f03694364cd
[dim. juin 11 06:52:14 2017] RSP: 002b:00007efb827eb8d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[dim. juin 11 06:52:14 2017] RAX: ffffffffffffffda RBX: 00007efadc1be518 RCX: 00007f03694364cd
[dim. juin 11 06:52:14 2017] RDX: 0000000000001000 RSI: 0000000000000002 RDI: 000000000000083e
[dim. juin 11 06:52:14 2017] RBP: 00007efadc1bf1f8 R08: 00007efadc1be538 R09: 00007efb827eb9f0
[dim. juin 11 06:52:14 2017] R10: 00007efb827eb9f0 R11: 0000000000000293 R12: 000000001f837ee0
[dim. juin 11 06:52:14 2017] R13: 0000000000000000 R14: 0000000000000000 R15: 00007efadc346028
[dim. juin 11 06:52:14 2017] INFO: task fbackup main:17986 blocked for more than 120 seconds.
[dim. juin 11 06:52:14 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:52:14 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:52:14 2017] fbackup main D 0 17986 19389 0x00000100
[dim. juin 11 06:52:14 2017] Call Trace:
[dim. juin 11 06:52:14 2017] __schedule+0x22f/0x700
[dim. juin 11 06:52:14 2017] schedule+0x3d/0x90
[dim. juin 11 06:52:14 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:52:14 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:52:14 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:52:14 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:52:14 2017] ? tsd_set+0x31a/0x4f0 [spl]
[dim. juin 11 06:52:14 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:52:14 2017] zfs_fsync+0x77/0xf0 [zfs]
[dim. juin 11 06:52:14 2017] zpl_fsync+0x68/0xa0 [zfs]
[dim. juin 11 06:52:14 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:52:14 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:52:14 2017] SyS_fsync+0x10/0x20
[dim. juin 11 06:52:14 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:52:14 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:52:14 2017] RIP: 0033:0x7f03694364cd
[dim. juin 11 06:52:14 2017] RSP: 002b:00007efa827ec310 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[dim. juin 11 06:52:14 2017] RAX: ffffffffffffffda RBX: 00007efa700d1608 RCX: 00007f03694364cd
[dim. juin 11 06:52:14 2017] RDX: 0000000000000020 RSI: 0000000000000002 RDI: 00000000000008a5
[dim. juin 11 06:52:14 2017] RBP: 00007efa7001a9f8 R08: 00007efa700d1628 R09: 00007efa827ec3c4
[dim. juin 11 06:52:14 2017] R10: 00007efa827ec3f8 R11: 0000000000000293 R12: 0000000000000022
[dim. juin 11 06:52:14 2017] R13: 0000000000000000 R14: 00007efa827ec3c0 R15: 00007efa70081b18
[dim. juin 11 06:54:17 2017] INFO: task fileindex write:43935 blocked for more than 120 seconds.
[dim. juin 11 06:54:17 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:54:17 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:54:17 2017] fileindex write D 0 43935 19389 0x00000100
[dim. juin 11 06:54:17 2017] Call Trace:
[dim. juin 11 06:54:17 2017] __schedule+0x22f/0x700
[dim. juin 11 06:54:17 2017] schedule+0x3d/0x90
[dim. juin 11 06:54:17 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:54:17 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:54:17 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:54:17 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:54:17 2017] ? rrw_exit+0x62/0x140 [zfs]
[dim. juin 11 06:54:17 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:54:17 2017] zpl_writepages+0xd6/0x170 [zfs]
[dim. juin 11 06:54:17 2017] do_writepages+0x1e/0x30
[dim. juin 11 06:54:17 2017] __filemap_fdatawrite_range+0xc6/0x100
[dim. juin 11 06:54:17 2017] filemap_write_and_wait_range+0x2a/0x70
[dim. juin 11 06:54:17 2017] zpl_fsync+0x3c/0xa0 [zfs]
[dim. juin 11 06:54:17 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:54:17 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:54:17 2017] SyS_fdatasync+0x13/0x20
[dim. juin 11 06:54:17 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:54:17 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:54:17 2017] RIP: 0033:0x7f0369165a4d
[dim. juin 11 06:54:17 2017] RSP: 002b:00007f034fffe9e0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
[dim. juin 11 06:54:17 2017] RAX: ffffffffffffffda RBX: 00000000027b8350 RCX: 00007f0369165a4d
[dim. juin 11 06:54:17 2017] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000017
[dim. juin 11 06:54:17 2017] RBP: 00000000027a2dc0 R08: 0000000000000001 R09: 00007f03489f5ac0
[dim. juin 11 06:54:17 2017] R10: 000000000c627000 R11: 0000000000000293 R12: 00000000000000e6
[dim. juin 11 06:54:17 2017] R13: 00007f0348021638 R14: 00007f034fffea70 R15: 0000000000000000
[dim. juin 11 06:54:17 2017] INFO: task files checkpoin:43945 blocked for more than 120 seconds.
[dim. juin 11 06:54:17 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:54:17 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:54:17 2017] files checkpoin D 0 43945 19389 0x00000100
[dim. juin 11 06:54:17 2017] Call Trace:
[dim. juin 11 06:54:17 2017] __schedule+0x22f/0x700
[dim. juin 11 06:54:17 2017] schedule+0x3d/0x90
[dim. juin 11 06:54:17 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:54:17 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:54:17 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:54:17 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:54:17 2017] ? tsd_set+0x31a/0x4f0 [spl]
[dim. juin 11 06:54:17 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:54:17 2017] zfs_fsync+0x77/0xf0 [zfs]
[dim. juin 11 06:54:17 2017] zpl_fsync+0x68/0xa0 [zfs]
[dim. juin 11 06:54:17 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:54:17 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:54:17 2017] SyS_fsync+0x10/0x20
[dim. juin 11 06:54:17 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:54:17 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:54:17 2017] RIP: 0033:0x7f03694364cd
[dim. juin 11 06:54:17 2017] RSP: 002b:00007f034cff8130 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[dim. juin 11 06:54:17 2017] RAX: ffffffffffffffda RBX: 00007f0330013c98 RCX: 00007f03694364cd
[dim. juin 11 06:54:17 2017] RDX: 000000000278b968 RSI: 0000000000000002 RDI: 00000000000007c1
[dim. juin 11 06:54:17 2017] RBP: 0000000000000005 R08: 0000000000000000 R09: 00007f03300fedf8
[dim. juin 11 06:54:17 2017] R10: 000000000000002d R11: 0000000000000293 R12: 00000000ffffffff
[dim. juin 11 06:54:17 2017] R13: 00007f036aefe010 R14: 0000000000000004 R15: 000000000003f15f
[dim. juin 11 06:54:17 2017] INFO: task fbackup write:37055 blocked for more than 120 seconds.
[dim. juin 11 06:54:17 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:54:17 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:54:17 2017] fbackup write D 0 37055 19389 0x00000100
[dim. juin 11 06:54:17 2017] Call Trace:
[dim. juin 11 06:54:17 2017] __schedule+0x22f/0x700
[dim. juin 11 06:54:17 2017] ? zfs_remove+0x49b/0x900 [zfs]
[dim. juin 11 06:54:17 2017] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[dim. juin 11 06:54:17 2017] schedule+0x3d/0x90
[dim. juin 11 06:54:17 2017] bit_wait+0x11/0x60
[dim. juin 11 06:54:17 2017] __wait_on_bit+0x58/0x90
[dim. juin 11 06:54:17 2017] ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[dim. juin 11 06:54:17 2017] __inode_wait_for_writeback+0xad/0xf0
[dim. juin 11 06:54:17 2017] ? autoremove_wake_function+0x40/0x40
[dim. juin 11 06:54:17 2017] inode_wait_for_writeback+0x26/0x40
[dim. juin 11 06:54:17 2017] evict+0xb3/0x190
[dim. juin 11 06:54:17 2017] iput+0x1c6/0x250
[dim. juin 11 06:54:17 2017] do_unlinkat+0x187/0x300
[dim. juin 11 06:54:17 2017] SyS_unlink+0x16/0x20
[dim. juin 11 06:54:17 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:54:17 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:54:17 2017] RIP: 0033:0x7f0369160dc7
[dim. juin 11 06:54:17 2017] RSP: 002b:00007efb26ffc1b8 EFLAGS: 00000206 ORIG_RAX: 0000000000000057
[dim. juin 11 06:54:17 2017] RAX: ffffffffffffffda RBX: 00007efb26ffc1d0 RCX: 00007f0369160dc7
[dim. juin 11 06:54:17 2017] RDX: 00007efb6c1c5aa0 RSI: 00007efb6c1784c0 RDI: 00007efb6c1c5aa0
[dim. juin 11 06:54:17 2017] RBP: 00007efb26ffc850 R08: 00007efb6c102f80 R09: 0000000000000080
[dim. juin 11 06:54:17 2017] R10: 00030f333a6f219f R11: 0000000000000206 R12: 00007efb26ffc7f0
[dim. juin 11 06:54:17 2017] R13: 0000000000449bc0 R14: 00007efb26ffc2b0 R15: 00007efb26ffcbe0
[dim. juin 11 06:54:17 2017] INFO: task fbackup main:31812 blocked for more than 120 seconds.
[dim. juin 11 06:54:17 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:54:17 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:54:18 2017] fbackup main D 0 31812 19389 0x00000100
[dim. juin 11 06:54:18 2017] Call Trace:
[dim. juin 11 06:54:18 2017] __schedule+0x22f/0x700
[dim. juin 11 06:54:18 2017] ? wb_queue_work+0x88/0xf0
[dim. juin 11 06:54:18 2017] schedule+0x3d/0x90
[dim. juin 11 06:54:18 2017] wb_wait_for_completion+0x5f/0x90
[dim. juin 11 06:54:18 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:54:18 2017] sync_inodes_sb+0xa9/0x2a0
[dim. juin 11 06:54:18 2017] ? __writeback_inodes_sb_nr+0x96/0xe0
[dim. juin 11 06:54:18 2017] sync_filesystem+0x5c/0xa0
[dim. juin 11 06:54:18 2017] SyS_syncfs+0x3e/0x70
[dim. juin 11 06:54:18 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:54:18 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:54:18 2017] RIP: 0033:0x7f0369165a87
[dim. juin 11 06:54:18 2017] RSP: 002b:00007efb25febc48 EFLAGS: 00000213 ORIG_RAX: 0000000000000132
[dim. juin 11 06:54:18 2017] RAX: ffffffffffffffda RBX: 00000000000007fd RCX: 00007f0369165a87
[dim. juin 11 06:54:18 2017] RDX: 00000000000007fd RSI: 0000000000080000 RDI: 00000000000007fd
[dim. juin 11 06:54:18 2017] RBP: 00007efb25ffab50 R08: 00007efaf01bf090 R09: 0000000000000000
[dim. juin 11 06:54:18 2017] R10: 002a6606490f54c8 R11: 0000000000000213 R12: 00007efb25ffa840
[dim. juin 11 06:54:18 2017] R13: 000000000001600d R14: 0000000000000001 R15: 00007f027010d890
[dim. juin 11 06:54:18 2017] INFO: task fbackup main:49433 blocked for more than 120 seconds.
[dim. juin 11 06:54:18 2017] Tainted: P O 4.10.13-1-ARCH #1
[dim. juin 11 06:54:18 2017] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[dim. juin 11 06:54:18 2017] fbackup main D 0 49433 19389 0x00000100
[dim. juin 11 06:54:18 2017] Call Trace:
[dim. juin 11 06:54:18 2017] __schedule+0x22f/0x700
[dim. juin 11 06:54:18 2017] schedule+0x3d/0x90
[dim. juin 11 06:54:18 2017] cv_wait_common+0x126/0x140 [spl]
[dim. juin 11 06:54:18 2017] ? wake_atomic_t_function+0x60/0x60
[dim. juin 11 06:54:18 2017] __cv_wait+0x15/0x20 [spl]
[dim. juin 11 06:54:18 2017] zil_commit.part.7+0x86/0x840 [zfs]
[dim. juin 11 06:54:18 2017] ? tsd_set+0x31a/0x4f0 [spl]
[dim. juin 11 06:54:18 2017] zil_commit+0x17/0x20 [zfs]
[dim. juin 11 06:54:18 2017] zfs_fsync+0x77/0xf0 [zfs]
[dim. juin 11 06:54:18 2017] zpl_fsync+0x68/0xa0 [zfs]
[dim. juin 11 06:54:18 2017] vfs_fsync_range+0x4b/0xb0
[dim. juin 11 06:54:18 2017] do_fsync+0x3d/0x70
[dim. juin 11 06:54:18 2017] SyS_fsync+0x10/0x20
[dim. juin 11 06:54:18 2017] do_syscall_64+0x54/0xc0
[dim. juin 11 06:54:18 2017] entry_SYSCALL64_slow_path+0x25/0x25
[dim. juin 11 06:54:18 2017] RIP: 0033:0x7f03694364cd
[dim. juin 11 06:54:18 2017] RSP: 002b:00007efb827eb8d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[dim. juin 11 06:54:18 2017] RAX: ffffffffffffffda RBX: 00007efadc1be518 RCX: 00007f03694364cd
[dim. juin 11 06:54:18 2017] RDX: 0000000000001000 RSI: 0000000000000002 RDI: 000000000000083e
[dim. juin 11 06:54:18 2017] RBP: 00007efadc1bf1f8 R08: 00007efadc1be538 R09: 00007efb827eb9f0
[dim. juin 11 06:54:18 2017] R10: 00007efb827eb9f0 R11: 0000000000000293 R12: 000000001f837ee0
[dim. juin 11 06:54:18 2017] R13: 0000000000000000 R14: 0000000000000000 R15: 00007efadc346028
[lun. juin 12 14:30:19 2017] InternetService[44127]: segfault at 27972d0 ip 00000000027972d0 sp 00007f032bffed58 error 15

I have one of the servers (Windows2008R2) experienced somewhat similar symptoms, but the 100% was for memory though. Server was running good initially and backups were completing good for about 1 month.

One day, the server decided to stop working and I realized that memory was at 100%. I was forced to perform a server reboot and everything went back to normal.

Could be a memory leak somewhere for my case.