Incremental Image Bug causing Kernel Panic

#1

Hi there! I’ve been using UrBackup for a long time. I have an incremental-forever ZFS COW image scheme set up. It has been running fine for months, but recently deletions of old backups during cleanup start causing kernel panics. I believe it is a bug in ZFS that has to do with the amount of clones that UrBackup uses:

https://github.com/zfsonlinux/zfs/issues/3959

I’m trying to delete a number of more recent incrementals (only very old incrementals seem to be affected) to see if that resolves the issue. I’m not sure if there’s a way to address this in UrBackup’s implementation.

Additionally, when deleting old backups, remnant snapshots are left over (I’m not sure if I can just delete these manually?). The backup that causes a kernel panic is always the oldest one:

rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190223-0227_Image_C      746M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190224-0230_Image_C      579M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190225-0235_Image_C      751M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190226-0238_Image_C      795M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190227-0242_Image_C      667M      -  29.4G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190228-0246_Image_C      756M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190303-0102_Image_C      688M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@ro                      10.7K      -  29.2G  -

Kernel stack trace example below:

Depth    Size   Location    (177 entries)
-----    ----   --------
  0)    15128      64   __zone_watermark_ok+0x5/0x150
  1)    15064     312   get_page_from_freelist+0x2c4/0x1400
  2)    14752     104   __alloc_pages_nodemask+0x11c/0x2c0
  3)    14648      48   alloc_pages_current+0x6a/0xe0
  4)    14600     112   new_slab+0x2ab/0x670
  5)    14488     192   ___slab_alloc+0x357/0x4b0
  6)    14296      32   __slab_alloc+0x20/0x40
  7)    14264      16   __kmalloc_node+0xbe/0x2c0
  8)    14248     128   spl_kmem_zalloc+0xdc/0x1a0 [spl]
  9)    14120     128   dbuf_dirty+0x20d/0x850 [zfs]
 10)    13992      56   dnode_setdirty+0xa7/0x100 [zfs]
 11)    13936     128   dbuf_dirty+0x3d2/0x850 [zfs]
 12)    13808      48   dmu_buf_will_dirty+0x11c/0x130 [zfs]
 13)    13760     200   zap_lockdir_impl+0x2ef/0x780 [zfs]
 14)    13560      88   zap_lockdir+0x8b/0xb0 [zfs]
 15)    13472     120   zap_update+0x62/0x210 [zfs]
 16)    13352      88   zap_update_int_key+0x66/0x90 [zfs]
 17)    13264      72   dle_enqueue_subobj.isra.5+0xa9/0x140 [zfs]
 18)    13192     200   dsl_deadlist_remove_key+0xce/0x180 [zfs]
 19)    12992      80   dsl_dataset_remove_clones_key.isra.2+0x1be/0x1f0 [zfs]
 20)    12912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 21)    12832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 22)    12752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 23)    12672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 24)    12592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 25)    12512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 26)    12432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 27)    12352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 28)    12272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 29)    12192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 30)    12112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 31)    12032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 32)    11952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 33)    11872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 34)    11792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 35)    11712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 36)    11632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 37)    11552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 38)    11472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 39)    11392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 40)    11312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 41)    11232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 42)    11152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 43)    11072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 44)    10992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 45)    10912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 46)    10832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 47)    10752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 48)    10672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 49)    10592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 50)    10512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 51)    10432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 52)    10352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 53)    10272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 54)    10192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 55)    10112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 56)    10032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 57)     9952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 58)     9872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 59)     9792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 60)     9712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 61)     9632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 62)     9552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 63)     9472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 64)     9392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 65)     9312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 66)     9232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 67)     9152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 68)     9072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 69)     8992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 70)     8912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 71)     8832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 72)     8752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 73)     8672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 74)     8592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 75)     8512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 76)     8432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 77)     8352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 78)     8272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 79)     8192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 80)     8112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 81)     8032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 82)     7952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 83)     7872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 84)     7792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 85)     7712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 86)     7632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 87)     7552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 88)     7472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 89)     7392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 90)     7312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 91)     7232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 92)     7152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 93)     7072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 94)     6992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 95)     6912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 96)     6832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 97)     6752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 98)     6672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 99)     6592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
100)     6512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
101)     6432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
102)     6352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
103)     6272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
104)     6192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
105)     6112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
106)     6032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
107)     5952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
108)     5872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
109)     5792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
110)     5712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
111)     5632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
112)     5552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
113)     5472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
114)     5392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
115)     5312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
116)     5232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
117)     5152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
118)     5072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
119)     4992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
120)     4912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
121)     4832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
122)     4752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
123)     4672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
124)     4592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
125)     4512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
126)     4432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
127)     4352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
128)     4272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
129)     4192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
130)     4112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
131)     4032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
132)     3952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
133)     3872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
134)     3792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
135)     3712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
136)     3632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
137)     3552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
138)     3472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
139)     3392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
140)     3312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
141)     3232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
142)     3152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
143)     3072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
144)     2992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
145)     2912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
146)     2832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
147)     2752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
148)     2672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
149)     2592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
150)     2512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
151)     2432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
152)     2352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
153)     2272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
154)     2192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
155)     2112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
156)     2032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
157)     1952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
158)     1872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
159)     1792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
160)     1712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
161)     1632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
162)     1552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
163)     1472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
164)     1392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
165)     1312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
166)     1232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
167)     1152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
168)     1072     152   dsl_destroy_snapshot_sync_impl+0x353/0xba0 [zfs]
169)      920      72   dsl_destroy_snapshot_sync+0x66/0x130 [zfs]
170)      848      48   dsl_sync_task_sync+0xb2/0x120 [zfs]
171)      800     120   dsl_pool_sync+0x302/0x430 [zfs]
172)      680     200   spa_sync+0x43e/0xd80 [zfs]
173)      480     184   txg_sync_thread+0x2cd/0x4a0 [zfs]
174)      296      48   thread_generic_wrapper+0x74/0x90 [spl]
175)      248      72   kthread+0x121/0x140
176)      176     176   ret_from_fork+0x35/0x40
#2

Hi,

I’m using the same scheme, what are your versions of OS / Kernel / UrBackup / ZFS ?

#3 
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

4.15.0-62-generic

ii urbackup-server                       2.3.8.0-1ubuntu1~bionic                     amd64

ii  libzfs2linux                          0.7.5-1ubuntu16.6                           amd64
ii  zfs-zed                               0.7.5-1ubuntu16.6                           amd64
ii  zfsutils-linux                        0.7.5-1ubuntu16.6                           amd64
#4

FYI, i dont have this problem for now, but my versions are newer :

debian version 9.9
ii  libzfs2linux                        0.7.12-2+deb10u1~bpo9+1           amd64        OpenZFS filesystem library for Linux
ii  zfs-dkms                            0.7.12-2+deb10u1~bpo9+1           all          OpenZFS filesystem kernel modules for Linux
ii  zfs-zed                             0.7.12-2+deb10u1~bpo9+1           amd64        OpenZFS Event Daemon
ii  zfsutils-linux                      0.7.12-2+deb10u1~bpo9+1           amd64        command-line tools to manage OpenZFS filesystems

Kernel : Linux version 4.19.0-0.bpo.5-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Debian 4.19.37-4~bpo9+1 (2019-06-19)

ii  urbackup-server                     2.3.8.0-1                         amd64

Maybe an issue fixed in ZFS since 0.7.5 ?

Regards,

#5

Hi.

Have you found a solution for this?
Running into same problems after 4 months of running Urbackup with incremental-forever ZFS.

I have almost same config as you:
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
4.15.0-66-generic

urbackup-server 2.4.11.0
libzfs2linux 0.7.5-1ubuntu16.6
zfsutils-linux 0.7.5-1ubuntu16.6
zfs-zed 0.7.5-1ubuntu16.6

#6

I never truly fixed it, but have found some reasonable options. This is a bug in ZFS that I think was permanently fixed in version 0.8.0-rc1: https://github.com/zfsonlinux/zfs/commit/c434d8806cfdf33a4a0b0ab24fb8e35a7b6811ee

Option A: You can upgrade to Ubuntu 19.10 (or later) which includes zfs 0.8.1 … I’m assuming this would fix the problem, but I haven’t tested it. In theory, anything earlier (even 19.04) would NOT work.

I did not do this because I like the LTS versions of Ubuntu (and I can’t get livepatch service on non-LTS distributions). The next LTS is still like a year away.

Option B (doesn’t work): I tried to build a modern version of ZFS on 18.04 LTS but this wasn’t really feasible due to many missing package dependencies and my concern for damaging the Ubuntu installation and affecting ability to do future upgrades. Plus there would be a need to recompile ZFS with every new kernel, which is annoying.

Option C: The kernel panic only seems to appear during cleanup when UrBackup deletes the oldest snapshot. Manually deleting the oldest snapshot also usually reproduces the problem. What you can do instead is to delete a bunch (20-30) of younger snapshots (a few weeks future to the oldest one) using the UrBackup web interface manually and the issue goes away temporarily after you’ve removed enough: the oldest snapshots are now removable without a kernel panic, and you’ve freed some storage space so UrBackup isn’t even trying to do that on routine cleanups.