Incremental Image Bug causing Kernel Panic

Hi there! I’ve been using UrBackup for a long time. I have an incremental-forever ZFS COW image scheme set up. It has been running fine for months, but recently deletions of old backups during cleanup start causing kernel panics. I believe it is a bug in ZFS that has to do with the amount of clones that UrBackup uses:

https://github.com/zfsonlinux/zfs/issues/3959

I’m trying to delete a number of more recent incrementals (only very old incrementals seem to be affected) to see if that resolves the issue. I’m not sure if there’s a way to address this in UrBackup’s implementation.

Additionally, when deleting old backups, remnant snapshots are left over (I’m not sure if I can just delete these manually?). The backup that causes a kernel panic is always the oldest one:

rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190223-0227_Image_C      746M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190224-0230_Image_C      579M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190225-0235_Image_C      751M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190226-0238_Image_C      795M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190227-0242_Image_C      667M      -  29.4G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190228-0246_Image_C      756M      -  29.3G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@190303-0102_Image_C      688M      -  29.2G  -
rpool/var/urbackup_files/images/HTPC/190304-0106_Image_C@ro                      10.7K      -  29.2G  -

Kernel stack trace example below:

Depth    Size   Location    (177 entries)
-----    ----   --------
  0)    15128      64   __zone_watermark_ok+0x5/0x150
  1)    15064     312   get_page_from_freelist+0x2c4/0x1400
  2)    14752     104   __alloc_pages_nodemask+0x11c/0x2c0
  3)    14648      48   alloc_pages_current+0x6a/0xe0
  4)    14600     112   new_slab+0x2ab/0x670
  5)    14488     192   ___slab_alloc+0x357/0x4b0
  6)    14296      32   __slab_alloc+0x20/0x40
  7)    14264      16   __kmalloc_node+0xbe/0x2c0
  8)    14248     128   spl_kmem_zalloc+0xdc/0x1a0 [spl]
  9)    14120     128   dbuf_dirty+0x20d/0x850 [zfs]
 10)    13992      56   dnode_setdirty+0xa7/0x100 [zfs]
 11)    13936     128   dbuf_dirty+0x3d2/0x850 [zfs]
 12)    13808      48   dmu_buf_will_dirty+0x11c/0x130 [zfs]
 13)    13760     200   zap_lockdir_impl+0x2ef/0x780 [zfs]
 14)    13560      88   zap_lockdir+0x8b/0xb0 [zfs]
 15)    13472     120   zap_update+0x62/0x210 [zfs]
 16)    13352      88   zap_update_int_key+0x66/0x90 [zfs]
 17)    13264      72   dle_enqueue_subobj.isra.5+0xa9/0x140 [zfs]
 18)    13192     200   dsl_deadlist_remove_key+0xce/0x180 [zfs]
 19)    12992      80   dsl_dataset_remove_clones_key.isra.2+0x1be/0x1f0 [zfs]
 20)    12912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 21)    12832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 22)    12752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 23)    12672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 24)    12592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 25)    12512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 26)    12432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 27)    12352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 28)    12272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 29)    12192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 30)    12112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 31)    12032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 32)    11952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 33)    11872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 34)    11792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 35)    11712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 36)    11632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 37)    11552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 38)    11472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 39)    11392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 40)    11312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 41)    11232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 42)    11152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 43)    11072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 44)    10992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 45)    10912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 46)    10832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 47)    10752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 48)    10672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 49)    10592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 50)    10512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 51)    10432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 52)    10352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 53)    10272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 54)    10192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 55)    10112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 56)    10032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 57)     9952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 58)     9872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 59)     9792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 60)     9712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 61)     9632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 62)     9552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 63)     9472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 64)     9392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 65)     9312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 66)     9232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 67)     9152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 68)     9072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 69)     8992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 70)     8912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 71)     8832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 72)     8752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 73)     8672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 74)     8592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 75)     8512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 76)     8432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 77)     8352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 78)     8272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 79)     8192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 80)     8112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 81)     8032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 82)     7952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 83)     7872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 84)     7792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 85)     7712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 86)     7632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 87)     7552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 88)     7472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 89)     7392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 90)     7312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 91)     7232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 92)     7152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 93)     7072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 94)     6992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 95)     6912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 96)     6832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 97)     6752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 98)     6672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
 99)     6592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
100)     6512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
101)     6432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
102)     6352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
103)     6272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
104)     6192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
105)     6112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
106)     6032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
107)     5952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
108)     5872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
109)     5792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
110)     5712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
111)     5632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
112)     5552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
113)     5472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
114)     5392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
115)     5312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
116)     5232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
117)     5152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
118)     5072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
119)     4992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
120)     4912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
121)     4832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
122)     4752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
123)     4672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
124)     4592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
125)     4512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
126)     4432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
127)     4352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
128)     4272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
129)     4192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
130)     4112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
131)     4032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
132)     3952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
133)     3872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
134)     3792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
135)     3712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
136)     3632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
137)     3552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
138)     3472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
139)     3392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
140)     3312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
141)     3232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
142)     3152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
143)     3072      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
144)     2992      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
145)     2912      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
146)     2832      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
147)     2752      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
148)     2672      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
149)     2592      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
150)     2512      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
151)     2432      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
152)     2352      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
153)     2272      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
154)     2192      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
155)     2112      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
156)     2032      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
157)     1952      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
158)     1872      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
159)     1792      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
160)     1712      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
161)     1632      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
162)     1552      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
163)     1472      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
164)     1392      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
165)     1312      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
166)     1232      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
167)     1152      80   dsl_dataset_remove_clones_key.isra.2+0x1d4/0x1f0 [zfs]
168)     1072     152   dsl_destroy_snapshot_sync_impl+0x353/0xba0 [zfs]
169)      920      72   dsl_destroy_snapshot_sync+0x66/0x130 [zfs]
170)      848      48   dsl_sync_task_sync+0xb2/0x120 [zfs]
171)      800     120   dsl_pool_sync+0x302/0x430 [zfs]
172)      680     200   spa_sync+0x43e/0xd80 [zfs]
173)      480     184   txg_sync_thread+0x2cd/0x4a0 [zfs]
174)      296      48   thread_generic_wrapper+0x74/0x90 [spl]
175)      248      72   kthread+0x121/0x140
176)      176     176   ret_from_fork+0x35/0x40

Hi,

I’m using the same scheme, what are your versions of OS / Kernel / UrBackup / ZFS ?

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

4.15.0-62-generic

ii urbackup-server                       2.3.8.0-1ubuntu1~bionic                     amd64

ii  libzfs2linux                          0.7.5-1ubuntu16.6                           amd64
ii  zfs-zed                               0.7.5-1ubuntu16.6                           amd64
ii  zfsutils-linux                        0.7.5-1ubuntu16.6                           amd64

FYI, i dont have this problem for now, but my versions are newer :

debian version 9.9
ii  libzfs2linux                        0.7.12-2+deb10u1~bpo9+1           amd64        OpenZFS filesystem library for Linux
ii  zfs-dkms                            0.7.12-2+deb10u1~bpo9+1           all          OpenZFS filesystem kernel modules for Linux
ii  zfs-zed                             0.7.12-2+deb10u1~bpo9+1           amd64        OpenZFS Event Daemon
ii  zfsutils-linux                      0.7.12-2+deb10u1~bpo9+1           amd64        command-line tools to manage OpenZFS filesystems

Kernel : Linux version 4.19.0-0.bpo.5-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Debian 4.19.37-4~bpo9+1 (2019-06-19)

ii  urbackup-server                     2.3.8.0-1                         amd64

Maybe an issue fixed in ZFS since 0.7.5 ?

Regards,