I think I have the same problem on the same setting of synology+docker+urbackup.
Image backups are failing to autocleanup, leading to space exaustion.
The shared directory for backups is btrfs, the backup image file format is RAW copy-on-write.
I get the same ioctl error here:
root@diskstation:/# urbackup_snapshot_helper test
Testing for btrfs…
Create subvolume ‘/backups/testA54hj5luZtlorr494/A’
Create a snapshot of ‘/backups/testA54hj5luZtlorr494/A’ in ‘/backups/testA54hj5luZtlorr494/B’
ERROR: cannot snapshot ‘/backups/testA54hj5luZtlorr494/A’: Inappropriate ioctl for device
TEST FAILED: Creating test snapshot failed
Delete subvolume (commit): ‘/backups/testA54hj5luZtlorr494/A’
ERROR: cannot access subvolume /backups/testA54hj5luZtlorr494/B: No such file or directory
TEST FAILED: Removing subvolume B failed
(does this mean that I should not use RAW copy-on-write?)
and when performing a manual cleanup even with user root, urbackupsrv doesnt manage to cleanup any image, but I get plenty of lines like this:
2019-11-03 15:09:23: Deleting image backup ( id=3196, path=/backups/6005-1/191015-0022_Image_SYSVOL/Image_SYSVOL_191015-0022.raw ) …
Unknown mode: -1
2019-11-03 15:09:23: Deleting image backup failed.
There seems to be no problem with file backups.
From other experiences, setting the backup image to VHD should make image backups work again.