[SOLVED] LVM snapshot across filesystems trouble

apagr · July 1, 2017, 4:05pm

I’m trying to backup linux client using snapshots. My installation of Jira and Confluence create enough traffic for indexing to fail because of constant file changes. Snapshots seem like holy grail to solve the problem, so I turned it on in snapshot.cfg. Together with volumes_mounted_locally=0 as it tried to snapshot thing like /proc and obviously failed.

My partition layout:

    /dev/mapper/wolumen-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    /dev/mapper/wolumen-var on /var type ext4 (rw,relatime,data=ordered)
    /dev/mapper/wolumen-tmp on /tmp type ext4 (rw,relatime,data=ordered)
    /dev/mapper/wolumen-home on /home type ext4 (rw,relatime,data=ordered)

Separate lv volumes in my case. The installation of Jira (and Confluence) is spread across filesystems:

/opt/jira 
/home/jira

So as far as I understand - urbackup client should create 2 snapshots, on two logical filesystems, and backup all files.
Ok, here’s my list-backupdirs:

root@jira:~# urbackupclientctl list-backupdirs
PATH                     NAME            FLAGS
------------------------ --------------- ------------------------------------------------------------------------------
/opt/jira                opt_jira        follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/home/jira               home_jira       follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/home/backup             home_backup     follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/home/confluence         home_confluence follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/opt/atlassian           opt_atlassian   follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/var/backups             var_backups     follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/var/spool/cron/crontabs crontabs        follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/etc                     etc             follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes
/root                    root            follow_symlinks,symlinks_optional,one_filesystem,require_snapshot,share_hashes

and the error:

2017-07-01 16:30:58: Indexing of "opt_jira" done. 10129 filesystem lookups 0 db lookups and 0 db updates
(I'm good here - / is snapshotted and indexed corectly)
2017-07-01 16:30:58: Creating shadowcopy of "home_jira" in indexDirs()
2017-07-01 16:30:58: orig_target=/home/jira volpath=/mnt/urbackup_snaps/70487ddbbf70c289aeecb93894a976903d8a0d1932522584
2017-07-01 16:30:58: Shadowcopy already present.
(wrong, /home is not the same snap, and we fail later on)
2017-07-01 16:30:58: done.
2017-07-01 16:30:58: Indexing "home_jira"...
2017-07-01 16:30:58: ERROR: Cannot access path to backup: "/mnt/urbackup_snaps/70487ddbbf70c289aeecb93894a976903d8a0d1932522584/home/jira" Errorcode: 2 - No such file or directory
2017-07-01 16:30:58: Deleting shadowcopy for path "/mnt/urbackup_snaps/70487ddbbf70c289aeecb93894a976903d8a0d1932522584/home/jira" -2
2017-07-01 16:30:58: Unmounting /dev/mapper/wolumen-urbackup_snap_70487ddbbf70c289aeecb93894a976903d8a0d1932522584 at /mnt/urbackup_snaps/70487ddbbf70c289aeecb93894a976903d8a0d1932522584...
2017-07-01 16:30:58: Destroying LVM snapshot /dev/mapper/wolumen-urbackup_snap_70487ddbbf70c289aeecb93894a976903d8a0d1932522584...
2017-07-01 16:30:58: Logical volume "urbackup_snap_70487ddbbf70c289aeecb93894a976903d8a0d1932522584" successfully removed
2017-07-01 16:30:58: Deleting Shadowcopy for dir "/"
2017-07-01 16:30:58: Script "/usr/local/etc/urbackup/postfileindex" does not exist
2017-07-01 16:30:58: Async index 9961b45165164dd243a9c60021dc3c9d finished with "error - index error"

referencing snapshot, that is /mnt/urbackup_snaps/70487ddbbf70c289aeecb93894a976903d8a0d1932522584
was created for /dev/mapper/wolumen-root (double checked during indexing process).
Again, as I understand - the client errously identified /home/jira as backup location belonging to the same snapshot, but since this snapshot was created for / not for /home - it fails.
Im my case all volumes should be snapshotted and either client should detect mountpoint or link subdirectories to recreate filesystem structure under /mnt/urbackup_snaps

Or is something I do not understand when using lvm snapshots?

uroni · July 1, 2017, 4:19pm

The currently undocumented volumes_mounted_locally=0 is for backing up paths that are not mounted locally, e.g. if you run the client in a hypervisor and you mount only the snapshot of a volume. I don’t think this is the case here? Removing that should enable the logic which determines same volumes.

When you are backing up via snapshots it shouldn’t even see the contents of /proc and therefore there shouldn’t be any problems there? Maybe you can add more details on that problem?

apagr · July 1, 2017, 4:46pm

ok, removed undocumented options and got this (by adding -x to script header)

0-1498926694-Following symbolic link at "/etc/systemd/system/timers.target.wants/certbot.timer" to "/lib/systemd/system/certbot.timer" confirms symlink backup target ".symlink_certbot.timer" to "/lib/systemd/system/certbot.timer"
0-1498926694-Indexing of "etc" done. 229 filesystem lookups 0 db lookups and 3 db updates
0-1498926694-Indexing of "root" done. 20 filesystem lookups 0 db lookups and 1 db updates
2-1498926694-Creating snapshot of "/run/network" failed
2-1498926694-+ set -e
2-1498926694-+ SNAP_SIZE=-l50%FREE
2-1498926694-++ dirname /usr/local/share/urbackup/lvm_create_filesystem_snapshot
2-1498926694-+ CDIR=/usr/local/share/urbackup
2-1498926694-+ mkdir -p /mnt/urbackup_snaps
2-1498926694-+ SNAP_ID=12c1bbe489915af6f9d7d4852a9f48fd4cfa80e14411ac06
2-1498926694-+ SNAP_MOUNTPOINT=/run
2-1498926694-+ SNAP_DEST=/mnt/urbackup_snaps/12c1bbe489915af6f9d7d4852a9f48fd4cfa80e14411ac06
2-1498926694-+ lsblk -r --output NAME,MOUNTPOINT --paths
2-1498926694-++ lsblk -r --output NAME,MOUNTPOINT --paths
2-1498926694-++ tr -s ' '
2-1498926694-++ egrep ' /run$$'
2-1498926694-++ cut '-d ' -f1
2-1498926694-++ head -n 1
2-1498926694-+ VOLNAME=
2-1498926694-++ df -T -P
2-1498926694-++ tr -s ' '
2-1498926694-++ egrep ' /run$$'
2-1498926694-++ cut '-d ' -f2
2-1498926694-++ head -n 1
2-1498926694-+ TYPE=tmpfs
2-1498926694-+ '[' xtmpfs = x ']'
2-1498926694-+ '[' xtmpfs = xbtrfs ']'
2-1498926694-+ '[' x = x ']'
2-1498926694-+ echo 'Could not find LVM volume for mountpoint /run'
2-1498926694-Could not find LVM volume for mountpoint /run
2-1498926694-+ exit 1
0-1498926694-Backing up ".symlink_network" without snapshot.
2-1498926694-Creating snapshot of "/proc/524/mounts" failed
2-1498926694-+ set -e
2-1498926694-+ SNAP_SIZE=-l50%FREE
2-1498926694-++ dirname /usr/local/share/urbackup/lvm_create_filesystem_snapshot
2-1498926694-+ CDIR=/usr/local/share/urbackup
2-1498926694-+ mkdir -p /mnt/urbackup_snaps
2-1498926694-+ SNAP_ID=b49b8d8ec3087ca5b585542bf3523d38d158a630c142a67e
2-1498926694-+ SNAP_MOUNTPOINT=/proc
2-1498926694-+ SNAP_DEST=/mnt/urbackup_snaps/b49b8d8ec3087ca5b585542bf3523d38d158a630c142a67e
2-1498926694-+ lsblk -r --output NAME,MOUNTPOINT --paths
2-1498926694-++ lsblk -r --output NAME,MOUNTPOINT --paths
2-1498926694-++ tr -s ' '
2-1498926694-++ egrep ' /proc$$'
2-1498926694-++ cut '-d ' -f1
2-1498926694-++ head -n 1
2-1498926694-+ VOLNAME=
2-1498926694-++ df -T -P
2-1498926694-++ tr -s ' '
2-1498926694-++ egrep ' /proc$$'
2-1498926694-++ cut '-d ' -f2
2-1498926694-++ head -n 1
2-1498926694-+ TYPE=
2-1498926694-+ '[' x = x ']'
2-1498926694-+ btrfs subvolume list -o /proc
2-1498926694-+ '[' x = xbtrfs ']'
2-1498926694-+ '[' x = x ']'
2-1498926694-+ echo 'Could not find LVM volume for mountpoint /proc'
2-1498926694-Could not find LVM volume for mountpoint /proc
2-1498926694-+ exit 1
0-1498926694-Backing up ".symlink_mounts_1" without snapshot.
0-1498926695-jira: Loading file list...
0-1498926695-jira: Calculating file tree differences...
0-1498926695-jira: Calculating tree difference size...
0-1498926696-jira: Linking unchanged and loading new files...

and got this:

Could not find LVM volume for mountpoint /proc
+ exit 1 (obviously, due to set -e)

apagr · July 1, 2017, 4:50pm

that’s strange… the mountpoints are searched with lsbkl,
here’s output of this :

root@jira:~# lsblk -r --output NAME,MOUNTPOINT --paths
NAME MOUNTPOINT
/dev/sr0
/dev/vda
/dev/vda1
/dev/mapper/wolumen-root /
/dev/mapper/wolumen-swap [SWAP]
/dev/mapper/wolumen-var /var
/dev/mapper/wolumen-tmp /tmp
/dev/mapper/wolumen-home /home
root@jira:~#

no /proc and /run but these are present in script debug output.

apagr · July 1, 2017, 5:17pm

digging a bit in debug found this:

2017-07-01 19:04:45: Creating shadowcopy of ".symlink_70-no-bitmaps.conf" in indexDirs()
2017-07-01 19:04:45: orig_target=/usr/share/fontconfig/conf.avail/70-no-bitmaps.conf volpath=/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974
2017-07-01 19:04:45: Shadowcopy already present.
2017-07-01 19:04:45: done.
2017-07-01 19:04:45: Indexing ".symlink_70-no-bitmaps.conf"...
2017-07-01 19:04:45: Hashing file "/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974/usr/share/fontconfig/conf.avail/70-no-bitmaps.conf"
2017-07-01 19:04:45: Creating shadowcopy of ".symlink_80-delicious.conf" in indexDirs()
2017-07-01 19:04:45: orig_target=/usr/share/fontconfig/conf.avail/80-delicious.conf volpath=/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974
2017-07-01 19:04:45: Shadowcopy already present.
2017-07-01 19:04:45: done.
2017-07-01 19:04:45: Indexing ".symlink_80-delicious.conf"...
2017-07-01 19:04:45: Hashing file "/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974/usr/share/fontconfig/conf.avail/80-delicious.conf"
2017-07-01 19:04:45: Creating shadowcopy of ".symlink_90-synthetic.conf" in indexDirs()
2017-07-01 19:04:45: orig_target=/usr/share/fontconfig/conf.avail/90-synthetic.conf volpath=/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974
2017-07-01 19:04:45: Shadowcopy already present.
2017-07-01 19:04:45: done.
2017-07-01 19:04:45: Indexing ".symlink_90-synthetic.conf"...
2017-07-01 19:04:45: Hashing file "/mnt/urbackup_snaps/3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974/usr/share/fontconfig/conf.avail/90-synthetic.conf"
2017-07-01 19:04:45: Creating shadowcopy of ".symlink_network" in indexDirs()
2017-07-01 19:04:45: ERROR: Creating snapshot of "/run/network" failed
2017-07-01 19:04:45: ERROR: + set -e
2017-07-01 19:04:45: ERROR: + SNAP_SIZE=-l50%FREE
2017-07-01 19:04:45: ERROR: ++ dirname /usr/local/share/urbackup/lvm_create_filesystem_snapshot
2017-07-01 19:04:45: ERROR: + CDIR=/usr/local/share/urbackup
2017-07-01 19:04:45: ERROR: + mkdir -p /mnt/urbackup_snaps
2017-07-01 19:04:45: ERROR: + SNAP_ID=5ddfbdc77f40bee6b6d0c432047596bbcb577209461ad006
2017-07-01 19:04:45: ERROR: + SNAP_MOUNTPOINT=/run
2017-07-01 19:04:45: ERROR: + SNAP_DEST=/mnt/urbackup_snaps/5ddfbdc77f40bee6b6d0c432047596bbcb577209461ad006
2017-07-01 19:04:45: ERROR: + case $SNAP_MOUNTPOINT in

which made me to find non-obvious symlink to /run/network

root@jira:/mnt/urbackup_snaps# ls -l /etc/network/run
lrwxrwxrwx 1 root root 12 kwi 20 12:27 /etc/network/run -> /run/network

and the same with proc:

2017-07-01 19:04:46: Creating shadowcopy of ".symlink_mounts_1" in indexDirs()
2017-07-01 19:04:46: ERROR: Creating snapshot of "/proc/524/mounts" failed
2017-07-01 19:04:46: ERROR: + set -e
2017-07-01 19:04:46: ERROR: + SNAP_SIZE=-l50%FREE
2017-07-01 19:04:46: ERROR: ++ dirname /usr/local/share/urbackup/lvm_create_filesystem_snapshot
2017-07-01 19:04:46: ERROR: + CDIR=/usr/local/share/urbackup
2017-07-01 19:04:46: ERROR: + mkdir -p /mnt/urbackup_snaps
2017-07-01 19:04:46: ERROR: + SNAP_ID=0414dd87d0e5bc6868aee7bcaf0e4617bb6905dca3470f65
2017-07-01 19:04:46: ERROR: + SNAP_MOUNTPOINT=/proc
2017-07-01 19:04:46: ERROR: + SNAP_DEST=/mnt/urbackup_snaps/0414dd87d0e5bc6868aee7bcaf0e4617bb6905dca3470f65
2017-07-01 19:04:46: ERROR: + case $SNAP_MOUNTPOINT in

guess what:

root@jira:/mnt/urbackup_snaps# find . -type l -exec ls -l  {} \; | grep proc | grep mounts
lrwxrwxrwx 1 root root 12 kwi 20 12:36 ./3cb914896f5e9f3ce9b32c1a5cbea7a21361979c61149974/etc/mtab -> /proc/mounts
root@jira:/mnt/urbackup_snaps#

and yes, no snapshot for these.

uroni · July 1, 2017, 5:24pm

I guess either disable following symlinks for those paths or exclude the files would fix it.

uroni · July 1, 2017, 5:30pm

I guess it should also not backup those files automatically because the symlink target is already automatically excluded (/proc) – bug

apagr · July 2, 2017, 4:52pm

I’d rather exclude these targets and it works. So, excluding: mtab and run does the job.
Probably disabling follow_symlinks would also work, but I’d rather have this option on.

I guess this kind of behaviour should be considered a bug, as symlinks to /proc /sys /dev and possibly other special dirs may possibly exists and cause backup to exit with non-obvious error. That’s why volumes_mounted_locally=0 seemed what I needed after first encounter of “unable to create snapshot for /proc”.

Thank you for supporting this case.