Virtual Appliance (hyperv) Server ran out of space. Now will not load web interface. Will not mount backup drive

infscape-appliance
1

I got this warning about freespace.

There is low free space on the appliance system volume (less than 1GB of free space).
Automatic action: Currently stopping backup processes and compressing system space to increase available free space.
Increase system volume size as soon as possible to guarantee error free functioning of the appliance.

Then the appliance when down the next day. When I rebooted (after freeing some space from apt-cache and increaseing the vm disk size) it said that there was an internal error starting up on the webinterface.

I ran a repair-database, defrage-database, remove unknown, and now I do not get an internal message warning, but it just forwards to wizard.htm or the UrBackup Appliance - Setup wizard.
It does not mount the backup’s

Please help.

3

Sorry you are having problems.

Could you have a look at the contents of /var/log/app.log ?

4

Thanks for taking the time to help. Sure thing!
app.log (926.6 KB)

5

If you look at the output of df -h or mount, do you see the /var/urbackup folder mounted? What does mount /var/urbackup say (and perhaps dmesg)?

6

df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.1G 13M 3.1G 1% /run
/dev/sda2 118G 64G 52G 56% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
tmpfs 16G 0 16G 0% /tmp
/dev/sda1 92M 64M 25M 73% /boot
/dev/sda2 118G 64G 52G 56% /var/urbackup
/dev/sda2 118G 64G 52G 56% /var/log
tmpfs 3.1G 0 3.1G 0% /run/user/1000

mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16032932k,nr_inodes=4008233,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3208796k,mode=755)
/dev/sda2 on / type btrfs (rw,noatime,compress-force=lzo,nossd,space_cache=v2,subvolid=11662,subvol=/sys_root_current)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16005)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/sda1 on /boot type ext2 (rw,noatime,nodiratime,errors=remount-ro)
/dev/sda2 on /var/urbackup type btrfs (rw,noatime,compress-force=lzo,nossd,space_cache=v2,subvolid=2630,subvol=/urbackup)
/dev/sda2 on /var/log type btrfs (rw,noatime,compress-force=lzo,nossd,space_cache=v2,subvolid=568,subvol=/log)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3208796k,mode=700,uid=1000,gid=1000)

I have seen the /media/backup/backups? I think in the past when the mount was correct, however its not mounting the backup drive. Reboots are really quick like there are things missing that I used to see at reboot.

7

No clue yet. Maybe /var/urbackup/app.db got damaged? Maybe by repair-database/defrag-database/maybe by the automatic compression. That is obviously not part of the appliance interface… For example deleting /var/urbackup/app.db-wal would cause those symptons.

Maybe you could try sqlite /var/urbackup/app.db then VACUUM; …

Before doing anything, create a snapshot:

mkdir -p /media/fs
mount /dev/sda2 /media/fs
btrfs subvol snap /media/fs/urbackup /media/fs/urbackup_old
8

Snapshot went well.
root@demi:/home/admin# mkdir -p /media/fs
root@demi:/home/admin# mount /dev/sda2 /media/fs
root@demi:/home/admin# btrfs subvol snap /media/fs/urbackup /media/fs/urbackup_o ld

Did a vacuum.
root@demi:/home/admin# sqlite3 /var/urbackup/app.db
SQLite version 3.16.2 2017-01-06 16:32:41
Enter “.help” for usage hints.
sqlite> VACUUM;
sqlite>

Rebooted Server and still the same thing.

9

What does

echo "SELECT * FROM misc; SELECT * FROM devices;" | sqlite3 /var/urbackup/app.db
echo "SELECT COUNT(*) FROM backups; SELECT COUNT(*) FROM backup_images;" | sqlite3 /var/urbackup/backup_server.db

say?

10

root@demi:/var/urbackup# echo “SELECT * FROM misc; SELECT * FROM devices;” | sqlite3 /var/urbackup/app.db
1|db_version|19
root@demi:/var/urbackup# echo “SELECT COUNT() FROM backups; SELECT COUNT() FROM backup_images;” | sqlite3 /var/urbackup/backup_server.db
2860
441

11

After

mkdir -p /media/fs
mount /dev/sda2 /media/fs

You probably don’t have a /media/fs/urbackup_ro_last, correct?

Do you have a backup of the encryption key, or was automatic upload enabled (you can check at https://app.urbackup.com/ )

12

root@demi:/media/fs# ls -l
total 0
drwxr-xr-x 1 root root 36 Apr 29 2020 cdcache
drwxr-xr-x 1 root root 4246 Nov 12 06:25 log
drwxr-xr-x 1 root root 3650 Sep 23 2019 sys_etc_v8.31
drwxr-xr-x 1 root root 3898 Oct 27 16:20 sys_etc_v9.21
drwxr-xr-x 1 root root 240 Nov 5 03:08 sys_root_current
drwxr-xr-x 1 root root 240 Sep 23 21:35 sys_root_last
drwxr-xr-x 1 root root 266 Mar 28 2020 sys_root_lastbak
drwxr-xr-x 1 root root 292 Sep 23 2019 sys_root_v8.31
drwxr-xr-x 1 root root 272 Nov 5 03:07 sys_root_v9.21
drwxr-xr-x 1 urbackup urbackup 333368 Nov 12 18:30 urbackup
drwxr-xr-x 1 urbackup urbackup 7020 May 4 2020 urbackup_before_import
drwxr-xr-x 1 root root 16 Nov 12 17:27 urbackup_old

I do have a backup of the key on the app.urbackup.com I belive I have always had to automatic upload enabled. Never knew about the site though. I was wondering about that.

14

So by default when I go to my urbackup webinterface. It forwards to the wizard, but never really loads anything. Just a blank page with the infoscape header on top.

15

Ok, then delete urbackup_before_import

btrfs subvol del /media/fs/urbackup_before_import/searchdb/*
btrfs subvol del /media/fs/urbackup_before_import

then follow the setup wizard, add the disk again (using the encryption key if it doesn’t retrieve it automatically). The danger is that it might start to delete backups that are not in the backed up database after import.

The danger is that that it starts removing unknown backups from backup storage after import+reboot (startup cleanup). So disable the server startup:

systemctl disable urbackup-server

Afterward, login again, stop the urbackup server, and copy all files except app.db* from urbackup_old to the new urbackup subvolume.

Untested…

mkdir -p /media/fs
mount /dev/sda2 /media/fs
systemctl stop urbackup-server
systemctl stop app-server
systemctl stop pystartup
cp /var/urbackup/app.db* /tmp/
btrfs subvol del /media/fs/urbackup
btrfs subvol snap /media/fs/urbackup_old /media/fs/urbackup
cp /tmp/app.db* /media/fs/urbackup/
#Check that backup db is properly in place >0 
echo "SELECT COUNT(*) FROM backups; SELECT COUNT(*) FROM backup_images;" | sqlite3 /media/fs/urbackup/backup_server.db
systemctl enable urbackup-server
init 6

Usually it’d just import the database backup from the backup storage but there is a bug where those don’t run if urbackup_before_import has subvolumes. Sorry about that. Will fix ASAP.

16

An alternative would perhaps be to copy it from an old backup on the backup storage:

systemctl stop urbackup-server
systemctl stop app-server
systemctl stop pystartup
cryptsetup luksOpen /dev/sdb1 backup_crypt
mount /dev/mapper/backup_crypt /media/backup
cp /media/backup/internal/DATETIME/app.db* /var/urbackup/*
init 6
17

I am unable to find **/media/backup/**internal…
after backup there is only
root@demi:/media/backup# systemctl stop urbackup-server
root@demi:/media/backup# systemctl stop app-server
root@demi:/media/backup# systemctl stop pystartup
root@demi:/media/backup# cryptsetup luksOpen /dev/sdb1 backup_crypt
Enter passphrase for /dev/sdb1:
root@demi:/media/backup# mount /dev/mapper/backup_crypt /media/backup
root@demi:/media/backup# ls -l

drwx------ 1 urbackup urbackup 888 Nov 9 02:46 backups
drwxr-x— 1 root root 60 Nov 9 02:46 clouddrive_cache
drwx------ 1 urbackup urbackup 38 Sep 13 03:06 meta
drwx------ 1 urbackup urbackup 0 Sep 12 23:06 tmp

18

No problem deleting urbackup_before_import. Stops me saying dir not empty.

root@demi:/media/backup/backups/urbackup_tmp_files# mkdir -p /media/fs

root@demi:/media/backup/backups/urbackup_tmp_files# mount /dev/sda2 /media/fs

root@demi:/media/backup/backups/urbackup_tmp_files# systemctl stop urbackup-server

root@demi:/media/backup/backups/urbackup_tmp_files# systemctl stop app-server

root@demi:/media/backup/backups/urbackup_tmp_files# systemctl stop pystartup

root@demi:/media/backup/backups/urbackup_tmp_files# cp /var/urbackup/app.db* /tmp/

root@demi:/media/backup/backups/urbackup_tmp_files# btrfs subvol del /media/fs/urbackup

Delete subvolume (no-commit): ‘/media/fs/urbackup’

ERROR: Could not destroy subvolume/snapshot: Directory not empty

19

Perhaps:

...
btrfs subvol del /media/fs/urbackup/searchdb/*
btrfs subvol del /media/fs/urbackup
btrfs subvol snap /media/fs/urbackup_old /media/fs/urbackup
cp /tmp/app.db* /media/fs/urbackup/
#Check that backup db is properly in place >0 
echo "SELECT COUNT(*) FROM backups; SELECT COUNT(*) FROM backup_images;" | sqlite3 /media/fs/urbackup/backup_server.db
systemctl enable urbackup-server
20

That snap went. had to add urbackup_old/urbackup for it to work. Rebooted still the wizard page with a balnk page and the infoscape header.
btrfs subvol snap /media/fs/urbackup_old/urbackup/ /media/fs/urbackup
Create a snapshot of ‘/media/fs/urbackup_old/urbackup/’ in ‘/media/fs/urbackup’

21

Did you do that?

22

then follow the setup wizard, add the disk again (using the encryption key if it doesn’t retrieve it automatically).

The setup wizard page is broken. It’s just blank.

It’s not impossible I didn’t systemctl disable urbackup-server, or I did just after starting the second time. However, I did enable it again when done. So I did disable it but maybe not as early as I should have.