Mac OS 10.13 HighSierra (APFS?) client fails - Sierra works

[Renato3]

Hello, fellows.

After some testing I can confirm that the Mac OS client really has a problem if it runs on Mac OS 10.13 High Sierra with APFS. I can not confirm if APFS is the problem or Mac OS 10.13 itself, as I only have 10.13 clients on APFS and one 10.12 client on a regular HFS harddisk.

On 10.12, the client runs perfectly, backups and restores are fine!

However, on 10.13 it fails to backup correctly - it looks like only the file metadata is transferred, BUT NOT THE FILES THEMSELVES. This symptom fools the web UI, everything looks perfect (although you might notice that making a full 40GB backup in 1 minute might be a bit fast for a server running on a raspberry pi ;-). However, if you try to restore files, you discover that alle files are empty. You’ll end up with files having the right size but filles with zero’s…

So, who is the Mac OS client ‘chef sache’? We need an update

Grtz,
R

[uroni]

Thanks for the heads up. You could help by uploading/sending a dtruss trace of a backup, as the issue is probably caused by some system calls not working as expected. For example HPFS did not support sparse files, but the client supports it (for Linux). If the sparse file calls do not work like in Linux there could be a problem where it detects the whole file as empty.

Another thing to do would be to integrate it with APFS snapshots.

[Renato3]

Hi @uroni, do you have a pointer on how to make a dtruss trace? Is this on the client or on the server? I never heard of it

Grtz,
R.

[Renato3]

Hey, @uroni!

I have done a 1 file full backup (look for the Simply Red mp3 file in the DTRUSS trace I cut out the DTRUSS info - here it is…I hope you can filter the problem. My guess is that there is some nice info and errors in it

Thanks in advance - keep up the good work!

BTW, it is an rtf file but I renamed it to a .txt.

Regard,
R.
dtruss-urbackup-simply-red-mp3.txt (58,5 KB)

[orogor]

trace: error on enabled probe ID 2165 (ID 956: syscall::write_nocancel:return): invalid kernel access in action #13 at DIF offset 92\

Looks like there a dtrace protection :

Error on enabled probe: syscall::open_nocancel:entry): invalid user access in action #2 at DIF

[Renato3]

So, does the file contain any hints?

It might be that there are some dtrace blockades in MacOS, but that there is still some usable information in the DTRUSS dump.

Let me know!

Grtz,
R.

[uroni]

Yes, the sparse file reading is messed up.

Relevant from the trace

5200/0x659d3: open("/Users/rene/Documents/UrBackupTest/Simply Red - Love Gave Me More.mp3\0", 0x1000000, 0x0) = 43 0
5200/0x63740: lstat64("/Users/rene/Documents/UrBackupTest/Simply Red - Love Gave Me More.mp3\0", 0x7000103984F0, 0x0) = 0 0
5200/0x659d3: fstat64(0x2B, 0x70000FEFCFE0, 0x0) = 0 0
5200/0x659d3: lseek(0x2B, 0x0, 0x4) = 0 0
5200/0x63740: lstat64("/Users/rene/Documents/UrBackupTest/Simply Red - Love Gave Me More.mp3\0", 0x7000103984A8, 0x0) = 0 0
5200/0x659d3: lseek(0x2B, 0x0, 0x3) = 7887081 0
5200/0x63740: stat64("/Users/rene/Documents/UrBackupTest/Simply Red - Love Gave Me More.mp3\0", 0x7000103984B8, 0x0) = 0 0
5200/0x659d3: lseek(0x2B, 0x7858E9, 0x4) = -1 Err#6\

Somehow this is the reverse from the Linux/POSIX API ( https://github.com/bminor/glibc/blob/bf7730194fed694a9ce821c306683266a5a7b78b/posix/unistd.h#L319 ). Does anyone have a support contract or something and can ask Apple about this?

[Falko_Trojahn]

Hi @uroni,
we have an Apple developer account in our company, not sure if this could help here somehow,
but I could try for sure.

Just can confirm now, too, that backups of files from Macs with 10.13 seem successful - but on restore,
all files which have been backed up after Upgrade to High Sierra … contain only zeros.
So this is a really high risk problem for us.

AFAIK the maintainers of Carbon Copy Cloner seem to have a lot of knowledge about APFS, e.g.

Preparing for the upgrade to macOS High Sierra | Carbon Copy Cloner | Bombich...

may be they answer your questions.

Apple doc about APFS seems not so much …

Introduction

Best regards,
Falko

[uroni]

Fixing this is trivial – just remove the sparse file support from the Apple client.

I was hinting at the other option – Apple fixing the API such that it works the same as in Linux/FreeBSD, because there is no reason to make it inverse.

[Falko_Trojahn]

That’d be great for now - so we have a reliable backup again. Are you sure this is only a sparse file problem?
When I made my restore tests, there were zip files containing zeros, too - and IMHO these are no sparse files. Or can this be another problem?

So, how could I test this, can we try the next client beta with stable server, or is it needed to upgrade both server and client to beta versions?

Thank you very much and have a nice sunday.

[Falko_Trojahn]

How to Skip Converting to APFS When Installing macOS High Sierra

may be a workaround for now. But no way back to HFS+ when on high sierra already …?

[uroni]

I don’t want to upgrade the mac to test on APFS because I have heard that MacOS 10.13 does not work in decembers

[Falko_Trojahn]

So how can we backup already upgraded High-Sierra-Macs using UrBackup atm?

[uroni]

I’ll publish a fixed version soon but won’t be able to test it.

Wrt. APFS: Can someone look into creating a script like this: https://raw.githubusercontent.com/uroni/urbackup_backend/dev/linux_snapshot/btrfs_create_filesystem_snapshot

And again I cannot find any documentation about this.

[Falko_Trojahn]

Well, found the following things for now that seem related:

Restoring from a snapshot with APFS — Mac Lovin'

Testing out snapshots in Apple’s next-generation APFS file system

But there’s no apfs_snapshot utility like mentioned in the last link:

bash-3.2# ls -l /System/Library/Filesystems/apfs.fs/Contents/Resources/
total 6240
-rwxr-xr-x 1 root wheel 526784 Oct 25 18:37 apfs.util
-rwxr-xr-x 1 root wheel 529952 Oct 25 18:37 apfs_invert
-rwxr-xr-x 1 root wheel 1271328 Oct 25 18:37 apfs_preflight_converter
-rwxr-xr-x 1 root wheel 24704 Oct 25 18:37 apfs_stats
-rwxr-xr-x 1 root wheel 694560 Oct 25 18:37 fsck_apfs
-rwxr-xr-x 1 root wheel 1265504 Oct 25 18:37 hfs_convert
-rwxr-xr-x 1 root wheel 34672 Oct 25 18:37 mount_apfs
-rwxr-xr-x 1 root wheel 559248 Oct 25 18:37 newfs_apfs
-rwxr-xr-x 1 root wheel 520736 Oct 25 18:37 slurpAPFSMeta

Using the tmutil snapshots would be cool, but … may be this works only if another Time Machine volume is configured:

APFS snapshots: where are they loc… | Apple Developer Forums

bash-3.2# /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util
Usage: apfs.util [[-p][-k] /dev/diskXsYsZ]|[-s /dev/diskXsY]|[-R SNAPSHOT][-M dir][-S dir][-O path]
-p : probe for volume name
-k : get volume UUID
-s : set volume UUID(s) and container UUID to new random values
-R SNAPSHOT : set the volume to revert to the snapshot named by SNAPSHOT on next mount.
-M dir : flag the named directory as “maintain-dir-stats”.
-S dir : get the directory stats from dir.
-O path [optional fs name] : override the fstypename for apfs to be “hfs” (or the optional name given)
apfs.util [-D [get | set=yes | set=no] /dev/diskXsY] -D get : Get defragmentation setting for volume or container
-D set=yes : Enable defragmentation for volume or container
-D set=no : Disable defragmentation for volume or container
apfs.util -B
-B : bootstrap the root filesystem

So maybe we can put together the apfs_create_filesystem_snapshot script using these cli tools.

Cannot do the tests on a production system, tho - but I*ll try to setup a test machine with High Sierra. If I read it correctly, the apfs tools have been available since Sierra, so one could create an apfs volume using Sierra and make some tests ,

[uroni]

Can someone try if this one fixes this problem: https://www.urbackup.org/downloads/Client/2.1.17/UrBackup%20Client%202.1.17.pkg

[Falko_Trojahn]

I’ve installed the 2.1.17 pkg on one client and just trying to get a backup started, will tell you later or tomorrow if it works.

To check if “/” is using apfs just used the following lines successfully:

TYPE=""
if [ x$(df -T apfs | egrep " ${SNAP_MOUNTPOINT}$" | head -n 1 | tr -s " " | cut -d" " -f9) == x${SNAP_MOUNTPOINT} ] ; then
TYPE=apfs
fi

[Falko_Trojahn]

Great! 2.1.17 seems to work - just compared/restored some files.

For the snapshot script maybe this works somehow:

if [[ $TYPE == “apfs” ]]
then

    tmutil localsnapshot
    # TODO: ... some funny things to check date and hour of snapshot?
    SNAP_DEST="tmutil listlocalsnapshots / | tail -n1"

else
echo “Cannot create snapshot of file system with type $TYPE”
exit 1
fi

here the output of tmutil commands regarding snapshots:

bash-3.2# tmutil | grep snap
Usage: tmutil delete snapshot_path …
tmutil compare [-@acdefghlmnstuEX] [-D depth] [-I name] snapshot_path
Usage: tmutil localsnapshot
Usage: tmutil listlocalsnapshots <mount_point>
Usage: tmutil listlocalsnapshotdates [<mount_point>]
Usage: tmutil deletelocalsnapshots <snapshot_date>
Usage: tmutil thinlocalsnapshots <mount_point> [purgeamount] [urgency]

Thank you very much for your quick client fix anyway!

[uroni]

You should be able to test this by creating a /Applications/UrBackup Client.app/Contents/MacOS/etc/urbackup/snapshot.cfg file with contents pointing to snapshot creation and destruction scripts:

create_filesystem_snapshot=/usr/local/etc/urbackup/create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/etc/urbackup/remove_filesystem_snapshot

I cannot see a command to remove a specific snapshot, though.

[Renato3]

Hi, I am on holidays now, but as soon as I return I will test the new client for MacOS.

By then, I guess you already know how it works, but alas, I will test it thoroughly