Exclude rules: Wrong characters(?) caused trouble!

general discussion
1

I tried to set up an exclude rule based on a tag in a folders name.

My initial idea was to use [FB!] and therefore my exclusion rule was

*\*[FB!]*\*;*/*[FB!]*/*

to cover Windows and Linux Clients and place it at any position in a folders name. I put this in the servers settings and the client adopted immediately.

I created

1Test\[FB!] [IB!] no backup\testfile.txt
1Test\testfile.txt

did an incremental backup and was wondering why 1Test was saved as an empty folder.
What i did not notice at that time: also every .exe and probably more was missing.

grafik
grafik1734×710 44.8 KB

Subsequently, i tried (if I remember correct)

*\:[FB!]:\*;*/:[FB!]:/*
*\:[nFB]:\*;*/:[nFB]:/*
*\*[nFB]*\*,*/*[nFB]*/* (maybe)

each with according testdata.
I finally noticed huge parts (e.g. the whole Documents folder) were saved as an empty folder.

After removing the exclusion, the next increment transfers almost everything again, and links to earlier saved files. I paused it.

|10.05.20 01:42  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 16% finished 400 KB/2.37958 MB at 52.368 KBit/s|
|---|---|---|
|10.05.20 01:43  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 32% finished 784 KB/2.37958 MB at 52.36 KBit/s|
|10.05.20 01:44  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 47% finished 1.14062 MB/2.37958 MB at 52.36 KBit/s|
|10.05.20 01:45  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 63% finished 1.51562 MB/2.37958 MB at 52.36 KBit/s|
|10.05.20 01:46  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 79% finished 1.89062 MB/2.37958 MB at 52.344 KBit/s|
|10.05.20 01:47  |DEBUG  |Loading "Pictures/SD/20101224/CIMG0508.JPG". 95% finished 2.26562 MB/2.37958 MB at 52.368 KBit/s|
|10.05.20 01:47  |DEBUG  |No old file for "CIMG0508.JPG" (2)|
|10.05.20 01:47  |DEBUG  |GT: Loaded file "CIMG0508.JPG"|
|10.05.20 01:47  |DEBUG  |Loading file "CIMG0509.JPG"|
|10.05.20 01:47  |DEBUG  |PT: Hashing file "CIMG0508.JPG"|
|10.05.20 01:47  |DEBUG  |HT: Copying file to "/media/mybook/urbackup/PcName/200510-0023/.hashes/Pictures/SD/20101224/CIMG0508.JPG"|
|10.05.20 01:47  |DEBUG  |HT: Linked file: "/media/mybook/urbackup/PcName/200510-0023/Pictures/SD/20101224/CIMG0508.JPG" (id=6583)|
  • Could you add a warning about characters not to use?
  • What would be the appropriate way to achieve this?
  • Similarly, I was planning to use $AllVolumes$\*\*[IB!]*\* /s in HKLM\…\FilesNotToSnapshot as proposed here: Exclude folder "c:\windows.old" on Image Backup Also a bad idea?
  • How can I avoid transferring 600GB redundantly? Just by deleting all faulty increments since I started my experiment?
  • Why is everything transferred again? I set Local incremental file backup transfer mode: Hashed and expected the client to hash and the server to notice that hash exists before copying the file. Do I really need Block differences - hashed to avoid this? Could also happen between different clients.
2

Yeah, you were quite unlucky. [] is part of the glob pattern matching … https://en.wikipedia.org/wiki/Glob_(programming) … so you were excluding all files with F, B or ! in the path.

I don’t find this usefull, so the best option might be to simply remove [], but at this point some people might rely on that feature… but yeah, it should be documented.

The way to escape them is to wrap them in [], so ;*/:[[]FB![]]:/* (the path separator gets handled automatically. This pattern is untested!).

1 Like
3

Thank you very much for such a quick support and providing UrBackup open source!

Since I know about it, this could also be really useful.

Does the first * match infinite subfolders or literally everything and : behaves like * in regular glob with UrBackup? Has ** a special meaning?
Should *[[]FB![]]:/* be equal to */:[[]FB![]]:/*?

A ‘/’ in a pathname cannot be matched by a ‘?’ or ‘*’
wildcard, or by a range like “[.-0]”.
glob(7) - Linux manual page

In all cases the path separator character ( / on Unix or \ on Windows) will never be matched.
glob (programming) - Wikipedia

I can test your provided escaping tomorrow.

Regarding retransmission when transfer mode is set to hashed: Would it be possible to search the database for the hash generated at the client before copying the file to the server? Similar to Block differences - hashed but only for complete files and with less network and CPU demand?

4

After inspecting my backups, I can confirm * matches subfolders and at least for Windows the exclude rules are NOT case sensitive.

I am still trying if this also works for image backups via registry as written above. To me the documentation is not clear about * matching subfolders.