Hi.
How come that by moving a file (FileABC) between two folders on client side (file ABC from C:/Folder1/FolderX/FileABC to C:/Folder1/FolderY/FileABC), results in a new transfer of the same file? Should the client/server not be intelligent enough from the hash to know its the same file? The file is exactly the same, so why is is transferred again.
The client is Windows 10, and the server is Linux/Debian.
I have tried with ext4 and btrfs format on the server, to no avail.
It shouldn’t, please post logs about file transfert, client+server
Whatever infos that lead you to think the whole file was transferred.
Your advanced setting for the specific host
Well, it s not really the info i asked for. But usually the transfert speed+volume when it shown like this is the amount of data which was backuped+the time it took, whatever the full file was transfered or not.
idem if you look at storage ued , it the apparent size of the backup (without accounting for the hardlinking)
If you put the server on debug and look at the transfert log from the activity tab , it will say if it transfet hashes or not.
Also that depends on the ADVANCED config tab ( the last one) , so, typically, if you do lan or wan backup (theses are 2 différent profiles)
I believe I uploaded all the things you asked for.
The logs on the server and client and the advanced settings?
Anyway, i tried to enable the two debug options inadvanced settings, and still the program does not recognize its the same file moved between two folders on the client, before AFTER uploading it to the server. Keeps uploading the same file everytime to the server, before finally deduplicating it, as it recognices its the same file.
Log1 server:
05/02/17 19:49 INFO Starting unscheduled incremental file backup…
05/02/17 19:49 DEBUG LAPTOP: Doing backup with hashes…
05/02/17 19:49 DEBUG LAPTOP: Doing backup with intra file diffs…
05/02/17 19:49 DEBUG LAPTOP: Connecting for filelist…
05/02/17 19:49 DEBUG LAPTOP: Waiting for filelist
05/02/17 19:49 DEBUG LAPTOP: Connecting for filelist (async)…
05/02/17 19:50 INFO Indexing of “FolderA” done. 2 filesystem lookups 1 db lookups and 2 db updates
05/02/17 19:50 DEBUG LAPTOP: Doing backup with hashed transfer…
05/02/17 19:50 INFO LAPTOP: Loading file list…
05/02/17 19:50 DEBUG LAPTOP Starting incremental backup…
05/02/17 19:50 INFO LAPTOP: Calculating file tree differences…
05/02/17 19:50 INFO LAPTOP: Calculating tree difference size…
05/02/17 19:50 INFO LAPTOP: Linking unchanged and loading new files…
05/02/17 19:50 INFO Waiting for file transfers…
05/02/17 19:50 DEBUG No old file for “200MB_Video.avi” (1)
05/02/17 19:50 DEBUG Loading file “200MB_Video.avi”
05/02/17 19:51 DEBUG Loading “FolderA/200MB_Video.avi”. 71% finished 142.937 MB/200.104 MB at 19.795 MBit/s
05/02/17 19:51 DEBUG GT: Loaded file “200MB_Video.avi”
05/02/17 19:51 DEBUG Loading file “FolderC” (metadata only)
05/02/17 19:51 DEBUG Loading file “FolderA” (metadata only)
05/02/17 19:51 DEBUG PT: Hashing file “200MB_Video.avi”
05/02/17 19:51 INFO Waiting for file hashing and copying threads…
05/02/17 19:51 DEBUG HT: Linked file: “/media/BACKUP/urbackup/LAPTOP/170502-1949/FolderA/200MB_Video.avi”
05/02/17 19:52 INFO Backup verification is enabled. Verifying file backup…
05/02/17 19:52 DEBUG Verified 1 files
05/02/17 19:52 INFO Backup verification ok
05/02/17 19:52 DEBUG Syncing file system…
05/02/17 19:52 DEBUG Creating symbolic links. -1
05/02/17 19:52 DEBUG Creating symbolic links. -2
05/02/17 19:52 DEBUG Symbolic links created.
05/02/17 19:52 INFO Creating user views…
05/02/17 19:52 INFO Transferred 200.112 MB - Average speed: 18.8752 MBit/s
05/02/17 19:52 DEBUG Script does not exist urbackup/post_incr_filebackup
05/02/17 19:52 INFO Time taken for backing up client LAPTOP: 2m 50s
05/02/17 19:52 INFO Backup succeeded
And this is second log AFTER moving the file between two folders on the client, and starting new incrementaal backup.
05/02/17 19:52 INFO Starting unscheduled incremental file backup…
05/02/17 19:52 DEBUG LAPTOP: Doing backup with hashes…
05/02/17 19:52 DEBUG LAPTOP: Doing backup with intra file diffs…
05/02/17 19:52 DEBUG LAPTOP: Connecting for filelist…
05/02/17 19:52 DEBUG LAPTOP: Waiting for filelist
05/02/17 19:52 DEBUG LAPTOP: Connecting for filelist (async)…
05/02/17 19:53 DEBUG LAPTOP: Doing backup with hashed transfer…
05/02/17 19:53 INFO LAPTOP: Loading file list…
05/02/17 19:53 DEBUG LAPTOP Starting incremental backup…
05/02/17 19:53 INFO LAPTOP: Calculating file tree differences…
05/02/17 19:53 INFO LAPTOP: Calculating tree difference size…
05/02/17 19:53 INFO LAPTOP: Linking unchanged and loading new files…
05/02/17 19:53 INFO Waiting for file transfers…
05/02/17 19:53 DEBUG No old file for “200MB_Video.avi” (1)
05/02/17 19:53 DEBUG Loading file “200MB_Video.avi”
05/02/17 19:54 DEBUG Loading “FolderA/FolderB/200MB_Video.avi”. 70% finished 140.39 MB/200.104 MB at 19.6282 MBit/s
05/02/17 19:54 DEBUG GT: Loaded file “200MB_Video.avi”
05/02/17 19:54 DEBUG Loading file “FolderB” (metadata only)
05/02/17 19:54 DEBUG PT: Hashing file “200MB_Video.avi”
05/02/17 19:54 DEBUG Loading file “FolderA” (metadata only)
05/02/17 19:54 INFO Waiting for file hashing and copying threads…
05/02/17 19:54 DEBUG HT: Linked file: “/media/BACKUP/urbackup/LAPTOP/170502-1952/FolderA/FolderB/200MB_Video.avi”
05/02/17 19:54 INFO Saving file metadata…
05/02/17 19:54 INFO Writing new file list…
05/02/17 19:54 INFO All metadata was present
05/02/17 19:54 INFO Backup verification is enabled. Verifying file backup…
05/02/17 19:54 DEBUG Verified 1 files
05/02/17 19:54 INFO Backup verification ok
05/02/17 19:54 DEBUG Syncing file system…
05/02/17 19:55 DEBUG Creating symbolic links. -1
05/02/17 19:55 DEBUG Creating symbolic links. -2
05/02/17 19:55 DEBUG Symbolic links created.
05/02/17 19:55 INFO Creating user views…
05/02/17 19:55 INFO Transferred 200.112 MB - Average speed: 19.6857 MBit/s
05/02/17 19:55 DEBUG Script does not exist urbackup/post_incr_filebackup
05/02/17 19:55 INFO Time taken for backing up client LAPTOP: 2m 10s
05/02/17 19:55 INFO Backup succeeded
Near the transfert speed/volume i get a line like:
53.2578 MB of files were already present on the server and did not need to be transferred
That s the advanced tab, it s in french, but if you look at the doc you find the meaning for raw, hashed and hashed by block transfert. Setting can be different for internet, local and full/inc.
IT IS EXACTLY THE SAME FILE. I AM MOVING ONE FILE OF 200.112 MB between two folders on client, and URBACKUP keeps sending the entire file EVERY time to the host.
This must be a bug in software. No matter which setting I choose for the transfer mode, it should not matter, as the file is exactly the same.
It seems there is absolutely not hash check on client end. All checks happens on the server, and this is a big no-no,
In internet mode there is calculate file hash on client option in internet tab.
I dont use local mode yet, so i dont know very well how it works.
Try to play with that and the checkbox just under :
“Connect to Internet backup server if connected to local backup server”
Maybe in local mode because transfert is so fast , it doesn’t use hashes.
Maybe the logic is that in local mode it s longuer to compute the hashes than to transfert the files.
After, i dont think i can help anymore, you d have to wait for @uroni insight.
Sorry to dig up an old thread, but I think I’m seeing the same issue as the OP:
I’ve just started evaluating UrBackup (looking like a nice project - thanks all!), and am running the server on a Synology NAS (hosted in a Docker container) connected on the LAN to a macOS client (running inside a VM for testing purposes).
Backups are running successfully - but I seem to be getting a full transfer when I wouldn’t expect it: to test, I’m duplicating the Xcode application (a 10.38GB bundle) and running an incremental file backup. I have Settings > Advanced > Local incremental file backup transfer mode set to Block differences - hashed - but every backup seems to transfer the full ‘new’ data.
I’ve also tried with Hashed - but in any case I believe this is related to the transfer method, rather than being a local version of Settings > Internet > Calculate file-hashes on the client.
Here’s the most-recent transfer log from the server:
Level Time Message
Info
23/01/18 12:47
Starting unscheduled incremental file backup...
Info
23/01/18 12:50
Backing up "Applications" without snapshot.
Info
23/01/18 12:50
Following symbolic link at "/Applications/App Store.app/Contents/Resources/AppStore.help/Contents/Resources/index.html" to "/Library/Documentation/Resources/Eagle/index.html" confirms symlink backup target ".symlink_index.html" to "/Library/Documentation/Resources/Eagle/index.html"
Info
23/01/18 12:50
Following symbolic link at "/Applications/Safari.app/Contents/MacOS/SafariForWebKitDevelopment" to "/Library/Application Support/Apple/Safari/SafariForWebKitDevelopment" confirms symlink backup target ".symlink_SafariForWebKitDevelopment" to "/Library/Application Support/Apple/Safari/SafariForWebKitDevelopment"
Info
23/01/18 12:50
Indexing of "Applications" done. 214472 filesystem lookups 0 db lookups and 0 db updates
Info
23/01/18 12:50
Backing up ".symlink_index.html" without snapshot.
Info
23/01/18 12:50
Backing up ".symlink_SafariForWebKitDevelopment" without snapshot.
Info
23/01/18 12:50
Tests-Mac.local: Loading file list...
Info
23/01/18 12:50
Tests-Mac.local: Calculating file tree differences...
Info
23/01/18 12:50
Tests-Mac.local: Calculating tree difference size...
Info
23/01/18 12:50
Tests-Mac.local: Linking unchanged and loading new files...
Info
23/01/18 12:50
Referencing snapshot on "Tests-Mac.local" for path "Applications" failed: FAILED
Warnings
23/01/18 12:52
Error getting complete file "2Y8M8qwBFjk4AarktE8i|Applications/UrBackup Client.app/Contents/MacOS/var/urbackup/data/filelist_new_0.ub" from Tests-Mac.local. Errorcode: CANNOT_OPEN_FILE (3)
Info
23/01/18 13:28
Waiting for file transfers...
Info
23/01/18 13:28
Waiting for file hashing and copying threads...
Info
23/01/18 13:28
Saving file metadata...
Warnings
23/01/18 13:33
Not all folder metadata could be applied. Metadata was inconsistent.
Info
23/01/18 13:33
Writing new file list...
Info
23/01/18 13:33
All metadata was present
Info
23/01/18 13:33
Transferred 11.6792 GB - Average speed: 44.3896 MBit/s
Info
23/01/18 13:33
Time taken for backing up client Tests-Mac.local: 46m 25s
Info
23/01/18 13:33
Backup succeeded
Like the OP in [request] Generate hashes on client for local network? I’m going to be backing up a number of machines with largely-static media files - some will be being moved around between folders - and it would be good to keep the network traffic to a minimum.
Is this currently expected behaviour?
EDIT: Sorry - forgot to include versions: Server 2.1.20, Client 2.1.17.
There is two issue specific to macOS client you’d need to check the beta/changelogs/forums to see if it s fixed. But appart from that, it should work.
As i remember one is with macsosx sparse files being backed up as empty files. And at this level it s almost a macos issue, but i admit, it s quite blocking.
The other one is with the client unicity (is that a word?). The unique key for client is the client name/computer name, but under macos, the computer name can change, thus it may create new clients. There is a thread on the urbackup forum about this, and on the macos forum, so it’s also not an actual urbackup issue.
To check if things are running as expected. Please monitor the bandwidth usage from outside. Like byte transferred from/to interface stat from ifconfig or whatever. You would need to look at the whole partition disk usage from df -h.
Because the stats from urbackup are inconsistant, even between différents parts of the monitoring disk usage or bandwidth is not that easy.
Like the transfert speed is like the backup speed or or ‘virtual transfert speed if files were transferred fully’. But in the stats page it can display on disk usage… or not, depending on the graph.
Because of dedup occurring if the same file already exists with hardlink/softlink .
If using btrfs as storage, there can also be reflink usage.
There can be sparse file usage.
if you have a lot of small files there can be some other things to consider as well.
Again, this is a general issue if you use a modern fs or try to go deep to optimize file usage.
So you either need to not think too much about it. Or know well how it works and dig very deep to check if there s an actual issue with how it s supposed to work.
Also, you can usually trust the urbackup beta. They fix a lot of issues, If it s like 2 weeks old and nobody reported a critical issue and thus @uroni didn’t released a new version… i would consider it as stable.
I think just install it on a few computers, actually try the restore. See if urbackup fits you.
You can also use the internet mode locally. I guess if it is running in docker you can simply not forward the ports for local lan mode.
You can also run the client in “internet only mode” (-i).
Many thanks for coming back to me with such a lot of information - much appreciated. I’ll bypass some of the comments considering uroni’s reply, but here is some further response which others might find useful:
