[request] Generate hashes on client for local network?

Feature requests
#1

I’m looking at Urbackup as an option for backing up a small studio, and we have a lot of raw captures and media that are very large and don’t change often, and I don’t want to have to transfer them over the network every time a full backup is ran.

Most of them reside on fileservers that have spare CPU cycles so calculating the hashes every time isn’t that much of a problem, but it seems there is only an option for client-side hashing under Internet options, any chance of it making it into the local network options? Otherwise is there any disadvantage to somehow forcing an “internet” connection over the LAN?

EDIT: It seems both the calculate hashes on client and connect to internet server even if local server available are missing from the Mac client, I’m assuming this is because it’s an early beta?

EDIT 2: It seems that the “Block differences - hashed” transfer mode also does what I need, but is there any way to enable it per client?

Moving file between 2 folders on client side => New file transfer to server, before deduplication?
#2

You can enable “Block differences - hashed” transfer mode per client on the server in the advanced tab (after checking use separate settings for this client).

#3

Awesome, thanks! I guess I should download/install it in a VM and start trying it out.

How stable are the beta versions? Where (approximately, obviously I know nothing is guaranteed) would you say they fall on a scale of “production ready” to “don’t even think about installing this on anything you care about”?

#4

I’m currently using server 2.0.21 on debian 8.3 and it is stable for what I’m doing so far (internet file & image backups). I’m planning on upgrading to 2.0.22 soon, but I would say it’s more or less production ready at this point.

Edit: I’m also using BTRFS for the data storage.

#5

I was looking this request cause I would like hash on client side in lan, but what i see on my server (2.1.20) for specific client (2.1.17) is “Local full file backup transfer mode” : Raw or hashed, do I misunderstand something ?

And Thanks for this excellent product.

#6

So I missed something :
Force my client to go through the “internet mode” just run on windows client enable_internet_only.bat
And now it’s ok

For other client I’ve not tried yet on Raspbian, but will do soon

#7

Sorry to grab that thread back up - but:

I have the same question: Full backup over internet is using hashes - only changed file are transferred actually. That works.

But currently (2.4.11) local fulls are ALWAYS transferring EVERYTHING. And I dont see any option which would change that in the advanced tab. I dont want to force the clients to internet mode really.

Is local full transfer mode hard coded? The posts above indicate, that it should be possible to change it? Am I not seeing it?

As I write - there is a translation error for Local full file backup transfer mode to german. German text states “Abbild” (image). But the string should be “Dateisicherung” (file backup).

@uroni I edited it at TransiFex! Please note that for the next update :slight_smile:

But that option is set to “Hashed”. Why does urbackup transfer the files anyway?

Example:

Info 13.10.22 19:35 Starting unscheduled full file backup…
Info 13.10.22 19:36 Selected no components to backup
Info 13.10.22 19:36 Indexing of “Users” done. 45666 filesystem lookups 0 db lookups and 0 db updates
Info 13.10.22 19:36 Robins-Blade: Loading file list…
Info 13.10.22 19:36 Robins-Blade: Started loading files…
Info 13.10.22 19:54 Waiting for file transfers…
Info 13.10.22 19:54 Waiting for file hashing and copying threads…
Info 13.10.22 20:03 Writing new file list…
Info 13.10.22 20:03 All metadata was present
Info 13.10.22 20:03 Transferred 24.9358 GB - Average speed: 196.838 MBit/s
Info 13.10.22 20:03 Time taken for backing up client Robins-Blade: 28m
Info 13.10.22 20:03 Backup succeeded

Directly after that I triggered another full backup:

Info 13.10.22 20:03 Starting unscheduled full file backup…
Info 13.10.22 20:04 Selected no components to backup
Info 13.10.22 20:04 Indexing of “Users” done. 45678 filesystem lookups 0 db lookups and 0 db updates
Info 13.10.22 20:04 Robins-Blade: Loading file list…
Info 13.10.22 20:04 Robins-Blade: Started loading files…
Info 13.10.22 20:23 Waiting for file transfers…
Info 13.10.22 20:23 Waiting for file hashing and copying threads…
Info 13.10.22 20:32 Writing new file list…
Info 13.10.22 20:32 All metadata was present
Info 13.10.22 20:32 Transferred 24.9358 GB - Average speed: 190.069 MBit/s
Info 13.10.22 20:32 Time taken for backing up client Robins-Blade: 28m 52s
Info 13.10.22 20:32 Backup succeeded

I triggered a internet full backup from another client:

[…]

Info 13.10.22 20:25 Transferred 612.392 KB - Average speed: 28.376 KBit/s
Info 13.10.22 20:25 (Before compression: 884.169 KB ratio: 1.4438)
Info 13.10.22 20:25 10.7966 GB of files were already present on the server and did not need to be transferred
Info 13.10.22 20:25 Time taken for backing up client s1.kluthr.de: 4m 40s
Info 13.10.22 20:25 Backup succeeded

But why?

The settings:

grafik
grafik885×99 2.25 KB

Can someone lead me in the right direction? :slight_smile:

#8

No one? :confused:

#9

@uroni Could you shed a light on this?

#10

Hashing takes time. If you have a fat LAN network pipe, why bother?

#11

Sry, this would need a new option. I’ll move it to feature requests…

2 Likes