I’d like the ability to have multiple urbackup servers operating in the same local network in a way where they can coordinate with each other, or at the very least be aware of and not interfere with each other.
- Each server discovers the others, and with administrator approval, share secret keys.
- Clients target one server at a time, and prefer to continue targeting that server when possible.
- Server can redirect a client to another server, in case of capacity issues, or for whatever reason
- Administrator can assign specific clients to specific servers with white-list, or disallow clients from servers with black-list
- Servers coordinate metadata
- Files stay put, unless an admin manually triggers a move.
- Clients can optionally transfer-speed test servers, and choose whichever works out fastest
- A client can have a designated “home” server, and if other servers receive an incoming backup from that client, once the backup is finished it’ll transfer to the “home” server for long term storage.
Use cases:
-
Multiple office locations, connected by VPN. Someone moving between these two locations with a laptop wouldn’t necessarily want to back up to an internet server, but a local server. If each office has an urbackup server that knows about the other servers, then the laptop could back up to the local site at high-speed, but then the servers could coordinate moving the backup to the laptops designated “home” server at a lower speed over a long time period.
-
Multiple servers operating at a single site for high-availability purposes. Personally I strongly prefer operating multiple instances of the same service when said service can be made cluster aware, to protect against downtime. Even things as simple as a DHCP server or DNS server, I prefer to have multiple, so I can turn things on or off on a whim without service interruption.
-
Front end urbackup server with fast, but low capacity, disks. Clients back up to this server, and the server transfers the backup to a backend urbackup server with slow but high capacity disks.
-
Cluster filesystem environments, such as CephFS. Multiple urbackup servers exposed to clients each sharing and coordinating a single storage environment.
I Would like this too (made a feature request for a meta server some time ago).
i think it s possible to gradually add features like :
- Each server discovers the others, and with administrator approval, share secret keys.
- they can coordinate with each other, or at the very least be aware of and not interfere with each other.
- Servers coordinate metadata
From that point on, a lot of features become availlable, like storage only server, migrate data between servers, user roaming between servers, etc… But already getting feature 1-2 would be nice.
When i made my feature request, I was thinking about an additional meta server, that would basically fetch the sqlite base of the other servers, then reconfigure them. I thought this would be easier to implement than to allow the servers to directly speak between each others.
Server to server interactions on Linux machines can grow to be quite feature rich.
For example, if both servers are using btrfs as the underlying storage, btrfs-send and btrfs-receive become quite interesting for transferring data between servers.
Personally my ideal situation is to have 3 urbackup servers in a “multi-master” configuration, where no specific server is “in charge” in any meaningful way. This would allow me to shut down one for maintenance without even a hint of concern at interrupting an active transaction, because there are two other servers available to handle things.
Of course, the servers would need to be aware of various cluster communication gotchas, such as the “split-brain” problem, where two groups of servers are isolated from each other, and each group believes that it’s the main part of the cluster and continues on doing work. When the isolation ends, the two halves of the cluster can’t reconcile the work that was done independently.
Generally this is why people recommend having an odd number of machines in a cluster, and configuring the software such that if it can’t reach strictly more than 50% of the total cluster, to stop processing until it can reach 50% or more of the cluster once more.
3 urbackup servers would avoid the split brain situation, in case a link somewhere is broken.
Some administrators may want to have multiple copies of each backup, one copy on two or more servers. I imagine they would want this for disaster recovery purposes. Having multiple servers working together would allow that to happen. The client sends the backup to server 1, server 1 sends a copy of the backup to server 2. The client only considers the backup finished when it’s stored in 2 locations.
Another thought is that with full server-to-server communications and transfers, Urbackup would be positioned (from an underlying capability standpoint, anyway) to support having clients be backup targets as well as backup sources.
An urbackup server coordinating multiple clients, where each client reserves some portion of disk space for storing copies of backups from other clients. Urbackup would, of course, want to avoid including the storage for these other clients in the backups made from the current client to the server, but that should be a relatively trivial matter to exclude the folder.
This would be very practical for home users, where dedicated server computers are unlikely, at best. Being able to have multiple computers form an ad-hoc backup group would be very attractive to lots of consumer-level computer users.