Staggered backup times

I have four things to backup. Ideally I’d like these to backup once a month. And start on different weeks to avoid either machine getting bogged down by running multiple backups at the same time.

Is there any way to set a calendar based backup?

disk1 - backup week 1
disk2 - backup week 2
disk3 - backup week 3
disk4 - backup week 4

All I can see is the ability to set a timer between backups and a day of the week.

When does this timer start? From when backup was added, or last time the server restarted? I am trying to work out a way to game those offsets on specific days of the week to attempt my ideal pattern. :slight_smile:

Are these all different clients? Or the same single client with multiple disks?

If you these are multiple systems (clients), you can specify a different backup window for each and set a 28 day interval so they only run every 4 weeks. I don’t believe it is possible to specify anything beyond day of week.

Client 1 - Mon/0-24
Client 2 - Tue/0-24
Client 3 - Wed/0-24
etc etc

How to define a backup window?

In the settings you can set a backup window. The server will only start backing up clients within this window. The clients can always start backups on their own, even outside the backup windows. If a backup is started it runs till it is finished and does not stop if the backup process does not complete within the backup window. A few examples for the backup window:

1-7/0-24: Allow backups on every day of the week on every hour.
Mon-Sun/0-24: An equivalent notation of the above
Mon-Fri/8:00-9:00, 19:30-20:30;Sat,Sun/0-24: On weekdays backup between 8 and 9 and between 19:30 and 20:30. On Saturday and Sunday the whole time.

As one can see a number can denote a day of the week (1-Monday, 2-Thuesday, 3-Wednesday, 4-Thursday, 5-Friday, 6-Saturday, 7-Sunday). You can also use the abbreviations of the days (Mon, Thues, Wed, Thurs, Fri, Sat, Sun). The times can either consist of only full hours or of hours with minutes. The hours are on the 24 hour clock. You can set multiple days and times per window definition, separated per ",". You can also set multiple window definitions. Separate them with ";".

If you’re referring to four separate disks all attached to a single system, then I don’t believe it’s possible to accomplish what you’re looking for. However, it shouldn’t be necessary, because you won’t be backing up all 4 disks at once. The files are backed up serially, regardless of the physical underlying disk. Nothing should be “bogged down” though the first full backup may take a while depending on the amount of data. There are beta settings that can make it multi-threaded, but that will still not run multiple jobs at once.

1 Like

I’ve split the disks with the multi-client system. So all four drives are in the list separate. I did this as one of the drives in that box needs a daily backup, others change rarely.

Yeah, have read the manual, and tried to find a way to split a slower backup. I had also got down to “pick a week day” option, but with a backup that can sometimes take over a day this still lead to overlap.

Previously I had combined this with a time between backups of “a month worth of hours”. So Monday, Wednesday, Friday and 672hours on the incremental. With a 60day full backup gap.

But this still leads to it all in the same week. I just need to stop trying to be neat. :grin:

This is possible with UrBackup, but requires a clever mix of inclusion/exclusion directives and scheduling policies, because UrBackup doesn’t natively support per-volume backup frequency…

However, with a bit of scripting or client-side override trickery, it’s completely doable…


:dart: What you want

Drive Backup Frequency Notes
Drive1 Daily Changes frequently (e.g. /mnt/D1)
Drive2 2nd Saturday monthly Slow but occasional (e.g. /mnt/D2)
Drive3 3rd Saturday monthly Same
Drive4 4th Saturday monthly Same

:white_check_mark: Solution Summary (Client-Side Control)

Use UrBackup’s add-backupdir and remove-backupdir client control features to dynamically control which drives are backed up, and pair that with a cron-based schedule…

:heavy_check_mark: What stays constant:

  • UrBackup server runs backups daily or weekly as per default
  • Client determines what is included in each scheduled run
  • Drive1 is always included
  • Drive2–4 are only added right before their designated Saturday
  • Drive2–4 are removed afterward to avoid backups on other days

:toolbox: Step-by-Step Setup

:jigsaw: 1. Add Drive1 permanently to the client

urbackupclientctl add-backupdir -x /mnt/D1

:spiral_calendar: 2. Create scripts to enable/disable the big drives

Example: /usr/local/bin/enable_drive2.sh

urbackupclientctl add-backupdir -x /mnt/D2

And /usr/local/bin/disable_drive2.sh

#!/bin/bash
urbackupclientctl remove-backupdir /mnt/D2

Repeat that step for Drive3 and Drive4.

Make all scripts executable:

chmod +x /usr/local/bin/enable_drive*.sh
chmod +x /usr/local/bin/disable_drive*.sh

:alarm_clock: 3. Add cron jobs

# Drive2 - 2nd Saturday
0 2 * * 6 [ $(date +\%m) -ne $(date -d "$(date +\%Y-\%m-01) +7 days" +\%m) ] && /usr/local/bin/enable_drive2.sh
0 23 * * 6 /usr/local/bin/disable_drive2.sh

# Drive3 - 3rd Saturday
0 2 * * 6 [ $(date +\%m) -ne $(date -d "$(date +\%Y-\%m-01) +14 days" +\%m) ] && /usr/local/bin/enable_drive3.sh
0 23 * * 6 /usr/local/bin/disable_drive3.sh

# Drive4 - 4th Saturday
0 2 * * 6 [ $(date +\%m) -ne $(date -d "$(date +\%Y-\%m-01) +21 days" +\%m) ] && /usr/local/bin/enable_drive4.sh
0 23 * * 6 /usr/local/bin/disable_drive4.sh

:brain: These cron lines:

  • Add the drive early Saturday (2 AM)
  • Remove the drive late Saturday (11 PM)
  • Only run on the nth Saturday by checking if the date calculation overflows into the next month

:lock: Optional Hardening

To avoid backup errors when the excluded paths are removed:

  • Ensure the UrBackup server does not predefine the paths
  • Let the client fully control included dirs (-x = only client defines)
  • Use:
urbackupclientctl list-backupdirs

to check what’s active


:soap: Summary

This gives you:

  • Daily backups of their important drive (Drive1)
  • Once-monthly staggered backups of large drives (Drives 2–4)
  • No server-side complexity — just client cron logic
  • Server doesn’t “see” the large drives except on their days
1 Like

@OnlyMe now this I like the look of. Looks pretty much like exactly what I need.

I am working at the moment so can’t test yet, but will be at this tomorrow to try it out.

I certainly like what i see. Totally understand the trick you are pulling here. I like it for its simplicity.

Currently my backups are split using “Additional virtual sub client names”. With a bit of massaging your script and reading the “urbackupclientctl” manual I am certain I can make this work.

Thank you. I will report back in a few days…

Hi @BatterPudding,

Just to be clear, the way I was planning it, the “primary” drive is backed up each day [as per the usual timings/schedule…the second drive is only backed up on the 2nd Saturday, the 3rd drive on the 3rd Saturday, and the 4th drive on the 4th Saturday - for months that happen to have a 5th Saturday, none of the “extra” drives are backed up…

1 Like

Yeah, that is understood. It fits the “once a month per drive” plan. Having a quiet “fifth week” was expected.

As my server already has four “sub-client” backups listed for this machine, I am going to work with tweaking that. I assume I can use urbackupclientctl to name which sub-client I am manipulating.

The only thing that puzzles me is when I come to recover. If I look at a backup in a “off week”, then won’t it be empty? Why would the server not just assume the drive was deleted?

I can’t find a man page of the options for urbackupclientctl, but I assume I can also just plain turn it on\off from the client too? With cron I think you have given me the perfect solution. I am now going to go and fiddle with urbackupclientctl options and see how it addresses the sub-clients.

Is there any online docs for urbackupclientctl ?

USAGE: 

   ./urbackupclientctl set-settings  [--proxy <url>] [--authkey <string>]
                                     [--name <string>] [--server-url <url>]
                                     [-n] [-v <setting value>] ...  [-k
                                     <setting key>] ...  [-c <hostname/IP>]
                                     [-p <path>] [--] [--version] [-h]


Where: 

   --proxy <url>
     HTTP CONNECT proxy to use to connect to server

   --authkey <string>
     Server authentication key for client

   --name <string>
     Client name

   --server-url <url>
     URL of server to connect to

   -n,  --no-merge
     Don't merge server and client settings if possible

   -v <setting value>,  --value <setting value>  (accepted multiple times)
     New value to set the setting to

   -k <setting key>,  --key <setting key>  (accepted multiple times)
     Key of the setting to set

  -c <hostname/IP>,  --client <hostname/IP>
     Start backup on this client

   -p <path>,  --pw-file <path>
     Use password in file

   --,  --ignore_rest
     Ignores the rest of the labeled arguments following this flag.

   --version
     Displays version information and exits.

   -h,  --help
     Displays usage information and exits.

   Set backup settings

I can see a -client-name in there, which should allow me to map to sub-clients. So I’ll be adding some cron commands shortly to match the @OnlyMe idea.

But I want to dig more, and want to tweak other parts of the settings. Currently hitting a dead end of how to setup “key” and “value” pairs which I assume is what I’d need to change other settings.
I just end up in a circle of references…

root@server:/usr/local/bin# ./urbackupclientctl set-settings -k --help
There need to be an equal amount of -k/--key and -v/--value arguments
root@server:/usr/local/bin# ./urbackupclientctl set-settings -v --help
There need to be an equal amount of -k/--key and -v/--value arguments
root@server:/usr/local/bin# 

What are the key\value pairs? I can’t find them in the documentation