Archive: Retroactive Retention Drift

Development
1

UrBackup fills a valuable niche as an open-source backup solution that’s genuinely approachable for home and small-business use. Archive retention is one of its more powerful features, and for most users it works well as long as the initial configuration is right. This proposal addresses a gap that surfaces when an administrator needs to correct or evolve their retention policy after the fact — something that becomes more likely the longer and more successfully the product is adopted.

TL;DR: UrBackup’s archive retention is currently immutable — timeouts are stamped once at archival and never change, even when the policy does. This proposal suggests a setting to toggle between immutable retention (current behavior, suitable for compliance where policy immutability is a requirement) and adaptive retention (re-evaluates existing backups when policies change, suitable for evolving environments). The re-evaluation itself is straightforward — for each archived backup, recalculate backuptime + new length and unarchive anything no longer claimed by a rule. The implementation is believed to be a small addition to updateArchiveSettings() (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L364-L444).

Problem: Archive policy changes don’t apply to existing backups

Note: Links are broken with spaces because of the forum’s new-user limitation: “Sorry, new users can only put 2 links in a post.” Remove the space before .com / .org to follow them.

When archive retention policies change, existing archived backups are not re-evaluated. Each backup’s archive_timeout is set correctly at archival time for the policy that was active at that moment. When the policy later changes, no mechanism exists to recalculate existing timeouts against the new policy. The archive_update flag — despite its name — only updates rule definitions, not existing backup timestamps.

Specifically:

  1. Archive timeout is set once, correctly, and never revisited. Backups are created with archived=0 (line 691, ServerBackupDao.cpp (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/dao/ServerBackupDao.cpp#L691)). A separate archive thread wakes hourly and, for each rule, checks if next_archival < now (line 134 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L134)). When a rule fires, it picks the newest non-archived backup (ORDER BY backuptime DESC LIMIT 1 at line 235 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L235)) and stamps it with getTimeSeconds() + length (line 265-268 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L265-L268)). This is correct for the policy active at that moment. The timeout is never recalculated when the policy changes.

  2. Shortening retention has no effect on existing backups. A backup with backuptime = 6 months ago, archived under length = 1 year, has timeout getTimeSeconds() + 1 year — roughly 6 months from now. Policy changes length to 1 month. The correct timeout would be backuptime + 1 month = 5 months ago — but no recalculation occurs, so the backup stays archived for at least another 6 months.

  3. Especially painful when correcting overly long retention. length = 2 years, corrected to 3 months. A backup with backuptime = 1 year ago has timeout getTimeSeconds() + 2 years — roughly 1 year from now. The correct timeout would be backuptime + 3 months = 9 months ago. The longer the original length was in place, the more storage is trapped.

  4. Lengthening retention also has no effect on existing backups. length = 1 year, extended to 2 years. A backup with backuptime = 6 months ago has timeout getTimeSeconds() + 1 year — roughly 6 months from now. The correct timeout would be backuptime + 2 years = 18 months from now. The backup expires roughly a year earlier than the new policy intends. In neither direction does a policy change have the expected effect on existing backups.

  5. Rooted in a reasonable design principle. Stamping archive expiry at creation time and treating it as immutable is a well-established pattern in backup systems — enterprise WORM storage and compliance-oriented tools deliberately prevent retroactive shortening. UrBackup’s current behavior follows this spirit. However, those systems are designed for regulated environments where immutability is a requirement, not a home or small-business NAS where the administrator is also the compliance authority. For the typical UrBackup user, the inability to retroactively correct a retention misconfiguration becomes a practical problem, not a safety feature.

  6. Manual correction doesn’t scale. Since this behavior has silently been in effect since the instance was first set up, the number of incorrectly locked backups tends to be large by the time anyone notices — typically when storage pressure forces a policy review. The web UI allows unarchiving individual backups one at a time, which is not practical across multiple clients with hundreds of backups each. The only alternative is direct SQL against the database with the server stopped, requiring knowledge of the archive rule matching logic, per-rule timeout calculations, and the distinction between manual and rule-assigned archives. A simple toggle would reduce this to a checkbox and a cleanup cycle.

  7. Scale of impact grows with history. The longer an instance has been running and the more clients it serves, the more storage becomes trapped behind stale timeouts after a policy change. In practice, the majority of archived backups may no longer match any current rule, with timeouts stretching years into the future.

Current behavior (server_archive.cpp (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp))

Three functions form the archive lifecycle, running hourly (line 41-49 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L41-L49)):

  1. archiveTimeoutFileBackups() (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L54-L70) — unarchives backups where archive_timeout < now. This is the only mechanism that clears the archived flag on rule-assigned backups.

  2. archiveBackups() (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L103-L196) — for each rule, if next_archival < now, selects the newest non-archived backup (WHERE archived=0 ORDER BY backuptime DESC LIMIT 1 at line 235 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L235)) and archives it with getTimeSeconds() + length (line 265-268 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L265-L268)). Then advances next_archival by the rule’s interval (line 205 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L205)).

  3. updateArchiveSettings() (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L364-L444) — triggered when archive_update=1 is set for a client. Updates the rule definitions in the automatic_archival table and clears the flag. Does not touch any backup’s archive_timeout.

When a policy changes, step 3 updates the rules but existing archived backups are untouched. They retain their original archive_timeout until it naturally expires via step 1. Step 2 only ever processes non-archived backups — it has no path to revisit or recalculate an already-archived backup’s timeout.

Additionally, step 2 archives only one backup per rule per cycle (LIMIT 1 at line 235, then next_archival advances by the full interval at line 205). This means re-population after any bulk unarchive is not viable — a daily rule takes 9 days to re-protect 9 backups, a bimonthly rule takes months, and a yearly rule takes years. Cleanup will delete unprotected backups far faster than the rules can re-claim them.

The archive_update flag’s name implies re-evaluation of existing backups, but it only updates rule parameters.

Expected behavior with proposed fix

When archive rules change and archive_update=1 is processed, existing backups should be re-evaluated against the new rules:

  • Backups still claimed by a rule: archive_timeout = backuptime + new length
  • Backups no longer claimed by any rule: archived=0, archive_timeout=0
  • Manual archives (archived=1, archive_timeout=0) are never touched
  • Example: backuptime = 6 months ago, length changed from 1 year to 1 month. Correct timeout is backuptime + 1 month = 5 months ago. Backup should be unarchived immediately.

Proposed Fix

New setting: archive_retroactive_eval (boolean, default: false)

When enabled and archive_update=1 is processed, updateArchiveSettings() also triggers a re-evaluation pass over existing archived backups. For each client, the re-evaluation:

  1. Iterates current archive rules and determines which backups each rule would claim (N most recent, spaced by interval)
  2. For claimed backups: sets archive_timeout = backuptime + length
  3. For unclaimed backups with archive_timeout > 0 (rule-assigned, not manual): sets archived=0, archive_timeout=0
  4. Never touches manual archives (archived=1, archive_timeout=0)

Default is false to preserve existing behavior and avoid surprises on upgrade.

In server_settings.h (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_settings.h):

bool archive_retroactive_eval;  // default: false (legacy: no re-evaluation)

In server_archive.cpp (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp), within updateArchiveSettings():

// After updating rule definitions in automatic_archival table:
if (settings.archive_retroactive_eval) {
    reevaluateExistingArchives(clientid);
}

Detection of manual vs rule-assigned archives:

-- Manual: archived=1 AND archive_timeout=0
-- Rule-assigned: archived=1 AND archive_timeout > 0
-- Never touch manual archives during re-evaluation

Interim Workaround

:warning: WARNING: There is no safe blanket SQL workaround. Unarchiving backups and running cleanup will cause irreversible data loss. The archive rules re-archive only one backup per rule per interval cycle (LIMIT 1 at line 235 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L235), then next_archival advances by the full interval at line 205 (github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp#L205)). A daily rule takes 9 days to re-protect 9 backups. A bimonthly rule takes months. A yearly rule takes years. Cleanup will delete unprotected backups long before the rules can re-claim them.

The only safe manual approach is to calculate which backups each rule would claim, set correct archive_timeout = backuptime + length on those specific backups, and only then unarchive the remainder. This requires replicating the rule matching logic externally — there is no built-in command that does this.

References

  • Forums user orogor (Sept 2017): “Archives are tagged as archive when the backup is taken, then the retention can not be changed.” — forums.urbackup .org/t/question-what-backup-retention-strategy-do-you-use/4024
  • Admin manual, section 11.3: “This is not possible while UrBackup is running, so you should […] stop the server run the cleanup separately by calling urbackupsrv cleanup --amount x” — urbackup .org/administration_manual.html
  • Admin manual, section 11.5: Archiving documentation — urbackup .org/administration_manual.html
  • Archive source code: github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_archive.cpp
  • Cleanup source code: github .com/uroni/urbackup_backend/blob/dev/urbackupserver/server_cleanup.cpp
  • No existing GitHub issue or forum thread for retroactive archive re-evaluation
  • cleanup --amount 0% does not address this — it only enforces min/max on non-archived backups