The off-site copy is the layer that survives the building. Its design has three requirements: the transport must cross a network, the credentials must not compromise the source, and the retention must be managed without human attention. The off-site backup script this article builds is a single shell file that satisfies all three: it pulls data from production hosts over SSH with restricted keys, rotates the copies on a retention policy, verifies the integrity with checksums, and runs from a timer.
The architecture and the pull model
The off-site backup host sits outside the production network, reachable only by SSH, holding no production credentials, and pulling data from the production hosts on a schedule. The pull direction is the security inversion: the source cannot erase the backup because the source has no write access to the backup host, the compromised production server cannot exfiltrate the backup because the backup host is on a different network.
The data flows through rsync over SSH, and the encryption is the SSH tunnel rather than a separate layer. The SSH keys are restricted to the rsync protocol alone, the command= directive in authorized_keys, and the key that the backup host holds opens the production hosts' data for reading, nothing more.
The script, rotate, copy, verify
The script is the article's product, and its structure is three phases in one file:
#!/bin/bash
set -euo pipefail
DEST=/srv/offsite
LOG=/var/log/offsite-backup.log
SOURCES=(prod01 prod02 db01)
exec >> $LOG 2>&1
echo offsite-start $(date -Is)
for src in ${SOURCES[@]}; do
OLD=$DEST/$src/$(date +%A -d 7-days-ago)
[ -d $OLD ] && rm -rf $OLD
done
for src in ${SOURCES[@]}; do
LATEST=$(ls -1t $DEST/$src/ 2>/dev/null | head -1)
LINKDEST=""
[ -n "$LATEST" ] && LINKDEST="--link-dest=$DEST/$src/$LATEST"
mkdir -p $DEST/$src/$(date +%A)
rsync -aHAX --numeric-ids $LINKDEST root@$src:/ $DEST/$src/$(date +%A)/
done
LOCAL_MD5=$(md5sum $DEST/prod01/$(date +%A)/etc/hostname | cut -d' ' -f1)
REMOTE_MD5=$(ssh root@prod01 md5sum /etc/hostname | cut -d' ' -f1)
[ "$LOCAL_MD5" = "$REMOTE_MD5" ] && echo verify-ok || exit 1
The three phases are the script's architecture. The rotation deletes the directory named after the day of the week from seven days ago. The copy uses the link-dest mechanism for hardlink deduplication. And the verify spot-checks one file per run.
The SSH keys and the restricted access
The backup host's SSH key is the credential that opens every source, and its restriction is the security architecture: the authorized_keys entry on each production host carries the command= directive that limits the key to the rsync protocol. The authorized_keys line:
command=rsync --server --sender -e. / ssh-ed25519 AAAA... backup-host-key
and the command= directive means the key can only run the rsync sender, the one operation the backup needs. The key without this restriction is a root credential on the production host.
Monitoring and the health signal
The off-site backup's health is measured by two signals: the newest copy's age and the script's exit code. The monitoring script reads the off-site directory for the most recent date-named subdirectory, compares the date against the wall clock, and alerts when the newest copy is older than the schedule allows.
Integrity and the deep verification
The script's spot-check verifies one file per run, and the deep verification is the monthly procedure that compares the full tree with rsync --dry-run, the zero-delta that proves the entire copy matches. The verification's importance increases with the copy's age: a copy that has not been read since creation may have degraded on the storage medium.
The comparison and the closing checklist
The off-site script is the simplest layer in the backup architecture. The rsync approach stores full copies with hardlink deduplication, browsable directories, and the restore that is a cp. The restic or Borg approach stores encrypted chunks with content deduplication, encryption at rest, and the restore that requires the tool. The off-site script is the right layer for the on-site-to-off-site link where the network is trusted and the restore must be instant without a tool.
The checklist: write the script with rotate, copy and verify phases. Restrict the SSH keys with the command= directive. Schedule through a cron entry. Monitor the newest copy's age. Run the deep verification monthly. Test the restore from the off-site copy on the quarterly restore test calendar.
The systemd timer and the monitoring integration
The scheduling follows the systemd timer pattern from the restic and Borg articles, and the two-unit pair is the service and the timer: the service unit runs the script with the environment sourced, the timer unit triggers it weekly or daily depending on the site's cadence. The monitoring integration is the exit code and the newest-copy age, the same two-signal pattern the backup articles established: the script's exit code is nonzero on failure, and the newest directory name under the off-site root is the timestamp the monitoring reads.
The alert that fires when the off-site copy is stale is the alert that catches the silent failure: the SSH key that expired, the network that changed, the disk that filled, the source that stopped responding. Each of these failures produces a backup that stops without a backup error, because the failure is in the transport rather than in the tool, and the age monitoring is the signal that catches it. The alert text names the source and the age, and the on-call engineer's first action is to check the log for the last successful rsync, the diagnostic that distinguishes a network problem from a source problem from a disk problem.
The deep verification and the restore rehearsal
The script's spot-check verifies one file per run, and the monthly deep verification compares the full tree with the same rsync invocation in dry-run mode: the zero-delta result that proves the entire copy matches the source, the same verification that the AIDE article uses to compare the filesystem against its baseline. The deep verification catches the bit rot and the silent corruption that a backend can inflict on data nobody reads, and the monthly cadence on the off-site copy is the schedule that the spinning-disk storage demands.
The restore rehearsal from the off-site copy is the quarterly test's off-site leg: the backup is restored from the off-site host, not from the on-site copy, and the restore path exercises the SSH transport, the rsync retrieval, and the application verification, the full chain from the remote host to a working system. The rehearsal is the test that proves the off-site copy is usable from the off-site location, the scenario where the building is gone and the only backup is the one that survived elsewhere.
The comparison with the chunk-deduplicating tools
The off-site rsync script is the simplest layer in the backup architecture, and its comparison against the chunk-deduplicating tools is a choice about what the layer is for. The rsync approach stores full copies with hardlink deduplication, browsable directories, and the restore that is a cp from a path. The restic or Borg approach stores encrypted chunks with content deduplication, encryption at rest, and the restore that requires the tool and the key.
The off-site script is the right layer for the on-site-to-off-site link where the network is trusted, the storage is the team's own hardware, and the restore must be instant without a tool or a key. The chunk-deduplicating repository is the right layer for the cloud-tier off-site copy where encryption matters, the storage cost must be managed by deduplication, and the custody requires the client-side encryption that the backup articles described. The two layers compose rather than compete, and the fleet that runs both has a backup architecture where the off-site link is simple and the cloud tier is encrypted, each layer solving the problem it was designed for.
The closing argument is the one that the backup industry repeats: the off-site copy is the layer that survives the worst event, and the off-site copy that has been tested from the off-site location is the layer that works. The script this article builds is the simplest implementation of that test, and its simplicity is its strength: the script reads like the procedure, the procedure is the runbook, and the runbook is what the on-call engineer follows at 3 a.m., which is the moment when simplicity is the most valuable property any backup system can have.
The off-site backup script also serves as the template for the site's broader off-site architecture: the same rotate-copy-verify pattern extends to multiple remote sources by adding entries to the SOURCES array, the same retention policy extends by changing the day-of-week calculation to a date-based calculation for longer retention, and the same monitoring pattern extends to the dashboard by exporting the newest-copy age as a metric. The script is the foundation, and the architecture grows from the same three-phase pattern.
The backup host that runs this script is the hardened machine from the bastion article, the minimal installation with the same SSH hardening, the same fail2ban jail, the same TOTP two-factor. The off-site host is the natural first deployment for that hardening stack because it is the machine that holds every other machine's data. The disk on the off-site host grows with the retention depth, and the sizing follows the same arithmetic as the rsnapshot article: the source's total size plus seven days of change, with the hardlink deduplication keeping the marginal cost of each additional day proportional to the change.
The script and the architecture it implements are the off-site layer that every backup architecture needs, and the layer's simplicity is its design: one script, one SSH key, one retention policy, one monitoring signal, and the off-site copy that survives the building because it was never in the building in the first place. The backup architecture that includes this layer is complete: the local browsable rotation, the encrypted chunk-deduplicated repository, and the off-site copy that survives everything, three layers answering three different questions about recovery, and the quarterly test that exercises all three is the discipline that makes the whole architecture real.
The off-site backup is also the layer where the backup tools from the earlier articles converge: the restic repository's off-site copy ships through the same SSH transport, the Borg repository's remote form is the same restricted-serve pattern, and the rsnapshot's rotation on the off-site host uses the same hardlink mechanism as its on-site counterpart. The off-site script is not a replacement for those tools but a transport for them, the layer that moves the encrypted repositories to the off-site host on the same schedule, and the off-site host that runs both the plain rsync copies and the encrypted repositories has a backup architecture that covers every data class in the fleet with the appropriate tool for each.
The maintenance of the off-site host is the same discipline as the production hosts, and the off-site host's own backup, the backup of the backup, is the layer that the backup industry calls the tertiary copy, the copy of the copies that survives even the off-site host's failure. The tertiary copy on a cloud tier, scheduled through the same rclone pattern from the earlier article, completes the backup architecture, and the architecture's layers now number four: the production filesystem, the local backup, the off-site copy and the cloud-tier tertiary, each layer answering the question that the layer below it cannot.
The closing observation about the off-site script is the same observation about every tool in this series: the script is not clever, it is not modern, and it does not use any technology that did not exist twenty years ago. It is a shell script that runs rsync, deletes old copies and verifies checksums, and the reason it survives every technology cycle is the same reason the tools it wraps survive: the backup problem is about discipline, not about technology, and the script implements the discipline that the retention policy names, the credentials restrict and the monitoring confirms, with the same quiet reliability that the rsync tool has provided for decades.