btrfs send backups

This setup backs up prod-srv by creating read-only btrfs snapshots of @rootfs and sending btrfs send streams to backup-srv.

Two remote storage modes are supported:

• BACKUP_MODE=encrypted-stream: encrypt each send stream on prod-srv and store opaque .btrfs.gpg files on backup-srv.
• BACKUP_MODE=remote-btrfs: have prod-srv unlock a LUKS volume on backup-srv, then use normal btrfs receive into that mounted filesystem.

In encrypted-stream mode, the backup server stores encrypted send-stream files, not received btrfs subvolumes. This is intentional: if backup-srv ran btrfs receive, its administrator could read the restored plaintext files. With this design, the decryption passphrase exists only on prod-srv.

This prevents backup-srv from decrypting the backups. It cannot prevent a root administrator of backup-srv from deleting encrypted files or wiping the disk, because that host controls its own storage.

remote-btrfs is more bandwidth and storage efficient after the first full send because backup-srv stores real btrfs snapshots with shared extents. It protects against offline disk theft, but while the LUKS volume is unlocked and mounted, root on backup-srv can read the plaintext backup data.

Files

• prod-btrfs-backup.sh: run from cron on prod-srv.
• backup-receive-btrfs-stream.sh: forced SSH receiver on backup-srv.
• backup-prune-btrfs-streams.sh: retention job run locally on backup-srv.

Setup on backup-srv

Create a restricted ingest user. Its password is locked and the backup SSH key will only be allowed to run the root-owned receiver through sudo.

sudo adduser --system --group --home /var/lib/btrfs-ingest --shell /bin/sh btrfs-ingest
sudo passwd -l btrfs-ingest
sudo usermod -aG ssh.app btrfs-ingest
sudo install -d -m 0700 -o btrfs-ingest -g btrfs-ingest /var/lib/btrfs-ingest/.ssh
sudo install -d -m 0750 -o root -g root /mnt/sdb1/btrfs-send-backups
sudo install -m 0755 -o root -g root backup-receive-btrfs-stream.sh /usr/local/sbin/btrfs-backup-receive
sudo install -m 0755 -o root -g root backup-prune-btrfs-streams.sh /usr/local/sbin/btrfs-backup-prune

Allow only the receiver command via sudo:

sudo visudo -f /etc/sudoers.d/btrfs-ingest

Add:

btrfs-ingest ALL=(root) NOPASSWD: /usr/local/sbin/btrfs-backup-receive

Generate an SSH key on prod-srv, then install the public key in /var/lib/btrfs-ingest/.ssh/authorized_keys on backup-srv with a forced command:

command="sudo /usr/local/sbin/btrfs-backup-receive \"$SSH_ORIGINAL_COMMAND\"",restrict ssh-ed25519 AAAA... prod-srv-btrfs-backup

The forced command means a compromise of prod-srv can upload new backup files using this key, but cannot run arbitrary commands or delete old backups through SSH. Stored files are root-owned and mode 0400.

On YunoHost, SSH login is restricted by SSH permission groups. For this local system account, add it to the local ssh.app group. The ssh.main group is managed through LDAP for normal YunoHost users and is not the right target for usermod.

For remote-btrfs, the same forced command also allows only the constrained open-luks, receive, prune-received, and close-luks actions. A compromised prod-srv with the SSH key and LUKS key can therefore open the remote backup volume during its backup window, but it still cannot run arbitrary commands on backup-srv.

Optional remote-btrfs LUKS setup

Use this mode when bandwidth is scarce and you accept that backup-srv can read plaintext during the backup window.

Generate a LUKS key file on prod-srv:

sudo sh -c 'umask 077; openssl rand -base64 64 > /root/.config/btrfs-send-backup/luks-key'

Temporarily copy that key to backup-srv over a trusted channel as /root/prod-srv-luks-key, create a LUKS container on the backup disk, and put btrfs inside it. This destroys existing data on /dev/sdb1.

sudo cryptsetup luksFormat /dev/sdb1 /root/prod-srv-luks-key
sudo cryptsetup luksOpen /dev/sdb1 btrfs_backup --key-file /root/prod-srv-luks-key
sudo mkfs.btrfs /dev/mapper/btrfs_backup
sudo install -d -m 0750 -o root -g root /mnt/btrfs-backup
sudo mount /dev/mapper/btrfs_backup /mnt/btrfs-backup
sudo install -d -m 0750 -o root -g root /mnt/btrfs-backup/received
sudo umount /mnt/btrfs-backup
sudo cryptsetup luksClose btrfs_backup
sudo shred -u /root/prod-srv-luks-key

Configure the receiver defaults on backup-srv:

sudo tee /etc/default/btrfs-backup-receive >/dev/null <<'EOF'
LUKS_DEVICE=/dev/sdb1
LUKS_NAME=btrfs_backup
LUKS_MOUNT=/mnt/btrfs-backup
RECEIVED_ROOT=/mnt/btrfs-backup/received
KEEP_DAILY=7
KEEP_WEEKLY=4
KEEP_MONTHLY=12
EOF

In remote-btrfs mode, pruning happens while prod-srv has unlocked the LUKS volume. The prune implementation runs on backup-srv and keeps the latest snapshot plus the configured daily, weekly, and monthly points. Unlike encrypted stream files, received btrfs snapshots do not require preserving every ancestor stream for restore.

For encrypted-stream mode, create the backup-server retention cron:

sudo tee /etc/cron.d/btrfs-backup-prune >/dev/null <<'EOF'
17 4 * * * root BACKUP_ROOT=/mnt/sdb1/btrfs-send-backups BACKUP_SET=prod-srv/rootfs KEEP_DAILY=7 KEEP_WEEKLY=4 KEEP_MONTHLY=12 /usr/local/sbin/btrfs-backup-prune
EOF

Setup on prod-srv

Install the producer script:

sudo install -m 0755 -o root -g root prod-btrfs-backup.sh /usr/local/sbin/btrfs-send-backup
sudo install -d -m 0700 -o root -g root /etc/btrfs-send-backup /root/.config/btrfs-send-backup

For encrypted-stream mode, create the encryption passphrase. Keep this off backup-srv; without it the backup files cannot be restored.

sudo sh -c 'umask 077; openssl rand -base64 48 > /root/.config/btrfs-send-backup/passphrase'

Create /etc/btrfs-send-backup/prod.conf:

SOURCE_TOP=/mnt/btrfs-root
SOURCE_SUBVOL=@rootfs
SNAPSHOT_DIR=/mnt/btrfs-root/@backup-snapshots
SNAPSHOT_PREFIX=prod-srv-rootfs
STATE_FILE=/var/lib/btrfs-send-backup/last-sent
BACKUP_SSH_TARGET=btrfs-ingest@backup-srv
# Optional. Leave empty for the default SSH port 22.
BACKUP_SSH_PORT=
# Optional. Set this when cron/root should use a dedicated backup key.
BACKUP_SSH_IDENTITY_FILE=
BACKUP_SET=prod-srv/rootfs
PASSPHRASE_FILE=/root/.config/btrfs-send-backup/passphrase
BACKUP_MODE=encrypted-stream
FULL_INTERVAL_DAYS=32
LOCAL_KEEP_DAILY=7
LOCAL_KEEP_WEEKLY=4
LOCAL_KEEP_MONTHLY=12

# Keep this list explicit. Include databases and application services whose
# files must be transactionally quiet while the btrfs snapshot is created.
SERVICES_TO_QUIESCE=(nginx php8.2-fpm mysql postgresql redis-server)

For the LUKS-backed receive mode, use this instead:

BACKUP_MODE=remote-btrfs
LUKS_KEY_FILE=/root/.config/btrfs-send-backup/luks-key
FULL_INTERVAL_DAYS=36500

FULL_INTERVAL_DAYS=36500 effectively avoids periodic full sends. Keep the local STATE_FILE and the newest local snapshot intact, because each new incremental send uses the previous successful snapshot as its parent.

Adjust SERVICES_TO_QUIESCE to the real services on your YunoHost system:

yunohost service status
systemctl list-units --type=service --state=running

Do not include ssh or the cron daemon. The services are stopped only for the few seconds needed to create the read-only btrfs snapshot, then restarted before the encrypted stream is transferred.

Create the cron job on prod-srv:

sudo tee /etc/cron.d/btrfs-send-backup >/dev/null <<'EOF'
05 2 * * * root /usr/local/sbin/btrfs-send-backup
EOF

First run and checks

Run once manually on prod-srv:

sudo /usr/local/sbin/btrfs-send-backup

Check backup-srv:

sudo find /mnt/sdb1/btrfs-send-backups/prod-srv/rootfs -maxdepth 1 -type f -ls

You should see files ending in .btrfs.gpg and matching .manifest files. The first backup is a full btrfs send stream. Later backups are incremental until FULL_INTERVAL_DAYS causes a new full stream. The prune job keeps the configured daily, weekly, and monthly restore points, plus any ancestor streams needed to restore those points.

For remote-btrfs, manually unlock the LUKS volume on backup-srv for inspection, then close it again:

sudo cryptsetup luksOpen /dev/sdb1 btrfs_backup --key-file /path/to/key-copy
sudo mount /dev/mapper/btrfs_backup /mnt/btrfs-backup
sudo btrfs subvolume list /mnt/btrfs-backup
sudo umount /mnt/btrfs-backup
sudo cryptsetup luksClose btrfs_backup

Restore outline for encrypted-stream

Copy the needed encrypted stream chain from backup-srv to a trusted machine that has the passphrase, then apply them in chronological order:

gpg --batch --pinentry-mode loopback --passphrase-file /root/.config/btrfs-send-backup/passphrase \
  --decrypt prod-srv-rootfs-YYYYmmddTHHMMSSZ.full.btrfs.gpg \
  | btrfs receive /mnt/restore-target

For incremental files, replay each stream after its parent has been received. The .manifest files record the snapshot and parent names.

Restore outline for remote-btrfs

Unlock and mount the LUKS volume on a trusted machine that has the key. The received snapshots are directly available as read-only btrfs subvolumes under received/prod-srv/rootfs; no stream chain replay is needed for an existing received snapshot.