Overview

In days of yore, the rust that spins (hard drives) were transcribed to the rust on tape (LTO). Current days, solid state storage is backed up to hard drives, which is still backed up to … LTO.

The more things change, the more they stay the same. Going to ignore LTO for now, because I don’t have that kind of coin, nor backup capacity requirements.

Iterations

• First iteration was done using raid 1 pairs of drives. This was more trouble than it was worth; any hiccup would require a full drive re-sync.
• Par2 recovery codes was done on next iteration. Viable solution, but increases time for backup considerably.
• Current iteration is simple brute force. Use dar’s internal sha512 checksum, results in a simple yes/no verification. Add resiliency via multiple backups.
• Enable par2 if Hail Mary recovery from single drive is needed.

Setup

fstab:

$ ag backup /etc/fstab 
11:LABEL=BACKUP    /mnt/backup  ext4 noatime,errors=remount-ro,noauto 0 0

Dar will be writing multiple 100G files, but nothing larger. No journal needed, as it’s sustained write, sync, and shelf. I simply place the ext4 filesystem on the raw block device; never will need to resize, boot from, or mount it on another OS. Mount by label works fine. Keep it simple.

mkfs.ext4 -L BACKUP /dev/sdX -m 0 -T largefile4 -O ^has_journal,^huge_file

Backup

backup.sh:

#!/bin/sh
cd /

set -eo pipefail


DEST=/mnt/backup
DATE=$(date +%Y%m%d-%H%M%S)
ARCHIVE=$(hostname)-${DATE}
LAST=$(hostname)-last

partprobe
mount ${DEST}

if [ -f ${DEST}/${LAST} ]; then
    REF="--ref $(cat ${DEST}/${LAST})"
fi

dar -Q -q \
    --create ${DEST}/${ARCHIVE}-data \
    --aux ${DEST}/${ARCHIVE}-index ${REF} \
    --key $(cat /root/.backup) \
    --fs-root / \
    --compression=zstd:9:256K \
    -Z '*.zip' -Z '*.mov' -Z '*.mp4' \
    --exclude-from-file /root/exclude.txt \
    --multi-thread=4 \
    --slice=100G --min-digits=2 \
    --alter=secu \
    --hash sha512 

# (
#   cd ${DEST}
#   par2 c -r10 -n1 -T1 ${ARCHIVE} ${ARCHIVE}-*.dar
# )

echo ${DEST}/${ARCHIVE}-index > ${DEST}/${LAST}
cp /root/contact.txt ${DEST}/

df -h ${DEST}

umount ${DEST}
sync

exclude.txt:

# mounts
/dev
/media
/mnt
/proc
/run
/sys
/tmp

# virt and cache
/var/cache
/var/lib/docker
/var/lib/libvirt

Dar command notes:

• --aux indexing and incremental backup support.
• --compression=zstd:9:256K able to keep the drives fed without much fuss.
• --alter=secu disable ctime check on incremental backups.