sg

← Writing

A Snapshot Paired with the Wrong Log

· systems · reliability

Fractal is a canvas that people draw on together and can zoom into. Its server is written in Go. It keeps each canvas in memory and saves it to a data directory on disk. I wanted an encrypted copy of that directory every hour, stored off the machine. I used the same tools as the laptop that runs our TV: restic (a backup tool), a Backblaze B2 bucket, and a systemd timer to run it on schedule.

Every file restic could copy out of that directory was safe on its own, but a backup made of those files could still restore the wrong canvas. It would load without a single error, so I'd only have found out on the day I needed it.

I found this while planning the backups, before an agent had written any of the scripts (how I work with agents). Nothing was lost. The bug only shows up if you read the server code in the same order restic reads the disk.

Each file on its own#

I started with the question of whether restic could ever copy a broken file. A canvas on disk is mostly two files.

canvas.snapshot holds the full canvas. The server rewrites it every 30 seconds while the canvas is active. It writes a temp file first and forces it to disk with fsync, then moves the current snapshot to canvas.snapshot.bak and renames the temp file into place. A snapshot file never changes after it's written, so if restic has one open, it reads a complete snapshot, even if a newer one replaces it partway through.

operations.log records each change to the canvas, such as a stroke, an undo, or a clear, which wipes the canvas. Each change is an operation, and it gets one line of JSON, added to the end of the file. The server applies an operation only after its line is safely on disk, and if the fsync after an append fails, it cuts the log back to where it was. When the server trims the log (more on that below), it uses the same temp file, fsync, and rename as the snapshot.

A reader that catches the log in the middle of a write sees everything before it, plus maybe a half-written last line. When the server loads the log, it cuts that line off, because it belonged to an operation that was never applied or confirmed to anyone.

So whatever restic opens, the server can load.

What a restore needs#

Loading a canvas takes both files at once, and they have to match. Every operation gets a sequence number, a running count. Each snapshot records OpSeq, the sequence number of the last operation it includes. To load a canvas, the server reads the snapshot, then replays every log record with a higher number.

The log can't grow forever, so after each snapshot write the server trims it. This trimming is called compaction. It keeps only the records newer than the previous snapshot, the one that just became .bak. That margin means .bak can still be replayed if the main snapshot is corrupt.

Here's an example with made-up numbers. Say three snapshot writes happen at OpSeq 100, 160, and 230:

Snapshot writecanvas.snapshot.bakoperations.log keeps
1st100previousnewer than previous
2nd160100101 onward
3rd230160161 onward

After the second write, a copy of the 100 snapshot still restores with the log. After the third, records 101 to 160 are gone from the log and exist only inside the newer snapshots.

The 100 snapshot needs 101 to 160, so paired with the log from after the third write, it can't get them. A snapshot and a log only belong together if restic read them close enough in time.

The order restic reads in#

restic doesn't freeze the files while it reads them, and it doesn't read a canvas's two files back to back. Its archiver lists each directory, sorts the names, and walks them in order. When it reaches a subdirectory, it goes in right away and finishes it before moving to the next name. It opens each file and hands it to a pool of workers that upload it, and when those workers are busy, the handoff waits.

The default canvas lives at the top of the data directory. Every other canvas lives under canvases/<id>/. Sorted, the top level looks like this (trimmed):

canvas.json
canvas.snapshot        <- opened here
canvas.snapshot.bak
canvases/              <- every other canvas: opened, queued for upload
operations.log         <- opened only after the canvases/ walk finishes
session.key

So restic opens the default canvas's snapshot, then walks every other canvas, and only then opens its log. With the example numbers from above, a live backup of the default canvas could go like this:

The walk through canvases/ slows down whenever the upload workers are busy. The first run uploads everything, and later runs skip files that haven't changed since the last backup, so the more data has changed under canvases/, the longer the walk takes.

Two snapshot writes during the walk are enough to break the pair. While someone is using the default canvas, that takes about 30 to 60 seconds, depending on where the 30-second timer was when restic opened the snapshot. Right after a clear it's shorter, because the server also writes a snapshot right away after each clear.

Why it would look fine#

That backup restores without a complaint. The snapshot reads fine and so does the log, and since no file is corrupt, every check that asks "does this load?" says yes.

The server replays the records newer than OpSeq, but the oldest of them are missing, so strokes drawn in that gap are just absent. A later delete or undo for one of those strokes finds nothing to remove, and replay skips it without logging anything.

A clear could make it worse, and nothing in the order of writes would stop it.1

A clear is a log record too, and the server writes an extra snapshot right after it. The compaction after that snapshot still keeps the clear record. If anyone draws afterwards, the next regular write comes within 30 seconds, and its compaction drops the clear record.2 A backup that paired the snapshot from before the clear with that log would bring back the canvas as it was before the clear, with the strokes drawn after it on top.

The fix: back up a copy, not the live directory#

The backup plan ruled out changes to the app, so the fix lives in the backup script and needed no change to the server. restic never sees the live directory. The script first copies each canvas directory into a local staging directory, then checks that the snapshot and log in the copy belong together.

bash
stage_dir() {
	local src=$1 dst=$2 attempt before
	for attempt in 1 2 3; do
		before=$(snapshot_id "$src")   # inode + mtime of canvas.snapshot
		if copy_dir_once "$src" "$dst" && [[ $before == "$(snapshot_id "$src")" ]]; then
			return 0
		fi
		echo "fractal-backup: $src changed during copy (attempt $attempt), retrying" >&2
	done
	die "could not stage a consistent copy of $src"
}

copy_dir_once copies every file in the directory, but not its subdirectories, *.tmp files or the log. Then it copies operations.log last.

The check is a bracket around that copy. It reads the identity of canvas.snapshot before and after: its inode (the file's ID on disk) and its modified time. If they match, no snapshot write finished during the copy, which means no compaction either, so the copied log still holds every record the copied snapshot needs.

If the identity changed, the script copies the directory again, and after three failures the run fails. Retries should be rare. Snapshot writes come 30 seconds apart, plus the extra one after each clear, and copying one directory locally takes far less time than uploading the whole tree over the network.

restic then backs up the staging directory, which nothing else writes to. A timer runs this every hour, and restic forget --prune thins out old backups, keeping 24 hourly, 7 daily, 4 weekly, and 6 monthly. The site's account data isn't in the canvas directories; it lives in SQLite and is copied separately with SQLite's .backup.

Reading restic's code for this turned up two smaller problems, and staging fixes both. First, restic quietly skips a file that disappears between listing a directory and opening it. A snapshot write has exactly that gap, between its two renames. In the staging copy, that moment either fails the copy or changes the snapshot's identity, so the directory is copied again. Second, half-written *.tmp files are skipped by name, so they never reach the backup.

Other fixes I turned down#

I looked at three other ways to keep the snapshot and log together. The most direct would be to make restic read them together. I can't control the order restic reads in: it sorts names itself, and when it opens a file depends on how fast uploads finish.

Stopping the server during the backup would make the copy consistent with no extra work. It would also disconnect everyone drawing, every hour.

LVM, Linux's disk volume manager, can take a snapshot that freezes a whole volume at one moment. It also adds mount and cleanup steps that can fail on their own, and I hadn't checked that the disk had spare space set aside for one. The staging copy gives the same guarantee for the files that matter.

What the restore check can't see#

A backup is only useful if it restores, so a second timer runs a restore check every day, and it passes. The check runs restic check to confirm the backup repository is intact, and it fails if the newest backup is more than two hours old. Then it restores that backup to a temp directory and starts a throwaway server on localhost. It connects to each canvas once over WebSocket and waits up to five seconds for the server to log that canvas's "ready" line.

That catches a broken backup repository, an old backup, or a snapshot that won't decode. It can't catch the bug in this post, because a snapshot paired with a log that was trimmed too far loads fine, and "ready" is all the check asks for.

The restored files do carry the evidence. Sequence numbers go up by one per operation, so a bad pair leaves a gap: the first log record newer than the snapshot has a sequence number above OpSeq + 1. In the example above, the 100 snapshot paired with the trimmed log would show its first record at 161.

A gap isn't always a bug, though. When the fsync after a log append fails and the server cuts the log back, that append has already used up its sequence numbers, so the next record skips ahead and leaves a gap with nothing missing.3

I haven't built the gap check either. The snapshot's OpSeq sits inside its binary header (Go's gob format), and the server never logs it. Reading it needs a small server change, and the backup plan ruled out app changes.

For now, the bracket around the staging copy is the only thing keeping each snapshot paired with a log that still holds what it needs, and nothing checks its work afterwards. If it ever lets a bad pair through, the daily restore check will still pass.

Footnotes

  1. I haven't seen this happen, and the live site can't hit it yet, because for now only a server started in dev mode can clear a canvas. ↩

  2. If nobody draws, that write is skipped, so the clear record stays in the log. ↩

  3. The same rollback is the one risk I accepted: a copy taken in that instant can keep a complete record that the server then rolls back. That takes a disk error, and a crash at the same moment would leave the same record behind anyway. ↩