The full disk that killed my database

Ten years of shipping software, most of them with an SRE (site reliability engineering) team and a DevOps rotation between me and the database. That is exactly why the service I launched had no database backup: backups, disk monitoring, and the rest of operational hygiene were things other people owned, and I had promised myself we would get to them once the service had been live for a while. Then, on a Saturday afternoon, the database behind the site shut down cleanly and refused to come back. The while had arrived before I was ready.

Postgres logged the same panic on every restart: could not locate a valid checkpoint record. Here is the shape of the log, trimmed to the lines that matter:

LOG:  database system was shut down at 2026-08-08 14:25:17 UTC
LOG:  invalid record length at 0/5F348C8: expected at least 24, got 0
LOG:  invalid checkpoint record
PANIC:  could not locate a valid checkpoint record

The failure that looks like a mystery

Postgres relies on a write-ahead log (WAL). Every change is appended there before it touches the real data files, and every so often the server writes a checkpoint record that marks a safe, consistent point. The control file remembers where that checkpoint lives. On startup, Postgres reads that location and expects to find a valid record.

Mine had zeros where the record should be. expected at least 24, got 0 means the WAL file had been truncated to nothing at exactly the block the database needed to boot. The clean shutdown makes it worse: the database stopped properly, so whatever damaged the file happened after it stopped.

You might read this and reach for the application code, the deploy, the migration that ran an hour before. None of those touch the WAL. When a database that shut down cleanly comes back with a corrupted checkpoint, that is a storage problem. The list of things that do this to files is short, and the first one to check is free disk space.

I recognized this signature the moment I saw it. A clean shutdown followed by a corrupted checkpoint is one of the few failures where the log names the guilty layer, and I had watched teams work through exactly this in incident reviews for years. Recognition was the part that cost me nothing. It also changed nothing: the fix was mine now, and the work that prevents this failure had been deferred by me.

The first place to look is disk

The mistake would have been to debug the app. The log was telling me the truth already, and the fastest way to confirm it is two commands on the host:

df -h
docker system df

My disk was at 100 percent. docker system df showed why: images nobody used anymore were holding most of the space.

In a team environment, someone would have had a disk dashboard open for weeks. I did not have a dashboard. I had df -h, and that was the whole monitoring story.

A full disk corrupts a database the way a half-finished letter corrupts a diary. The filesystem does not stop writing when it runs out of room; it writes until it cannot, and the write that does not finish leaves a file with zeros where real data should be. Postgres appends to the WAL constantly. When the append cannot complete, the file is wrong, and on the next boot Postgres treats a file that is wrong as a file that is dangerous.

Notice what a full disk does not do. It does not alert you. The machine keeps answering SSH, the web server keeps serving pages it already has in memory, and the filesystem quietly fails the writes that do not fit. The database is the first thing that notices, because it is the thing writing constantly. By the time it panics, the disk has been full for a while.

The recovery, in the order that matters

Once the cause was clear, the order of operations mattered more than any single command. I did not have a backup, so the plan was to protect the copy I did have, prove it worked, and only then point the live service at it.

I had read a hundred incident reviews about recoveries like this one, always from the observer's seat. Reading about a recovery and doing one are different skills, and the doing was mine now.

Know your containers and volumes first

Before touching anything, get the exact names: the database container, the image it runs, and the volume it stores data in. Panic time is not when you want to guess.

docker ps --format 'table {{.Names}}\t{{.Image}}'            # find the db container
docker inspect <db-container> --format '{{.Config.Image}}'   # the exact image tag it runs
docker inspect <db-container> \
  --format '{{range .Mounts}}{{.Type}} {{.Name}} -> {{.Destination}}{{println}}{{end}}'

The last command is the one that matters. It shows how the database is attached to disk: a named volume, and where inside the container it is mounted. On the official Postgres images that path is /var/lib/postgresql/data. Everything that follows is a variation on one theme: copy that volume, verify the copy, swap it in.

Record the Postgres major version before you rebuild anything. A data directory can only be read by the same major version, and the fresh image has to match:

docker run --rm -v <db-volume>:/pgdata alpine cat /pgdata/PG_VERSION

1. Stop the restart loop

A crash-looping database keeps writing to a damaged data directory. Freeze it, and stop your orchestration from auto-restarting it for the duration.

docker stop <db-container>

2. Snapshot the volume before touching it

This was not a restore-from-clean-copy backup; it was the raw material. Tar the existing volume while it is still readable. It gives you something to work from and a way back if a later step makes things worse.

docker run --rm -v <db-volume>:/pgdata -v "$PWD:/backup" alpine \
  tar czf /backup/db-recovery-$(date +%F).tar.gz -C /pgdata .

If the disk is full - our situation - the tarball has to land somewhere with room. Attach a spare volume for it, or stream it to another machine:

docker run --rm -v <db-volume>:/pgdata alpine tar czf - -C /pgdata . \
  | ssh you@backup-host 'cat > db-recovery-$(date +%F).tar.gz'

3. Bring the volume up on a fresh instance

Recovery runs against a copy, never against the live data. Create a fresh volume, fill it from the tarball, and start a new container from a fresh image of the same major version you recorded in the first step.

docker volume create db-recovery
docker run --rm -v db-recovery:/pgdata -v "$PWD:/backup" alpine \
  tar xzf /backup/db-recovery-$(date +%F).tar.gz -C /pgdata

docker run -d --name db-recovery \
  -v db-recovery:/var/lib/postgresql/data \
  <same-image-tag-from-inspect>

Because the data directory already exists, Postgres skips initialization and uses it as-is. The POSTGRES_USER and POSTGRES_PASSWORD variables only matter for an empty directory, so the credentials are whatever the database already had.

4. Run verification queries

A green boot is not the goal; evidence is. Read the log, list the tables, count the rows in a table that must not lose data, and read recent rows to confirm you are looking at current data rather than an older copy.

docker logs db-recovery --tail 20          # look for "ready to accept connections", no PANIC
docker exec db-recovery psql -U <db-user> -d <db-name> -c '\dt'
docker exec db-recovery psql -U <db-user> -d <db-name> \
  -c 'SELECT count(*) FROM <a table that must not lose data>;'
docker exec db-recovery psql -U <db-user> -d <db-name> \
  -c 'SELECT * FROM <recent-data-table> ORDER BY <created_at> DESC LIMIT 5;'

5. Free the disk on the VPS

With a working copy in hand, prune the images and build cache that filled the disk (the commands in the "Prune on a schedule" section below). This is the calm version of the job, done while the database is still down. Confirm the disk has room before the swap.

6. Swap the live service to the verified volume

Stop the live container, then point the service at the verified volume. There are two ways, and which one you use depends on how the service is defined.

Point the service config at the recovery volume. If the database service is defined in a compose file or a deploy platform's service config, edit the volume reference so it mounts db-recovery instead of the old volume, then recreate:

# edit docker-compose.yml: the db service now mounts db-recovery instead of <db-volume>
docker compose up -d db

The container is recreated with the new volume. This is what I did: the live service now mounts the volume I verified.

Or adopt the verified data into the volume name the service already expects. If you would rather not touch the config, rebuild the original volume name from the verified data, then recreate the container:

docker rm <db-container>                   # container is already stopped
docker volume rm <db-volume>               # the broken volume - the tarball is safe
docker volume create <db-volume>
docker run --rm -v db-recovery:/from -v <db-volume>:/to alpine \
  sh -c 'tar -C /from -cf - . | tar -C /to -xf -'
docker compose up -d db

Whichever route you take, re-run the verification queries against the live container, and keep the recovery container around for a few days. Do not destroy the only working copy while you are still proving the swap.

The fallback I did not need. If the recovered volume had refused to boot, the next tool would have been pg_resetwal, which clears the write-ahead log and control information so a database with a corrupted checkpoint can start. It is blunt: the Postgres docs call it a last resort, warn that the recovered database may contain inconsistent data, and tell you to dump it, rebuild, and restore as soon as it is up. It refuses to run while the server is live, and it only works with the same major version. This time the volume came up on a fresh instance and verified clean, so I never reached for it.

Why the disk was full

The database was the victim. The cause was the other half of the disk bill: the Docker image store. Every deploy builds a new image. The old image and every intermediate layer that built it stay on disk until something removes them. After enough deploys, the machine is holding the current version, every version before it, and a pile of layers that are not even tagged. Those are the hanging images, and they do not clean up after themselves.

The command that makes this visible:

docker system df

That prints a table with columns for images, containers, volumes, and build cache, plus a reclaimable figure for each. The reclaimable number is your pending housekeeping. On my VPS it was most of the disk.

Prune on a schedule, not in a crisis

The one-off prune I did was a reset, not a fix. The fix is a cron job that runs the same housekeeping every week so the disk never sneaks up on you again. The five commands below cover the parts of a Docker VPS that grow without anyone noticing: stopped containers, old images, build cache, unused networks, and the systemd journal.

docker container prune -f
docker image prune -af --filter "until=168h"
docker builder prune -af --filter "until=168h"
docker network prune -f
journalctl --vacuum-time=14d

Put them in a script, then register it in the root crontab. A weekly Sunday run is a good cadence:

0 3 * * 0 /usr/local/bin/server-maintenance.sh >> /var/log/server-maintenance.log 2>&1

Two details keep this safe. The until=168h filters protect a brand-new image you might still roll back to, so the prune never deletes yesterday's release. And none of the commands touch Docker volumes: a volume is data, not cruft, and pruning should never sweep it by accident.

What changed after

A full disk is a database outage, not a sysadmin chore. Three habits stop the sequel. The pattern behind them is the honest summary of this post: I treated operational work as something to start once the service had been live for a while, and the while expired without consulting me. If you are the whole team, deferring it hands the same work to a future version of you, under worse conditions.

• Alert on disk before it is full. A disk at 100 percent is already an incident. Alert at 70 and you act on a Tuesday instead of a Saturday. Uptime monitors will not catch this: the site stays up while the disk silently fills. Monitor the disk directly.
• Prune on a schedule. The cron job above keeps the reclaimable number small so the disk never becomes the thing that decides when your site goes down.
• Back up the database and restore it once. I did not have a backup that afternoon; the data came back because the volume was still readable and I verified it before pointing the live service at it. That is luck, and you should not buy it twice. A scheduled dump plus a restore drill takes an hour, and it turns a near-miss into a footnote.

What the incident taught me

• The log tells you where to look. A clean shutdown followed by a corrupted checkpoint is a storage signature; it says nothing about your code, so do not spend the first hour debugging your code.
• Disk space is a database reliability concern. The database writes constantly, so it is the first casualty of a full disk.
• Hanging images are deferred disk. Prune them on a schedule, not when something breaks.
• Reading about recovery and doing it are different skills. I had sat through a hundred postmortems and still improvised under pressure. A scheduled backup plus a restore drill turns the same move into routine.
• "Once it has been live for a while" is not a plan; it is a promise with no date. The incident picks its own, and its default pick is right after launch, when you are shipping the most.

If you want those habits before your own Saturday afternoon, that is what Deploy & Ship covers: monitoring the numbers that matter, restoring a database backup for real, and keeping a VPS from quietly filling up.