Backups and upgrades

Backups

provision.sh installs a nightly backup. Each run writes a compressed dump into /opt/crewhall/backups and keeps a month.

./backup.sh                              # run one now
systemctl list-timers 'crewhall-*'       # what is scheduled

Those dumps are on the same disk as the database: they survive a mistake, not a lost server. Copy them somewhere else as well — restic or rclone to object storage — and keep the master key separate from them, in a password manager.

Test the restore before you need it

On a second machine, with the same .env (above all the same CREWHALL_MASTER_KEY):

./setup.sh && ./setup.sh
docker compose -f compose.prod.yml stop api
docker compose -f compose.prod.yml exec -T db \
  pg_restore -U crewhall -d crewhall --clean --if-exists < backups/crewhall-<date>.dump
docker compose -f compose.prod.yml start api

Then sign in, open an agent and check that a model provider still answers. That last step is the point: it proves the master key matches the data.

Upgrades

./upgrade.sh v0.2.0     # backup, pull, restart, health check
./upgrade.sh v0.1.0     # back to the previous version

upgrade.sh always takes a backup first and tells you how to go back if the new version does not answer.

Database migrations run when the API starts and are not undone by going back to an older image. If a version that migrated the database fails, restore the dump as well.

Deploying by itself

Every five minutes the server asks the registry whether the tag in .env points at a new image, and upgrades when it does.

journalctl -u crewhall-autodeploy --since today       what it has done
systemctl disable --now crewhall-autodeploy.timer     stop deploying by itself

Follow a version tag in production (CREWHALL_IMAGE_TAG=0.1.0), so a release goes live when you tag it. The tag edge follows every push to the main branch and belongs on a staging server.

Keeping an eye on it

docker compose -f compose.prod.yml ps
docker compose -f compose.prod.yml logs -f api
curl -s https://your-domain/api/health

An external uptime check on /api/health is worth the five minutes it takes to set up: it is the difference between knowing at 09:00 and hearing from a colleague at 11:00.