Operations and maintenance

This document is for whoever manages an Orafila installation: checks, updates, backup and restore, incidents, and supporting users. The deploy/... commands are typed on the server, from the orafila folder.

Who does what

Checks

Docker restarts a service that stops by itself; the health checks of the app (every 30 seconds) and of the mail worker (a signal every 15 seconds) mark it as "unhealthy" if it does not respond.

Updates

Orafila:

deploy/nova.sh update

The command checks the memory, downloads the new version from the release server and verifies its digital signature, takes a safety backup, replaces Orafila's files, rebuilds the image and restarts; the database migrations are applied automatically at start-up. deploy/.env, keys and backups are not touched. It refuses to start if someone has edited Orafila's files on the server by hand (it lists them) and never goes back to an earlier version by itself. deploy/nova.sh version tells which version is installed and whether there is a new one. The service is interrupted for a few moments while the containers restart.

Operating system and Docker: security updates are installed automatically every day. If *** System restart required *** appears when you log in, restart (sudo reboot): after a couple of minutes everything starts again by itself; check with deploy/nova.sh status.

Once a month, the base images (database, Caddy, Node.js), which contain security fixes:

cd deploy && docker compose pull db caddy && docker compose build --pull && cd ..
deploy/nova.sh up

Disk space: up (and therefore update) deletes the previous version of the image every time, and the build cache when less than 4 GB is left free. To free space sooner: docker builder prune -af.

Never use docker compose down -v, docker volume rm or docker system prune --volumes: they delete the database.

Backups

• Automatic every night at 02:30 (Italian time, plus up to 15 random minutes), after deploy/nova.sh install-timer; if the server was off, it runs when the server is switched back on.
• By hand at any time: deploy/nova.sh backup. List: deploy/nova.sh backups.
• Each backup is the complete database (all companies), encrypted with age, verified and then copied to the bucket if one is configured. The last 7 are kept on the server and 30 days in the bucket (values can be changed).
• The names are the UTC date and time: nova-20261020T013000Z.dump.age.
• Backups contain the data; not the database users: after a restore on a new server, the database passwords can be new ones.

Safekeeping of the keys (in the password manager, off the server):

1. the backups' private key (deploy/secrets/backup.agekey);
2. APP_ENCRYPTION_KEY;
3. recommended: the whole deploy/.env, updated after every change.

Restore

On the same server (for example to go back to an earlier state):

deploy/nova.sh restore nova-YYYYMMDDTHHMMSSZ.dump.age --yes [--key ~/saved-key.txt]

The restore first takes a safety backup of the current state, stops the app and the mail worker, and loads the backup into a temporary database that replaces the current one only if loading succeeds (with an external database, in a single transaction); it then applies the migrations newer than the backup, restarts and waits for the services to be healthy. If the key does not open the backup, it stops before changing anything. All the companies in the installation go back to the state of the backup.

On a new server (server lost or broken):

1. new server, DNS record updated to the new IP, package downloaded, sudo deploy/setup-host.sh;
2. the backups' private key put back in deploy/secrets/backup.agekey and deploy/.env recreated from the password manager (at least APP_ENCRYPTION_KEY and the bucket details);
3. deploy/nova.sh init --domain ... --email ... to fill in the missing values, then deploy/nova.sh up;
4. deploy/nova.sh backups and deploy/nova.sh restore <latest file in the bucket> --yes: the file is downloaded from the bucket, checked and restored;
5. deploy/nova.sh install-timer.

Users sign in with their password and their verification code, as before. If nobody can complete two-step verification, APP_ENCRYPTION_KEY is not the one from the original server. The detailed steps are in the guides for Oracle (in Italian) and DigitalOcean (in Italian).

Test the restore when the service starts, with the copy of the key kept in the password manager: a backup that has never been restored guarantees nothing.

If you suspect an intrusion into the server

Consider everything that was on the server compromised: deploy/.env (database passwords, bucket and AI keys, APP_ENCRYPTION_KEY) and the backup key.

1. Stop the server from the provider's console, without deleting it.
2. Revoke the bucket access key, the AI providers' keys and, if the server had them, the release access credentials (the Orafila provider revokes them); create new ones.
3. Rebuild on a new server with new database passwords, gateway secret and backup key (keeping APP_ENCRYPTION_KEY, which is needed to read the data) and restore the last backup from before the intrusion, opened with the old backup key.
4. In the application: change the mailbox passwords (first at the mail provider, then in Queues and mail) and have two-step verification reset for all users (Settings › Users), and their access too if appropriate.
5. Assess with your privacy adviser whether the breach must be notified (GDPR Articles 33-34).

Platform operator tasks

These operations appear in the company's activity log with “System” as the author.

Supporting users (company administrators)

Common problems