Server down or restoring a backup

Situation #

The Dispatch server stays silent, answers safe_mode, or reports damaged data after a power cut, a crash or a disk fault.

What keeps working #

• Your records stay on disk. A forced stop leaves the database consistent, because every change is written as a whole transaction.
• Reading, exporting and backing up keep working in safe mode.
• Drivers' queued writes wait on their phones and send again with their idempotency keys when the server returns, so nothing posts twice. See "Safe retries (idempotency)".
• Deliveries to webhooks wait in the queue and send after the server returns.

Steps #

1. Check the server computer. Is it on, connected and out of disk space? Open /health from the server itself.
2. Look at the answer.
   • No answer: start the server (dispatch-server serve). Read the first lines of its log. If the data folder, port or licence settings are wrong, the message says which.
   • ok: true: the server is well. Check the office network and the address people use.
   • safe_mode: true: the server crashed three times in five minutes. Keep the newest file in the crash folder of the data folder, and go to step 4 after five quiet minutes.
   • ok: false with the banner "The app needs your help": the safety check found a problem. Go to step 4.
3. Give dispatchers a fallback while you work: take calls on paper with the pickup address, phone number and time, and enter them when the server returns.
4. Restore the newest good backup.
   1. Stop the server.
   2. Open the backups folder in the data folder. Pick the newest folder by time.
   3. Run dispatch-server restore --from <that folder> --data-dir <data folder>.
   4. Read the message. It names each owner and gives their reset-password command.
5. Run reset-password for each owner, then for everyone else. Each person gets a one-time password and chooses their own at sign-in.
6. Start the server. Owners and managers set up the authenticator app again. Make a new signing secret for each webhook endpoint.
7. If the restore reports a failed check, try the next older backup. Your data from before the restore stays safe as the before-restore backup.

How to confirm it worked #

• /health answers ok: true and safe_mode: false.
• The owner signs in and GET /tow/board shows the calls you expect.
• Open an invoice: its lines and payments are in step, because every invoice passes an integrity check when it is read.
• Take a call and see it on the board.

Afterwards #

1. Enter the calls you took on paper.
2. Anything done after the backup time is missing. Ask drivers for their queued work and dispatchers for recent calls.
3. Find the cause: disk space, a power cut, hardware. Read the newest crash log.
4. Make a backup now, and check that tomorrow's nightly backup appears.
5. Put the data folder on a disk with a nightly copy. See "Setting up the office server".