Disaster recovery: rebuilding Nazeel from nothing

This is the sequence to rehearse, and the one to follow when the cluster is gone. It assumes the worst realistic case: the cluster no longer exists, and all you have is a backup file and your git server.

It is deliberately a runbook rather than a command. A restore that can be run against a live installation is a way to destroy one, and a disaster recovery plan a bank can sign off is a sequence its own people can rehearse, not a button.

hack/e2e-dr.sh performs exactly these steps on a fresh cluster, so the procedure cannot rot without the build failing.

What you need before you start

The keys are the thing to check now, not later. They live in the nazeel-api-keys Secret, they are deliberately not in the backup — a stolen volume snapshot should not yield both the data and the key that decrypts it — and the backup command says so every time it runs. Back that Secret up somewhere your database backup tooling does not reach.

The Secret holds three keys: backup decrypts the backups, session signs sign-in tokens and encrypts stored second-factor secrets, and audit-export signs evidence packages. Losing session costs everyone a re-enrolment. Losing backup costs you the backups, and nothing can be done about it afterwards.

The sequence

1. Install onto the new cluster

Follow the install guide as for a first install, with one difference: do not enter the licence key or configure anything yet. You are creating an empty installation to restore into.

helm install nazeel oci://registry.nazeel.sa/charts/nazeel --version <the version you were running> \
  --namespace nazeel-system --create-namespace \
  --set ingress.domain=dev.bank.internal \
  --set api.persistence.enabled=true

Install the same version you were running. A backup restores into its own schema; a newer version will migrate it forward on first start, but that is a second change happening during an incident, and one change at a time is the rule.

2. Put the keys back before the database

The API's signing keys live in the Secret nazeel-api-keys, not in the backup. Restore that Secret from wherever you keep it:

kubectl apply -f ./nazeel-api-keys.yaml       # the Secret as you backed it up
kubectl rollout restart deploy/nazeel-api -n nazeel-system

Without this Secret the backup in step 3 cannot be decrypted at all, which is why it comes first.

Keys first, because the API mints new ones on first start if it finds none — and a restored database whose stored TOTP secrets were encrypted under the old session key would then be undecryptable, locking out every user with a second factor.

If the keys are gone: the database still restores and the product works. Everyone signs in again, users with a second factor must re-enrol (Users → Reset second factor), and evidence packages exported before the loss can no longer be verified against this installation's public key — they remain verifiable against the old public key, which is inside those packages, so keep the packages themselves.

3. Restore the database

Follow Backup and restore in the admin guide, from step 1. In short: scale the API to zero, decrypt the backup, copy it over nazeel.db in the data volume from a pod mounting the same claim, scale back to one.

Decrypt it with the backup key from the Secret you restored in step 2:

kubectl -n nazeel-system get secret nazeel-api-keys -o jsonpath='{.data.backup}' | base64 -d > ./backup.key
nazeel-api verify-backup  --in ./nazeel-2026-09-12.db.enc --key-file ./backup.key
nazeel-api decrypt-backup --in ./nazeel-2026-09-12.db.enc --out ./nazeel.db --key-file ./backup.key

verify-backup proves the key is right and the file opens, before you overwrite anything. decrypt-backup then writes the plain database to copy into place. Remove that plain copy, and backup.key, once the restore is done.

4. Check what came back before going further

kubectl -n nazeel-system port-forward svc/nazeel-api 8080:8080 &
curl -s -H "Authorization: Bearer $TOKEN" localhost:8080/api/v1/audit/verify | jq

intact: true and a headSeq matching what you had is the signal that this is the same installation and not a new one wearing its name. Users, workspaces, repository definitions, settings and the licence are all in that database and should all be present. Applications and policies are not — they are Kubernetes objects, and step 6 deals with them.

5. Re-enter repository credentials

Credentials are in Kubernetes Secrets, not in the database, so they did not come back with it. Under Repositories, re-enter the credential for each repository. Everything else about the repository — its URL, ref, and the application built from it — is restored already.

6. Restore the Kubernetes objects

Applications (Baselines), Environments and Policies live in the cluster, not in the database. They are Custom Resources, and they went with the cluster. This is the step most likely to go wrong, and the one to rehearse.

If you have a CRD dump — from Velero, an etcd snapshot, or a scheduled kubectl get baselines,policies -A -o yaml — restore Baselines and Policies from it:

kubectl apply -f ./nazeel-baselines.yaml
kubectl apply -f ./nazeel-policies.yaml

Take that dump on a schedule. It is two objects per team and it is the difference between a ten-minute recovery and re-entering every team's quotas, TTL limits and terminal settings by hand:

kubectl get baselines,policies -A -o yaml > nazeel-crds-$(date +%F).yaml

If you do not have one, re-create them through the dashboard: connect each application again under Applications (its repository and credentials are restored, so this is filling in the same form), and set each team's Policy again under Quotas & cost. Write down what your policies were before you need this.

Environments are not worth restoring: a personal environment is a copy of a branch, and a developer re-creates theirs with nazeel up in a minute. Per-PR environments reappear on the next webhook event from an open pull request.

7. Confirm end to end

Create one environment, open its preview URL, check the audit log shows the whole sequence. The installation is recovered when a developer can do their ordinary work.

What you get back, and what you do not

Terminal recordings are the item worth thinking about in advance. They live on the data volume, so on this installation they are lost with the cluster — and they are evidence. The audit entries for those sessions are in the database and do come back; only the replayable files are gone. If your retention policy requires the recordings themselves to survive the cluster, copy them off the volume on a schedule of your own.

If you have no backup

The audit log is gone, and nothing recreates it: that is the point of a hash-chained log. What you can rebuild is the working product — install fresh, enter the licence, re-create workspaces and re-connect repositories. Users re-appear as they sign in through your identity provider.

Say so plainly in your incident report rather than quietly starting a new log. An auditor who finds a gap they were not told about draws a worse conclusion than one who reads that the cluster was lost on a known date and the log restarts there.

Rehearsing it

Run hack/e2e-dr.sh against a scratch cluster, or do the same by hand once a year against a restored copy of your own backup. The parts that go wrong in practice are always the same three: the nazeel-api-keys Secret nobody had backed up, repository credentials nobody had written down, and a chart version that no longer matched the backup. All three are cheap to check today and expensive to discover during an incident.