ADR 0007: The build sandbox, and where built images go

Date: 2026-09-11 Status: accepted

Context

Today a developer must build and push an image before nazeel up can use it. That is CI's job in production and nobody's job in a preview loop, and it is the gap between Nazeel and the thing it replaces: nobody runs a pipeline before docker compose up.

Closing it means running a build inside the customer's cluster, which means executing code the customer's developers wrote, on infrastructure a bank's security team signed off. Two questions follow, and this ADR is about both: what the build runs inside, and where its output goes.

Decision

Rootless BuildKit in a Job, not a daemon

No Docker daemon anywhere. A mounted Docker socket is root on the node, and no amount of surrounding policy makes that acceptable for something running repository code. Kaniko is archived upstream. Rootless BuildKit as a one-shot Kubernetes Job is what is left, and it is the right answer anyway: a build is a batch job, and Kubernetes already knows how to bound, schedule and clean up batch jobs.

One namespace, at privileged, and this is the real cost

nazeel-build holds builder Jobs and nothing else, and it is the only namespace Nazeel runs at the privileged Pod Security Standard.

This was not the plan. The plan said baseline, on the reasoning that rootless BuildKit needs unshare and nothing more. Running it showed otherwise, twice:

• With RuntimeDefault seccomp, BuildKit fails at fork/exec /proc/self/exe: operation not permitted. It needs Unconfined, which baseline forbids outright. So privileged it is.
• With allowPrivilegeEscalation: false, newuidmap fails with Could not set caps: the kernel strips that binary's file capabilities. It also needs SETUID and SETGID.

Each of those was tried, failed on a real cluster, and is now pinned by a test whose comment explains why, because every one of those changes looks like a security improvement and the failure appears minutes later inside a Job as a socket timeout.

Being honest about it is the point. A whitepaper that claimed everything runs at restricted would be wrong, and a security team would find that out themselves. The section in the whitepaper states it and then lists what contains it.

What contains it

The relaxation is affordable because it is surrounded:

• No service account token. A build cannot reach the Kubernetes API at all. This is the single most important control: it is the first thing anything hostile looks for.
• uid 1000, all capabilities dropped but two, no host paths, no privileged container.
• NetworkPolicy allow-list: DNS, the git server, the registry, and the team's declared dependency mirrors. Nothing else. A fetch from the public internet fails loudly rather than quietly succeeding and leaving a customer believing they are isolated. Link-local is carved out of every allowed range, because cloud instance metadata hands credentials to anything that can reach it.
• A deadline and a resource cap, per team.
• A separate checkout container. The git credential is used by an init container and is never visible where repository code runs. A Dockerfile can run arbitrary commands, and that token is exactly what one would look for.
• Admission screening of build arguments (they are recorded in the image history and readable by anyone who can pull it) and of context and Dockerfile paths (a path climbing out of the checkout would read the Job's own filesystem).

The customer's registry, not ours and not none

A built image has to go somewhere every node can pull from: a node cannot run an image that exists only on another node's disk. "No registry at all" is not possible on multi-node Kubernetes and is not offered.

The customer's own registry is the default, because every air-gapped install already operates one for mirroring. A bundled in-cluster registry is a quick-start option for an evaluation; it holds preview images only, and it requires trusting its CA on the nodes, which the install guide documents as a one-time step rather than hiding.

One function, ociprobe.ImageRef, decides an image's name. The builder pushes it, the Environment runs it, and the garbage collector deletes it; a disagreement between those three shows up as an ImagePullBackOff long after the build succeeded.

Commit, not branch. Digest, not tag.

A build is always of one resolved commit. Two builds of "main" producing different images under one name is the kind of thing nobody can debug, and a branch moves under a build that takes minutes.

A successful build is deployed by digest. A tag can be moved after the fact and the pod would quietly run something else; a digest cannot.

A build is not a release artefact

Preview images are not scanned for vulnerabilities and not signed. They are never promoted, they are deleted with the environment, and the release pipeline already scans and signs everything that actually ships. Adding an offline vulnerability database to every build namespace is a real operational cost for an artefact whose lifetime is hours. When a preview image can become something a customer runs, that changes, and this decision should be revisited then.

Consequences

• helm install creates a namespace at the privileged Pod Security Standard. Some customers will ask about it; the whitepaper answers before they have to.
• A cluster with a policy engine forbidding privileged namespaces outright cannot build. build.enabled: false removes the namespace, and image overrides keep working, which is exactly what the product did before this phase.
• Builds are visible, countable and killable as ordinary Jobs in one namespace, which is what a platform team wants when something is consuming a cluster.
• Egress is a per-team allow-list, so a team whose dependency mirror is not listed sees its builds fail at the fetch. That is the intended failure: the alternative is a build that reaches the internet from inside a bank.