ADR 0001: per-environment namespaces and header-based routing

Date: 2026-09-09. Status: accepted.

Context

An Environment redeploys only the services a developer changed and shares the rest with the Baseline. The copies must be isolated for quota and cleanup, yet still reach shared services by their normal in-cluster names, and traffic for one developer must reach that developer's copies without touching anyone else's.

Decision

1. One namespace per Environment, named <workspace>-<application>-<environment>. The operator labels it, applies a ResourceQuota from the matching Policy, and deletes the whole namespace when the Environment is deleted. Nothing else needs cleaning up.
2. Shared services are ExternalName aliases in the environment namespace pointing at the baseline instance in the application namespace. Copies keep calling ledger and nothing in the application has to change.
3. Routing is by HTTP header X-Nazeel-Env: <environment>. A Go reverse proxy owned by Nazeel will sit behind every baseline Service name and send requests carrying the header to <service>.<environment-namespace>, everything else to the baseline. The preview host <environment>--<application>.<domain> sets the header at the edge. Applications must forward the header hop to hop, like trace context; W3C baggage: nazeel-env= is accepted as an alternative carrier. Raw TCP dependencies are never copied.
4. Shared dependencies cannot be overridden. Databases and queues stay on the baseline.
5. API group is nazeel.sa, version v1alpha1, module github.com/nazeel/nazeel/operator.

Consequences

• Cluster admins see one namespace per developer environment, which maps cleanly to their existing quota, NetworkPolicy and audit tooling.
• The router is a single extra component with no mesh or Gateway API requirement.
• Async traffic (queues) is not isolated per environment in v0.
• Cross-namespace owner references are not allowed, so the Environment carries a finalizer that deletes its namespace instead.

Addendum 2026-09-09: router implementation decisions

Approved with the router binary (operator/cmd/router). All of these are enforced in code.

• Edge trust boundary. North-south requests arrive from the Ingress on the router's edge port. The environment is derived only from the Host header (<env>--<app>.<domain> or <app>.<domain>). Any client-supplied X-Nazeel-Env or baggage: nazeel-env= is discarded before forwarding, counted in nazeel_router_edge_headers_dropped_total, and logged at debug level. East-west requests on per-service listener ports trust the header, because only in-cluster callers can reach those ports.
• Fail-open to the baseline. A request naming an unknown environment is served by the baseline with no routing key forwarded. Each occurrence increments nazeel_router_unknown_env_total{source="edge"|"internal"} and emits a Kubernetes Event UnknownEnvironment on the Baseline, throttled to one per environment name per minute. The API step ingests these Events into the audit log.
• Ingress TLS. Preview hosts are one label under the domain, so a single wildcard certificate for *.<domain> covers every environment and the baseline. The operator references a customer-provided Secret through --ingress-tls-secret; the Secret must exist in each application namespace. Without it, preview URLs are plain HTTP and the Baseline carries the condition IngressTLS=False/NoTLSSecret. There is no cert-manager or ACME integration, by design: the product never reaches the internet.
• Per-service listener ports. The router identifies the target service by the port the request arrived on, never by Host. Ports are allocated from 20000 upward and recorded in Baseline.status.routes; an allocation never moves while the service exists. This caps routable service ports at 10000 per Baseline, far above any real application. Should a customer approach it, the fallback is Host-based identification on a single listener, which the table already supports through ParseHost.
• Selector flip. When the router first becomes available, the operator repoints each routable Service from its pods to the router in one update. kube-proxy reprograms endpoints within seconds; in-flight connections to the old endpoints may be reset. The Baseline reports RoutingReady=False/RouterSwitching while this happens and RouterReady afterwards. Accepted for v0.
• One image, two binaries. /manager and /router ship in the same image; the operator passes its own image to the router Deployment through ROUTER_IMAGE, so an air-gapped mirror has one artefact to carry.
• Deferred: pod-IP inference. Applications that propagate neither the header nor baggage lose the environment on their outbound calls. Inferring it from the caller's pod IP needs a pod watch and is left for a follow-up.
• TODO (API step): authenticate preview URLs. Resolved 2026-09-11, see ADR 0003. The API mints a short-lived signed cookie after a session check and the router verifies it in the request path, so the control plane is not in front of developer traffic.

Addendum 2026-09-09: rendering services from git

The Baseline's spec.source is rendered by operator/internal/source into the same normalized service list that inline spec.services provides. Decisions:

• Only the customer's git server is contacted. go-git clones into the operator's cache volume with credentials from a Secret (token+username, or ssh-privatekey+known_hosts, optional ca.crt for a private CA). SSH host keys are never trusted blindly; there is no insecure flag. Helm dependencies must be vendored under charts/; a missing dependency is a hard render error, never a repository fetch. Kustomize runs against an in-memory copy of the checkout so no base can leave the repository, and any URL-shaped reference is rejected before the build. Compose files are parsed without the host environment.
• Render results are cached by inputs hash (commit, path, kind, values, operator render version) as JSON on the cache volume and in memory. A branch ref is re-resolved only when the source spec changes, on an opt-in pollInterval, or as a retry after a failure.
• Rendering is asynchronous. A bounded worker pool renders while the reconciler reports Rendered=False/Rendering and requeues. What is already deployed is left untouched until a new list is ready.
• The effective service list lives in status.resolvedServices, merged with inline overrides from spec.services (shared flag, image, replicas, port protocols). Both reconcilers and the router read it through EffectiveServices(), so the rendered list is inspectable with kubectl and the Environment reconciler needs no git access.
• Warnings are first class. Everything the renderer drops or guesses is listed in status.renderWarnings (capped at 50, full list in the operator log) and summarised by the Rendered condition reason RenderedWithWarnings.
• Not carried over in this version: volumes, probes, init containers, sidecars beyond the first container, envFrom, and the contents of ConfigMaps and Secrets. Env references to ConfigMaps and Secrets are preserved and warned about. Supporting objects need a model change and are a follow-up.
• Operator RBAC now reads Secrets cluster-wide. Required to load git credentials from application namespaces. Credential values are never logged; error messages are scrubbed.
• SBOM additions: go-git, Helm SDK v3 (loader, chartutil, engine only; no cluster client), kustomize api and kyaml, compose-go v2.

Addendum 2026-09-09: TTL, idle sleep and wake

• Expiry is creationTimestamp + ttl, where ttl is spec.ttl or the Policy default, capped at the Policy maximum. A cap is never silent: condition TTL=False/CappedByPolicy and a TTLCapped Event. At expiry copies scale to zero and the phase is Expired (replacing Expiring, which never had a meaning). After Policy.spec.expiredGracePeriod (default 24h) the operator emits ExpiredDeleted and deletes the Environment object; its finalizer removes the namespace. Extension is raising spec.ttl; the operator scales the copies back up and records ExpiryExtended. Who may extend is enforced by the API, which knows users; the operator does not.
• Idle is measured from status.lastActivity, written by the router, or from the Ready transition when no request ever arrived. After sleepAfterIdle copies scale to zero, phase Sleeping, Event Sleeping. Environments without copies never sleep.
• Wake travels through status: the router writes status.wakeRequestedAt (throttled to one write per 10 s per environment) and holds the request for up to 90 s, polling its routing table until the operator reports Ready. On timeout it answers 503 with Retry-After and a self-contained bilingual page (Arabic RTL and English, no external assets), or JSON for API clients, and records WakeTimeout. Expired environments answer 410 at the edge and fall back to the baseline on east-west hops.
• Every transition is an Event written directly through events.k8s.io by the shared internal/events recorder, one per transition, because recorder aggregation would merge a day of Sleeping events into one series. Reasons: Expired, ExpiryExtended, TTLCapped, ExpiredDeleted, Sleeping, Woken, WakeTimeout, UnknownEnvironment.
• Time is injected (k8s.io/utils/clock) so lifecycle is tested with a fake clock.
• No polling loop: each reconcile requeues exactly at the next deadline among expiry, expiry plus grace, and idle sleep.

Addendum 2026-09-09: admission webhook and webhook certificates

• Policy is enforced at admission, not only by the reconciler. A validating webhook on Environment CREATE and UPDATE denies: unknown or shared-dependency overrides, images from registries outside allowedRegistries, TTL above maxTTL or non-positive durations, per-user and per-workspace environment counts at the Policy maximum, spec.terminal when the Policy disables terminals, and changes to baselineRef, owner or workspace. The reconciler keeps its own TTL cap and override checks as defense in depth for objects that predate a Policy change. Failure policy is Fail: an unreachable operator blocks writes rather than admitting them unchecked.
• Errors are message keys, not prose. Every violation is one StatusCause on a 422: field is the JSON path, message is <key> <json params>. The catalog api/v1alpha1.Messages maps each key to English and Arabic templates; a test asserts both exist for every key. The API, CLI and dashboard render from the catalog.
• Quotas count per namespace (Policies are namespaced) and include Expired and Failed environments until they are deleted. Cross-namespace team quotas belong to the API.
• Certificates need no external issuer. By default the operator generates a P-256 CA (10 years) and a serving certificate (1 year, renewed 30 days early, daily check), stores them in the Secret webhook-server-cert in its namespace, writes them to the webhook cert directory before the manager starts, and patches the CA bundle into its ValidatingWebhookConfiguration. Renewal runs on the leader; other replicas reload from the Secret through the file watcher. --webhook-cert-mode=cert-manager disables all of this for clusters that already run cert-manager. No ACME, no internet.
• New RBAC: a Role in the operator namespace for the certificate Secret, and cluster-wide patch on ValidatingWebhookConfigurations.

Addendum 2026-09-10: narrowing the operator's Secret access

The rendering addendum recorded cluster-wide Secret read as an accepted v0 compromise. It is now narrowed: the operator reads only Secrets labelled nazeel.sa/git-credentials=true, and reports SourceResolved=False/ResolveFailed with the exact instruction when a referenced Secret lacks the label. Connecting an application therefore never grants the operator sight of unrelated application secrets, which is what a buyer's security team asks about first. The chart's ClusterRole keeps the get/list/watch verbs because label-based restriction is enforced by the admission of the request, not by RBAC alone; sites that need RBAC-level enforcement can replace the ClusterRole with per-namespace Roles listing the Secret names.

Addendum 2026-09-12: terminal session limits and recording retention

The terminal was specified as "every session is recorded and audited". That was right and is unchanged, but it left two things unsaid, and both turned out to matter more than the recording itself.

Nothing closed a session. A shell left open at a prompt overnight is the classic privileged-access finding, and the product had no answer to it. Sessions are now bounded by the team's Policy: terminalIdleTimeout (15m by default) and terminalMaxDuration (4h). Either may be set to "0" to disable it, which has to be an explicit decision rather than the state of a fresh install. Why the reason is recorded: "the shell exited" and "we closed it because it sat idle for an hour" are different facts to whoever reads the log afterwards, and only the second answers the review question.

Nothing removed a recording. Recordings share the volume with the SQLite database, so they were the one thing on the install that grew without bound, and filling that volume makes the database read-only — which stops the audit log, the one thing this product promises will always be there. terminalRecordingRetentionDays (90 by default) deletes recordings older than that; 0 keeps them forever, for sites whose regulator asks for it, and the volume gauge and dashboard banner exist so that choice is made with the consequence visible.

Retention deletes the recording, never the audit entry. The entry saying a session happened, who opened it and why it ended is permanent; only the replayable file ages out, and its expiry is itself audited as actor system. A retention policy that could remove audit entries would be a way of erasing history rather than of managing disk, and the append-only triggers on the audit table would refuse it anyway.

A recording is capped at 20 MiB, and reaching the cap ends the session. The alternative — carrying on with the recording stopped — would leave an unrecorded shell running, which is the one state the terminal must never be in. A runaway process reaches the cap in seconds, which is the case it exists for; a person typing will not reach it in a working day.