Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR 0002 · Reconcile CalibanTask → agent-sandbox Sandbox (+ per-task SA & NetworkPolicy)

  • Status: accepted
  • Date: 2026-07-04
  • Amended by: ADR 0005 — decision 6 (status gains prospero's AgentsSettled condition as an input, via server-side apply field ownership)
  • Source: k8s system-design spec (§"caliban-operator", §"agent-sandbox integration") in the caliban-ai docs hub · caliban #283 · epic #274 · builds on ADR 0001

Context

ADR 0001 fixed the controller stack (kube 4.0 / k8s-openapi 0.28 / v1_32) and the CalibanTask API. #282 shipped a placeholder reconcile that only initialized .status.phase = Pending. #283 makes the reconcile real: turn a CalibanTask into a running, sandboxed caliband pod reachable at a stable DNS, with per-task RBAC and network isolation, and drive .status from the backing Sandbox.

The spec is explicit that the operator is a thin caliban-semantics layer over agent-sandbox — it must not reimplement pods/isolation. So the reconcile composes existing objects rather than managing pods directly.

Forces:

  • agent-sandbox owns the Sandbox CRD (agents.x-k8s.io/v1beta1). We consume it — we must not generate, install, or drift-guard its schema. Verified v0.5.0 SandboxSpec = { podTemplate{metadata,spec: core.PodSpec} (required), service bool, operatingMode enum(Running|Suspended)=Running, shutdownPolicy enum(Delete|Retain)=Retain, shutdownTime, volumeClaimTemplates []core.PVC }; SandboxStatus.serviceFQDN is the pod's stable DNS.
  • Reconcile must be idempotent and self-healing — re-running on the same CalibanTask must converge, not duplicate, and children must be garbage-collected when the CalibanTask is deleted.
  • Cluster-agnostic (the eventual charts are public): no home-cluster identifiers baked into generated objects.
  • k3s v1.31 target vs v1_32 compile surface (ADR 0001): stick to long-stable PodSpec/NetworkPolicy fields.
  • The caliband pod needs zero Kubernetes API access; its identity exists only to be a NetworkPolicy/PodSecurity subject.

Decision

  1. Consume Sandbox as a foreign typed resource. Hand-author a minimal Rust Sandbox via #[derive(CustomResource)] pointed at group = "agents.x-k8s.io", version = "v1beta1", kind = "Sandbox", namespaced, status = "SandboxStatus", declaring only the fields we set or read (podTemplate, service, operatingMode, shutdownPolicy, shutdownTime, volumeClaimTemplates; status serviceFQDN, conditions). podTemplate reuses k8s_openapi PodTemplateSpec; volumeClaimTemplates reuses PersistentVolumeClaim. We never call Sandbox::crd(), emit its YAML, or add it to the drift-sync test — agent-sandbox installs its own CRD. Extra SandboxSpec fields we omit are pruned by the API server's structural schema; this is a read/write view, not the schema of record.

  2. Reconcile = pure builders + a thin apply loop. The mapping CalibanTask → {Sandbox, ServiceAccount, NetworkPolicy} and the status derivation are pure functions (build_sandbox, build_service_account, build_network_policy, derive_status), unit-tested with no cluster. The async reconcile only: server-side-applies each child, reads back the Sandbox, and patches CalibanTask.status.

  3. Idempotency via server-side apply + owner references. Children are applied with Patch::Apply under field manager caliban-operator (force), so re-reconciles converge on one object. Every child carries a controller OwnerReference to its CalibanTask, so deleting the task cascades. Child names are deterministic: <task>-sbx (Sandbox), <task>-sa (ServiceAccount), <task>-netpol (NetworkPolicy).

  4. Least-privilege RBAC = a dedicated, token-less ServiceAccount with no bound Role. The caliband pod is given its own per-task ServiceAccount with automountServiceAccountToken: false, referenced from the podTemplate. Because no Role/RoleBinding is created, the pod holds zero API permissions — strictly the least privilege. An empty Role would be noise; the token-less SA is the RBAC posture. (The operator's own cluster RBAC — the right to create Sandboxes/SAs/NetworkPolicies — is deploy-time chart concern #284, not this reconcile.)

  5. NetworkPolicy: default-deny with the minimal working allowances. The per-task NetworkPolicy selects the Sandbox's pod labels and sets policyTypes: [Ingress, Egress] with: deny-all by default, allow DNS egress (UDP/TCP 53), allow general egress (git clone + provider APIs; the empty-to rule allows egress to all destinations at MVP, in-cluster included — restricting it to non-cluster CIDRs is deferred because it would require an environment-specific cluster CIDR, a home-cluster identifier we don't bake in), and allow ingress on the caliband port (8443) from the task's namespace. The spec's finer rules (prospero→caliband, pod→gonzalo) require cross-namespace identity the CR does not carry generically at MVP; they are parameterized in a later iteration. The MVP ships a real, cluster-agnostic default-deny posture (selectors are the pod's own labels — no home-cluster identifiers).

  6. Status is derived from the Sandbox, not initialized-and-frozen. This evolves #282's init-once no-op: derive_status(task, sandbox) returns Provisioning once the Sandbox is applied but has no serviceFQDN yet, Running once serviceFQDN is populated, and sets calibandEndpoint = "<serviceFQDN>:8443" and sandboxRef accordingly. Pending remains the pre-provision default (a CalibanTask observed before its first successful apply). The status patch is skipped when the derived status equals the observed one (no-op churn avoidance, same discipline as #282).

  7. Explicit deferrals (out of #283 scope, tracked upstream): the caliban-aware drain finalizer (needs caliband's checkpoint gRPC — deferred with the transport #314), idle→pause/resume mapping, SandboxTemplate/SandboxWarmPool (Phase 4), and full Secret/ConfigMap credential projection. #283 projects only what the acceptance path needs: the workspace PVC, the caliband image + runtimeClass, and the gonzalo endpoint / model-router refs as env, so the pod comes up reachable.

Consequences

  • A CalibanTask now materializes real cluster objects. Applying one yields a Sandbox (→ caliband pod on the configured RuntimeClass, workspace PVC, stable DNS), a token-less SA, and a default-deny NetworkPolicy; .status reports Provisioning → Running with calibandEndpoint. This satisfies #283's acceptance and unblocks prospero's K8sFleet (#64) reading the endpoint.
  • agent-sandbox must be installed in the cluster for the applies to succeed (its Sandbox CRD must exist). This is the cluster prerequisite the umbrella chart bundles by default (helm-charts#6); without it the operator's Sandbox apply fails and the task stays Provisioning — an honest, retried error, not a crash.
  • The operator needs cluster RBAC to create these children — it can create sandboxes.agents.x-k8s.io, serviceaccounts, and networkpolicies.networking.k8s.io, and patch calibantasks/status. That ClusterRole ships with the operator chart (#284); this ADR fixes the exact verb set that chart must grant.
  • Deferring the drain finalizer means a deleted CalibanTask tears down its Sandbox immediately (owner-ref GC) without checkpointing in-flight agents. This is acceptable for the MVP and revisited when caliband checkpoint gRPC lands.
  • The foreign Sandbox view can rot if agent-sandbox changes its v1beta1 schema. We pin the consumed version (v1beta1) and cover the fields we use with round-trip tests against the published CRD's shape; a breaking upstream change surfaces as a deserialize/apply failure, caught in integration.
  • serviceFQDN is not readiness — resolved (#45). Decision 6's "Running once serviceFQDN is populated" reported Ready=True as soon as the Service object existed, before caliband had bound its port; a crash-looping caliband could present as Ready indefinitely and prosperod dialed a dead endpoint. The caliband container now carries TCP readiness and startup probes on the control port (no liveness probe — a restart kills every supervised agent), and derive_status gates Running on agent-sandbox's own Sandbox Ready condition, which v0.5.0 sets True only once the pod passes its readiness probe, has an IP, and the Service is ready. An FQDN without that condition stays Provisioning with no endpoint, and a reported not-ready surfaces as Ready=False / SandboxNotReady carrying agent-sandbox's message.
  • Allow-all egress could not be tightened — resolved (#58). The default egress rule above still allows every destination, and remains the default. A Workspace may now declare spec.egress.allow (CIDR + optional TCP ports), which replaces allow-all with DNS plus exactly those destinations; an empty list is DNS-only. It is a Workspace-level field rather than part of the shared IsolationSpec, because CalibanTaskSpec.isolation is a per-run override that wins over the workspace default and would otherwise let a task widen the restriction. Malformed CIDRs and out-of-range ports fail the Workspace.
  • volumeClaimTemplates could not reuse PersistentVolumeClaim — superseded (#7/#8). Decision 1 says the Sandbox view's volumeClaimTemplates reuses k8s_openapi's PersistentVolumeClaim. It cannot: that type serializes its group/version/kind, and agent-sandbox v0.5.0's structural schema rejects those on server-side apply (.spec.volumeClaimTemplates[].apiVersion: field not declared in schema), failing every reconcile. src/sandbox.rs declares a bare VolumeClaimTemplate (metadata + spec) instead, which is what the upstream schema actually describes. podTemplate does still reuse PodTemplateSpec as decision 1 says.
  • The Sandbox view omits shutdownPolicy and shutdownTime — not implemented. Decision 1 lists both among the fields the view declares; it declares podTemplate, service, operatingMode and volumeClaimTemplates only. Nothing sets a shutdown policy today, and the view is deliberately a read/write subset — fields it omits are pruned by the API server, so the gap is inert rather than a defect. It is recorded here because decision 1 reads as a complete list and is not one.