Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR 0007 · Test the control plane against an in-process fake caliban

Context

End-to-end testing of Prospero would normally require a real caliband, real agents, API keys, and live LLM calls — slow, non-deterministic, expensive, and awkward in CI. Yet the behavior worth testing (spawn defaults, poll reconciliation, attach/normalize, replay-then- tail, resilience to dropped streams and refused sockets) is exactly the control-plane logic that sits between Prospero and caliban.

Because the only coupling to caliban is the NDJSON wire format (0003), we can substitute anything that speaks that protocol.

Decision

Build an in-process fake caliban as the cornerstone of the test strategy: a harness (shipped in prospero-core behind a testkit feature) that listens on a real Unix socket and speaks the same NDJSON control + per-agent stream protocol as the real daemon. Tests drive the real FleetManager / CalibandClient against this fake.

This enables deterministic, end-to-end testing of the whole control plane — including the CLI-through-HTTP path — with no real caliban, no API keys, and no LLM calls.

Consequences

  • Positive: the full stack is tested fast and deterministically in CI; scripted frames make event sequences, resilience cases, and seq recovery reproducible. Pure units (framing, normalizer, discovery resolution, store, reconciliation) are tested directly; the fake is reserved for integration-level behavior that needs the protocol.
  • Negative: the fake must track the wire protocol — if caliban's protocol drifts, the fake and the mirrored client must be updated together (the same explicit seam noted in 0003). Tests against a real caliban binary + live model remain out of scope for the first stab (manual / CI-gated later), so the fake covers the wire contract, not caliban's own correctness.
  • Revisit if: the fake and the real caliban diverge in behavior the wire contract doesn't capture (green tests, broken integration) — that gap would justify a real-caliban smoke test in CI alongside the fake.