/ overview
From a trace to a release decision.
ControlSurface connects observability with the work that follows an agent failure: investigation, incident review, regression design, and a release decision. The aim is to preserve evidence across that entire path.
I built the Python SDK and control plane, the storage and projection path, and the Next.js investigation interface. The public repository describes a local, self-hosted, single-owner system with multiple projects.
/ problem
A trace is the beginning of the investigation.
Agent failures cross model calls, tools, retrieval, retries, and changing state. A raw span can show an error without explaining its relationship to the rest of the run.
The engineering problem is to derive useful structure without losing the source evidence, then make reviewed failures reproducible enough to inform the next release.
/ architecture
A durable boundary before derived intelligence.
The authenticated OTLP receiver validates, redacts, bounds, and compresses incoming telemetry. Acknowledgement follows a durable PostgreSQL inbox write. A separate worker projects spans and agent run graphs into ClickHouse.
PostgreSQL owns project configuration, reviewed incidents, regression cases, and release evidence. ClickHouse owns source span facts and their derived projections. The API and Next.js interface expose those layers progressively.
- Project identity is derived from a hashed bearer key.
- PostgreSQL is the durable ingestion boundary; V1 does not add a queue broker.
- Derived graph nodes link back to original trace and span identifiers.
- Evaluation runs locally or in CI; it is not a distributed execution service.
Inspect the evidenceWhy acknowledgment and projection are separate
The receiver acknowledges after writing to PostgreSQL. A worker projects accepted batches into ClickHouse. A ClickHouse outage pauses projection; a PostgreSQL outage prevents acceptance.
This describes the documented V1 boundary. It does not establish high availability or production capacity.
/ Engineering decisions
Explicit boundaries make the system easier to trust.
Keeping the compressed inbox in PostgreSQL gives V1 recovery, isolation, and backlog accounting without operating another service. It also makes acknowledgement and projection separate events that can be measured independently.
Deterministic signatures group recurring failures. Nearby change events provide an inspectable association score, which should be read as a heuristic rather than causal proof.
Release gates compare paired baseline and candidate cases with a manifest of revisions, hashes, model configuration, and policy. The resulting evidence bundle is content-hashed and append-only at the API layer.
Inspect the evidenceWhen a gate cannot make a valid comparison
The release implementation validates required manifest fields and rejects mismatched or duplicate case IDs. Only valid paired evidence reaches the configured policy checks.
Invalid inputs raise a validation error. They are distinct from a valid evaluation whose regressions cause a block decision.
/ challenges
Outages and late evidence are normal inputs.
A ClickHouse outage pauses projection while durable batches remain in the inbox. A PostgreSQL outage prevents acceptance. A bounded full inbox returns a retryable response.
Late spans can change the interpretation of a run. Per-project, per-trace advisory locking and versioned replacement prevent competing workers from replacing a more complete projection with an older one.
- Retry and quarantine repeatedly failing batches.
- Keep original span attributes separate from inferred labels.
- Bound incoming payloads and retained inbox data.
/ results
A working failure-to-release workflow.
The public implementation includes OTLP ingestion, graph-based investigation, failure clustering, reviewed incident workflows, paired regression evaluation, and pass/block release gates.
A published September 2026 local benchmark measured 1,245.94 spans per second and a 32.52 ms p95 for 50-row trace-list requests. It used synthetic spans on a single WSL2 / Docker node; the results do not establish sustained production capacity.
Setup instructions, recovery guidance, and an example agent workflow make the system inspectable locally. Evidence is append-only through the API; operators still control the underlying database.
Inspect the evidenceWhat the benchmark actually measured
The published 27 September run used 10,000 synthetic throughput spans, batches of 100, and concurrency 4 on one WSL2 / Docker Compose node. The trace-list p95 applies to a 50-row API request.
These are repository-reported local results, not a new benchmark run by this portfolio. They do not establish sustained load, WAN performance, or production sizing.
/ learnings
Reliability needs a chain of evidence.
An observability product becomes more useful when its outputs can become inputs to the engineering process. Preserving the link from source span to reviewed case to release evidence is the central design idea.
The documented roadmap includes deeper contract intelligence, persistent SLO windows, and selected framework integrations. Those remain future work rather than current capabilities.
Source record
Sources & technical context
Architecture summaries are based on the linked documentation and source. Current implementation details and limits live in the repository.