Personal Project
PausedSelf-Improving Multi-Agent Harness
While building a personal multi-agent harness capable of self-improvement, I used actual operational data to investigate which agent limitations engineering can reduce, which are irreducible, and how cognitive tasks should be allocated across models.
- Role
- System Designer / Engineer (sole responsibility for design, implementation, and operation)
- Responsibility
- Sole responsibility for harness design, implementation, operation, and observation/documentation of limitations
- Period
- 2026.05–2026.06 · Paused
- Claude
- Codex
- Bash orchestration
- Node.js (ESM)
- Markdown/YAML contracts
Project Overview
I began by asking how judgments made while working with agents could compound over time
The starting problem was that decisions, trial and error, and lessons learned while working with AI disappeared at the end of each session, leading to the same mistakes in subsequent tasks.
The harness is a runtime that runs role agents such as ceo, tl, and qa in parallel in tmux panes. A bash router aggregates each worker's output (outbox) to detect knowledge gaps, while memory and knowledge subsystems accumulate and distill judgments and inject them into later tasks.
Initially, the memory system was introduced only declaratively and never connected to actual execution, leaving it largely inert. I therefore redefined the harness as a runtime that repeatedly produces, structures, and reuses judgments rather than a framework for accumulating features. This research case documents the structural agent limitations observed in that process.
Rendering the diagram.
Ownership
My direct responsibilities
- Defined disappearing judgments and the declaration≠execution disconnect as the problem, reframing the repository from a collection of settings into a runtime that accumulates judgments
- Separated the self-improvement loop into detection (automated in code) and mutation (human-gated), making the safety boundary explicit
- Structured limitations observed in actual project operation into a taxonomy distinguishing limitations reduced through engineering from irreducible limitations
- Designed allocation policies for providers, model tiers, and reasoning depth (effort) based on role complexity, cost, and benefit
- Discovered and documented a routing gap in code between configured model capability and actual application (intent≠application)
Constraints
Constraints to address first
- Because these were observations during actual operation rather than a controlled benchmark (n>1, control), figures had to be presented only as natural-experiment observations and self-measurements.
- Model IDs and provider policies depend on the point in time, so I needed to state the observation date.
- I needed to show plainly that mutation is blocked by a human gate, without overstating the system as autonomous self-improving AI.
- I needed to center the findings about limitations rather than list system size, such as subsystem or role counts, as achievements.
Implementation Timeline
The research is organized along four axes, from self-improvement design to assessment of limitations
I organized observations from building the self-improving harness along four axes: safety-boundary design, model-capability allocation, implemented verification artifacts, and the limitations distinguished through them. These are axes of observation, not sequential development stages; readers can select the relevant axis to inspect its evidence. Select a phase to see the problem identified at that stage and how it led to the architecture and verification criteria.
Separating Detection from Mutation
Development History
Founding decisions are recorded in ADRs and CHARTER; maturation in a six-day hardening sprint. Development stopped in 2026-06
The direction is recorded in two sources. Founding decisions, such as the redefinition as a judgment runtime and memory wiring, are in pre-Git ADRs and CHARTER. Subsequent hardening is recorded in commits from 2026-05-29 to 2026-06-03. Development stopped after the final commit (c27a98f, 2026-06-03).
The Git history (50 commits) records an operational hardening sprint, not the founding period. Founding work, including memory wiring in ADR-0008 and the runtime redefinition in CHARTER, predates Git; deriving the start date from commits would distort the history. I therefore cite ADRs and CHARTER for founding decisions and commits for phase-specific hardening.
| Phase | Direction | Evidence |
|---|---|---|
| P0 · Founding (pre-Git) | Redefined the repository from a collection of settings into a judgment runtime; connected memory to the runtime | ADR-0008, CHARTER (~2026-05) |
| P1 · Provider & Backlog Tools (05-31) | Established provider routing and the backlog closure-track foundation | b15c6e3 registered the auto-growth closure track |
| P2 · Directory SSOT Migration (05-30) | Unified the repository layout under runtime/… as the SSOT | 4bfd571 migrate repo-wide layout |
| P3 · auto-growth closure·ADR(05-31) | Raised readiness-scan defaults; closed the B-096 verify gate | 8af28cd(B-096 close), c992260(readiness 40→60) |
| P4 · Provider Dispatch Hardening (06-01) | Hardened provider-dispatch failure paths | fb3b114 harden provider dispatch |
| P5 · Role-Based Effort & ultracode (06-03) | Injected role-specific reasoning depth and ultracode profiles (B-131); development stopped afterward | 4032c37 injection; c27a98f documented design and rationale (final commit) |
Outcome
Verifiable outcomes
- An operational taxonomy distinguishing reduced from irreducible agent limitations (2 reduced / 1 partial / 2 irreducible)
- Bootstrap Paradox — identified an irreducible human-in-the-loop floor in self-referential governance
- Derived a model-capability allocation policy: judgment=claude/parallelism; execution=codex/model depth
- Intent≠application gap — discovered in code that routing can nullify configured model capabilities
- Expressed the actual boundary of self-improvement without exaggeration: detection=automatic, mutation=human-gated
Limitations
Limitations and unimplemented scope
- This is an n=1 self-report study. There is no control group, and the observer is also the operator, limiting generalizability.
- Figures such as functioning rate 15%→60% and tokens down 71% are operational observations and repository self-measurements, not controlled evaluations.
- Model IDs (Opus/Sonnet/Haiku) and provider policies are as of 2026-06; allocation judgments change as model generations change.
- Development has been stopped since 2026-06 (final commit c27a98f), so next steps such as verifying the import gate's effectiveness have not been pursued.