Lee ChanyangBackend Engineer

Language

← Back to project list

Personal Project

Paused

Self-Improving Multi-Agent Harness

While building a personal multi-agent harness capable of self-improvement, I used actual operational data to investigate which agent limitations engineering can reduce, which are irreducible, and how cognitive tasks should be allocated across models.

Role
System Designer / Engineer (sole responsibility for design, implementation, and operation)
Responsibility
Sole responsibility for harness design, implementation, operation, and observation/documentation of limitations
Period
2026.05–2026.06 · Paused
  • Claude
  • Codex
  • Bash orchestration
  • Node.js (ESM)
  • Markdown/YAML contracts

Project Overview

I began by asking how judgments made while working with agents could compound over time

The starting problem was that decisions, trial and error, and lessons learned while working with AI disappeared at the end of each session, leading to the same mistakes in subsequent tasks.

The harness is a runtime that runs role agents such as ceo, tl, and qa in parallel in tmux panes. A bash router aggregates each worker's output (outbox) to detect knowledge gaps, while memory and knowledge subsystems accumulate and distill judgments and inject them into later tasks.

Initially, the memory system was introduced only declaratively and never connected to actual execution, leaving it largely inert. I therefore redefined the harness as a runtime that repeatedly produces, structures, and reuses judgments rather than a framework for accumulating features. This research case documents the structural agent limitations observed in that process.

Self-Improving Harness at a Glance — Parallel Roles → Aggregation → Memory Accumulation → ReinjectionMermaid

Rendering the diagram.

Ownership

My direct responsibilities

  • Defined disappearing judgments and the declaration≠execution disconnect as the problem, reframing the repository from a collection of settings into a runtime that accumulates judgments
  • Separated the self-improvement loop into detection (automated in code) and mutation (human-gated), making the safety boundary explicit
  • Structured limitations observed in actual project operation into a taxonomy distinguishing limitations reduced through engineering from irreducible limitations
  • Designed allocation policies for providers, model tiers, and reasoning depth (effort) based on role complexity, cost, and benefit
  • Discovered and documented a routing gap in code between configured model capability and actual application (intent≠application)

Constraints

Constraints to address first

  • Because these were observations during actual operation rather than a controlled benchmark (n>1, control), figures had to be presented only as natural-experiment observations and self-measurements.
  • Model IDs and provider policies depend on the point in time, so I needed to state the observation date.
  • I needed to show plainly that mutation is blocked by a human gate, without overstating the system as autonomous self-improving AI.
  • I needed to center the findings about limitations rather than list system size, such as subsystem or role counts, as achievements.

Implementation Timeline

The research is organized along four axes, from self-improvement design to assessment of limitations

I organized observations from building the self-improving harness along four axes: safety-boundary design, model-capability allocation, implemented verification artifacts, and the limitations distinguished through them. These are axes of observation, not sequential development stages; readers can select the relevant axis to inspect its evidence. Select a phase to see the problem identified at that stage and how it led to the architecture and verification criteria.

Phase 01 · Separating Detection from Mutation

Separating Detection from Mutation

Phase 01 · Self-Growth Loop

I automated detection in code and placed mutation behind a human gate

Separating self-improvement into detection and mutation is the core safety boundary. Detection is automated, but mutations that actually change memory must pass human ratification.

Self-Improvement Loop — Detection (Automatic) vs. Mutation (Human-Gated)Mermaid

Rendering the diagram.

  1. 01

    Detection Automated in Code

    When a worker leaves markers such as [KNOWLEDGE_GAP] in its outbox, the router increments counters. clo-trigger-check scans for thresholds (K3=3, K4=5) being exceeded and automatically records the result in the CLO inbox.

  2. 02

    Mutation Ratified by a Human

    Mutations that actually change memory (brain) are permitted only after CLO consolidation, human ratification, and a sentinel. Without the sentinel, the PreToolUse hook blocks the edit itself.

  3. 03

    Why I Drew the Boundary Here

    Automatically overwriting memory can silently accumulate incorrect learning, and humans may miss validation failures in self-referential governance. I therefore fixed the maturity boundary at detection by code and mutation through a human gate.

Do Not Make Mutation Autonomous

Choice

I automated only detection and escalation in code; brain mutation requires CLO consolidation, human ratification, and a sentinel.

Problem
When detection signals exceed thresholds, the system can automatically generate improvement candidates.
Rationale
Automatically overwriting memory can silently accumulate incorrect learning, and humans may miss validation failures in self-referential governance.
Safeguards
  • The PreToolUse hook blocks brain edits without a sentinel
  • Mutations prioritize appending; unauthorized deletion is prohibited

Trade-offHuman involvement is a bottleneck compared with full autonomy, but it preserves a clear, safe maturity boundary.

Development History

Founding decisions are recorded in ADRs and CHARTER; maturation in a six-day hardening sprint. Development stopped in 2026-06

The direction is recorded in two sources. Founding decisions, such as the redefinition as a judgment runtime and memory wiring, are in pre-Git ADRs and CHARTER. Subsequent hardening is recorded in commits from 2026-05-29 to 2026-06-03. Development stopped after the final commit (c27a98f, 2026-06-03).

The Git history (50 commits) records an operational hardening sprint, not the founding period. Founding work, including memory wiring in ADR-0008 and the runtime redefinition in CHARTER, predates Git; deriving the start date from commits would distort the history. I therefore cite ADRs and CHARTER for founding decisions and commits for phase-specific hardening.

Development History — Direction by Phase and Supporting Commits
PhaseDirectionEvidence
P0 · Founding (pre-Git)Redefined the repository from a collection of settings into a judgment runtime; connected memory to the runtimeADR-0008, CHARTER (~2026-05)
P1 · Provider & Backlog Tools (05-31)Established provider routing and the backlog closure-track foundationb15c6e3 registered the auto-growth closure track
P2 · Directory SSOT Migration (05-30)Unified the repository layout under runtime/… as the SSOT4bfd571 migrate repo-wide layout
P3 · auto-growth closure·ADR(05-31)Raised readiness-scan defaults; closed the B-096 verify gate8af28cd(B-096 close), c992260(readiness 40→60)
P4 · Provider Dispatch Hardening (06-01)Hardened provider-dispatch failure pathsfb3b114 harden provider dispatch
P5 · Role-Based Effort & ultracode (06-03)Injected role-specific reasoning depth and ultracode profiles (B-131); development stopped afterward4032c37 injection; c27a98f documented design and rationale (final commit)
Commit hashes and dates were verified directly in the ai-workflows repository. Only P0 predates Git and uses ADRs and CHARTER as evidence.

Outcome

Verifiable outcomes

  • An operational taxonomy distinguishing reduced from irreducible agent limitations (2 reduced / 1 partial / 2 irreducible)
  • Bootstrap Paradox — identified an irreducible human-in-the-loop floor in self-referential governance
  • Derived a model-capability allocation policy: judgment=claude/parallelism; execution=codex/model depth
  • Intent≠application gap — discovered in code that routing can nullify configured model capabilities
  • Expressed the actual boundary of self-improvement without exaggeration: detection=automatic, mutation=human-gated

Limitations

Limitations and unimplemented scope

  • This is an n=1 self-report study. There is no control group, and the observer is also the operator, limiting generalizability.
  • Figures such as functioning rate 15%→60% and tokens down 71% are operational observations and repository self-measurements, not controlled evaluations.
  • Model IDs (Opus/Sonnet/Haiku) and provider policies are as of 2026-06; allocation judgments change as model generations change.
  • Development has been stopped since 2026-06 (final commit c27a98f), so next steps such as verifying the import gate's effectiveness have not been pursued.