# Autoresearch Guide: Self-Improving Command Prompts > **Guide Origin**: Official | **ArcKit Version**: [VERSION] ArcKit includes an autonomous prompt optimisation system inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch). It lets Claude Code iteratively improve any command prompt by running it, scoring the output, tweaking the prompt, and keeping or discarding the change based on whether quality improved. You start a run and walk away. The system loops until a stop condition fires — score target hit, iteration budget exhausted, or double plateau detected. See [Stopping Conditions](#stopping-conditions) below for the defaults and how to tune them. --- ## Quick Start In the ArcKit repo, tell Claude Code: ```text read scripts/autoresearch/program.md and optimise the requirements command ``` Replace `requirements` with any command name (e.g., `adr`, `backlog`, `risk`, `stakeholders`). Claude will: 1. Create a git worktree (`../autoresearch-requirements`) on a new branch -- your main checkout stays clean 2. Set up a scratch project with fixture data inside the worktree 3. Run the command, score the output, log the baseline 4. Enter the experiment loop -- tweaking, re-running, keeping or discarding To stop early: interrupt Claude at any time. Otherwise the loop self-terminates on a stop condition (see [Stopping Conditions](#stopping-conditions)). Either way, the worktree has the best prompt. ### Self-Harness enhanced autoresearch For command, agent, hook, or full harness improvement, use the Self-Harness enhanced program: ```text read scripts/autoresearch/program-selfharness.md and optimize the requirements command ``` The Self-Harness program extends the prompt-only loop with trace collection, verifier-grounded weakness mining, held-in/held-out fixture validation, and modes for improving commands, agents, hooks, or broader harness configuration. It is useful when the failure is not just wording in a command prompt, but the surrounding harness behavior: tool use, runtime setup, hook behavior, verification, or agent instructions. Common targets: - `optimize command requirements` - prompt-only command optimization, compatible with the standard loop - `optimize command requirements mode:full` - full harness optimization for a command - `optimize agent research` - agent definition optimization - `optimize hook graph-inject` - hook behavior optimization See [`scripts/autoresearch/program-selfharness.md`](../../scripts/autoresearch/program-selfharness.md) for the detailed draft workflow. ### Watching progress without blocking the main session For long overnight runs, open a second Claude Code session in the worktree directory and ask it to `Monitor` the results log: ```text Monitor scripts/autoresearch/runs/*.log and summarise each new iteration ``` The `Monitor` tool (Claude Code v2.1.98+) tails stdout from a background `tail -F` and delivers each new scoring line to Claude as a notification. You get progress updates on demand without interrupting the experiment loop in the main session. Requires Claude Code v2.1.98 or later. ### Phone pings via Remote Control For overnight runs where you actually walk away from the laptop, pair the autoresearch session with [Claude Code Remote Control](https://code.claude.com/docs/en/remote-control): ```bash claude remote-control ``` Drive the worktree session from claude.ai/code or the mobile app, then enable `/config → Push when Claude decides` so your phone pings when an iteration keeps a change above the score threshold or when the loop hits a decision point. Combined with `ENABLE_PROMPT_CACHING_1H=1` (see Tips below), you can run autoresearch overnight and check progress from anywhere. Caveats: Pro/Max plans only (no API keys, no Bedrock/Vertex/Foundry), push is a single on/off, and the local `claude` process must keep running. ArcKit's minimum Claude Code floor (v2.1.287) already covers the v2.1.110 RC requirement. --- ## How It Works The system adapts autoresearch's ML experiment loop to prompt engineering: - **autoresearch**: modifies `train.py` to minimise `val_bpb` - **ArcKit autoresearch**: modifies a command `.md` file to maximise a quality score - **ArcKit Self-Harness**: modifies prompts, agents, hooks, runtime configuration, or verification harnesses, then accepts only changes that generalize across held-in and held-out tasks ### The Loop Each iteration follows the same cycle: 1. **Read** the current prompt and results history 2. **Identify** one specific improvement 3. **Edit** the command `.md` file 4. **Commit** the change to git 5. **Clean** the scratch project (delete generated artifacts, keep fixtures) 6. **Execute** the command against the scratch project 7. **Score** the output (structural checks then LLM-as-judge) 8. **Compare** to the previous best score 9. **Keep or discard** based on a minimum delta threshold (>= 0.3) 10. **Log** the result to `results.tsv` 11. **Check stop conditions** — exit if score target hit, iteration budget exhausted, or double plateau detected (see [Stopping Conditions](#stopping-conditions)) If discarded, the prompt is reverted via `git checkout` to the previous best version. The full history (including discards) is preserved in `results.tsv`. ### Status Line After each iteration, a status line is printed: ```text [iter 3] score: 9.2 (best: 8.8) | status: keep | keeps: 3 discards: 1 | streak: 0/5 to plateau ``` This gives live terminal visibility without needing to read files. --- ## Evaluation: Two Layers ### Layer 1: Structural Gate (pass/fail) Eight checks that must all pass: 1. Document Control table with all 14 required fields 2. Document ID follows `ARC-NNN-TYPE-vX.Y` pattern 3. Revision History table present 4. Standard footer present 5. All major template sections present 6. File written to correct path 7. Domain-specific IDs correct (BR-xxx, FR-xxx, etc.) 8. Wardley Map math validation (WARD commands only): stage-evolution alignment, coordinate range [0.00-1.00], OWM-to-table consistency If any check fails, the iteration scores `FAIL 0.0` and is discarded immediately. ### Layer 2: LLM-as-Judge (1.0-10.0) Five dimensions, each scored 1-10: - **Completeness** -- all sections substantively filled - **Specificity** -- references actual project context, not generic boilerplate - **Traceability** -- cross-references between artifacts present and correct - **Actionability** -- a vendor or review board could use this document as-is - **Clarity** -- well-structured, no contradictions The combined score is the arithmetic mean, rounded to one decimal place. Scoring uses an adversarial reviewer persona to prevent self-evaluation bias. --- ## Self-Harness Extension Self-Harness is an enhanced autoresearch mode based on "Self-Harness: Harnesses That Improve Themselves" (Zhang et al., 2026, arXiv:2606.09498v1). The ArcKit draft implementation keeps the original prompt-only workflow intact and adds a broader harness improvement loop for cases where the command prompt is not the only variable. ### What It Adds - **Multi-dimensional optimization**: can improve command prompts, agent definitions, hook behavior, runtime wiring, and verifier logic depending on the selected mode - **Trace collection**: records execution traces under `.arckit/autoresearch-traces///` - **Weakness mining**: clusters repeated verifier failures and trace signatures so the next proposal addresses a concrete failure mode - **Harness proposals**: uses `harness-proposer.mjs` to generate scoped candidate improvements - **Held-in/held-out validation**: requires improvements to hold on the optimization fixtures and not regress on reserved fixtures - **Regression validation**: uses `harness-validator.mjs` to reject candidates that overfit or exceed the allowed edit scope ### Execution Modes | Mode | Typical target | Editable scope | |------|----------------|----------------| | `prompt` | `command requirements` | One command `.md` file | | `full` | `command requirements mode:full` | Command, hooks, templates, runtime, and verification files declared in the run | | `agent` | `agent research` | One agent definition | | `hook` | `hook graph-inject` | One hook implementation | Use the standard `program.md` when you want a tight prompt-only optimization. Use `program-selfharness.md` when you need evidence about whether failures come from the prompt, execution harness, supporting tools, or validation layer. ### Acceptance Rule The Self-Harness loop is intentionally conservative: 1. Score the baseline across held-in and held-out fixtures. 2. Mine failures from traces and verifier output. 3. Propose a scoped harness change. 4. Validate the candidate on held-in fixtures. 5. Re-run held-out fixtures before accepting. 6. Keep only changes that improve held-in performance without degrading held-out performance beyond the configured tolerance. This prevents the loop from optimizing around one fixture while making the command, agent, or hook worse in real use. --- ## What Gets Modified (and What Doesn't) For the standard prompt-only loop: **Editable** (the only variable): - `plugins/arckit-claude/commands/.md` -- the prompt being optimised **Read-only** (the fixed benchmark): - Template file (defines expected output structure) - Quality checklist (evaluation standard) - Scratch project fixtures (controlled input) - Test argument (`$ARGUMENTS = "001"`) - Evaluation rubric This mirrors autoresearch's design: `prepare.py` is read-only, `train.py` is the only file modified. For Self-Harness runs, the editable scope depends on the selected mode and is declared at setup time in `program-selfharness.md`. Keep that scope narrow for each run. A full harness run can touch more than one file, but it should still treat fixtures, scoring standards, and held-out tasks as read-only controls. --- ## Constraints - **One change per iteration** -- isolate variables - **Minimum delta of 0.3** -- filters noise from non-deterministic evaluation - **Simplicity criterion** -- marginal improvement + added complexity = not worth it; simplification + same score = keep - **Log everything** -- every iteration gets a row in `results.tsv` - **No git reset --hard** -- use targeted `git checkout` + revert commits - **Session-wide budgets apply** -- Claude Code caps each session at 200 WebSearch calls (v2.1.212) and 20 concurrently-running subagents (v2.1.217). The 200-spawns-per-session cap was removed in v2.1.224; `CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION` is still accepted but never read. A long autoresearch loop that repeatedly re-runs a research-heavy command can exhaust the search budget mid-run, and that cap soft-degrades rather than failing -- searches return a "not performed" result and the loop carries on with thinner evidence. Raise it with `CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION`, or let the loop start fresh sessions -- `/clear` resets it. `WebFetch` is uncounted but cached per URL for 15 minutes (`CLAUDE_CODE_WEBFETCH_CACHE_TTL_MS`, v2.1.233+), so a loop that re-reads the same page inside that window gets the cached copy. --- ## Results Results are logged to `results.tsv` (tab-separated): ```text commit structural score status description a1b2c3d PASS 8.4 keep baseline b2c3d4e PASS 8.8 keep expand NFR subcategories c3d4e5f PASS 9.0 discard derive NFR targets from context (delta < 0.3) d4e5f6g PASS 9.2 keep add use case structure instruction ``` Status values: `keep`, `discard`, `plateau`, `crash`, `complete`, `budget-exhausted`. ### Plateau Detection If 15 consecutive iterations are discarded, the system shifts strategy: - Re-reads the template for unaddressed sections - Reviews the quality checklist for uncovered criteria - Tries prompt simplification - Combines ideas from previous near-misses A `plateau` row is logged when the trigger fires; the strategy shift resets the discard streak. ### Stopping Conditions The loop runs until one of three explicit stop conditions fires (no "run forever" mode). All three are tunable inline at the top of step 12 in `program.md`: | Condition | Default | Final status row | Meaning | |-----------|---------|------------------|---------| | Score target | `best >= 9.5` | `complete` | Quality target reached; further iteration unlikely to pay off | | Iteration budget | `iter >= 30` | `budget-exhausted` | Token budget bounded; rerun with higher cap if more headroom is wanted | | Double plateau | 2 `plateau` rows within 10 iterations | `complete` | Strategy shift exhausted; diminishing returns | The final status row carries the reason in its `description` field so `results.tsv` self-documents why the run ended. To extend a run, edit the constants in `program.md` step 12 and re-launch — the constraints in section 4 prohibit the LLM from negotiating past a hit threshold mid-run. --- ## Practical Results Commands optimised so far and their score improvements: - **requirements**: 8.4 to 9.2 (+0.8) in 3 iterations - Expanded NFR subcategories (5 generic to 7 specific with sub-prefixes) - Added explicit use case structure (UC-xxx with main/alt/exception flows) - **adr**: 8.6 to 9.0 (+0.4) in 1 iteration - Strengthened Consequences section with project-specific metrics, mitigation owners, after-action review - **backlog**: 8.0 to 8.8 (+0.8) in 1 iteration - Strengthened acceptance criteria rules (banned vague phrases, required measurable thresholds) Net prompt changes are typically 3-8 lines. The improvements are small and high-leverage. --- ## Which Commands to Optimise Good candidates: - Commands that produce long, structured documents (requirements, backlog, sobc, risk) - Commands with detailed templates that the prompt may not fully leverage - Commands where output quality varies between runs Not suitable: - Agent-delegated commands without a direct-execution fallback (research, datascout, aws-research, azure-research, gcp-research, framework) -- the prompt is a thin wrapper - Simple utility commands (customize, init, health) -- too short to benefit --- ## File Structure ```text scripts/autoresearch/ program.md # The instruction file Claude follows program-selfharness.md # Self-Harness enhanced instruction file fixtures/ 000-global/ ARC-000-PRIN-v1.0.md # Architecture principles (6 principles) 001-test-project/ README.md # Project description ARC-001-STKE-v1.0.md # Stakeholder analysis (4 stakeholders) ``` The experiment runs in a **git worktree** (`../autoresearch-`), keeping the main checkout clean. The scratch project, results TSV, and all experiment commits live in the worktree. The branch tip (the improved command `.md`) is the deliverable. Self-Harness runs also use trace and candidate outputs: ```text .arckit/autoresearch-traces/ / / iteration-N.json scripts/autoresearch/runs/ / results.tsv ``` --- ## Tips - **Run overnight** -- each iteration takes 2-3 minutes, so you get 20-30 experiments per hour - **Extend the prompt cache TTL for overnight runs** -- set `ENABLE_PROMPT_CACHING_1H=1` (Claude Code v2.1.108+) before launching Claude. The default 5-minute prompt cache expires between iterations once Claude pauses to think, score, and write `results.tsv`; the 1-hour TTL keeps the template, fixtures, and accumulated `results.tsv` warm across the full overnight run, materially reducing token cost. Pair with `ANTHROPIC_API_KEY` billing dashboards to confirm cache-read rates climb. - **Configure a fallback model** -- launch with `--fallback-model ` so that if the primary model is briefly unavailable mid-run (e.g. Opus overloaded), Claude Code switches to the fallback instead of failing every request (Claude Code v2.1.152+). Keeps an unattended overnight loop alive through a transient capacity blip rather than stalling on iteration 7. From **v2.1.166** you can set a `fallbackModel` in settings listing **up to three** models tried in order, `--fallback-model` also applies to **interactive** sessions (so a long interactive `/arckit:research` survives a blip too), and Claude Code automatically **retries a turn once on the fallback model** when the API returns an unexpected non-retryable error (auth, rate-limit, request-size, and transport errors still surface immediately). - **Review the results.tsv** -- the discard history tells you what didn't work, which is as valuable as what did - **Check against standards** -- before starting a run, review relevant external standards (e.g., UK Gov ADR Framework for ADRs, GDS Service Standard for assessments) to prime the system with specific gaps to target - **Create a PR for the prompt change only** -- the experiment branch has noise (scratch files, results, reverts); cherry-pick the kept commits onto a clean branch - **One command per worktree** -- each optimisation run gets its own `../autoresearch-` worktree - **Cleanup** -- after cherry-picking results, remove the worktree: `git worktree remove ../autoresearch-` --- ## Limitations - **Self-evaluation bias** -- the same model generates and judges; adversarial scoring instructions mitigate but don't eliminate this - **Fast convergence** -- most commands reach 8.8-9.2 within 2-3 kept changes; further improvements are incremental - **Fixture dependency** -- a thin test project constrains what improvements the system can discover; richer fixtures reveal more gaps - **Non-deterministic scoring** -- the 0.3 delta threshold is empirically chosen; genuine improvements near the threshold may be discarded