Back to Guides

Guide Origin: Official | ArcKit Version: [VERSION]

ArcKit includes an autonomous prompt optimisation system inspired by karpathy/autoresearch. It lets Claude Code iteratively improve any command prompt by running it, scoring the output, tweaking the prompt, and keeping or discarding the change based on whether quality improved.

You start a run and walk away. The system loops until a stop condition fires — score target hit, iteration budget exhausted, or double plateau detected. See Stopping Conditions below for the defaults and how to tune them.

Quick Start

In the ArcKit repo, tell Claude Code:

read scripts/autoresearch/program.md and optimise the requirements command

Replace requirements with any command name (e.g., adr, backlog, risk, stakeholders).

Claude will:

  1. Create a git worktree (../autoresearch-requirements) on a new branch -- your main checkout stays clean
  2. Set up a scratch project with fixture data inside the worktree
  3. Run the command, score the output, log the baseline
  4. Enter the experiment loop -- tweaking, re-running, keeping or discarding

To stop early: interrupt Claude at any time. Otherwise the loop self-terminates on a stop condition (see Stopping Conditions). Either way, the worktree has the best prompt.

Self-Harness enhanced autoresearch

For command, agent, hook, or full harness improvement, use the Self-Harness enhanced program:

read scripts/autoresearch/program-selfharness.md and optimize the requirements command

The Self-Harness program extends the prompt-only loop with trace collection, verifier-grounded weakness mining, held-in/held-out fixture validation, and modes for improving commands, agents, hooks, or broader harness configuration. It is useful when the failure is not just wording in a command prompt, but the surrounding harness behavior: tool use, runtime setup, hook behavior, verification, or agent instructions.

Common targets:

  • optimize command requirements - prompt-only command optimization, compatible with the standard loop
  • optimize command requirements mode:full - full harness optimization for a command
  • optimize agent research - agent definition optimization
  • optimize hook graph-inject - hook behavior optimization

See scripts/autoresearch/program-selfharness.md for the detailed draft workflow.

Watching progress without blocking the main session

For long overnight runs, open a second Claude Code session in the worktree directory and ask it to Monitor the results log:

Monitor scripts/autoresearch/runs/*.log and summarise each new iteration

The Monitor tool (Claude Code v2.1.98+) tails stdout from a background tail -F and delivers each new scoring line to Claude as a notification. You get progress updates on demand without interrupting the experiment loop in the main session. Requires Claude Code v2.1.98 or later.

Phone pings via Remote Control

For overnight runs where you actually walk away from the laptop, pair the autoresearch session with Claude Code Remote Control:

claude remote-control

Drive the worktree session from claude.ai/code or the mobile app, then enable /config → Push when Claude decides so your phone pings when an iteration keeps a change above the score threshold or when the loop hits a decision point. Combined with ENABLE_PROMPT_CACHING_1H=1 (see Tips below), you can run autoresearch overnight and check progress from anywhere.

Caveats: Pro/Max plans only (no API keys, no Bedrock/Vertex/Foundry), push is a single on/off, and the local claude process must keep running. ArcKit's minimum Claude Code floor (v2.1.287) already covers the v2.1.110 RC requirement.

How It Works

The system adapts autoresearch's ML experiment loop to prompt engineering:

  • autoresearch: modifies train.py to minimise val_bpb
  • ArcKit autoresearch: modifies a command .md file to maximise a quality score
  • ArcKit Self-Harness: modifies prompts, agents, hooks, runtime configuration, or verification harnesses, then accepts only changes that generalize across held-in and held-out tasks

The Loop

Each iteration follows the same cycle:

  1. Read the current prompt and results history
  2. Identify one specific improvement
  3. Edit the command .md file
  4. Commit the change to git
  5. Clean the scratch project (delete generated artifacts, keep fixtures)
  6. Execute the command against the scratch project
  7. Score the output (structural checks then LLM-as-judge)
  8. Compare to the previous best score
  9. Keep or discard based on a minimum delta threshold (>= 0.3)
  10. Log the result to results.tsv
  11. Check stop conditions — exit if score target hit, iteration budget exhausted, or double plateau detected (see Stopping Conditions)

If discarded, the prompt is reverted via git checkout to the previous best version. The full history (including discards) is preserved in results.tsv.

Status Line

After each iteration, a status line is printed:

[iter 3] score: 9.2 (best: 8.8) | status: keep | keeps: 3 discards: 1 | streak: 0/5 to plateau

This gives live terminal visibility without needing to read files.

Evaluation: Two Layers

Layer 1: Structural Gate (pass/fail)

Eight checks that must all pass:

  1. Document Control table with all 14 required fields
  2. Document ID follows ARC-NNN-TYPE-vX.Y pattern
  3. Revision History table present
  4. Standard footer present
  5. All major template sections present
  6. File written to correct path
  7. Domain-specific IDs correct (BR-xxx, FR-xxx, etc.)
  8. Wardley Map math validation (WARD commands only): stage-evolution alignment, coordinate range [0.00-1.00], OWM-to-table consistency

If any check fails, the iteration scores FAIL 0.0 and is discarded immediately.

Layer 2: LLM-as-Judge (1.0-10.0)

Five dimensions, each scored 1-10:

  • Completeness -- all sections substantively filled
  • Specificity -- references actual project context, not generic boilerplate
  • Traceability -- cross-references between artifacts present and correct
  • Actionability -- a vendor or review board could use this document as-is
  • Clarity -- well-structured, no contradictions

The combined score is the arithmetic mean, rounded to one decimal place.

Scoring uses an adversarial reviewer persona to prevent self-evaluation bias.

Self-Harness Extension

Self-Harness is an enhanced autoresearch mode based on "Self-Harness: Harnesses That Improve Themselves" (Zhang et al., 2026, arXiv:2606.09498v1). The ArcKit draft implementation keeps the original prompt-only workflow intact and adds a broader harness improvement loop for cases where the command prompt is not the only variable.

What It Adds

  • Multi-dimensional optimization: can improve command prompts, agent definitions, hook behavior, runtime wiring, and verifier logic depending on the selected mode
  • Trace collection: records execution traces under .arckit/autoresearch-traces/<target>/<mode>/
  • Weakness mining: clusters repeated verifier failures and trace signatures so the next proposal addresses a concrete failure mode
  • Harness proposals: uses harness-proposer.mjs to generate scoped candidate improvements
  • Held-in/held-out validation: requires improvements to hold on the optimization fixtures and not regress on reserved fixtures
  • Regression validation: uses harness-validator.mjs to reject candidates that overfit or exceed the allowed edit scope

Execution Modes

Mode Typical target Editable scope
prompt command requirements One command .md file
full command requirements mode:full Command, hooks, templates, runtime, and verification files declared in the run
agent agent research One agent definition
hook hook graph-inject One hook implementation

Use the standard program.md when you want a tight prompt-only optimization. Use program-selfharness.md when you need evidence about whether failures come from the prompt, execution harness, supporting tools, or validation layer.

Acceptance Rule

The Self-Harness loop is intentionally conservative:

  1. Score the baseline across held-in and held-out fixtures.
  2. Mine failures from traces and verifier output.
  3. Propose a scoped harness change.
  4. Validate the candidate on held-in fixtures.
  5. Re-run held-out fixtures before accepting.
  6. Keep only changes that improve held-in performance without degrading held-out performance beyond the configured tolerance.

This prevents the loop from optimizing around one fixture while making the command, agent, or hook worse in real use.

What Gets Modified (and What Doesn't)

For the standard prompt-only loop:

Editable (the only variable):

  • plugins/arckit-claude/commands/<command>.md -- the prompt being optimised

Read-only (the fixed benchmark):

  • Template file (defines expected output structure)
  • Quality checklist (evaluation standard)
  • Scratch project fixtures (controlled input)
  • Test argument ($ARGUMENTS = "001")
  • Evaluation rubric

This mirrors autoresearch's design: prepare.py is read-only, train.py is the only file modified.

For Self-Harness runs, the editable scope depends on the selected mode and is declared at setup time in program-selfharness.md. Keep that scope narrow for each run. A full harness run can touch more than one file, but it should still treat fixtures, scoring standards, and held-out tasks as read-only controls.

Constraints

  • One change per iteration -- isolate variables
  • Minimum delta of 0.3 -- filters noise from non-deterministic evaluation
  • Simplicity criterion -- marginal improvement + added complexity = not worth it; simplification + same score = keep
  • Log everything -- every iteration gets a row in results.tsv
  • No git reset --hard -- use targeted git checkout + revert commits
  • Session-wide budgets apply -- Claude Code caps each session at 200 WebSearch calls (v2.1.212) and 20 concurrently-running subagents (v2.1.217). The 200-spawns-per-session cap was removed in v2.1.224; CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION is still accepted but never read. A long autoresearch loop that repeatedly re-runs a research-heavy command can exhaust the search budget mid-run, and that cap soft-degrades rather than failing -- searches return a "not performed" result and the loop carries on with thinner evidence. Raise it with CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION, or let the loop start fresh sessions -- /clear resets it. WebFetch is uncounted but cached per URL for 15 minutes (CLAUDE_CODE_WEBFETCH_CACHE_TTL_MS, v2.1.233+), so a loop that re-reads the same page inside that window gets the cached copy.

Results

Results are logged to results.tsv (tab-separated):

commit  structural  score  status   description
a1b2c3d PASS        8.4    keep     baseline
b2c3d4e PASS        8.8    keep     expand NFR subcategories
c3d4e5f PASS        9.0    discard  derive NFR targets from context (delta < 0.3)
d4e5f6g PASS        9.2    keep     add use case structure instruction

Status values: keep, discard, plateau, crash, complete, budget-exhausted.

Plateau Detection

If 15 consecutive iterations are discarded, the system shifts strategy:

  • Re-reads the template for unaddressed sections
  • Reviews the quality checklist for uncovered criteria
  • Tries prompt simplification
  • Combines ideas from previous near-misses

A plateau row is logged when the trigger fires; the strategy shift resets the discard streak.

Stopping Conditions

The loop runs until one of three explicit stop conditions fires (no "run forever" mode). All three are tunable inline at the top of step 12 in program.md:

Condition Default Final status row Meaning
Score target best >= 9.5 complete Quality target reached; further iteration unlikely to pay off
Iteration budget iter >= 30 budget-exhausted Token budget bounded; rerun with higher cap if more headroom is wanted
Double plateau 2 plateau rows within 10 iterations complete Strategy shift exhausted; diminishing returns

The final status row carries the reason in its description field so results.tsv self-documents why the run ended. To extend a run, edit the constants in program.md step 12 and re-launch — the constraints in section 4 prohibit the LLM from negotiating past a hit threshold mid-run.

Practical Results

Commands optimised so far and their score improvements:

  • requirements: 8.4 to 9.2 (+0.8) in 3 iterations
    • Expanded NFR subcategories (5 generic to 7 specific with sub-prefixes)
    • Added explicit use case structure (UC-xxx with main/alt/exception flows)
  • adr: 8.6 to 9.0 (+0.4) in 1 iteration
    • Strengthened Consequences section with project-specific metrics, mitigation owners, after-action review
  • backlog: 8.0 to 8.8 (+0.8) in 1 iteration
    • Strengthened acceptance criteria rules (banned vague phrases, required measurable thresholds)

Net prompt changes are typically 3-8 lines. The improvements are small and high-leverage.

Which Commands to Optimise

Good candidates:

  • Commands that produce long, structured documents (requirements, backlog, sobc, risk)
  • Commands with detailed templates that the prompt may not fully leverage
  • Commands where output quality varies between runs

Not suitable:

  • Agent-delegated commands without a direct-execution fallback (research, datascout, aws-research, azure-research, gcp-research, framework) -- the prompt is a thin wrapper
  • Simple utility commands (customize, init, health) -- too short to benefit

File Structure

scripts/autoresearch/
  program.md              # The instruction file Claude follows
  program-selfharness.md  # Self-Harness enhanced instruction file
  fixtures/
    000-global/
      ARC-000-PRIN-v1.0.md    # Architecture principles (6 principles)
    001-test-project/
      README.md                # Project description
      ARC-001-STKE-v1.0.md    # Stakeholder analysis (4 stakeholders)

The experiment runs in a git worktree (../autoresearch-<command>), keeping the main checkout clean. The scratch project, results TSV, and all experiment commits live in the worktree. The branch tip (the improved command .md) is the deliverable.

Self-Harness runs also use trace and candidate outputs:

.arckit/autoresearch-traces/
  <target>/
    <mode>/
      iteration-N.json

scripts/autoresearch/runs/
  <target>/
    results.tsv

Tips

  • Run overnight -- each iteration takes 2-3 minutes, so you get 20-30 experiments per hour
  • Extend the prompt cache TTL for overnight runs -- set ENABLE_PROMPT_CACHING_1H=1 (Claude Code v2.1.108+) before launching Claude. The default 5-minute prompt cache expires between iterations once Claude pauses to think, score, and write results.tsv; the 1-hour TTL keeps the template, fixtures, and accumulated results.tsv warm across the full overnight run, materially reducing token cost. Pair with ANTHROPIC_API_KEY billing dashboards to confirm cache-read rates climb.
  • Configure a fallback model -- launch with --fallback-model <model> so that if the primary model is briefly unavailable mid-run (e.g. Opus overloaded), Claude Code switches to the fallback instead of failing every request (Claude Code v2.1.152+). Keeps an unattended overnight loop alive through a transient capacity blip rather than stalling on iteration 7. From v2.1.166 you can set a fallbackModel in settings listing up to three models tried in order, --fallback-model also applies to interactive sessions (so a long interactive /arckit:research survives a blip too), and Claude Code automatically retries a turn once on the fallback model when the API returns an unexpected non-retryable error (auth, rate-limit, request-size, and transport errors still surface immediately).
  • Review the results.tsv -- the discard history tells you what didn't work, which is as valuable as what did
  • Check against standards -- before starting a run, review relevant external standards (e.g., UK Gov ADR Framework for ADRs, GDS Service Standard for assessments) to prime the system with specific gaps to target
  • Create a PR for the prompt change only -- the experiment branch has noise (scratch files, results, reverts); cherry-pick the kept commits onto a clean branch
  • One command per worktree -- each optimisation run gets its own ../autoresearch-<command> worktree
  • Cleanup -- after cherry-picking results, remove the worktree: git worktree remove ../autoresearch-<command>

Limitations

  • Self-evaluation bias -- the same model generates and judges; adversarial scoring instructions mitigate but don't eliminate this
  • Fast convergence -- most commands reach 8.8-9.2 within 2-3 kept changes; further improvements are incremental
  • Fixture dependency -- a thin test project constrains what improvements the system can discover; richer fixtures reveal more gaps
  • Non-deterministic scoring -- the 0.3 delta threshold is empirically chosen; genuine improvements near the threshold may be discarded