Evaluations

Memory systems should be evaluated by the behavior they improve. Codifix includes a repeatable evaluation harness for testing whether memory changes agent outcomes, retrieval quality, cost, and latency.

What the eval harness protects against

Overclaiming from a demo.
Confusing retrieval success with autonomous coding success.
Ignoring token and latency cost.
Treating stale or wrong memories as harmless.

Run an evaluation

Always dry-run first, then run the real suite only after reviewing scripts and fixtures.

# dry run
codifix eval run --suite evals/examples/memory-smoke \
  --condition full-memory --profile offline --dry-run

# paired run: no-memory vs full-memory
codifix eval run --suite evals/suites/memory-improvement-v1 \
  --condition no-memory --condition full-memory --allow-shell --repeat 5

# compare
codifix eval compare \
  --baseline  'target/memory-evals/*no-memory*.json' \
  --candidate 'target/memory-evals/*full-memory*.json' --text

Use --allow-shell only after reviewing suite scripts and fixtures. Shell-executing evals are code execution inputs, not passive data files.

Suite structure

An eval suite is a directory with:

File	Purpose
`suite.toml`	Suite name, project, fixture, default profile, repeat count, label status, and minimum item count.
`items.jsonl`	One evaluation item per line.
optional `scripts/`	Commands used by command or agent-build items.
optional `fixtures/`	Source workspaces, seeded data, or static inputs.

Common item types:

Type	Measures
`retrieval_qa`	Whether expected memories are returned.
`grounded_answer`	Whether answers include required facts and avoid forbidden claims.
`resume_quality`	Whether briefings cover expected context.
`command_task`	Whether a command succeeds.
`agent_build_task`	Whether an agent completes a fixture task under controlled conditions.
`agent_build_sequence`	Whether an agent maintains continuity across ordered work on one workspace.

External retrievers

Plug in your own retrieval backend for comparison:

codifix eval run --suite evals/suites/memory-improvement-v1 \
  --condition full-memory --retriever-cmd './my-retriever' --allow-shell

Ablation tests

Compare no-memory and memory-enabled variants item by item. Pair variants on the same suite, commit, and model to isolate what memory contributes.

Metrics

Metric	Meaning
Success rate	Whether the task met its expected outcome.
Recall@K	Whether relevant items appear in the top K results.
MRR	How early the first relevant result appears.
nDCG	Whether useful results rank near the top.
Assertion recall	Whether expected factual assertions were recovered.
Token cost	Model context or generation cost used by a run.
Latency	How long retrieval, answer generation, or eval work took.

Metric improvement is evidence for a bounded claim about the suite, model, and configuration used. It is not universal proof that every future agent task will improve.

Reproducibility

Tie evaluation claims to artifacts, suite version, commit, model/provider, and configuration. Keep raw JSON outputs under target/memory-evals/ and compare item-level results before making claims.

Current benchmark claim

The 2026-05-03 full memory-improvement-v1 report records five paired repeats and 155 comparison units: success moved from 0.0% to 18.1%, and total tokens from 22,069,461 to 12,970,186. Its valid artifact batch is 20260503062007, run group 1a4b75bf-ca1d-4c73-839e-4936c4c51fc0; earlier same-day batches are excluded. The hidden memory-only facts deliberately favored access to the seeded memory. These are historical suite results, not a current rc.1 performance guarantee. The sequence runs lacked required verified query artifacts, so the report does not establish improved long-running autonomous coding continuity. No new paid evaluation was run for this documentation refresh.

When publishing a claim, include:

suite name and version
git commit
model and provider
conditions compared
repeat count
item types
success, retrieval, token, and latency deltas
known shortcomings

The memory-quality canary

Beyond the improvement benchmark, evals/suites/memory-quality-v1 guards recall quality itself: retrieval items check the right memory surfaces, grounded items check answers state the right facts, and adversarial-stale items seed superseded facts, vague rumors, and unanswerable questions, asserting the pipeline refuses or prefers the fresh fact. Its gate (evals/gates/memory-quality-v1.toml) enforces absolute floors — at least 90% overall success and zero adversarial failures — independent of baseline deltas.

Gate status belongs to a specific run. The checked-in 2026-07-06 comparison and re-baseline records an offline/dev fixture regression, then a provenance-stable re-baseline of 18/26 overall and 0/7 adversarial items; that run failed the gate. Other pages may describe later fixes, but a current passing claim needs its matching run artifacts, commit, suite checksum and configuration. This refresh did not rerun the suite or establish current gate status. See Memory quality for the mechanisms.

Common mistakes

Mistake	Why it weakens the result
Running only one condition	There is no baseline.
Using one lucky repeat	LLM variance is hidden.
Ignoring token cost	Memory may improve quality while increasing cost, or the reverse.
Publishing unreviewed labels	The scoring target may be wrong.
Allowing shell without review	The suite can execute arbitrary code.
Treating retrieval metrics as coding proof	Finding a memory is not the same as completing a software task.

Read CLI eval reference, How it works, or Operations.