Tool descriptions. Skill files. Configuration. Everything around the model that decides what it sees and what it can do. The harness.
Code goes through review, tests, CI. These files ship straight to production.
02
The trust loop
Trust is a loop you run
Plan. Decide what good looks like before anything runs
Validate. Check the files the agent reads
Test. Run the checks on every change, before it ships
Eval. Watch and score what it actually does in production
Iterate. Every failure becomes a new check
Re-harness. Retire checks the new model no longer needs
03
Plan
PlanValidateTestEvalIterateRe-harness
Move the judgement upstream
Put the human at the few early points where an hour of thinking is worth days of correction.
Product review, before anything is built
System architecture: which components exist and how they talk
Program design: inside each component, each function's inputs and outputs
Slices reviewed a hundred lines at a time
The same amount of judgement, applied earlier.
04
Validate · Failure one
PlanValidateTestEvalIterateRe-harness
The agent picks the wrong tool and you blame the model
The reasoning was fine. The interface was ambiguous. So you add instructions on top, and the real fix was in the description the whole time.
05
Validate
PlanValidateTestEvalIterateRe-harness
The description is the interface
Ambiguous
"List items on a Miro board."
Transparent
"List items on a Miro board (max 50). For ALL items with auto-pagination, use miro_list_all_items. For text search, use miro_search_board. USE WHEN: what's on the board, show all stickies, list shapes."
Both are correct. Only one says when not to use it. It ships in my Miro MCP server today, word for word.
06
Validate
PlanValidateTestEvalIterateRe-harness
Cap what your tools return
A tool that returns everything makes the agent read everything. You pay on every call, in tokens and in accuracy.
Return the fields the job needs. Reliability and cost are the same lever.
Validate
PlanValidateTestEvalIterateRe-harness
Do not confuse your agents
So I built SkillCheck: a validator for the files agents read. Line caps, token budgets, dead file references, missing stop conditions. It runs on every change.
getskillcheck.com
$ skillcheck check sift
CRITICAL body exceeds 800 lines (got 1277)
CRITICAL body exceeds 8000 tokens (15356)
WARNING referenced file missing on disk
WARNING no explicit stop conditions
1 critical, 7 warnings more · 70/109 passed
Status: FAIL
07
Validate
PlanValidateTestEvalIterateRe-harness
Tools in Miro's official MCP server
0
Tools in my Miro MCP server
0
Both are right. Theirs asks what a board means. Mine asks what you want to do to it.
Describability is the part you control.
08
Test
PlanValidateTestEvalIterateRe-harness
Point the pipeline at the files agents read
Linting, so the standard lives in a check instead of someone's memory
Continuous integration, so nobody has to run it
Code health gates, so decay is visible early
Dependency triage, so stale things get chased
The machinery is decades old. The target is the new part: tool descriptions, skill files, configuration.
09
Test · Eval
PlanValidateTestEvalIterateRe-harness
Test the harness. Eval the behavior.
Test
Runs on the files, on every change, before anything ships. Deterministic: the same file fails the same way twice. The verdict is pass or fail, and fail blocks the merge.
Eval
Runs on the traces, after the agent acted. The same prompt can go five different ways. The verdict is a score over many runs, and you watch which way it moves.
The loop connects them: what an eval catches in production becomes a test on the next change.
Eval
PlanValidateTestEvalIterateRe-harness
Trust is earned unattended
Scheduled scans that post only what changed since yesterday. A delta you stop reading is the same as no check at all.
A real delta, posted by my dependency triage run while I slept.
delta · 10-08-2026 · scheduled run
NEW (1) miro-mcp-server#98 deps High
RESOLVED (1) mcp-otel-go#31 merged 09-08
NEWLY STALE 0
CARRIED 0
10
Eval
PlanValidateTestEvalIterateRe-harness
Three altitudes of eval
Capability. What the model can do on a benchmark
Context. Whether it works inside your domain and workflow
Value. Whether it moved the number the business cares about
Benchmarks help you pick a model. Trusting a product takes the other two. Pilots die when value never gets measured.
Eval
PlanValidateTestEvalIterateRe-harness
Read the traces before you write checks
You do not know what to check until you read what it actually did. The criteria come out of the traces themselves. Reading them is how every one of these failures was found.
Hamel Husain calls this error analysis before evals. It is the least glamorous step and the highest yield.
Eval · Failure two
PlanValidateTestEvalIterateRe-harness
Correct reasoning, wrong answer
One of my scheduled agents reported a security update as resolved. It was still open. The reasoning was perfect. The data it reasoned over was wrong for about thirty seconds.
What it reported
security update: resolved
What a second read showed
still open
The fix: every "resolved" claim now gets verified against a second source.
12
Eval · Failure three
PlanValidateTestEvalIterateRe-harness
Absence is not evidence
Another agent of mine audits pull requests. One report claimed a change had zero reviewers.
What the agent reported
reviewers: 0
What was actually true
reviewers: NOT_CHECKED
The agent never queried that field. Your tooling has to distinguish "unchecked" from "zero", and say which one it is.
Eval
PlanValidateTestEvalIterateRe-harness
Most checks do not need AI
Invalid JSON: schema validation
Missing field: a keyword check
Wrong tool picked: assert on the tool name
Broken SQL: parse it
Save the LLM judge for genuinely subjective calls: binary pass or fail with a critique, checked against examples you labelled yourself.
Eval
PlanValidateTestEvalIterateRe-harness
The ceiling of automated evaluation
0%
The best of three commercial eval agents caught 87 percent of the failures human reviewers had flagged. All three missed the same class: the trace reads like a success, and the business goal was missed.
The last 13 percent is your product judgement. Saha and Husain, Parlance Labs, 07-2026.
Eval
PlanValidateTestEvalIterateRe-harness
Give every metric a card
A definition in one sentence
A unit and a cadence: per answer, aggregated weekly
Thresholds that say what good looks like
A named owner and an action when it drops
A groundedness card: above 0.8 excellent, below 0.6 investigate retrieval, below 0.5 an alert fires. Thresholds are "we know what good looks like" written in numbers.
Iterate
PlanValidateTestEvalIterateRe-harness
Every failure becomes a check
Write down what broke, with the date
Turn it into a rule the machine can enforce
Add it to the pipeline, so it runs without you
All three failures run as automated checks today. That is the whole iteration loop: pay for a lesson once, then stop paying.
delta · 13-08-2026 · scheduled run
RESOLVED (4)
mediawiki-mcp-server#112 merged
nordic-registry#62 merged
ridge#53 · miro-cli#18 merged
"verified by direct read after the
search index returned zero hits"
14
Re-harness
PlanValidateTestEvalIterateRe-harness
Every check is a bet that expires
Every check exists because a model failed once. Models improve, so the bets expire: last year's harness can block this year's model.
At every model upgrade, take one check out
Run without it and watch for the old failure
It comes back: the check stays. It stays quiet: retire it
Researchers call this ablation: take a part out on purpose to learn whether it still earns its place.
15
The part that stays human
A linter enforces a standard. Choosing one is your job.
How many tools is too many
When a plain command line beats a server
What belongs in context and what is noise
Which mistakes are worth preventing
Every one of these is a product decision. The machine holds whatever line you draw. Drawing it stays with you.
16
The part that stays human
To trust your agents you have to trust yourself, and that means knowing what good looks like.
Every check in the loop is a judgement someone made, written down and made cheap to repeat. The expertise you already have is the scarce input.
Take this home
Four checks you can add this week
Describe each tool: exactly what it does, and what it does not
Cap what comes back before it costs you
Verify the one kind of claim that would embarrass you
Make "I did not check this" visible in the output
Are agents ready for enterprise adoption
Ready where you can say what good looks like
And check it cheaply. Where you cannot say what correct output is, an agent is a claim you cannot audit.
