WAYS OF WORKING · DEMO 13 NINO CHAVEZ
Three ways to make a meeting archive answer a question. Three pass/fail gates, each committed to git before the model weights finished downloading. Three refusals in one day — and the feature that shipped is the one the evidence admitted.
Registered, before the run
"False passages returned: 0–2. Product work continues only if every clause passes."
→ or scroll to advance · ← to go back
02 WHY THE GATES GOT FROZEN
The subject is a local meeting-notes app I run — recordings stay on the machine, and the build in question was learning to answer questions about past meetings. Its first search probe passed: 10 of 10. Three of those ten passed by a margin under 0.04 — a coin balanced on its edge, reported as a win.
The passing demo
Semantic retrieval finds the right meeting, 10 of 10.
A result that dies when you change the ruler was never a result. After you've seen the score, every adjustment feels like fixing the measurement; before, the same adjustment is just design. So the ruler moved to the only place it's safe: before the run. Medicine learned this the hard way — trials register their endpoints before anyone unblinds the data. Same instinct, applied to an AI feature.
03 THE METHOD
"Registered 2026-08-08, before the fixture was embedded or any score was observed. Nothing in this section reports a result." opening line of the probe document, committed before inference
The fixture & the model
A synthetic test set — invented companies, invented meetings, zero real recordings — plus the exact model files, pinned by checksum. Nobody can quietly swap either after seeing a score.
The threshold & the prediction
The pass bar comes from previously committed evidence, never from the run it will judge. The prediction is two-sided — it names how many false positives are tolerable, not just how many hits are hoped for.
The consequence
What happens on failure is written down too: stop, record the receipt, build nothing. No re-runs, no threshold-chasing. The agent can't negotiate with a commit that predates its own output.
Placement is the trick — outside the loop, earlier in time.
Why this matters for agents specifically: an agent is a relentless completer. If the gate lives anywhere the loop can edit, the loop will satisfy the gate instead of the goal — tuning until the number passes is what "helpful" looks like from inside. The only placement that survives is earlier in time and outside the editable surface.
04 PROBE ONE · THE SIMILARITY FLOOR
Rank the meeting windows by similarity to the question; return only what clears a floor fixed in advance. Registered prediction: 0–2 false passages. Eight minutes after the gate was committed, the refusal was on the record:
The texture of the failure is the point. Asked about a dental plan that appears nowhere in the test set, the system confidently returned three passages anyway. Topical, plausible, wrong — exactly the failure a happy-path demo never surfaces, because nobody demos the question with no answer.
05 THE FACT TUNING WOULD HAVE HIDDEN
The lowest true answer scored 0.4195. A passage that explicitly answers nothing scored 0.6096. Any floor high enough to block the impostor also deletes a real answer.
"Raising the floor cannot fix this result. […] The bi-encoder score measures topical similarity. It does not establish that a passage answers the question. The fixed floor is left unchanged, and no alternative is fitted to these scores." the probe document's own close, written into the failure record
This is the verdict only a frozen gate can deliver. A tuner would have slid the threshold until this fixture passed and shipped a feature that fails on the next real question — "needs tuning" and "cannot work as specced" produce identical demos and opposite decisions. The frozen gate is what tells them apart.
06 PROBES TWO AND THREE
The obvious next move each time was a smarter component. Each got its own preregistered gate — committed before its checkpoint finished downloading — and each probe document opens by restating that the feature remains stopped.
Probe two · a pairwise scorer reads question and passage together
Registered: accept every true answer, refuse the registered control set and all four unanswerable questions.
Probe three · an extractive model with an explicit no-answer option
Registered: the no-answer head should finally make refusal honest.
Three mechanisms, three refusals, one afternoon — each failure on the record within minutes of its gate. Refusal was cheap because the protocol made it cheap. That's the part worth stealing: the discipline isn't willpower, it's a harness that makes stopping the default outcome of a failed gate.
07 THE SAME WEEK, ELSEWHERE
Long-form transcription
A preregistered pipeline for transcribing long meetings — synthetic audio, gates on fidelity, seam integrity, and runaway repetition. Measured: 22% word error, 21 lost seam anchors, 634 runaway four-word repetitions. Three gates failed.
Refused. The production transcription path was not changed.
Speaker naming
What shipped is the honest subset: you type who was in the meeting, one name per voice slot, exact and operator-authored. Automatic voice identification — the demo-friendly version — stayed unbuilt after its probe failed the same style of gate.
Shipped: the version that can't be confidently wrong.
Four refusals in one week, from the same operator's agents, none of them argued after the fact. Every argument had already happened — in a commit, before the run, when nobody knew the answer and everyone could still be honest.
08 WHAT SHIPPED INSTEAD
The app still shipped that week — twice. The release that went out answers questions the only way the evidence admitted: exact retained words, cited to the turns they came from, with refusal when nothing clears the bar. No generated summaries, because generation never passed a gate.
The release ledger, same week
0.5.0 shipped "the first build that can be asked a question" — then a routine check found a common word returning nothing at five meetings.
Most people have never watched a system tell its owner "no" and the owner ship the no. But that's what the week's output actually was: three features that don't exist, one that does, and receipts for all four. A refusal with a receipt is a deliverable. The alternative was three demos that worked and a product that lied.
09 YOUR VERSION OF THIS
If you never touch code
Before you look at any AI result — a report, a screening, a recommendation — write down what would make you reject it. One sentence, before. After you've seen the output, you will negotiate with it; everyone does. The sentence you wrote first is the only one that can't.
If you're technical
Preregister the gate: fixture, model checksums, threshold, a two-sided prediction, and the consequence of failure — committed before first inference. One held-out observation. A failed gate means stop, not tune; "needs tuning" and "can't work" demo identically.
If you build systems
Put the gate where the loop can't edit it — earlier in time, outside the agent's writable surface. Make receipts content-free so they can live in the repo forever. And make refusal cheap: minutes from failed gate to recorded stop is what keeps the discipline alive.