A gate somewhere in your pipeline is rejecting four inputs in five, and the fix that suggests itself is the threshold. It is the cheapest change available, it will work immediately, and it is almost always wrong. What follows is four days of being wrong about exactly that, with the measurement that ended each suspicion — and the one habit that would have got there on the first day instead of the fourth.
This is The Vision Log, a series about one long-running agent that reads, changes and verifies its own source code, and this part follows four wrong diagnoses inside it. Vision is not an assistant and not an automation framework: it is designed as a digital counterpart to the developer who built it, and autonomy is a means it uses rather than the point of the exercise. Like every part of the series, this one states only what the project’s own engineering log records, including the diagnoses that turned out to be wrong. Part 0 explains what the project is and the wish it came from.
The symptom: most researched topics never reached the writer
If topics keep dying at a gate in your own pipeline, the shape of this one will look familiar. The pipeline this article is about is an ordinary one. A run collects candidate topics, searches for material on each, hands the collected documents to an extractor that returns propositions with the source URLs that back them, and then a verifiability check asks whether enough of those propositions are attested. Topics that fail that check are held: no article is written, no writing tokens are spent, nothing is published.
By early September the hold rate was the dominant fact of every run. Across three sampled runs the log records 66, 59 and 62 topics held out of 77 researched (engineering log, 2026-09-07). Around half of those holds were not marginal calls. The dossier contained zero propositions at all: 38, 31 and 32 of them in the same three runs (engineering log, 2026-09-07).
The obvious first move is to reach for the threshold. That is exactly the move worth distrusting, and the four suspicions below are what it cost to stop reaching for it. Read them as four things to rule out before you touch a threshold of your own.
Suspicion 1: the gate is too strict
Your first instinct at a gate that rejects most of what reaches it is to loosen the gate; count the material first instead. Across 2,094 research dossiers, the number of independent source domains had a median of 1 and a mean of 1.1: 734 dossiers had no source domain, 927 had exactly one, and 416, or 20%, had two or more (engineering log, 2026-09-07).
That single number disposes of the suspicion. The gate’s main verification path is cross-checking, which requires two independent domains supporting the same proposition. Four out of five topics could not satisfy it at any threshold, because the material was one source deep. Even the articles that did get published averaged a cross-check ratio of 0.11 (engineering log, 2026-09-07).
The downstream failures were the same shortage wearing two different masks. Either the writer refused to assert unverified propositions and hedged every paragraph, which failed a repetition check on hedging language in 19 of the most recent 65 verdicts, or it dropped the procedures and figures entirely and told the reader to consult the original, which failed the intent check (engineering log, 2026-09-07). Two unrelated-looking gate failures, one cause underneath.
Takeaway. A pass rate is a joint property of the threshold and the input distribution. If most inputs are structurally incapable of passing, the threshold was never the variable. Measure the distribution of what arrives at your gate before you touch the number on it.
Suspicion 2: there is only one source, so go find the second
The second thing you are likely to try is to go and find another source. This one was right about the shortage and wrong about the cure. A corroboration round was built (engineering log, 2026-09-07): after the first search round, every proposition that was neither cross-checked nor attested by its own subject gets its statement used as a second query, with every domain already in the corpus excluded, capped at three propositions per candidate, and a re-extraction only when a genuinely new domain came back. Exclusion by domain is what makes the second hit independent rather than a second page on the same site.
It shipped, the stack restarted, and the next run measured essentially flat. The share of dossiers with two or more independent source domains went from 13.0% to 13.2%, and the hold rate from 85.7% to 73.5% (engineering log, 2026-09-11).
The reason is worth sitting with. A second query derived by the same broken procedure searches for the same wrong things, so it returns the same corpus minus the domains already seen. The round was correctly built, it is still in the pipeline, and it did almost nothing on its own, because it was downstream of the actual defect.
Takeaway. Record a before and after for every change, including the ones you are sure about. A plausible fix that measures flat is information: it tells you the bottleneck is upstream of where you just worked.
Suspicion 3: the subject’s own page is not being recognized
The third thing you will doubt is your own definition of what counts as evidence. The third suspicion was that attestation was too narrow. A claim could count as primary-attested only when the announcing domain was the candidate’s own source domain, so a roundup article citing a repository’s own page for a fact about that repository got no credit, even though the subject was describing itself on its own hosting page.
That was a real defect, and it was fixed by adding a claim-level attestation path on top of the candidate-level rule, with a vocabulary filter so that evaluative, comparative and predictive sentences can never self-attest (engineering log, 2026-09-07).
But it was measured before it was believed. Against the holds it was meant to rescue, the subject-hosting assumption recovered 4 of 66 held topics under the candidate-level rule and 8 of 66 under the new claim-level one, with 3 and 6 in each of the other two runs, while a control group of opinion, prediction and question topics false-passed zero times in all three runs (engineering log, 2026-09-07).
Call it six percent of the problem. Worth shipping, and not the cause.
Takeaway. A fix with a ceiling of six percent is not a fix for a problem of eighty. So before you ship one, put a number on its reach: what share of the failures you have actually observed can this change explain, at most?
Suspicion 4: it must be the extractor, the prompt or the model
By this point you will want to blame the model. With half the holds containing zero propositions, the extractor was the natural defendant, and its model the natural sentence. The log could not settle it. A dossier recorded only its surviving source domains and its gaps, the findings and the raw extractor output were discarded, and extractor exceptions were swallowed by a bare handler that returned an empty tuple (engineering log, 2026-09-07).
So the next change was not a fix at all. It was instrumentation: the extractor’s return type began carrying how many documents it considered, how many raw propositions it returned before any were dropped, how many were dropped for citing no URL in the corpus, the exception class if it raised, and the lengths of the prompt text and of the response. All with defaults, so old records still load. The extractor’s behaviour was deliberately not touched (engineering log, 2026-09-07).
The instrumentation immediately killed the suspicion. Every zero-proposition hold had research results present; the count of cases where the search returned nothing was zero. In one run, the 38 zero-proposition holds had between 25 and 61 independent domains among their findings, with a median around 47 (engineering log, 2026-09-07). The material was there, in volume, and the model was still returning an empty array.
Takeaway. An empty result has to record why it was empty. Until it does, every explanation of it is a guess, and swallowed exceptions guarantee the guess will be wrong for a while. Make your own empty results say why they are empty, and you will not spend a week explaining one.
The real cause: a version number shredded into three queries, inside a window of 24
You find this one only by replaying the inputs. The answer came from reproducing six headlines against the real search providers and looking at exactly what the extractor was shown. Of the 20 documents in the window, the number actually related to the topic was 0, 0, 0, 0, 1 and 1, and the number of raw propositions returned was 0, 0, 0, 0, 0 and 5 (engineering log, 2026-09-07).
That is not a model failing. That is a model correctly declining to invent propositions about documents that have nothing to do with the topic. And the one case that had an on-topic source in the window produced five propositions immediately.
The reproduction put the cause on the screen in one line. A headline about CodeQL 2.26.4 left the query builder as three separate searches, 2, 26 and 4: the version number had been shredded into its digits, and those digits were what the search providers were asked. Every suspicion above had been an explanation of the answers to that question.
Four defects, all upstream, all in code that nobody had suspected because it had no failure of its own to report (engineering log, 2026-09-07):
- Query derivation shredded entities. The salient-term extractor split multi-word proper nouns and version numbers into separate tokens, as the version above shows, and it was not confined to versions:
Blender Lab Activity ReportproducedLab,ActivityandReport. The search engine was being asked the wrong questions. - The relevance filter passed on one shared token. A document survived the filter if it shared a single token with a query, so everything published in
SeptemberaboutAIwas relevant to everything. - One class of provider skipped the filter entirely. Community sources went into the corpus unfiltered, and encyclopedia lead paragraphs are long, so they occupied the window.
- The window truncated by arrival order. The extractor sees at most 24 documents, and the list was cut without being sorted, so the candidate’s own source article could fall outside it. In one reproduction it sat at position 19.
The repairs match the defects one to one: pull quoted phrases, capitalized runs and name-plus-version pairs as single queries; require two distinct token matches, with an exception for names whose shape is distinctive, such as internal capitals or digits; wrap the community providers in the same relevance filter; and sort by topical relevance before truncating, with the candidate’s own source URLs pinned to the front of the window.
Note what is not in that list. The window stayed at 24. The extractor’s prompt was not rewritten. The model was not changed. No threshold in the gate was lowered. The inputs were repaired and everything downstream was left alone, which is the order worth trying on a pipeline of your own: repair what arrives at a stage before you touch the stage.
Holds fell from 79.7% to 38.0% with no threshold touched
If you would rather check this repair than take it, the three figures below are the ones to replay. They were measured for this article directly from the publication record, by replaying the same verifiability function over every dossier each run recorded (engineering log, 2026-09-11). "Before" is the three runs of 2026-09-07 that ran before the repair reached the live stack; "after" is the three runs of the same day that followed it, and the third column extends that to every run through 2026-09-10.
| What was measured, per researched topic | Before: 3 runs, 182 topics | After: 3 runs, 131 topics | Through 2026-09-10: 363 topics |
|---|---|---|---|
| Research ended with zero propositions | 44.5% | 15.3% | 12.9% |
| Topic held before any writing | 79.7% | 53.4% | 38.0% |
| Dossier had two or more independent source domains | 14.3% | 29.0% | 34.2% |
At the boundary itself, comparing the last run before the repair with the first run after it, zero-proposition research went from 47.1% to 17.6%, holds from 73.5% to 47.1%, and two-or-more-domain dossiers from 13.2% to 29.4% (engineering log, 2026-09-11).
Two honesty notes, because this is a production pipeline and not an experiment. Each run draws a different pool of candidate topics, so run-to-run variation is real and none of these numbers is a controlled measurement. And the boundary between the two windows is not an assumption: the third defect above, the unfiltered community provider, leaves a fingerprint. The median corpus size fell from 153 documents to 45, and the share of propositions resting on an encyclopedia domain fell from 17.5% to 5.3%, in exactly the two adjacent runs (engineering log, 2026-09-11). Take the pair of windows as a before and after you could reproduce rather than as a controlled result: if your own numbers move like that, look for the input that changed shape before you look for the threshold somebody moved.
What changes for you
You can take the whole of this article as one habit, and it would have found the defect on the first day: a low pass rate at a gate is a measurement of the inputs, not a verdict on the gate. No threshold in the gate was lowered and zero-proposition research still fell from 44.5% to 12.9%, which is what repairing the inputs instead of the boundary looks like. Loosening the gate does the opposite: it converts a visible refusal into an invisible bad output, which is a worse failure and a much harder one to find later.
From there the method is to trace backwards to where the failing input is manufactured. In this case the chain ran gate, extractor, corpus, relevance filter, query derivation, and the defect was at the far end, four steps upstream of where the symptom appeared. Three smaller rules made that walk survivable. Instrument the step that produces nothing, because an empty output with no recorded cause supports every theory equally and is the most expensive thing in a pipeline. Put a ceiling on a fix before you ship it: a change that can explain at most six percent of the failures is not the answer even when it is correct. And in any retrieval window, order before you truncate and pin the document you already know is relevant, because a window is a ranking problem wearing a size limit.
Four of those days were spent on explanations that were reasonable, cheaply testable, and wrong. The reason this article can be written at all is that each one was recorded with its measurement instead of being quietly overwritten by the next suspicion, which is the same practice that lets you tell a flat result from a fix.
Where this goes next
Part 4 — where this stands, against the standard it was given closes the series, and it turns the same method on the project itself: the six things Vision was originally described as, the loops that would make it act first and are switched off, and a charter that rules out almost every number this series has published.
Part 0, on what Vision is and the wish it was built from, is here. Part 1, on the two systems built and abandoned before this one, is here. Part 2, on what breaks when an agent edits its own repository, is here.