A pass bar written down before any detector ran, and a study allowed to come back empty

The Situation

Finance research reads corporate disclosure as text, looking at tone, hedging, readability and how much a filing changed since last year. All of it rests on an assumption, which used to be free, that a person wrote the words. If a language model revised them, the measure is reading the model’s style, and a company that looks newly cautious may be a company that started running its footnotes through an assistant.

Nobody can say how often that happens, for the narrow reason that no published detector has been validated on financial disclosure. The ones that exist were trained to separate web prose from model output, and a footnote is neither. So the first question is not whether AI-generated language predicts returns but whether it can be seen at all in this kind of writing, and that has to be settled against labeled data before a filing is scored.

What I Did

I rebuilt the measurement instrument, which has four parts.

Passages are located by the filer’s own tags. The first version found the notes to the financial statements with a regular expression over the document text. The extractor now reads the index the filer publishes alongside the filing and takes the sections the filer tagged as notes, which came to 4,653 passages from 1,324 notes across 74 filings, drawn from a 40-company sample of which 37 had filings in the window. All of them are dated before 2020, so the human labels are certain by construction.

The labels are built, not assumed. The validation set holds 200 human passages, 600 model-revised versions of them and 50 wholly generated passages for the upper bound. Every revision is paired with its own original, so a detector can be asked whether it scores a passage as more machine-like than the text it came from.

The instrument reports a Detectability Curve, not a score. Accuracy is measured at three rungs of edit intensity (light, moderate, heavy), and a passage’s rung is measured afterwards from edit distance and n-gram overlap against its original, never inferred from the prompt that asked for it, because a prompt is a request and edit intensity is what happened.

The Gate. A detector may score real filings only if it reaches an AUC of at least 0.70 at the moderate rung and holds its false-positive rate at or below 5% against genuine pre-2020 footnotes. Four were named in advance: Fast-DetectGPT, Binoculars, a stylometric floor with no neural weights behind it, and the detector the withdrawn pipeline scored with, kept as the incumbent baseline. If none clears the bar, the study stops at the validation stage and reports that LLM editing of filing footnotes is not detectable with current public methods, and that is the finding rather than a failure to find one. The bar is not lowered, the ladder is not re-cut, and the panel is not scored in the hope that scale compensates.

What Changed

What changed is the thing I would lead with in an interview, which is that I withdrew my own published result. A report I compiled in June 2026 carried three headline findings. I re-examined it against the source data and against live filings, found three defects, and superseded it.

The score was a ranking wearing a probability’s clothes. The final quantity was a within-sample z-score through a sigmoid, and a filing was flagged if it landed in the top quartile. That flags exactly a quarter of any sample by construction, whatever the text says, and means adding one company changes the score of every company already scored. It was reported as though it were the probability that a filing was machine-written.

The extractor was reading the wrong text. Tested against six live filings, five landed on a cross-reference or a table-of-contents line rather than on the notes. One company ranked near the top for AI language had been scored on 4,109 characters of exhibit index and signature page. The heading pattern the extractor searched for matched zero times in all six, because a modern filing renders a note heading as “1. Significant Accounting Policies”, never as “Note 1 —”.

The trend appeared at dates where nothing had happened. I re-ran the specification with a fabricated break in November of every year from 2017 to 2024, and every one came back significant. Adding a linear time trend removed the effect, and so did a separate specification carrying firm fixed effects and a form-type control. The score drifts smoothly upward across the decade, and the date I had picked is a point on that line rather than a break in it.

So I wrote a correction, dated it, said the report was superseded and should not be cited, and recorded that it go to anyone who had received it. The pipeline was then rebuilt from scratch, with a provenance record on every passage giving its accession number, source URL, note and character offsets. Any number in the write-up now traces back to text a reader can open, which is the discipline whose absence let a signature page be reported as a finding.

Pending Result Whether AI-generated language in filings predicts abnormal returns is not answered here. The panel is not scored and is not authorised to be scored until a detector clears the Gate. If none does, the reportable finding is non-detectability, with the curve attached so that “filers are not doing this” stays distinguishable from “this instrument cannot see it”.

As for where the rest of it stands, the study is faculty-advised, and that is a working relationship rather than a name on a cover page. An application for research funding has been submitted but has not been decided, and it has not been awarded. A conference is the intended venue, whichever way the Gate falls. None of the three is an outcome.

What I Learned

The uncomfortable part is that the defects were not carelessness, since each one was a choice made with the answer already on the screen (which break date, which heading pattern, which cut-off counted as high), and none was recorded as a choice because at the time none of them felt like one.

So the remedy is not more care, because care is what produced it, aimed in the direction of the result I expected. The remedy is to commit to the decisions before the numbers exist, in a file with a date on it, which is why The Gate can disqualify a detector I would rather keep. The narrower version of the lesson is that a result that has survived no placebo is not a result. The check that cost me the finding took an afternoon, and I had not run it because the finding agreed with what I expected.

Where Else This Applies

A score defined by a ranking is not a measurement. Anything defined as the top quartile, such as vendor risk tiers, performance bands, lead scores or accounts flagged for review, identifies that share of the population whatever the population does, and adding a case changes the score of every case already in it. The damage comes when a sorting rule is escalated as a probability.

Check what the extractor actually read. Any process pulling fields out of documents, whether for diligence, compliance review or invoice and contract automation, reports what it found and almost never what it read instead. The cheapest check in this study was opening six documents and looking at the characters that had been scored.

Fix the pass bar before the measurement. Pilot evaluations, model selection and vendor bake-offs reach the same fork, where the success criteria are written down either before the numbers arrive or afterwards, in which case they are a description of the numbers. Fixing it first is what makes a negative result reportable rather than embarrassing, and a negative result nobody can report is a budget that keeps getting spent.