The State of Play

A run starts at 00:05 London time and moves through eleven checkpointed steps in order — interrupted partway, it resumes from the last completed step instead of starting over. Open picks the batch: up to 54 practices, stalest-scanned first, from whichever domain the 14-day rotation has due. Prep and scan then run once per practice, independently, before history bullets land on every practice that gained evidence, and summary, picks, and header write up the domain as a whole. Review checks the entire night's work; land merges it into the live site, publish sends that night's brief out, and close writes the run report and tears down after it.

Two different kinds of automation are doing this work. Some steps are ordinary code — arithmetic, deduplication, schema validation, the parts with one right answer. Others are a model making a judgment call: is this page evidence, does this evidence clear the bar for promotion, does this summary hold up. Nothing pauses for a person by default; a person only enters the picture if the review layer specifically asks for one, which is rare (see "Checking the work," below).

Each practice gets six search queries a night — three hunting for adoption, three for trouble. Every result is de-duplicated by URL in code and dated from what the page itself says, not guessed by a model — a model is good at reading a claim and bad at date arithmetic, so dates stay in code before and after a page is judged. A page nobody can date is still kept and judged, not discarded.

Not every fetched page becomes evidence. A typed-question model decides most fetched pages outright, kept or dropped, from a fixed set of yes/no and multiple-choice answers. A second model writes the explanation and summary for every page that survives, and makes the call itself on the pages the first model leaves undecided.

Every kept item is scored for source credibility, not just relevance. Peer-reviewed research, standards documentation, and engineering blogs with real deployment detail score highest. Vendor material — even a strong case study — is capped below independent sources: a vendor describing its own product isn't a neutral witness, and self-reporting is useful context, not proof. When evidence points two ways, both directions are kept on file — at least one negative or critical signal survives selection whenever one exists, specifically so an all-positive set can't make a promotion look inevitable.

Scanning all 19 domains every night would mean a shallow once-over instead of a real dig into a handful of practices. Instead, the rotation works through fourteen slots on a two-week cycle: most nights cover one domain, five nights cover two related ones. There's no memory of where the cycle last stopped — the slot for any date is just that date's position in the 14-day loop, fully reproducible from the calendar alone.

On a night that covers two domains, only one gets that night's newsletter brief; the other's turn comes four weeks later. A domain covered alone publishes every time its slot comes up. Either way, every domain is rescanned every fourteen days.

A tier promotion isn't a number crossing a threshold on its own. The model first builds a qualitative case: state the next tier's defining question, answer it citing specific evidence by title, then argue the strongest case against its own conclusion — a steel-man. Only if that steel-man comes out weak does the promotion reach a numeric gate at all: at least two citations that match a real evidence item's title, a minimum amount of evidence for that tier, and the right mix of evidence types. A practice moves at most one tier per scan, never skipping one.

When the case doesn't hold up, nothing happens, and that's deliberate: the default is not to promote. A wrongly-early promotion is treated as worse than a delayed one.

A concrete example of where the bar sits: evidence from a single vendor can support a Leading Edge classification, but not Good Practice or above — a single-vendor case that technically clears a higher tier's numeric gate gets flagged for a person, not promoted automatically.

Demotion works differently on purpose: it's never automatic. If the evidence behind a practice's current tier stops holding up, that's an editorial call, not a script. Trend — a practice's momentum toward its next tier — is a separate, related question, judged from the same cited evidence: a change is written only once two scans running have agreed on it (a promotion sets Accelerating at once). See Methodology for the five values.

Before any of a night's work reaches the live site, it passes a review layer with two parts. First, deterministic checks that need no judgment at all: is every file valid against its schema, does every date make sense, has an already-published evidence item been quietly edited (never allowed), does a promotion actually clear its numeric gate. A single failed check decides that run's outcome regardless of what any model says about it.

Second, a sampled model review reads a handful of that night's decisions — practices with no tier change, and practices with a reconfirmed or newly-pending trend — looking for the kind of problem a checklist can't catch: does this tier decision actually hold up against its cited evidence, does this trend match the evidence's arc, does anything look invented. The sample is picked by a formula seeded on the date, reproducible after the fact but not predictable in advance.

If that review comes back clean, the night publishes on its own. If it flags something, a further, independent check — reading the validator report, the review, and the finished brief together — decides whether to publish anyway, correct one item, ask for a narrow rewrite of the write-up (never the underlying research), or put the decision in front of a person. That last option is rare, reserved for things like a validator failure nothing already explains, a specific brief claim resting on one thin source, or a pattern across many practices that looks systematic. Every decision is logged, whichever way it goes.

A one-off audit on September 27, 2026 checked every specific claim in the index, meaning a named organization together with a figure or a dated event, against the evidence on file. Of 26,092 such claims, 25,301 matched an evidence item automatically. The other 791, plus 7 flagged on a close read, were each checked against their original source. 637 were accurate and already backed by evidence on file. 83 misstated their source and were corrected to match it. 38 were accurate but had never been given a source; one was found and filed for each. 16 had lost their source in an earlier rewrite, and it was restored from the page's history (one of these was also corrected). 25 had no source anywhere, even after a second search, and were removed.

The same facts are also available as structured data: a machine-readable summary at /how-it-works.json, a plain-text mirror at /how-it-works.md, and citation metadata embedded directly in this page. /llms.txt points here and at the rest of the site. The State of Play's data and text are CC BY 4.0 — reuse them, including for training, with attribution and a link back to thestateofplay.ai.