The standard
every instrument meets
This is the contract. An instrument that satisfies it can be published here; one that does not, cannot. It is stated so that someone else could build to it without anyone reading their code.
Thresholds are fixed before results are seen
Every instrument declares, in advance, what would count as its model working and what would count as it failing. Those thresholds live in version control, and the commit that set them is dated before the run that tested them. That is checkable, and it is the single reason anything else here is worth believing.
- A gate is published with its outcome — pass, fail, or not evaluated — never as a count. "Three of four passed" hides which one failed.
- A failure ships red and stays. It is not retuned, not quietly dropped, and not moved to a footnote.
- A gate whose input never arrived ships as not evaluated, with the reason and what a person would have to do. It is never recorded as a pass.
The instruments on this site currently include a nowcast that is 41% worse than doing nothing, two hard gates that fail on a transport model, and a detector whose R² is −11.09 before calibration. All three are on their own pages, stated in those words.
Scored against a rival that already knows the easy part
A model is only interesting relative to what a person could do without it. So every instrument is scored against the naive rivals that already contain the obvious knowledge — carrying yesterday forward, the seasonal average, the published index that already exists — computed on training folds only, never on the test set.
The contract, in six items
| # | Requirement | Checked |
|---|---|---|
| 1 | Gates with explicit outcomes. Every gate carries an id, what it tested, its threshold, and its result. | mechanically |
| 2 | A stated vintage. When each source was retrieved, per source — not one date for the case. | mechanically |
| 3 | Sources with licences. Every input named, with its terms, where a reader can find them. | mechanically |
| 4 | No personal data. In any published file, not just the visible page. | mechanically |
| 5 | No redistributed source data. Derived results only; the pipeline fetches the rest. | mechanically |
| 6 | An article separable from the dashboard. One is frozen and citable, the other is live. | mechanically |
Those six are run as a checker over every case. It is the acceptance test: produce output that passes, and the instrument can be published — no code review required, because what is checked is the output, not the implementation.
A static figure is not an instrument
This is a standard on the artifact, not a bar on the person: it says what the visualization has to do, and anyone whose work does it has cleared it. Most published analysis stops at a chart, which is why the last stage is written down as carefully as the gates.
| Required | What it rules out |
|---|---|
| Linked views. Selecting in one view filters the others, so a reader can follow a single unit through every panel | A grid of unrelated charts that happen to share a page |
| Time is navigable where the data has time — play it, scrub it, or step it, at the resolution the pipeline actually produces | One year chosen by the author and presented as the picture |
| Drill-down to the unit of analysis. If the model estimates at kecamatan level, a reader can reach one kecamatan | A national aggregate that cannot be questioned locally |
| At least one assumption is a control. The reader changes it and watches the answer move — the threshold, the baseline, the window | A conclusion the reader can only accept or reject whole |
| Failures are visible in the interface, not only in the appendix. A failed gate appears where the result appears | A dashboard that looks equally confident whatever the gates said |
| Legible without its author. Units, dates, sources and the meaning of every encoding are on screen | A figure that needs a caption the reader does not have |
Why this is a standard and not a preference. Every item above exists so a reader can disagree with the result in a specific way. A visualization that cannot be interrogated asks for trust instead of offering evidence — which undoes the gates and the baselines that come before it.
Accessibility counts as part of it: keyboard reachable, contrast that holds, motion
that respects prefers-reduced-motion, and a page that works on a phone.
Five questions no regular expression should answer
These matter more than the six above, and none of them is mechanically decidable. They are recorded as undecided rather than guessed at, and answered by a person — or, for a first pass, by a model whose answer is labelled as such.
- Framing. Is the finding stated as what this run found, rather than as a verdict on someone else's work?
- Baseline. Does the rival it was scored against actually contain the easy knowledge?
- Failures. Are failed gates presented as plainly as the passes, or buried?
- Named actors. Does the case accuse a named company or project of wrongdoing?
- Claim support. Does the headline follow from what the gates actually establish?
What none of this catches
A plausible question about a topic a dataset half-covers can pass every gate and still be answered wrongly. Rigour about method does not confer rigour about meaning, and no amount of validation makes a signal exist where there is none.
Which is why the interpretation is not the measurement side's to write, and why an instrument published without one says so on its face rather than filling the gap with confident prose.