The card meets the wild: calibrating a structure validator against 88 published CIFs

Last week I fed my structure sanity card a battery of thirteen synthetic CIF corruptions and mapped where it was blind. That left the obvious question hanging: synthetic sins are one thing, but how often does the card cry wolf on real, published files? A validator without a measured false-positive rate is just an opinion with a CLI.

So I sampled the wild. 88 inorganic structures from the Crystallography Open Database, eight stratified years from 1991 to 2024, uniform random within each year, organics filtered out. No cherry-picking: whatever the COD's index served, I carded. (One aside from the sampling itself: the inorganic share of the COD collapses from 81% in 1991 to about 5% in recent years. The database of record for crystal structures is now overwhelmingly organic and metal-organic chemistry. Make of that what you will.)

The headline: of 88 files, the card fully cleared one. 63 came back CHECK, 21 FAIL, and 3 could not be parsed at all. The single clean pass was SrTiO3, five atoms in Pm-3m, which is to say the one entry that looks exactly like the card's tuning set. A validator that only trusts what resembles its training examples is not validating. It is pattern-matching familiarity.

My standing rule is that an anomalous result is a bug until proven otherwise, and a 99% accusation rate against the published literature is as anomalous as it gets. So before believing any of it, I checked every accusation by hand.

The min-pair gate: 10 FAILs, nine and a half false. Every "physically impossible" overlap turned out to be a disorder representation, not corruption. Split oxygen sites at 0.15 Å with 50% occupancy each. A barium/lead mixed cation site summing to exactly 1.0. These atoms are alternatives in a probability distribution, not simultaneous residents, and the gate had no concept of that. The one survivor: a 2011 polyoxometalate (COD 4500668) with a partially occupied tungsten sitting 0.33 Å from a fully occupied oxygen. That one is genuinely questionable modeling, and the gate was right to scream.

The reference matcher: 15 FAILs, all mine. Every displacement storm traced to the matcher's own limitations: non-standard space group settings (Pcab instead of Pbca, P 21/n instead of P 21/c), triclinic P-1 cells where there is no symmetry to refine against, and large disordered cells. Version 4.2 taught the matcher to survive origin shifts. The wild speaks in whole setting changes, and against those it still compares apples to a rotated orchard. Zero actual coordinate corruption found.

The metadata gate: an 82-entry CHECK storm of convention noise. Declared formula versus parsed composition sounds simple until you learn that CIF authors variously report the formula unit, the reduced formula, or the full cell contents, and that hydrogens are routinely declared in the formula but never given coordinates (an honest, standard practice: the H was not located in the refinement). Worse, two of the failures were my own bugs wearing the file's clothes. The v4.2 disordered-surrogate swap deletes minority species before the formula comparison, so mixed sites guarantee a mismatch. And the space group label check compares strings: it flagged "Pbnm" as inconsistent with detected "Pnma", which are the same group, number 62, in different axis settings. That is not a file sin. That is the card failing crystallography.

One gate read the wild exactly right. The symmetry fragility sweep flagged nine structures, and they are a pleasure to look at: BiFeO3 climbing R3c to R3m to Pm-3m as tolerance loosens, which is precisely the distorted-perovskite-to-cubic-parent story; Ag3SI sitting fragile between R3 and R3m. Real physics, correctly described.

The wild did serve up some genuine mess. Three files were unparseable by pymatgen at all: malformed or occupancy-inconsistent atom_site loops. One of them is a 2024 NMC-111 battery cathode CIF (COD 3000558) whose oxygen rows are missing a field, so the standard tool of our field refuses to build the structure. Published this year, in a real journal, describing one of the most studied cathode materials on Earth. Real sloppiness exists. It is just rare, specific, and looks nothing like what my gates were tuned to catch.

The full per-entry ledger, with gate levels and my manual classification of every FAIL, is in the companion dataset. The card's v4.4 fix list writes itself: occupancy-aware pair distances, a matcher that normalizes settings before comparing (or bows out on triclinic), formula checks against the raw parse rather than the surrogate, and space group labels compared by IT number instead of by string. Until then, the honest summary of this exercise is the one I should have expected: the battery proved the card catches authored corruption, and the wild proved it cannot yet tell corruption from honest weirdness. Both facts were worth measuring. Only one of them was comfortable.