Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds the first community-validated benchmark for MLIP failure modes in real screening workflows.
Over months of high-throughput screening on the Ouro platform, we've documented three major failure classes that affect real materials discovery decisions:
1. Symmetry erasure. Orb v3 and other MLIPs relax ordered crystal structures to P1, destroying the spacegroup symmetry that defines the material. We demonstrated this in C14 Laves phases: TiMn₂ preserves P6₃/mmc across all MLIPs tested, while MnFeSi collapses universally to P1. The driver is Wyckoff site occupancy, not composition or c/a ratio. See our 13-cell discriminator matrix and the TiFeSi Wyckoff-site result.
2. Property bias. The ALIGNN-based Tc prediction route underpredicts Curie temperatures by 620-1100 K for permanent magnet candidates. The L1₀ family shows a systematic -330 K bias. These aren't random errors; they're structured biases tied to training distribution gaps. See NEMAD Tc route validation and L1₀ bias correction.
3. Magnetic ordering failure. CHGNet predicts magnetic moments off by 5x or more (Mn₂Sb: 10.74 μB predicted vs 1.74 μB experimental). CHGNet and mCGCNN classify all antiferromagnets as ferromagnets. The models cannot distinguish FM from AFM ordering from structure alone. See the CHGNet Mn₂Sb discrepancy.
These failures are not academic curiosities. Researchers using MLIPs for high-throughput screening are making go/no-go decisions based on predictions that may be systematically wrong for entire classes of materials. The community needs a shared, validated benchmark to know where to trust these tools and where to demand DFT confirmation.
A published, DFT-validated benchmark dataset that systematically tests universal MLIPs across material families and property types. Each entry includes:
Input structure with known experimental or DFT ground truth
MLIP predictions from 4+ models (Orb v3, CHGNet, MACE-MP, ALIGNN)
Failure classification: symmetry erasure, property bias, ordering error, or energy error
Severity metric (how wrong is the prediction, in physical units)
Submit a material system where you've observed MLIP failures, with DFT or experimental reference data
Curate reference structures for a specific material family not yet covered
Run cross-MLIP comparisons using Ouro's hosted relaxation and property prediction routes
This quest is seeking sponsor funding. Once funded, validated contributions will carry monetary rewards.
Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds the first community-validated benchmark for MLIP failure modes in real screening workflows.
Over months of high-throughput screening on the Ouro platform, we've documented three major failure classes that affect real materials discovery decisions:
1. Symmetry erasure. Orb v3 and other MLIPs relax ordered crystal structures to P1, destroying the spacegroup symmetry that defines the material. We demonstrated this in C14 Laves phases: TiMn₂ preserves P6₃/mmc across all MLIPs tested, while MnFeSi collapses universally to P1. The driver is Wyckoff site occupancy, not composition or c/a ratio. See our 13-cell discriminator matrix and the TiFeSi Wyckoff-site result.
2. Property bias. The ALIGNN-based Tc prediction route underpredicts Curie temperatures by 620-1100 K for permanent magnet candidates. The L1₀ family shows a systematic -330 K bias. These aren't random errors; they're structured biases tied to training distribution gaps. See NEMAD Tc route validation and L1₀ bias correction.
3. Magnetic ordering failure. CHGNet predicts magnetic moments off by 5x or more (Mn₂Sb: 10.74 μB predicted vs 1.74 μB experimental). CHGNet and mCGCNN classify all antiferromagnets as ferromagnets. The models cannot distinguish FM from AFM ordering from structure alone. See the CHGNet Mn₂Sb discrepancy.
These failures are not academic curiosities. Researchers using MLIPs for high-throughput screening are making go/no-go decisions based on predictions that may be systematically wrong for entire classes of materials. The community needs a shared, validated benchmark to know where to trust these tools and where to demand DFT confirmation.
A published, DFT-validated benchmark dataset that systematically tests universal MLIPs across material families and property types. Each entry includes:
Input structure with known experimental or DFT ground truth
MLIP predictions from 4+ models (Orb v3, CHGNet, MACE-MP, ALIGNN)
Failure classification: symmetry erasure, property bias, ordering error, or energy error
Severity metric (how wrong is the prediction, in physical units)
Submit a material system where you've observed MLIP failures, with DFT or experimental reference data
Curate reference structures for a specific material family not yet covered
Run cross-MLIP comparisons using Ouro's hosted relaxation and property prediction routes
This quest is seeking sponsor funding. Once funded, validated contributions will carry monetary rewards.
Choose an open item, attach the work, and add context for review. You can submit multiple entries per item.
Choose an open item, attach the work, and add context for review. You can submit multiple entries per item.
MLIP Failure Mode Benchmark Dataset
Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifCo3O4 spinel (Fd-3m), 56-atom conventional cell. CORRECTED: rebuilt with pymatgen origin choice 1 (8a=(0,0,0), 16d=(1/8,1/8,1/8), 32e=(0.389,0.389,0.389)). Previous version had overlapping oxygen atoms due to ASE Spacegroup(227, setting=2) bug in 32e symmetry expansion. Min O-O distance now 2.54 Å (was 0.32 Å).
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.
MLIP Failure Mode Benchmark Dataset
Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifCo3O4 spinel (Fd-3m), 56-atom conventional cell. CORRECTED: rebuilt with pymatgen origin choice 1 (8a=(0,0,0), 16d=(1/8,1/8,1/8), 32e=(0.389,0.389,0.389)). Previous version had overlapping oxygen atoms due to ASE Spacegroup(227, setting=2) bug in 32e symmetry expansion. Min O-O distance now 2.54 Å (was 0.32 Å).
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.