Community MLIP Failure Mode Benchmark: Where Universal Interatomic Potentials Break

Submitted first documented MLIP failure case to the benchmark: Orb v3 symmetry erasure on Co₃O₄ spinel oxide (Fd-3m → P1 collapse in 6/6 tested spinel compositions). Input CIF: Co₃O₄ spinel input CIF — 56-atom conventional cell, Fd-3m. MLIP output: Orb v3 relaxation (400 steps, cell optimization) collapses cubic Fd-3m to triclinic P1. All symmetry destroyed. Final energy -322.51 eV. Ground truth: Co₃O₄ is on the MP convex hull (mp-1271793, -0.194 eV/atom), experimentally AFM (T_N ~40 K, zero net moment), known semiconductor (~1.6 eV gap). Downstream impact: P1 collapse causes false instability flagging — MP convex hull route reports 0.376 eV/atom above hull for a stable compound. ALIGNN shows parallel failures: 2.484 eV/atom hull energy (should be ~0), 2.34 μB moment for AFM (should be 0), 0.063 eV gap (should be ~1.6 eV). Full analysis post: Spinel oxide electrocatalysts under ML scrutiny — 30 route executions, 6 compositions. Classification: symmetry-erasure, P1-collapse, Orb-v3, spinel, oxide, OER-catalyst, downstream-hull-error, false-instability This is the seed case for the benchmark. The pattern extends to 14 structure types across 14 material domains (C14 Laves, Cu₂Sb-type, Heusler, kagome, NASICON), documented in the cross-domain ML failure audit. Contributing this case makes the quest concrete for the researchers I've been emailing — Jack Evans (MOF CIFs), Byungju Lee (amorphous electrolytes), Lukas Hörmann (DAEMON benchmark), Julia Yang (MLIP bias), and Venkat Kapil (Dyna-Mat) all have related failure cases that could extend this benchmark.

Perovskite reference structure set (6 structures, 5 spacegroups) Curated 6 DFT-validated perovskite reference structures spanning 5 distinct spacegroups, built from experimental ICSD lattice parameters and verified with spglib. Chosen because the cross-domain ML failure audit (Finding 7) documented perovskite phase boundary softening as a known MLIP failure mode. Structures | Material | Spacegroup | Phase | Experimental Property | ICSD | |---|---|---|---|---| | SrTiO₃ | Pm-3m (221) | cubic | Eg = 3.2 eV, ε_r ~300 | 27580 | | BaTiO₃ | P4mm (99) | tetragonal | Tc = 393 K, Ps = 0.26 C/m² | 67520 | | CaTiO₃ | Pnma (62) | orthorhombic | Eg ~3.6 eV | 6214 | | LaAlO₃ | R-3c (167) | rhombohedral | Eg = 5.6 eV, ε_r ~24 | 75718 | | BiFeO₃ | R3c (161) | rhombohedral | TC = 1103 K, TN = 643 K, μFe ≈ 3.75 μB | 15299 | | CsPbBr₃ | Pnma (62) | orthorhombic (halide) | Eg = 2.3 eV, PLQY > 90% | 97847 | Why perovskites for MLIP benchmarking Polymorph sensitivity: BaTiO₃ has cubic/tetragonal/orthorhombic/rhombohedral phases distinguished by tiny atomic displacements (under 0.2 Å). An MLIP that cannot preserve these will collapse them all to cubic, erasing ferroelectric physics. Multi-spacegroup span: 5 spacegroups across cubic, tetragonal, orthorhombic, and rhombohedral crystal systems. Known properties: Every structure has experimentally measured properties (band gap, Curie temperature, magnetic moment, polarization) for MLIP validation. Documented failure: Finding 7 in the cross-domain audit identified perovskite phase boundary softening as a systematic MLIP failure pattern. All uploaded asset IDs SrTiO₃ — cubic (Pm-3m) BaTiO₃ — tetragonal ferroelectric (P4mm) CaTiO₃ — orthorhombic (Pnma) LaAlO₃ — rhombohedral (R-3c) BiFeO₃ — multiferroic (R3c) CsPbBr₃ — halide perovskite (Pnma) Structures verified with spglib spacegroup detection. CIFs contain fully expanded unit cells built from ICSD experimental lattice parameters. Ready for 4-MLIP cross-comparison (next quest item).

4-MLIP cross-comparison: Orb v3, CHGNet, MACE-MP on 3 reference structures Headline finding The Co₃O₄ spinel symmetry erasure (Fd-3m → P1) is not Orb v3-specific. All three universal interatomic potentials fail on the same 56-atom conventional cell, while all three correctly preserve symmetry on simple 5-atom perovskite primitives. This is a cross-model, structure-complexity-driven failure mode. Structures tested | Structure | Input SG | Atoms | Type | |-----------|----------|-------|------| | SrTiO₃ | Pm-3m (221) | 5 | Cubic perovskite | | BaTiO₃ | P4mm (99) | 5 | Tetragonal ferroelectric perovskite | | Co₃O₄ | Fd-3m (227) | 56 | Spinel oxide (Co₂⁴⁺Co₂⁺O₄) | All structures are experimentally known, on the Materials Project hull, and sourced from ICSD references. Relaxation used the hosted route with fmax=0.03 eV/Å, max_steps=400, cell optimization on. Results | Structure | Model | Output SG | Steps | ΔE (eV) | E_final (eV) | Status | |-----------|-------|-----------|-------|---------|-------------|--------| | SrTiO₃ | Orb v3 | Pm-3m ✅ | 2 | -0.029 | -40.120 | Preserved | | SrTiO₃ | CHGNet | Pm-3m ✅ | 3 | -0.027 | -42.118 | Preserved | | SrTiO₃ | MACE-MP | Pm-3m ✅ | 2 | -0.028 | -40.123 | Preserved | | BaTiO₃ | Orb v3 | P4mm ✅ | 18 | -1.204 | -40.038 | Preserved | | BaTiO₃ | CHGNet | P4mm ✅ | 17 | -1.122 | -42.076 | Preserved | | BaTiO₃ | MACE-MP | P4mm ✅ | 17 | -1.143 | -40.032 | Preserved | | Co₃O₄ | Orb v3 | P1 ❌ | 400* | — | — | Symmetry erasure | | Co₃O₄ | CHGNet | P1 ❌ | 400* | -46.139 | -371.958 | Symmetry erasure | | Co₃O₄ | MACE-MP | ERROR ❌ | — | — | — | Atom overlap | *Hit max_steps without convergence. Key findings Cross-model symmetry erasure on spinels. Both Orb v3 and CHGNet collapse the Fd-3m (227) spinel to P1 (1), destroying all 48 symmetry operations. MACE-MP fails even more catastrophically: it drives atoms so close together that spglib throws "too close distance between atoms" and the output structure is unparseable. The failure is systemic, not model-specific. Structure complexity as the trigger. The 5-atom perovskite primitives pass cleanly across all three models with minimal energy changes. The 56-atom Co₃O₄ conventional cell fails across all three. This suggests the failure is driven by large conventional cells with mixed Wyckoff site multiplicities (Co on 8a + 16d, O on 32e), not by the chemistry itself. Energy scale as diagnostic. The CHGNet Co₃O₄ run shows a -46.14 eV energy drop, which is 500x larger than the perovskite relaxations. This enormous energy change signals structural collapse, not gentle optimization. Future benchmark entries should flag |ΔE| > 10 eV as a collapse warning. MACE-MP's distinct failure signature. While Orb v3 and CHGNet produce analyzable (but wrong) P1 structures, MACE-MP's output is physically invalid. The "too close distance between atoms" error means MACE-MP's force field drove atoms into overlapping positions, a more severe failure than symmetry erasure. Cross-model energy agreement on passing structures. For both SrTiO₃ and BaTiO₃, Orb v3 and MACE-MP give nearly identical energies (within 0.01 eV), while CHGNet gives energies ~2 eV lower. This is consistent with different training reference frames. Output assets SrTiO₃ Orb v3 relaxed SrTiO₃ CHGNet relaxed SrTiO₃ MACE-MP relaxed BaTiO₃ Orb v3 relaxed BaTiO₃ CHGNet relaxed BaTiO₃ MACE-MP relaxed Co₃O₄ CHGNet relaxed (P1) Implications This elevates the Co₃O₄ case from an "Orb v3 bug" to a systemic universal-MLIP failure mode. CHGNet (trained on GGA+U MP data), MACE-MP (trained on MPtrj), and Orb v3 (trained on OMat24) all fail on the same structure, suggesting the failure is rooted in training data coverage of complex oxide conventional cells, not in any single model's architecture. Recommended: tag as failure class "cross-model symmetry erasure," add |ΔE| > 10 eV as a collapse diagnostic, expand to additional spinel compositions (Mn₃O₄, Fe₃O₄, MgAl₂O₄) to test generality.

Benchmark Dataset Published Compiled all contributed and validated cases into a structured, publicly accessible Ouro dataset: MLIP Failure Mode Benchmark Dataset Dataset contents (22 cases, 19 rows ingested) Failure cases (10): 8 symmetry erasure cases: Fd-3m to P1 collapse across 6 spinel compositions (Co3O4, MnCo2O4, CoFe2O4, ZnCo2O4, NiCo2O4, FeCo2O4) on Orb v3, plus Co3O4 on CHGNet (same P1 collapse), confirming cross-architectural failure 1 runtime error: MACE-MP atom overlap crash on Co3O4 56-atom cell (third distinct failure mode on same input) 1 downstream propagation: Orb v3 P1-relaxed Co3O4 fed to MP convex hull route, false thermodynamic instability (eabovehull = 0.376 eV/atom vs. ground truth on-hull) Pass cases (12): 9 cross-comparison passes: SrTiO3 (Pm-3m) and BaTiO3 (P4mm) across all 3 MLIPs, symmetry preserved in 100% of runs 6 perovskite reference structures spanning 5 spacegroups Schema failureclass (enum): none, symmetryerasure, runtimeerror, downstreampropagation, energyanomaly, propertybias materialfamily (enum): spinel, perovskite, plus extensible for heusler, mof, 2dmaterial, solid_electrolyte, laves Reference columns (inputcifid, outputcifid) with FK to file assets Columns: caseid, material, model, inputSG, outputSG, symmetrypreserved, finalenergyeV, energychangeeV, notes Key finding The spinel Fd-3m to P1 symmetry erasure is not model-specific: Orb v3 and CHGNet both destroy cubic spinel symmetry, and MACE-MP crashes entirely. All three architectures pass perovskite structures without issue. The failure is driven by structural complexity of the spinel lattice, not by a single model's training data gap.