Triaging hypothetical crystals for stability

High-throughput materials search invents far more candidate crystals than anyone can afford to validate with density functional theory. This benchmark asks a model to look at more than 200,000 unrelaxed hypothetical structures and say which ones are thermodynamically stable, so that the expensive calculations can be spent on the right ones.

Task

For each structure in the WBM test set, predict its energy and classify it as stable, meaning it lies on or below the Materials Project convex hull, or unstable. The hull is fixed: it is built from the Materials Project's own DFT entries as queried on 2023-02-07 (release v2022.10.28). Models may use any training data; the site's "Compliant models" filter shows only open models trained exclusively on Materials Project-anchored data. The headline score is the $F_1$ of the stable class, the harmonic mean of precision and recall. Most entries are machine-learning force fields that relax each structure first and then read off its energy, the protocol the site labels IS2RE-SR.

Background

The test structures come from Wang, Botti and Marques, who generated 257,487 candidates by repeated chemically-similar element substitution on Materials Project crystals, in five rounds, and relaxed them all with Materials Project-compatible DFT settings. Riebesell and co-workers turned that into a benchmark in 2023: because the substitutions move progressively away from the training distribution, the set measures extrapolation rather than interpolation, and because the metric is a classification score it reflects what a screening campaign actually cares about, which is the hit rate among the candidates it chooses to compute.

Data and evaluation

Data: the WBM test set and the Materials Project reference energies, as registered files on figshare (https://figshare.com/articles/dataset/22715158) and fetched by the matbench_discovery package. After cleaning, 256,963 structures remain of the 257,487 generated.
Split: there is no cross-validation; the whole WBM set is the test set. The default leaderboard view is the unique-prototype subset, 215,488 structures left after removing prototypes that already occur in the Materials Project and keeping only the lowest-energy structure per prototype; the full set is the other view.
Metric: $F_1$ for the stable class, higher is better. The site also reports discovery acceleration factor, precision, recall, energy MAE and $R^2$ , and a combined score CPS that weights $F_1$ at 50%, phonon $\kappa_{\mathrm{SRME}}$ at 40% and geometry-optimisation RMSD at 10%.
Official evaluation code: https://github.com/janosh/matbench-discovery (MIT); a model is submitted as a pull request with its predictions and a metadata file. The data files on figshare are CC BY 4.0.

What is known

Top of the leaderboard on unique prototypes: EquiformerV3+DeNS-OAM at $F_1 = 0.931$ , with TECE-OAM-RRA-1.0 and EquFlashV2 at 0.929 and Prophet-OAME-MBD at 0.928. The four are within 0.003 of each other.
Prophet-OAME-MBD leads the combined score at CPS 0.912, mostly on the strength of its phonon term.
The first generation of universal potentials is far behind: MACE-MP-0 at 0.669, CHGNet at 0.613 and M3GNet at 0.569. Much of the gain since then came from training data: the same architecture trained on OMat24 and Alexandria gains sharply over its MPtrj-only variant, for example eSEN-30M at 0.925 against 0.831.
The best compliant model, EquiformerV3+DeNS-MP, is trained only on MPtrj and reaches $F_1 = 0.863$ , so part of the gap at the top reflects training data rather than method.

What would count as progress

An $F_1$ above 0.931 on the unique-prototype set under the official protocol, submitted with the prediction files so precision, recall and the energy metrics can be recomputed.
A compliant model above 0.863, trained only on Materials Project-anchored data, which isolates the method from the data.
Code and a checkpoint that reproduce the run, plus the geometry and phonon metrics, since a model that wins on $F_1$ alone can still fail the combined score.

References

J. Riebesell, R. E. A. Goodall, P. Benner et al., "A framework to evaluate machine learning crystal stability predictions", Nature Machine Intelligence 7, 836–847 (2025). https://doi.org/10.1038/s42256-025-01055-1
H.-C. Wang, S. Botti and M. A. L. Marques, "Predicting stable crystalline compounds using chemical similarity", npj Computational Materials 7, 12 (2021). https://doi.org/10.1038/s41524-020-00481-6
A. Jain et al., "The Materials Project: A materials genome approach to accelerating materials innovation", APL Materials 1, 011002 (2013). https://doi.org/10.1063/1.4812323
B. Póta, P. Ahlawat, G. Csányi and M. Simoncelli, "Thermal Conductivity Predictions with Foundation Atomistic Models" (2024), the source of the $\kappa_{\mathrm{SRME}}$ term in CPS. https://arxiv.org/abs/2408.00755

Source

Imported from Matbench Discovery, crystal stability, curated by Janosh Riebesell et al. Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.