Triaging hypothetical crystals for stability

Classify 215,488 hypothetical crystals as stable or not against a fixed DFT convex hull, scored by F1.

shoal State the problem, from Matbench Discovery a40cb4e · 2周前历史

76 行 · 5.3 KB 原始文件永久链接历史

1	# Triaging hypothetical crystals for stability
2
3	High-throughput materials search invents far more candidate crystals than anyone can afford to validate with
4	density functional theory. This benchmark asks a model to look at more than 200,000 unrelaxed hypothetical structures
5	and say which ones are thermodynamically stable, so that the expensive calculations can be spent on the right ones.
6
7	## Task
8
9	For each structure in the WBM test set, predict its energy and classify it as stable, meaning it lies on or below
10	the Materials Project convex hull, or unstable. The hull is fixed: it is built from the Materials Project's own
11	DFT entries as queried on 2023-02-07 (release v2022.10.28). Models may use any training data; the site's
12	"Compliant models" filter shows only open models trained exclusively on Materials Project-anchored data.
13	The headline score is the $F_1$ of the stable class, the harmonic mean of precision and recall. Most entries are
14	machine-learning force fields that relax each structure first and then read off its energy, the protocol the site
15	labels IS2RE-SR.
16
17	## Background
18
19	The test structures come from Wang, Botti and Marques, who generated 257,487 candidates by repeated
20	chemically-similar element substitution on Materials Project crystals, in five rounds, and relaxed them all with
21	Materials Project-compatible DFT settings. Riebesell and co-workers turned that into a benchmark in 2023: because
22	the substitutions move progressively away from the training distribution, the set measures extrapolation rather
23	than interpolation, and because the metric is a classification score it reflects what a screening campaign
24	actually cares about, which is the hit rate among the candidates it chooses to compute.
25
26	## Data and evaluation
27
28	- Data: the WBM test set and the Materials Project reference energies, as registered files on figshare
29	(<https://figshare.com/articles/dataset/22715158>) and fetched by the `matbench_discovery` package. After
30	cleaning, 256,963 structures remain of the 257,487 generated.
31	- Split: there is no cross-validation; the whole WBM set is the test set. The default leaderboard view is the unique-prototype subset, 215,488 structures
32	left after removing prototypes that already occur in the Materials Project and keeping only the lowest-energy
33	structure per prototype; the full set is the other view.
34	- Metric: $F_1$ for the stable class, higher is better. The site also reports discovery acceleration factor,
35	precision, recall, energy MAE and $R^2$, and a combined score CPS that weights $F_1$ at 50%, phonon
36	$\kappa_{\mathrm{SRME}}$ at 40% and geometry-optimisation RMSD at 10%.
37	- Official evaluation code: <https://github.com/janosh/matbench-discovery> (MIT); a model is submitted as a pull
38	request with its predictions and a metadata file. The data files on figshare are CC BY 4.0.
39
40	## What is known
41
42	- Top of the leaderboard on unique prototypes: EquiformerV3+DeNS-OAM at $F_1 = 0.931$, with TECE-OAM-RRA-1.0 and
43	EquFlashV2 at 0.929 and Prophet-OAME-MBD at 0.928. The four are within 0.003 of each other.
44	- Prophet-OAME-MBD leads the combined score at CPS 0.912, mostly on the strength of its phonon term.
45	- The first generation of universal potentials is far behind: MACE-MP-0 at 0.669, CHGNet at 0.613 and M3GNet
46	at 0.569.
47	Much of the gain since then came from training data: the same architecture trained on OMat24 and Alexandria
48	gains sharply over its MPtrj-only variant, for example eSEN-30M at 0.925 against 0.831.
49	- The best compliant model, EquiformerV3+DeNS-MP, is trained only on MPtrj and reaches $F_1 = 0.863$, so part of
50	the gap at the top reflects training data rather than method.
51
52	## What would count as progress
53
54	- An $F_1$ above 0.931 on the unique-prototype set under the official protocol, submitted with the prediction
55	files so precision, recall and the energy metrics can be recomputed.
56	- A compliant model above 0.863, trained only on Materials Project-anchored data, which isolates the method from
57	the data.
58	- Code and a checkpoint that reproduce the run, plus the geometry and phonon metrics, since a model that wins on
59	$F_1$ alone can still fail the combined score.
60
61	## References
62
63	1. J. Riebesell, R. E. A. Goodall, P. Benner et al., "A framework to evaluate machine learning crystal stability
64	predictions", Nature Machine Intelligence 7, 836–847 (2025). <https://doi.org/10.1038/s42256-025-01055-1>
65	2. H.-C. Wang, S. Botti and M. A. L. Marques, "Predicting stable crystalline compounds using chemical
66	similarity", npj Computational Materials 7, 12 (2021). <https://doi.org/10.1038/s41524-020-00481-6>
67	3. A. Jain et al., "The Materials Project: A materials genome approach to accelerating materials innovation",
68	APL Materials 1, 011002 (2013). <https://doi.org/10.1063/1.4812323>
69	4. B. Póta, P. Ahlawat, G. Csányi and M. Simoncelli, "Thermal Conductivity Predictions with Foundation Atomistic
70	Models" (2024), the source of the $\kappa_{\mathrm{SRME}}$ term in CPS. <https://arxiv.org/abs/2408.00755>
71
72	## Source
73
74	Imported from [Matbench Discovery, crystal stability](https://matbench-discovery.materialsproject.org/benchmarks/discovery), curated by Janosh Riebesell et al. Open according
75	to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original
76	wording, history and discussion.