Predicting DFT band gaps from crystal structure

Band gap decides whether a crystal is useful as a semiconductor, a transparent conductor or a photocatalyst, and computing it costs a DFT run per material. This task asks for the computed gap from the structure, on 106,113 Materials Project entries.

Task

Given a crystal structure, predict the PBE band gap in electronvolts as reported by the Materials Project. Scoring is the mean absolute error over the five fixed folds. More than 40% of the entries are metals with zero gap, so the target is a spike at zero plus a broad tail.

Background

The labels are PBE gaps from the Materials Project, filtered by Matbench to entries within 150 meV of the hull and without noble gases. It is the second-largest task in the suite, large enough that graph networks clearly beat descriptor-based models. The task is also a reminder of a limit: PBE systematically underestimates measured gaps, so a model that predicts these labels perfectly still would not predict experiment, which is what the suite's matbench_expt_gap task is for.

Data and evaluation

• Data: matbench_mp_gap.json.gz from MPContribs-ML (106,113 records, each a pymatgen Structure and a PBE gap in eV), at https://ml.materialsproject.org/projects/matbench_mp_gap or via task.load() in the matbench package; the gaps are Materials Project calculations.
• Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's KFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
• Metric: mean absolute error in eV over the five test folds, lower is better. The leaderboard also lists the fold-to-fold standard deviation, RMSE and worst single-sample error.
• Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_mp_gap.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

• Top of the task leaderboard: coGN, mean MAE 0.1559 eV (standard deviation 0.0017).
• Runners-up: DeeperGATGNN 0.1694, coNGN 0.1697 and ALIGNN 0.1861 eV.
• A dummy regressor scores 1.3272 eV against a mean absolute deviation of 1.3271.
• The leaders' worst-case errors stay around 7 eV, so some compounds are still missed by several electronvolts.
• The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

• A mean MAE below 0.1559 eV on the five fixed folds, beating coGN, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
• Code and a pinned environment that reproduce the run from the fixed folds.
• Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

1. A. Jain et al., "The Materials Project: A materials genome approach to accelerating materials innovation", APL Materials 1, 011002 (2013). https://doi.org/10.1063/1.4812323
2. A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
3. R. Ruff, P. Reiser, J. Stühmer and P. Friederich, "Connectivity optimized nested line graph networks for crystal structures", Digital Discovery 3, 594–601 (2024). https://doi.org/10.1039/D4DD00018H

Source

Imported from Matbench v0.1 matbench_mp_gap, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.