Predicting refractive index from crystal structure

The refractive index controls how light slows and bends inside a material, and it is the first number an optical designer asks for. This task asks for the computed index from the crystal structure, on 4,764 inorganic compounds.

Task

Given a crystal structure, predict the refractive index $n$ (dimensionless). Scoring is the mean absolute error over the five fixed folds. The target has a heavy tail, and a handful of high-index outliers dominate the squared-error metrics the leaderboard also reports.

Background

The values were computed by Petousis and co-workers in a high-throughput density functional perturbation theory screen of dielectric and optical materials, then adapted for Matbench: entries more than 150 meV from the hull, indices below 1 and noble-gas compounds were removed. Dielectric response is harder to learn than formation energy because it depends on the electronic structure rather than on bonding geometry alone, and it is one of the tasks where a feature-based model still beats the graph networks.

Data and evaluation

• Data: matbench_dielectric.json.gz from MPContribs-ML (4,764 records, each a pymatgen Structure and a refractive index), at https://ml.materialsproject.org/projects/matbench_dielectric or via task.load() in the matbench package; the dielectric calculations are from Petousis and co-workers.
• Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's KFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
• Metric: mean absolute error in unitless over the five test folds, lower is better. The leaderboard also lists the fold-to-fold standard deviation, RMSE and worst single-sample error.
• Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_dielectric.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

• Top of the task leaderboard: MODNet v0.1.12, mean MAE 0.2711 (standard deviation 0.0714).
• Runners-up: MODNet v0.1.10 at 0.2970, then the graph networks coGN at 0.3088 and coNGN at 0.3142.
• A dummy regressor scores 0.8088 (mean absolute deviation 0.8085).
• The leading submissions all report a worst-case error near 59, caused by the same few very-high-index structures; the tail, not the bulk, is where this task is unsolved.
• Unmerged claim: pull request #366 (September 2026) reports 0.2473 and 0.2512 for two SciKG variants, which correct MODNet's predictions with evidence from the literature and databases published before 2020. It has not been reviewed or added to the leaderboard.
• The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

• A mean MAE below 0.2711 unitless on the five fixed folds, beating MODNet v0.1.12, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
• Code and a pinned environment that reproduce the run from the fixed folds.
• Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

1. I. Petousis et al., "High-throughput screening of inorganic compounds for the discovery of novel dielectric and optical materials", Scientific Data 4, 160134 (2017). https://doi.org/10.1038/sdata.2016.134
2. A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
3. P.-P. De Breuck, M. L. Evans and G.-M. Rignanese, "Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet", Journal of Physics: Condensed Matter 33, 404002 (2021). https://doi.org/10.1088/1361-648X/ac1280
4. R. Ruff, P. Reiser, J. Stühmer and P. Friederich, "Connectivity optimized nested line graph networks for crystal structures", Digital Discovery 3, 594–601 (2024). https://doi.org/10.1039/D4DD00018H

Source

Imported from Matbench v0.1 matbench_dielectric, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.