Predicting steel yield strength from composition

Alloy designers would like to know the yield strength of a steel before making it. This task asks for that number from the chemical formula alone, with no processing history and no microstructure, on a small set of 312 measured alloys.

Task

Given a chemical composition as a formula string, predict the experimentally measured yield strength in megapascals. Scoring is the mean absolute error over the five fixed folds, so a result is one number per test alloy and nothing else. With 312 alloys in total, each test fold holds only 62 or 63 of them.

Background

The data was retrieved from Citrine Informatics and deduplicated for Matbench v0.1, the 13-task suite Dunn and co-workers assembled in 2020 so that materials-ML methods are compared on the same data and splits. It is the smallest task in the suite and the only one about a mechanical property of an engineering alloy, which makes it the suite's test of whether a method can learn from a few hundred samples without overfitting. Yield strength depends strongly on processing, so composition cannot explain all of it; the interesting question is how much it can explain.

Data and evaluation

Data: matbench_steels.json.gz from MPContribs-ML (312 records, each a formula string and a yield strength in MPa), at https://ml.materialsproject.org/projects/matbench_steels or via task.load() in the matbench package; the set is derived from a Citrine Informatics dataset.
Split: five folds fixed in matbench_v0.1_validation.json, built with scikit-learn's KFold(n_splits=5, shuffle=True, random_state=18012019). Train on each fold's training set, tune inside it only, predict its test set, report the mean. Changing the folds invalidates the comparison.
Metric: mean absolute error in MPa over the five test folds, lower is better. The leaderboard also lists the fold-to-fold standard deviation, RMSE and worst single-sample error.
Official evaluation code: the matbench package (MIT), https://github.com/materialsproject/matbench; the task's record(fold, predictions), as in mb.matbench_steels.record(...) on a MatbenchBenchmark mb, scores a fold and mb.to_file(...) writes the submission file.

What is known

Top of the task leaderboard: TPOT-Mat, mean MAE 79.9468 MPa over the five folds (fold-to-fold standard deviation 13.5883).
Runners-up: AutoML-Mat 82.3043 MPa and MODNet v0.1.12 87.7627 MPa. The first two were submitted for this task only; MODNet v0.1.12, which was run on all 13 tasks, is the best of the site's general-purpose entries.
A mean-predicting dummy scores 229.7445 MPa, close to the target's mean absolute deviation of 229.3743 MPa, so the best model cuts the error by about two thirds.
AutoML pipelines lead, ahead of the neural composition model CrabNet (107.3160 MPa) and the fine-tuned language models Darwin (123.2932 MPa) and gptchem (143.0028 MPa). Graph networks cannot enter, since the task gives no structures.
Unmerged claim: pull request #371 (October 2026) reports 75.5426 MPa for Atomic-AffQRF, a composition-only ensemble of extra-trees models; it has not been reviewed or added to the leaderboard.
The leaderboard's repository has merged no submission since 20 January 2024; newer ones wait as open pull requests, so the official numbers may lag what has been achieved.

What would count as progress

A mean MAE below 79.9468 MPa on the five fixed folds, beating TPOT-Mat, submitted as the matbench benchmark file so every fold's predictions can be re-scored.
Code and a pinned environment that reproduce the run from the fixed folds.
Per-fold numbers, not just the mean, so that a gain can be compared with the leader fold by fold on the same folds; a small gain should hold on each fold or over repeated training runs. Reproducing the current leader counts too, since the board has not moved since 2024.

References

R. S. Olson, N. Bartley, R. J. Urbanowicz and J. H. Moore, "Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science", GECCO '16, 485–492 (2016). https://doi.org/10.1145/2908812.2908918
A. Dunn, Q. Wang, A. Ganose, D. Dopp and A. Jain, "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm", npj Computational Materials 6, 138 (2020). https://doi.org/10.1038/s41524-020-00406-3
P.-P. De Breuck, M. L. Evans and G.-M. Rignanese, "Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet", Journal of Physics: Condensed Matter 33, 404002 (2021). https://doi.org/10.1088/1361-648X/ac1280

Source

Imported from Matbench v0.1 matbench_steels, curated by Materials Project (Dunn et al.). Open according to the source as of 2026-10-20. The statement above is written in our own words; see the source for the original wording, history and discussion.