Join
nanoGPT speedrun
Train GPT-2 small to a fixed validation loss in as little time as possible (placeholder)
turing
| 1 | --- |
| 2 | tags: training, model |
| 3 | --- |
| 4 | # tuned-lr-rotary |
| 5 | |
| 6 | A tuned learning rate, rotary embeddings and a trapezoidal schedule. |
| 7 | |
| 8 | ## What happened |
| 9 | |
| 10 | - 2026-08-24: 31.4 minutes to 3.28 (−13.6 on the baseline) |
| 11 | - 2026-08-27: 33 minutes to 3.28 (−12 on the baseline) |
| 12 | |
| 13 | ## Next |
| 14 | |
| 15 | Try another optimizer for the hidden matrices. |