加入
nanoGPT speedrun
Train GPT-2 small to a fixed validation loss in as little time as possible (placeholder)
| # | minutes to 3.28 ↓ | 进步 | 改了什么 |
|---|---|---|---|
| 11 | 3.95 | ↓ 0.71 | 这条路线上后来的提交 |
| 10 | 4.66 | ↓ 0.37 | Block-sparse attention with a batch-size schedule |
| 9 | 5.03 | ↓ 2.17 | FP8 matmul for the language-model head |
| 8 | 7.2 | ↓ 1 | 这条路线上后来的提交 |
| 7 | 8.2 | ↓ 2.6 | FlexAttention with a growing sliding window |
| 6 | 10.8 | ↓ 1.2 | 这条路线上后来的提交 |
| 5 | 12 | ↓ 3.2 | U-net skip connections and value embeddings |
| 4 | 15.2 | ↓ 7.1 | QK-norm, ReLU² and untied embeddings |
| 3 | 22.3 | ↓ 2.6 | 这条路线上后来的提交 |
| 2 | 24.9 | ↓ 6.5 | Muon, orthogonalized momentum, for the hidden matrices and AdamW for the rest |
| 1 | 31.4 | ↓ 13.6 | A tuned learning rate, rotary embeddings and a trapezoidal schedule |
| 0 | 45 | 基线：llm.c GPT-2 (124M) |
