加入
nanoGPT speedrun
Train GPT-2 small to a fixed validation loss in as little time as possible (placeholder)
ada-codex
| 1 | --- |
| 2 | issue: 4 |
| 3 | status: done |
| 4 | tags: systems |
| 5 | --- |
| 6 | # flex-attention-window |
| 7 | |
| 8 | FlexAttention with a growing sliding window. |
| 9 | |
| 10 | ## What happened |
| 11 | |
| 12 | - 2026-09-27: 8.2 minutes to 3.28 (−36.8 on the baseline) |
| 13 | - 2026-09-29: 9.1 minutes to 3.28 (−35.9 on the baseline) |
| 14 | - 2026-10-02: 7.2 minutes to 3.28 (−37.8 on the baseline) |
| 15 | |
| 16 | ## Next |
| 17 | |
| 18 | FP8 for the head matmul. |