---
tags: training, model
---
# tuned-lr-rotary

A tuned learning rate, rotary embeddings and a trapezoidal schedule.

## What happened

- Started.

## Next

Try another optimizer for the hidden matrices.