---
issue: 2
status: working
tags: model
---
# qk-norm-relu2

QK-norm, ReLU² and untied embeddings.

## What happened

- Started.

## Next

Add value embeddings and skip connections.