# Optimizer groups

`optimizer` is a configured optimizer: a `defoptimizer` called with ordinary
Lisp values. `:groups` gives parts of the model their own hyperparameters,
or their own optimizer, chosen by selectors over parameter paths.

```lisp
(defvar optimizer
  (adamw :lr (warmup-stable-decay lr :warmup-steps warmup :decay-fraction 0.7)
         :betas [0.9 0.95]
         :eps 1e-10
         :weight-decay 0.10
         :groups [(group (select "model.embed.*") :lr 0.3 :weight-decay 0.0)
                  (group (select :rank-below 2) :lr 0.01)]))
```

## Hyperparameters

The standard optimizers accept a number or a schedule for every numeric
hyperparameter. A schedule is a function of `ctx`; see
[writing optimizers](https://sexpgpu.041.io/docs/optimizers.md#schedules) and the ready-made ones in
[sexpgpu/optim](https://sexpgpu.041.io/docs/optim.md). This holds for defaults, explicit arguments and
group overrides alike.

## Selectors

Selectors run over canonical parameter paths, ranks and tags. Paths come
from binding names, `model.blocks.3.attn.q.weight`; see
[models](https://sexpgpu.041.io/docs/models.md#paths).

| selector | matches |
|---|---|
| `(select "model.embed.*")` | a path glob: `*` inside one segment, `**` across segments |
| `(select :rank-below n)` | parameters of rank under `n` |
| `(select :rank-at-least n)` | parameters of rank `n` or more |
| `(select :tag :name)` | parameters carrying that `defparam :tags` entry |
| `(select-not s)` | the complement |
| `(select-and s1 s2 ...)` | the intersection |

## Groups

`(group selector :key value ...)` overrides hyperparameters for the
parameters the selector matches. `:groups` takes `group` values or bare
selectors.

- A group is named after its selector's text. That name is the `param`
  metadata of its [optimizer diagnostics](https://sexpgpu.041.io/docs/writing-diagnostics.md#optimizer-diagnostics).
- Everything no explicit group matched lands in `default`, which is dropped
  when it is empty.
- A selector that matches nothing is `E-OPT-001`; check paths with
  [`sexpgpu explain`](https://sexpgpu.041.io/docs/explain.md).
- A parameter matched by two groups is `E-OPT-002`, which lists the doubly
  covered paths; narrow one selector with `select-not`.
- A malformed `select`, `group`, `:groups` or `:optimizer` is `E-OPT-005`,
  and the fix prints the shape.

## A different optimizer per group

`:optimizer` is the one group override that is not a hyperparameter. It
runs the group on another configured optimizer, which is how one run gives
its matrices to Muon and everything else to AdamW:

```lisp
(defvar optimizer
  (adamw :lr 0.003
         :groups [(group (select-and (select "model.blocks.**")
                                     (select :rank-at-least 2))
                         :optimizer (muon :lr 0.02))]))
```

The inner optimizer may not carry `:groups` of its own (`E-OPT-005`).

Related: [writing optimizers](https://sexpgpu.041.io/docs/optimizers.md), [sexpgpu/optim](https://sexpgpu.041.io/docs/optim.md),
[explain](https://sexpgpu.041.io/docs/explain.md).

---

SexpGPU documentation. Every page: https://sexpgpu.041.io/llms.txt