Optimizer groups

optimizer is a configured optimizer: a defoptimizer called with ordinary Lisp values. :groups gives parts of the model their own hyperparameters, or their own optimizer, chosen by selectors over parameter paths.

(defvar optimizer
  (adamw :lr (warmup-stable-decay lr :warmup-steps warmup :decay-fraction 0.7)
         :betas [0.9 0.95]
         :eps 1e-10
         :weight-decay 0.10
         :groups [(group (select "model.embed.*") :lr 0.3 :weight-decay 0.0)
                  (group (select :rank-below 2) :lr 0.01)]))

Hyperparameters

The standard optimizers accept a number or a schedule for every numeric hyperparameter. A schedule is a function of ctx; see writing optimizers and the ready-made ones in sexpgpu/optim. This holds for defaults, explicit arguments and group overrides alike.

Selectors

Selectors run over canonical parameter paths, ranks and tags. Paths come from binding names, model.blocks.3.attn.q.weight; see models.

selector	matches
`(select "model.embed.*")`	a path glob: `` inside one segment, `*` across segments
`(select :rank-below n)`	parameters of rank under `n`
`(select :rank-at-least n)`	parameters of rank `n` or more
`(select :tag :name)`	parameters carrying that `defparam :tags` entry
`(select-not s)`	the complement
`(select-and s1 s2 ...)`	the intersection

Groups

(group selector :key value ...) overrides hyperparameters for the parameters the selector matches. :groups takes group values or bare selectors.

A group is named after its selector's text. That name is the param metadata of its optimizer diagnostics.
Everything no explicit group matched lands in default, which is dropped when it is empty.
A selector that matches nothing is E-OPT-001; check paths with sexpgpu explain.
A parameter matched by two groups is E-OPT-002, which lists the doubly covered paths; narrow one selector with select-not.
A malformed select, group, :groups or :optimizer is E-OPT-005, and the fix prints the shape.

A different optimizer per group

:optimizer is the one group override that is not a hyperparameter. It runs the group on another configured optimizer, which is how one run gives its matrices to Muon and everything else to AdamW:

(defvar optimizer
  (adamw :lr 0.003
         :groups [(group (select-and (select "model.blocks.**")
                                     (select :rank-at-least 2))
                         :optimizer (muon :lr 0.02))]))

The inner optimizer may not carry :groups of its own (E-OPT-005).