Knobs, variants and sweeps

A run declares everything it may vary, in itself. Every tool selects a point before the file is read, and the selection is recorded in the manifest, the metrics metadata, the status file and the run's slug.

(defvariant small (layers 2) (width 128))
(defvariant smoke (layers 2) (width 128) (seq 64))

(defknobs
  (lr 0.0015 "peak learning rate") ; name, default, optional doc
  (layers 4)
  (optimizer :adamw))              ; any ordinary value

(defsweep seeds (seed [1 2 3]))
(defsweep deep-lr :variant small (lr [0.002 0.004]))

defknobs

(defknobs (name default ["doc"]) ...) binds each knob as an ordinary top-level variable, so the rest of the file reads lr and knows nothing about where the number came from. A knob is declared once (E-KNOB-001) and before it is used; a knob read before its defknobs is E-EVAL-003.

defvariant

(defvariant name (knob value) ...) is a named set of overrides.

• Declare variants before the first defknobs. Late or duplicate variants are E-KNOB-006.
• Override expressions evaluate once, when the declaration executes, in the declaring lexical environment. Macros and loops may generate variants; see macros.
• A variant may only set declared knobs (E-KNOB-003), checked after all declarations have executed.
• Variants do not compose: a selection names one or none.

defsweep

(defsweep name [:variant v] (knob [values...]) ...) is a cross product, the last axis varying fastest. seed is a legal axis in every file, because every run has defrun :seed. :variant names the base variant of every point. Run one with sexpgpu sweep.

The selection

source	flag	wins over
--set knob=value	repeatable	the variant and the default
--variant name	once	the default
the declared default	—	—

Every source is recorded per knob (default, variant, set, and override for --steps, --eval-every and --diagnostics). A --variant or --set naming something the file does not declare is E-KNOB-004 or E-KNOB-005, never silence. --set seed=N reaches defrun :seed in a file that declares no seed knob. How the selection meets the environment and defrun is in environment.

check, explain, diff, watch, ir, run and bundle take the selection; see the command line.

Seeing them

$ sexpgpu variants my-run.sx
knobs:
  lr      0.0015  peak learning rate of the block matrices, the default group
  layers  4       transformer blocks
  width   256     model width
  seq     1024    sequence length in tokens; a window holds seq + 1
  seed    1       the run seed; every random stream folds it in
variants:
  small  layers=2 width=128
  smoke  layers=2 width=128 seq=64
sweeps:
  seeds  3 points over seed

Change a knob with --set or a variant; copy the file with sexpgpu new only when the change is not a knob.

Related: run, sweep, explain.