The run file

An experiment file is recognized by top-level bindings with conventional names. Anything else in the file is an ordinary definition and is ignored by the contract. The file is read once and evaluated once, in order.

The order of a file

(require ...)          standard modules and library files the file uses
(defvariant ...)       named sets of knob overrides, before defknobs
(defknobs ...)         everything this run may vary
(defsweep ...)         named cross products over knob values
(defvar ...)           ordinary constants
train-loader           required: where the training records come from
(defeval ...)          optional, any number: named evaluation passes
prepare                required: batch -> (list :inputs ... :targets ...)
eval-prepare           optional: defaults to prepare
model                  required: a model value
objective              required: prediction, targets -> one float scalar
evaluate               required when a pass exists: emits metrics
optimizer              required: a configured optimizer
curriculum             optional: stage, ctx, observations -> new stage
(defrun ...)           required: steps, microbatches, cadence, precision, seed

name	page
require	modules and require
defvariant, defknobs, defsweep	knobs, variants and sweeps
train-loader, prepare, eval-prepare	loaders and prepare
defeval	evaluation passes
model	models and parameters
objective, evaluate	objective and evaluate
optimizer	optimizer groups, writing optimizers
curriculum	curriculum and ctx
defrun	defrun

Missing a required name is E-CONTRACT-001; binding one to the wrong kind of value is E-CONTRACT-002. The components are checked independently, so a mistake in objective never hides a missing optimizer. The contract names are looked up in the run file itself, so a library that defines model gives the run none.

A complete minimal run

Six names, no variation, no evaluation.

;;; The smallest file the contract accepts: six names and nothing else.

(require "sexpgpu/nn" cross-entropy gpt)
(require "sexpgpu/optim" sgd)

(defvar train-loader
  (loader :sources [(files ["data/tiny-train.jsonl.gz"] :encoding "jsonl.gz")]
          :fields [(field :tokens :from "input_ids" :dtype :i32 :shape [17])]
          :batch-size 4
          :infinite true))

(defun prepare (batch)
  (let ((tokens (field batch :tokens)))
    (list :inputs (slice tokens 1 0 16) :targets (slice tokens 1 1 17))))

(defvar model (gpt :vocab 32 :layers 2 :dim 32 :heads 2))

(defun objective (prediction targets) (cross-entropy prediction targets))

(defvar optimizer (sgd :lr 0.5))

(defrun :steps 20 :microbatches 2 :precision :f32 :seed 1)

$ sexpgpu check minimal.sx
ok: 32 parameters, 65 graphs, 20 steps
$ sexpgpu run minimal.sx --steps 3
...
done: 3 steps, train/loss 3.1645
  counter records 24

A complete realistic run

A realistic run adds knobs and variants (including a smoke variant small enough for --device cpu), an evaluation pass, a counter, a reported per-token loss beside the summed objective, and optimizer groups with a schedule each. Its selection and its optimizer:

(require "sexpgpu/nn" cross-entropy gpt)
(require "sexpgpu/optim" adamw warmup-stable-decay)

(defvariant small (layers 2) (width 128))
(defvariant smoke (layers 2) (width 128) (seq 64))

(defknobs
  (lr 0.0015 "peak learning rate of the block matrices, the default group")
  (layers 4 "transformer blocks")
  (width 256 "model width")
  (seq 1024 "sequence length in tokens; a window holds seq + 1")
  (seed 1 "the run seed; every random stream folds it in"))

(defsweep seeds (seed [1 2 3]))

(defvar warmup 25)

(defun schedule (peak)
  (warmup-stable-decay peak :warmup-steps warmup :decay-fraction 0.7))

(defvar optimizer
  (adamw :lr (schedule lr)
         :betas [0.9 0.95]
         :eps 1e-10
         :weight-decay 0.10
         :groups [(group (select "model.embed.*")
                         :lr (schedule 0.3)
                         :betas [0.8 0.95]
                         :weight-decay 0.0)
                  (group (select "model.head.weight")
                         :lr (schedule (/ 1.0 320.0))
                         :betas [0.8 0.95]
                         :weight-decay 0.0)
                  (group (select :rank-below 2)
                         :lr (schedule 0.01)
                         :betas [0.8 0.95]
                         :weight-decay 0.0)]))

explain --variant smoke reads it back in those terms: the parameters and their element count, the four update groups (the two named ones, :rank-below 2, and default with the remaining block matrices) and which parameters each holds, each group's m and v states and the constants its schedule folded to, and the microbatches of one step.

Move reusable definitions into a library to keep the run file short, and format it with sexpgpu fmt.