Models and parameters

A model is a set of parameters plus the function that uses them. defmodel defines a constructor; calling it creates fresh parameters and returns the model; applying a model is calling it.

(defmodel linear (in out &key (bias true) (init-std nil))
  (defparam weight (normal [in out] :std (or init-std (/ 1.0 (sqrt in)))))
  (when bias (defparam b (zeros [out])))
  (lambda (x) (if bias (+ (matmul x weight) b) (matmul x weight))))

(defvar model (linear 32 8))

defmodel

The body is ordinary Lisp: let sub-models, loop, branch on shapes.
The body must end in a function, a builtin or another model (E-PARAM-004); return (lambda (x) ...) last.
Each call to a constructor creates its own parameters. Importing a definition imports no state.
The contract's model must be a model value, the result of calling a constructor; see the run file.
Weights are stored [in, out]: (matmul x weight). A PyTorch Linear stores the transpose.

defparam

(defparam name initializer &key tags numerics) declares persistent trainable state, inside a model only (E-PARAM-002).

The initializer is a closed graph of its own: shapes, constants and random nodes, never another value (E-PARAM-003). The parameter's shape and dtype are its output's, so (normal [in out] :std s) says everything once.
Random initializers fold in defrun :seed; gensym names never affect the initial values.

keyword	value	effect
`:tags`	a list or vector of names	`(select :tag :name)` finds the parameter; see optimizer groups
`:numerics`	`:high`	under `:bf16` the parameter stays an `f32` master weight read without a cast down; see numerics

Both evaluate their argument expressions. Any other option, or a :numerics other than :high, is E-PARAM-006.

Paths

A parameter's canonical path comes from binding names:

the root is the top-level name the model is bound to, normally model;
inside a defmodel, a model bound by let takes its binding name;
a list or vector of models takes one index per element;
a defparam takes its own name.

(let ((blocks (repeat 12 (lambda (i) (block dim heads))))) ...)
;; model.blocks.3.attn.q.weight

(named "segment" value) overrides the segment of a model or parameter (E-PARAM-005 on anything else).
A model never bound to a name gets anon.N and the warning W-PARAM-001.
Gensym bindings never name models; a parameter declared with a gensym gets a stable param.N within its model.

Paths are what selectors match, what explain lists and what checkpoints store. Read them with sexpgpu explain before writing a selector.

Weight tying

Tying is value sharing: use the same parameter, or the same model, in two places and it is one parameter, updated once.

(defmodel tied-lm (vocab dim)
  (defparam table (normal [vocab dim] :std 0.02))
  (lambda (ids) (matmul (index table ids) (transpose table [1 0]))))

model.table is one parameter; its gradient sums both uses.