# Models and parameters

A model is a set of parameters plus the function that uses them.
`defmodel` defines a constructor; calling it creates fresh parameters and
returns the model; applying a model is calling it.

```lisp
(defmodel linear (in out &key (bias true) (init-std nil))
  (defparam weight (normal [in out] :std (or init-std (/ 1.0 (sqrt in)))))
  (when bias (defparam b (zeros [out])))
  (lambda (x) (if bias (+ (matmul x weight) b) (matmul x weight))))

(defvar model (linear 32 8))
```

## defmodel

- The body is ordinary Lisp: `let` sub-models, loop, branch on shapes.
- The body must end in a function, a builtin or another model
  (`E-PARAM-004`); return `(lambda (x) ...)` last.
- Each call to a constructor creates its own parameters. Importing a
  definition imports no state.
- The contract's `model` must be a model value, the result of calling a
  constructor; see [the run file](https://sexpgpu.041.io/docs/run-file.md).
- Weights are stored `[in, out]`: `(matmul x weight)`. A PyTorch
  `Linear` stores the transpose.

## defparam

`(defparam name initializer &key tags numerics)` declares persistent
trainable state, inside a model only (`E-PARAM-002`).

- The initializer is a closed graph of its own: shapes, constants and
  random nodes, never another value (`E-PARAM-003`). The parameter's shape
  and dtype are its output's, so `(normal [in out] :std s)` says everything
  once.
- Random initializers fold in `defrun :seed`; gensym names never affect
  the initial values.

| keyword | value | effect |
|---|---|---|
| `:tags` | a list or vector of names | `(select :tag :name)` finds the parameter; see [optimizer groups](https://sexpgpu.041.io/docs/optimizer-groups.md) |
| `:numerics` | `:high` | under `:bf16` the parameter stays an `f32` master weight read without a cast down; see [numerics](https://sexpgpu.041.io/docs/numerics.md) |

Both evaluate their argument expressions. Any other option, or a
`:numerics` other than `:high`, is `E-PARAM-006`.

## Paths

A parameter's canonical path comes from binding names:

- the root is the top-level name the model is bound to, normally `model`;
- inside a `defmodel`, a model bound by `let` takes its binding name;
- a list or vector of models takes one index per element;
- a `defparam` takes its own name.

```lisp
(let ((blocks (repeat 12 (lambda (i) (block dim heads))))) ...)
;; model.blocks.3.attn.q.weight
```

- `(named "segment" value)` overrides the segment of a model or parameter
  (`E-PARAM-005` on anything else).
- A model never bound to a name gets `anon.N` and the warning
  `W-PARAM-001`.
- Gensym bindings never name models; a parameter declared with a gensym
  gets a stable `param.N` within its model.

Paths are what [selectors](https://sexpgpu.041.io/docs/optimizer-groups.md#selectors) match, what
[`explain`](https://sexpgpu.041.io/docs/explain.md) lists and what checkpoints store. Read them with
`sexpgpu explain` before writing a selector.

## Weight tying

Tying is value sharing: use the same parameter, or the same model, in two
places and it is one parameter, updated once.

```lisp
(defmodel tied-lm (vocab dim)
  (defparam table (normal [vocab dim] :std 0.02))
  (lambda (ids) (matmul (index table ids) (transpose table [1 0]))))
```

`model.table` is one parameter; its gradient sums both uses.

Related: [sexpgpu/nn](https://sexpgpu.041.io/docs/nn.md), [numerics](https://sexpgpu.041.io/docs/numerics.md), [tensor operations](https://sexpgpu.041.io/docs/tensors.md).

---

SexpGPU documentation. Every page: https://sexpgpu.041.io/llms.txt