Loaders and prepare

train-loader says where training records come from; prepare turns one batch of them into the model's inputs and the objective's targets. Shapes are static, so every field declares its shape.

(defvar train-loader
  (loader :sources [(files ["data/train.parquet"])]
          :fields [(field :tokens :from "input_ids" :dtype :i32 :shape [1025])]
          :batch-size 4
          :infinite true))

(defun prepare (batch)
  (let ((tokens (field batch :tokens)))
    (counter :tokens (numel tokens))
    (list :inputs (slice tokens 1 0 seq)
          :targets (slice tokens 1 1 (+ seq 1)))))

The forms

form	keywords	notes
`loader`	`:sources` `:stages` `:fields` `:batch-size` `:shuffle` `:infinite`	`:sources`, `:shuffle` and `:infinite` describe the single main stage; needs `:fields` and at least one source or stage
`stage`	`:name` `:sources` `:shuffle` `:infinite`	`:name` defaults to `main`
`files`	a vector of paths, `:encoding`	one URI per string, no globbing; the encoding is inferred from the name when absent: `parquet`, `jsonl.gz`, `msgpack.zstf`
`manifest`	a URI, `:split` `:weight` `:encoding`	an S3 data manifest
`field`	a name, `:from` `:dtype` `:shape`	`:from` defaults to the name, `:dtype` to `:f32`, `:shape` to a scalar

An unknown keyword anywhere here is E-CONTRACT-003, and the fix lists the accepted ones.
:shuffle N is a buffer of N rows.
:batch-size belongs to the loader, not the stage: every stage has the same batch size because shapes are static.
A relative path in files resolves against the experiment file's directory. Absolute paths and s3:// URIs pass through untouched; S3 credentials are in environment.
Use :stages [(stage :name "easy" ...) (stage :name "hard" ...)] when a curriculum moves between them.
A loader read by an evaluation pass without :batches must be finite. See evaluation passes.

prepare

prepare receives the batch and returns a plist with :inputs and :targets (E-CONTRACT-004 otherwise).

(field batch :name) reads one field, shaped [batch-size] ++ field-shape. A name the loader does not declare is E-CONTRACT-005, and the fix lists what it declares.
:inputs and :targets are each a tensor or a list of tensors; a list of inputs is spread over the model's parameters.
Any other key of the returned plist is passed to objective and evaluate as a third metadata argument, when they are written with three parameters.
eval-prepare, when defined, replaces prepare on every evaluation pass; otherwise prepare serves both.

prepare is traced into the training graph, so a metric in it behaves as one in objective.

Counters

(counter :name scalar) declares a progress counter. It belongs in prepare, where records are counted (E-CONTRACT-010 elsewhere). The runtime sums it over microbatches and keeps it for the whole run and the current stage:

ctx offers it as :name and :stage-name to schedules and the curriculum; see ctx.
every step emits counter/<name> and rate/<name> (per second); see metrics.
the status file and the summary carry the total.

A counter is not a metric: (counter :tokens (numel tokens)) gives a counter/tokens curve and tokens per second without reporting either. The runtime itself counts records.