# Loaders and prepare

`train-loader` says where training records come from; `prepare` turns one
batch of them into the model's inputs and the objective's targets. Shapes
are static, so every field declares its shape.

```lisp
(defvar train-loader
  (loader :sources [(files ["data/train.parquet"])]
          :fields [(field :tokens :from "input_ids" :dtype :i32 :shape [1025])]
          :batch-size 4
          :infinite true))

(defun prepare (batch)
  (let ((tokens (field batch :tokens)))
    (counter :tokens (numel tokens))
    (list :inputs (slice tokens 1 0 seq)
          :targets (slice tokens 1 1 (+ seq 1)))))
```

## The forms

| form | keywords | notes |
|---|---|---|
| `loader` | `:sources` `:stages` `:fields` `:batch-size` `:shuffle` `:infinite` | `:sources`, `:shuffle` and `:infinite` describe the single main stage; needs `:fields` and at least one source or stage |
| `stage` | `:name` `:sources` `:shuffle` `:infinite` | `:name` defaults to `main` |
| `files` | a vector of paths, `:encoding` | one URI per string, no globbing; the encoding is inferred from the name when absent: `parquet`, `jsonl.gz`, `msgpack.zstf` |
| `manifest` | a URI, `:split` `:weight` `:encoding` | an S3 data manifest |
| `field` | a name, `:from` `:dtype` `:shape` | `:from` defaults to the name, `:dtype` to `:f32`, `:shape` to a scalar |

- An unknown keyword anywhere here is `E-CONTRACT-003`, and the fix lists
  the accepted ones.
- `:shuffle N` is a buffer of `N` rows.
- `:batch-size` belongs to the loader, not the stage: every stage has the
  same batch size because shapes are static.
- A relative path in `files` resolves against the experiment file's
  directory. Absolute paths and `s3://` URIs pass through untouched; S3
  credentials are in [environment](https://sexpgpu.041.io/docs/environment.md#s3-credentials).
- Use `:stages [(stage :name "easy" ...) (stage :name "hard" ...)]` when a
  [curriculum](https://sexpgpu.041.io/docs/curriculum.md) moves between them.
- A loader read by an evaluation pass without `:batches` must be finite.
  See [evaluation passes](https://sexpgpu.041.io/docs/evaluation.md).

## prepare

`prepare` receives the batch and returns a plist with `:inputs` and
`:targets` (`E-CONTRACT-004` otherwise).

- `(field batch :name)` reads one field, shaped `[batch-size] ++ field-shape`.
  A name the loader does not declare is `E-CONTRACT-005`, and the fix lists
  what it declares.
- `:inputs` and `:targets` are each a tensor or a list of tensors; a list of
  inputs is spread over the model's parameters.
- Any other key of the returned plist is passed to `objective` and
  `evaluate` as a third `metadata` argument, when they are written with
  three parameters.
- `eval-prepare`, when defined, replaces `prepare` on every evaluation
  pass; otherwise `prepare` serves both.

`prepare` is traced into the training graph, so a [`metric`](https://sexpgpu.041.io/docs/metrics.md)
in it behaves as one in `objective`.

## Counters

`(counter :name scalar)` declares a progress counter. It belongs in
`prepare`, where records are counted (`E-CONTRACT-010` elsewhere). The
runtime sums it over microbatches and keeps it for the whole run and the
current stage:

- `ctx` offers it as `:name` and `:stage-name` to schedules and the
  curriculum; see [ctx](https://sexpgpu.041.io/docs/curriculum.md#ctx).
- every step emits `counter/<name>` and `rate/<name>` (per second); see
  [metrics](https://sexpgpu.041.io/docs/metrics.md#metric-names).
- the status file and the summary carry the total.

A counter is not a metric: `(counter :tokens (numel tokens))` gives a
`counter/tokens` curve and tokens per second without reporting either. The
runtime itself counts `records`.

Related: [the run file](https://sexpgpu.041.io/docs/run-file.md), [tensor operations](https://sexpgpu.041.io/docs/tensors.md).

---

SexpGPU documentation. Every page: https://sexpgpu.041.io/llms.txt