Evaluation passes

A pass is a named loader and a cadence. Every pass runs eval-prepare (or prepare), the model and evaluate over its own loader. A file declares as many passes as it needs.

(defeval eval
  :loader (loader :sources [(files ["data/val.parquet"])]
                  :fields [(field :tokens :from "input_ids" :dtype :i32 :shape [1025])]
                  :batch-size 4)
  :every 25)

(defeval probe :loader val-loader :every 5 :batches 16)

keyword	default	meaning
:loader	required	the records the pass reads; see loaders
:every	none	steps between runs; without it the pass runs only after the last step
:batches	the whole loader	read this many batches from the start; without it the loader must be finite

When a pass runs

• At step 0, every :every steps, and once more after the last step.
• Passes due at the same step run in the order the file declares them.
• A pass reopens its loader every time, so a limited pass reads the same records each time.
• --eval-every N replaces every pass's :every for one session; 0 leaves only the pass after the last step, so a curriculum never runs. See run.

The pass named eval

The pass named eval is the one the summary, the status file, the progress line and sweep's table report; a file without one reports its first pass. Every observation from a pass carries pass metadata, so eval/loss from eval and from probe are two series. See metrics.

Errors

code	when
E-CONTRACT-013	two passes share a name
E-CONTRACT-014	an evaluate or a curriculum in a file with no pass, so nothing would run it

Related: curriculum, the status file.