S-exp GPU
All pages
Docs · Start hereMarkdown

Quickstart

Five commands: write a file, check it, read what the compiler made of it, train a few steps on the CPU, and pack it for a GPU box.

1. Write the experiment

Save as minimal.sx. It is the smallest file the contract accepts: six names and nothing else (the run file explains each).

(require "sexpgpu/nn" cross-entropy gpt)
(require "sexpgpu/optim" sgd)

(defvar train-loader
  (loader :sources [(files ["data/tiny-train.jsonl.gz"] :encoding "jsonl.gz")]
          :fields [(field :tokens :from "input_ids" :dtype :i32 :shape [17])]
          :batch-size 4
          :infinite true))

(defun prepare (batch)
  (let ((tokens (field batch :tokens)))
    (list :inputs (slice tokens 1 0 16) :targets (slice tokens 1 1 17))))

(defvar model (gpt :vocab 32 :layers 2 :dim 32 :heads 2))

(defun objective (prediction targets) (cross-entropy prediction targets))

(defvar optimizer (sgd :lr 0.5))

(defrun :steps 20 :microbatches 2 :precision :f32 :seed 1)

The loader reads data/tiny-train.jsonl.gz next to the file: one JSON object per line with an input_ids array of 17 token ids below 32, each a counting sequence the model can learn. Download a 512-line sample:

mkdir -p data
curl -fsSL https://sexpgpu.041.io/data/tiny-train.jsonl.gz -o data/tiny-train.jsonl.gz

Any shape works once the :shape, the slices and :vocab agree; see loaders.

2. Check it

$ sexpgpu check minimal.sx
ok: 32 parameters, 65 graphs, 20 steps

check needs no data and no GPU. Break a shape and it names your line; see check.

3. Read what it compiled to

sexpgpu explain minimal.sx

Parameters with their optimizer groups, every graph, one training step and the peak activation memory. See explain.

4. Train a few steps

$ sexpgpu run minimal.sx --steps 3 --device cpu
...
done: 3 steps, train/loss 3.1645
  counter records 24

--device cpu is the IR interpreter: every machine has it, and it holds toy sizes only (devices). On a GPU machine, --device cuda --metrics jsonl:minimal.jsonl trains at full speed and writes every number; see run and metrics.

5. Pack it for a GPU box

sexpgpu bundle minimal.sx -o out/minimal --binary ./sexpgpu-linux-cuda
scp -r out/minimal gpu-box: && ssh gpu-box 'SEXPGPU_DEVICE=cuda minimal/run.sh'

See bundle.

Next