# Tensor operations

A tensor is a symbolic value: an operation on tensors records a node in the
graph being traced and returns a new tensor with a known shape and dtype.
Tensors exist only inside a traced function; see
[how it works](https://sexpgpu.041.io/docs/how-it-works.md#where-graphs-come-from).

## Shapes, dtypes, broadcasting

- Shapes are vectors of non-negative integers, fixed at compile time.
- Dtypes are `:f32` `:bf16` `:i32` `:i64` `:bool`; anything else is
  `E-DIM-002`.
- An axis may be negative and counts from the end.
- Broadcasting follows NumPy: right aligned, each dimension equal or 1.
  Every broadcast appears as an explicit `broadcast` node in
  [`sexpgpu ir`](https://sexpgpu.041.io/docs/explain.md#ir).
- `+ - * / pow sqrt exp log max` and the six comparisons are overloaded:
  the tensor reading wins when any argument is a tensor, and a number mixed
  with a tensor becomes a broadcast constant. Plain numbers at the front of
  an n-ary `*` fold first, so `(* lr scale tensor)` is one constant times
  the tensor.

## Operations

| builtin | keywords | result |
|---|---|---|
| `(neg x)` `(rsqrt x)` `(sin x)` `(cos x)` `(tanh x)` `(abs x)` | — | tensor-only unaries; `neg` and `abs` also take integers |
| `(acos x)` `(asin x)` `(atan2 y x)` | — | angles in radians; `acos` and `asin` are NaN outside [-1, 1]; `atan2` broadcasts and returns the angle of `(x, y)` in [-pi, pi] |
| `(maximum a b)` `(minimum a b)` | — | elementwise pair, broadcast |
| `(where mask a b)` | — | `mask` is `:bool`; all three broadcast to one shape |
| `(sum x :axes [-1] :keepdims false)` | `:axes` `:keepdims` | every axis by default; `(sum xs)` over a list of numbers adds them |
| `(max x :axes [..] :keepdims ..)` | `:axes` `:keepdims` | the same, maximum |
| `(reshape x [..])` | — | same element count and dtype |
| `(broadcast-to x [..])` | — | explicit broadcast |
| `(transpose x [0 2 1 3])` | — | a full permutation |
| `(slice x axis start stop)` | — | non-negative integer bounds, `start < stop <= dim` |
| `(concat axis [a b ...])` | — | equal ranks and equal dims off `axis` |
| `(matmul a b)` | — | batched; leading dims broadcast, inner dims must agree |
| `(index table ids)` | — | rows of `[n, rest...]` by an integer tensor, giving `[ids..., rest...]` |
| `(index-add base ids values)` | — | `base` with each row of `values` added at the row its id names, repeated ids accumulating: the adjoint of `index`, so `values` is `[ids..., rest...]` |
| `(gather x idx :axis -1)` | `:axis` | take along one axis; equal ranks, the result has `idx`'s shape |
| `(scatter-add base idx values :axis -1)` | `:axis` | `base` with `values` added along one axis at `idx`, repeated positions accumulating: the adjoint of `gather`; `idx` and `values` have one shape |
| `(linear-scan a b :axis 0 :reverse false)` | `:axis` `:reverse` | `h[t] = a[t] * h[t-1] + b[t]` along `:axis` from `h = 0`, so `h[0] = b[0]`; `:reverse true` runs from the end. `a` and `b` are floats of one dtype and broadcast to one shape; computed in `f32` under `:bf16`. `cumsum` in [sexpgpu/nn](https://sexpgpu.041.io/docs/nn.md) is built on it |
| `(cast x :bf16)` | — | same shape; a cast to the same dtype is the identity |
| `(iota [n n] :axis 1 :dtype :i32)` | `:axis` `:dtype` | indices along an axis; `:axis 0`, `:i32` by default; integer dtypes only |
| `(zeros [..])` `(ones [..])` | `:dtype` | `:f32` by default |
| `(full [..] value)` | `:dtype` | one repeated value |
| `(zeros-like x)` `(ones-like x)` | — | `x`'s shape and dtype |
| `(normal [..])` | `:std` `:mean` `:dtype` | `:std 1.0 :mean 0.0 :f32` |
| `(uniform [..])` | `:low` `:high` `:dtype` | `:low 0.0 :high 1.0 :f32` |
| `(shape x)` `(rank x)` `(dim x axis)` `(numel x)` `(dtype x)` | — | ordinary values, for compile-time decisions |

A degenerate spread (`:std 0.0`, or `:low` equal to `:high`) emits a
constant instead of a random node, so a zero-initialized projection costs
no random stream. Random streams fold in `defrun :seed`.

Reporting operations (`metric`, `diagnostic`, `tap-gradient`, `counter`)
are in [metrics](https://sexpgpu.041.io/docs/metrics.md) and [writing diagnostics](https://sexpgpu.041.io/docs/writing-diagnostics.md).
Higher-level functions (`mean`, `softmax`, `rms-norm`, `cross-entropy`,
attention) are in [sexpgpu/nn](https://sexpgpu.041.io/docs/nn.md).

## Errors

| code | when |
|---|---|
| `E-DIM-001` | shapes do not fit; the notes name both operands as you wrote them, with shape and dtype, and any broadcast that would have applied |
| `E-DIM-002` | dtypes differ, or a keyword is not a dtype |
| `E-DIM-003` | a tensor was expected |
| `E-FLOW-001` | `if` on a tensor; use `where`. See [staging](https://sexpgpu.041.io/docs/control.md#truth-and-staging) |
| `E-FLOW-002` | a tensor from another graph; recompute it here |
| `E-FLOW-003` | a tensor operation with no graph being traced |

```console
error[E-DIM-001]: matmul: inner dims differ: [2, 16] x [32, 8]
  --> bad-shape.sx:7:15
   |
 7 |   (lambda (x) (matmul x weight)))
   |               ^^^^^^^^^^^^^^^^^
note: operands: `x` [2, 16] f32, `weight` [32, 8] f32
```

Related: [precision and numerics](https://sexpgpu.041.io/docs/numerics.md), [models](https://sexpgpu.041.io/docs/models.md).

---

SexpGPU documentation. Every page: https://sexpgpu.041.io/llms.txt