a perpetual, ever-growing dump of everything i read, learn, and (mis)understand about AI
Note
this page is a permanent work in progress. notes will get added, rewritten, and corrected as my understanding shifts. anything wrong here is a wip on the way of being less wrong.
Training Basics
-
MFU - Model FLOPS Utilization
MFU = useful FLOPs per sec / peak FLOPs per sec- More precisely,
MFU = (model FLOPs per token × tokens/sec) / (num_GPUs × peak FLOP/s per GPU) - For a dense transformer model with
Nparameters, roughly,MFU = (6 × N × tokens per second per GPU) / peak FLOP/s per GPU
-
MFU vs HFU
- HFU (Hardware FLOPS Utilization) counts all operations actually performed including recompute
- MFU counts only the forward+backward ops required by the model, excluding recompute (implementation-agnostic). A GPU with enough memory to skip recompute does fewer ops (lower HFU) but may train faster — MFU rewards that, HFU punishes it. What ultimately matters is wall-clock-to-train, so MFU is the more honest cross-system metric.
-
Arithmetic Intensity (AI)
- Ratio of arithmetic operations to memory operations in a kernel (FLOPs/byte)
AI = FLOPs / bytes
Collective Communication Primitives
- All-reduce - Sum everyone's tensor, give the full sum to everyone.
- Reduce-scatter - Sum everyone's tensor, but give each rank only its shard of the result.
- All-gather - Collect everyone's shard, give the full tensor to everyone.
- Broadcast - One rank holds the tensor; copy it out so everyone ends up with the same full tensor.
- Scatter - One rank holds the full tensor; split it into shards and send each rank only its shard.