Notes

A perpetual, ever-growing dump of everything I read, learn, and (mis)understand about AI

Training Basics

• MFU - Model FLOPS Utilization

  • MFU = useful FLOPs per sec / peak FLOPs per sec
  • More precisely, MFU = (model FLOPs per token × tokens/sec) / (num_GPUs × peak FLOP/s per GPU)
  • For a dense transformer model with N parameters, roughly, MFU = (6 × N × tokens per second per GPU) / peak FLOP/s per GPU

• MFU vs HFU

  • HFU (Hardware FLOPS Utilization) counts all operations actually performed including recompute
  • MFU counts only the forward+backward ops required by the model, excluding recompute (implementation-agnostic). A GPU with enough memory to skip recompute does fewer ops (lower HFU) but may train faster — MFU rewards that, HFU punishes it. What ultimately matters is wall-clock-to-train, so MFU is the more honest cross-system metric.

• Arithmetic Intensity (AI)

  • Ratio of arithmetic operations to memory operations in a kernel (FLOPs/byte)
  • AI = FLOPs / bytes

• Roofline model

  • Plots attainable FLOP/s against arithmetic intensity on log-log axes: below the ridge point (peak FLOP/s ÷ memory bandwidth) a kernel is memory-bound (perf = bandwidth × AI); above it, compute-bound (perf = peak FLOP/s)

The explorer below is deliberately simplified: each preset pairs theoretical dense FP16 Tensor Core throughput with FP32 accumulation and peak memory bandwidth. It is useful for intuition, not hardware benchmarking; real kernels and non-Tensor-Core paths can run below the line.

The word dense matters here. NVIDIA's spec tables sometimes also show a 2x rate "with sparsity": that assumes a supported structured pattern lets Tensor Cores skip zeros. It does not make every sparse-looking model twice as fast, and it does not change memory bandwidth. This explorer uses the non-sparse rate, so turning on a genuinely supported sparse path would raise the flat compute ceiling and move the ridge point to the right.

Preset sources: NVIDIA's HGX B200 specifications and DGX B200 specifications, H100 specifications, A100 specifications, and Ada GPU architecture whitepaper for the RTX 4090. The data-centre tables mark structured-sparsity figures separately; the dense values plotted here leave that multiplier out. The B200 preset converts NVIDIA's eight-GPU system totals to per-GPU values: 2.25 PFLOP/s dense FP16/BF16 and 8 TB/s of HBM3e bandwidth.

Collective Communication Primitives

• All-reduce - Sum everyone's tensor, give the full sum to everyone.
• Reduce - Sum everyone's tensor, give the full sum to one root rank.
• Reduce-scatter - Sum everyone's tensor, but give each rank only its shard of the result.
• All-gather - Collect everyone's shard, give the full tensor to everyone.
• Broadcast - One rank holds the tensor; copy it out so everyone ends up with the same full tensor.
• Scatter - One rank holds the full tensor; split it into shards and send each rank only its shard.
• All-to-all - Every rank splits its tensor into shards and sends shard j to rank j — a distributed transpose.

These are conceptual state transitions across four ranks, not literal message schedules, network topologies, or performance models. Real implementations choose algorithms such as rings or trees based on the hardware and payload.

The frames show logical collective outputs, not physical buffer lifetime. A source chunk staying on screen during a transfer is a teaching aid: real libraries may use in-place or out-of-place buffers, and bytes can still exist locally even when they are no longer part of that rank's defined collective output.