The core idea
Inference is a numerical program.
Terms such as neural network, transformer, and attention name organizations of algorithms. They do not replace the algorithms. Underneath them are embedding lookups, matrix multiplications, normalization, nonlinear functions, residual addition, probability calculation, and sampling. The surprising behavior comes from the architecture, the learned values, the training process, and the scale at which those ordinary operations are composed.
At a glance
From a prompt to one more token
Google describes Gemma 4 12B Unified as a dense, decoder-only model with 11.95 billion parameters and 48 transformer layers. The same layers are reused for every prompt and every generated token.
“The dog runs”
[42, 107, 88]
[tokens, features]
attention + MLP
append “fast”
token ID 314
one score per token
Vocabulary first
Matrix, tensor, or slice?
A tensor is the general computing term for a multidimensional array. A matrix is specifically a two-dimensional tensor. A 3×3 array can therefore be an entire rank-2 tensor, or it can be a two-dimensional slice selected from a larger tensor.
7Scalarrank 0[7, 4, 2]Vectorrank 1[[…], […], […]]Matrixrank 2[[[…]]]Tensorrank 3+In an LLM, a matrix may contain permanent weights learned during training, or temporary activations created while processing the current prompt. Shape does not tell us which one it is; its role does.
Interactive model
A 3×3 laboratory for attention
The toy model has three tokens and three features. Its attention matrix A controls how the value activations V are mixed into context C. Each GPU worker below owns one output cell. Choose a worker to inspect its complete row-by-column calculation.
Worker 1: C[0,0] = Σ A[0,k] × V[k,0]
(.47 × 1) + (.26 × 0) + (.26 × 1) = .73
Inside one transformer layer
Attention builds a temporary token-to-token matrix
The input activation tensor flows through the layer. The layer’s learned weight tensors stay in place and transform it into queries, keys, and values.
[3,3][3,3][3,3][3,3]X × WQX × WKX × WVsoftmax(QKᵀ / √d)A × V feature j
0 1 2
i=0 [1 0 1]
i=1 [0 1 1]
i=2 [1 1 0]
output j
0 1 2
k=0 [1 0 0]
k=1 [0 1 0]
k=2 [0 0 1]
feature j
0 1 2
i=0 [1 0 1]
i=1 [0 1 1]
i=2 [1 1 0]
Identity weights make the first example easy to audit. Learned weights normally rotate, scale, combine, suppress, and amplify features instead of copying them unchanged.
A useful bridge
What PageRank helps us see
PageRank and transformer attention are not the same algorithm, but they share a useful linear-algebraic intuition: a matrix controls how information is redistributed among entities.
Pages distribute rank
rₜ₊₁ = Pᵀrₜ
A link-derived transition matrix repeatedly moves a rank vector until it approaches a stable distribution.
Tokens distribute context
C = softmax(QKᵀ/√d)V
A prompt-derived attention matrix mixes value vectors. It is rebuilt for the current activations in each layer.
The boundary of the analogy
PageRank converges. A transformer progresses.
PageRank normally applies a mostly fixed transition matrix repeatedly to seek a stable ranking. A transformer applies different learned layers in sequence, constructs attention from the current input, and then samples from vocabulary scores. The common ground is matrix-based information movement—not identical purpose or behavior.
Why GPUs fit
Every output cell is a small independent job
Matrix multiplication repeats the same multiply-and-accumulate operation across many indexes. GPUs contain many execution units designed to perform these jobs concurrently.
C[i,j] += A[i,k] × V[k,j]A real inference engine batches far larger matrices and schedules vast numbers of these operations across GPU cores. Specialized tensor hardware accelerates the same family of operations at lower numerical precision.
Optional detail
Open only what you need
The main argument stands on its own. These sections tighten the vocabulary and explain where common simplifications stop being accurate.
01Where INT4 fits
INT4 usually compresses learned weight matrices, not token IDs, layer pointers, or the entire inference process. Each four-bit code selects one of 16 representable levels. A scale—and sometimes a zero point—shared by a group of weights maps those codes back into useful numerical ranges.
stored code: q ∈ {0 … 15}
approximate weight: w ≈ scale × (q − zero_point)
token ID → commonly a wider integer
layer reference → normal program index or pointer
weight value → may be packed into INT402Weights do not move token by token
The model’s weight tensors are loaded into memory and reused. Activation tensors represent the current request and move conceptually from operation to operation. During generation, the KV cache retains earlier key and value activations so the model does not recompute the entire history for every new token.
03Matrix multiplication is the core of XLA but it is not the entire algorithm.
Calling inference “matrix multiplication” is directionally useful but incomplete. An implementation also performs lookups, normalization, position encoding, element-wise nonlinear functions, masking, residual addition, softmax, caching, and sampling. The computation remains ordinary mathematics and algorithms; it is not one matrix operation repeated without structure.
04Why 48 layers do not mean 48 tensors
A transformer layer is a container for computation. It owns multiple weight tensors for attention, normalization, and the feed-forward network. It also creates temporary activation tensors. Gemma’s 48 transformer layers therefore correspond to hundreds of named tensors in a checkpoint, not one tensor per layer.
The useful mental model
Weights stay. Activations flow. Algorithms transform.
A token is not a tiny object traveling through a mysterious machine. Its ID selects numbers from an embedding table. Those numbers become activation tensors. Each transformer layer applies learned weight tensors and explicit algorithms. The final values become vocabulary scores, and a sampling rule chooses what comes next.
Primary references
- Google AI for Developers — Gemma 4 model card
- Page, Brin, Motwani, and Winograd — The PageRank Citation Ranking: Bringing Order to the Web
- Lawrence Page — Method for Node Ranking in a Linked Database (US Patent 6,285,999 B1)
The matrices and token IDs in this article are deliberately small, illustrative examples. They explain indexing and data flow, not Gemma’s production tensor dimensions or learned values.
Continue learning
Ready to move beyond the 3×3 model?
Review the demonstration’s limits, prerequisite mathematics, algorithm map, and recommended learning paths.