# Transformer Explainer: LLM Model Visualized

> See how GPT-2 (small) predicts text with self-attention and MLP layers.
- Title: Transformer Explainer: LLM Model Visualized · Github
- Summary: See how GPT-2 (small) predicts text with self-attention and MLP layers. GPT-2 (small) uses self-attention and MLP layers to predict the next word in a…
- Keywords: transformer, gpt2, self-attention, mlp, llm, text-generation, neural-network, embedding, positional-encoding, attention-mechanism, probability-distribution, temperature-parameter
- Source: Github — https://poloclub.github.io/transformer-explainer
- Read time: 1 min
- Topics: transformer, gpt2, self-attention, mlp, llm, text-generation, neural-network, embedding, positional-encoding, attention-mechanism, probability-distribution, temperature-parameter
## What is a Transformer?
Transformer is a neural network architecture that changed AI approaches since 2017.

> Attention is All You Need
## Transformer Models Power Text
GPT-2, Llama, and Gemini use transformers for text generation and other applications.
## Next-Token Prediction
Transformers predict the most probable next token after a user prompt.
## Self-Attention Mechanism
Self-attention captures long-range dependencies more effectively than previous architectures.
## GPT-2 (Small) Model
Transformer Explainer uses GPT-2 (small) with 124 million parameters.
## Embedding Layer
Embedding converts text into numerical vectors for model processing.
## Tokenization Process
Tokenization splits text into tokens like words or subwords with unique IDs.
## Token Embeddings
GPT-2 (small) uses 768-dimensional vectors for each token in its vocabulary.
## Positional Encoding
Positional encoding adds information about token positions in the input sequence.
## Final Embedding
Final embedding combines token and positional encodings for full representation.
## Transformer Block
Transformer blocks process input through self-attention and MLP layers.
## Multi-Head Self-Attention
Multi-head attention captures relationships from different perspectives.
## Query, Key, Value Matrices
Query is search text, key is web page titles, value is web page content.
## Multi-Head Splitting
GPT-2 (small) splits attention into 12 heads for diverse relationships.
## Masked Self-Attention
Masked self-attention prevents future token access during prediction.
## MLP Layer
MLP expands and compresses token representations for richer patterns.
## Output Probabilities
Output probabilities determine next token likelihood through softmax.
## Temperature Parameter
Temperature controls randomness in predictions with temperature < 1 for confidence.
## Interactive Features
Users can explore attention weights, temperature, and sampling methods.
## Model Implementation
Transformer Explainer uses live GPT-2 (small) in browser with ONNX Runtime.
## Key takeaway

GPT-2 (small) uses self-attention and MLP layers to predict the next word in a sequence.