What is a Transformer?

Transformer is a neural network architecture that changed AI approaches since 2017.

“Attention is All You Need”

Transformer Models Power Text

GPT-2, Llama, and Gemini use transformers for text generation and other applications.

Next-Token Prediction

Transformers predict the most probable next token after a user prompt.

Self-Attention Mechanism

Self-attention captures long-range dependencies more effectively than previous architectures.

GPT-2 (Small) Model

Transformer Explainer uses GPT-2 (small) with 124 million parameters.

Embedding Layer

Embedding converts text into numerical vectors for model processing.

Tokenization Process

Tokenization splits text into tokens like words or subwords with unique IDs.

Token Embeddings

GPT-2 (small) uses 768-dimensional vectors for each token in its vocabulary.

Positional Encoding

Positional encoding adds information about token positions in the input sequence.

Final Embedding

Final embedding combines token and positional encodings for full representation.

Transformer Block

Transformer blocks process input through self-attention and MLP layers.

Multi-Head Self-Attention

Multi-head attention captures relationships from different perspectives.

Query, Key, Value Matrices

Query is search text, key is web page titles, value is web page content.

Multi-Head Splitting

GPT-2 (small) splits attention into 12 heads for diverse relationships.

Masked Self-Attention

Masked self-attention prevents future token access during prediction.

MLP Layer

MLP expands and compresses token representations for richer patterns.

Output Probabilities

Output probabilities determine next token likelihood through softmax.

Temperature Parameter

Temperature controls randomness in predictions with temperature < 1 for confidence.

Interactive Features

Users can explore attention weights, temperature, and sampling methods.

Model Implementation

Transformer Explainer uses live GPT-2 (small) in browser with ONNX Runtime.