Skip to content

Vivid Labs AI

Now Watching

Paws & Parameters

Paws & Parameters /From Text Data to a Bigram Language Model

From Text Data to a Bigram Language Model

Prepare a text corpus, tokenize it into model-ready sequences, create training and validation splits, and build the first bigram language-model baseline.

Queue

8 videos in this course

Paws & Parameters

Jump across the full chapter sequence without leaving the player.

01

From Text Data to a Bigram Language Model

Prepare a text corpus, tokenize it into model-ready sequences, create training and validation splits, and build the first bigram language-model baseline.

02

The Tensor and Matrix Operations Behind the Model

Understand the tensor shapes, indexing, broadcasting, and matrix operations that turn token sequences into predictions the model can train against.

03

Training, Generating, and Evaluating the Baseline

Train the baseline model, generate sample text, interpret loss, and use validation results to separate genuine learning from memorization.

04

From Nonlinearity to the Transformer Architecture

Add nonlinear computation and trace the architectural steps that move a simple language model toward a transformer capable of richer context.

05

Position Information and the GPT Forward Pass

Encode token position and follow the GPT forward pass from embeddings through contextual representations to next-token logits.

06

Building Attention Blocks and Residual Paths

Build self-attention, feed-forward layers, normalization, and residual connections into reusable transformer blocks.

07

Scaling Data and Operating Model Checkpoints

Scale the dataset and training loop while using device selection, checkpoints, and repeatable operating practices to keep experiments recoverable.

08

Generation Boundaries, Fine-Tuning, and Next Steps

Recognize the model's generation limits, compare fine-tuning paths, and identify the evaluation and engineering steps needed to continue responsibly.