PyTorch: the training loop

Dataset and DataLoader, loss functions, the SGD and Adam optimizers, the full training loop, eval mode and accuracy, saving a model with state_dict — by training a small classifier on synthetic data.

  • University
  • 25 min
  • 10 questions
  • Lesson 64 of 66

In this lesson you will learn

  • Split data into batches with Dataset and DataLoader
  • Choose a loss function and an optimizer that fit the task
  • Write the full training loop and evaluate the model on validation/test data
  • Save a model with state_dict and load it back

We now have all the pieces: tensors, autograd and nn.Module. Let us assemble them into the standard recipe that practitioners write every day: data → DataLoader → model → loss → optimizer → training loop → evaluation → saving. Remarkably, this skeleton is almost the same for a tiny classifier and for a language model with billions of parameters — only the size of the data and the model changes.

Dataset and DataLoader

A Dataset is an object that behaves like a list of examples: __len__ returns the number of examples and __getitem__(i) returns the i-th (x, y) pair. If your data is already in tensors, the ready-made TensorDataset is enough; to read from files (images, texts) you write your own class. A DataLoader builds batches from a Dataset, shuffles the examples every epoch if asked (shuffle=True) and can load data in parallel with num_workers.

Python
import torch
from torch.utils.data import Dataset, DataLoader

class SquaresDataset(Dataset):
    def __init__(self, n):
        self.x = torch.arange(n, dtype=torch.float32)

    def __len__(self):
        return len(self.x)

    def __getitem__(self, i):
        return self.x[i], self.x[i] ** 2

ds = SquaresDataset(10)
print(len(ds), ds[3])
loader = DataLoader(ds, batch_size=4)
for xb, yb in loader:
    print(xb.tolist(), yb.tolist())

Expected output

10 (tensor(3.), tensor(9.))
[0.0, 1.0, 2.0, 3.0] [0.0, 1.0, 4.0, 9.0]
[4.0, 5.0, 6.0, 7.0] [16.0, 25.0, 36.0, 49.0]
[8.0, 9.0] [64.0, 81.0]
10 examples, batch size 4: two full batches and a last batch of 2 examples. The DataLoader stacks the individual examples into tensors for you.

Batch, step (iteration) and epochDefinition

A batch is a group of examples processed together for one update. A step, or iteration, is one parameter update on one batch. An epoch is one pass in which every training example is “seen” once. Training usually lasts tens of epochs, and every epoch consists of many steps.

steps per epoch = ⌈N / B⌉updates = epochs · ⌈N / B⌉steps per epoch = ⌈N / B⌉updates = epochs · ⌈N / B⌉

Where

N
number of training examples
B
batch size
⌈ ⌉
rounding up: the last incomplete batch is a step too

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

Loss functions and optimizers

TaskLast layerLossTargets
Regressionnn.Linear(h, 1)nn.MSELossfloat32 numbers
Binary classificationnn.Linear(h, 1)nn.BCEWithLogitsLoss0.0 or 1.0 (float32)
K-class classificationnn.Linear(h, K)nn.CrossEntropyLossclass indices 0…K−1 (int64)
L = −zₜ + ln ∑ⱼ₌₁ᴷ eᶻʲ

Where

z
the model's raw logits
t
index of the true class

nn.CrossEntropyLoss combines log-softmax and cross-entropy: −ln softmax(z)ₜ = −zₜ + ln ∑ eᶻʲ. For a batch, the losses are averaged.

Example 1: the loss from logits

Problem

a) Logits z = (2, 0.5, −1), true class 0. What is the CrossEntropyLoss?
b) Predictions (2.5, 0), true values (3, −0.5). What is the MSELoss?

Show solutionHide solution

Solution

a) e² + √e + e⁻¹ ≈ 7.389 + 1.649 + 0.368 = 9.406; ln 9.406 ≈ 2.2413.
L = −2 + 2.2413 = 0.2413.
b) The errors are −0.5 and 0.5, their squares 0.25 and 0.25; the mean is 0.25.
Python
import torch
from torch import nn

logits = torch.tensor([[2.0, 0.5, -1.0]])
target = torch.tensor([0])
print(round(nn.CrossEntropyLoss()(logits, target).item(), 4))
pred = torch.tensor([2.5, 0.0])
true = torch.tensor([3.0, -0.5])
print(round(nn.MSELoss()(pred, true).item(), 4))

Expected output

0.2413
0.25

An optimizer updates the parameters using their gradients. The simplest is torch.optim.SGD: its step() applies θ ← θ − η · g. The code below repeats the “one step” example from the first lesson (x = 2, y = 6, w = 1, η = 0.1) and gets the same w = 2.6. zero_grad() resets the gradients to None.

Python
import torch

w = torch.tensor([1.0], requires_grad=True)
optimizer = torch.optim.SGD([w], lr=0.1)
loss = ((2 * w - 6) ** 2).sum()
loss.backward()
print(w.grad)
optimizer.step()
print(w)
optimizer.zero_grad()
print(w.grad)

Expected output

tensor([-16.])
tensor([2.6000], requires_grad=True)
None
mₜ = β₁·mₜ₋₁ + (1 − β₁)·gₜvₜ = β₂·vₜ₋₁ + (1 − β₂)·gₜ²θₜ = θₜ₋₁ − η · m̂ₜ / (√v̂ₜ + ε)mₜ = β₁·mₜ₋₁ + (1 − β₁)·gₜvₜ = β₂·vₜ₋₁ + (1 − β₂)·gₜ²θₜ = θₜ₋₁ − η · m̂ₜ / (√v̂ₜ + ε)

Where

gₜ
the gradient at step t
mₜ, vₜ
running averages of the gradient and of its square (momentum and scale)
m̂ₜ, v̂ₜ
bias-corrected values: mₜ/(1 − β₁ᵗ), vₜ/(1 − β₂ᵗ)
β₁, β₂, ε, η
PyTorch defaults 0.9, 0.999, 10⁻⁸, 0.001

The Adam optimizer. Every parameter gets its own step size: a parameter with consistently large gradients is updated cautiously, one with small gradients more boldly. SGD with momentum works as v ← μ·v + g, θ ← θ − η·v.

The training loop

Now to a real task. We generate points shaped like two interleaving half-moons (“two moons”): 500 points per class that cannot be separated by a straight line — so a non-linear model is needed. We split the data into 800 training and 200 test examples.

Python
import math
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader, random_split

torch.manual_seed(42)

def make_moons(n, noise=0.2):
    t = torch.rand(n) * math.pi
    upper = torch.stack([torch.cos(t), torch.sin(t)], dim=1)
    lower = torch.stack([1 - torch.cos(t), 0.5 - torch.sin(t)], dim=1)
    X = torch.cat([upper, lower]) + noise * torch.randn(2 * n, 2)
    y = torch.cat([torch.zeros(n), torch.ones(n)]).long()
    return X, y

X, y = make_moons(500)
train_set, test_set = random_split(TensorDataset(X, y), [800, 200])
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
test_loader = DataLoader(test_set, batch_size=100)
xb, yb = next(iter(train_loader))
print(len(train_set), len(test_set), len(train_loader))
print(xb.shape, yb.shape, yb.dtype)

Expected output

800 200 25
torch.Size([32, 2]) torch.Size([32]) torch.int64
800 / 32 = 25 steps per epoch. Labels are int64 because CrossEntropyLoss expects class indices. The test data is not shuffled.
  1. Zero the gradients

    optimizer.zero_grad() — clear the gradients from the previous step.

  2. Forward pass

    logits = model(xb) — predictions for the batch.

  3. Loss

    loss = loss_fn(logits, yb) — a single scalar.

  4. Backward pass

    loss.backward() — gradients for all parameters.

  5. Step

    optimizer.step() — update the parameters.

Python
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = nn.Sequential(
    nn.Linear(2, 16), nn.ReLU(),
    nn.Linear(16, 16), nn.ReLU(),
    nn.Linear(16, 2),
).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(1, 21):
    model.train()
    total = 0.0
    for xb, yb in train_loader:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        total += loss.item() * len(xb)
    if epoch == 1 or epoch % 5 == 0:
        print(f'epoch {epoch:2d}  loss {total / len(train_set):.4f}')

Expected output

epoch  1  loss 0.4741
epoch  5  loss 0.1200
epoch 10  loss 0.0864
epoch 15  loss 0.0715
epoch 20  loss 0.0823
This continues the previous code. The output was produced with PyTorch 2.14 on a CPU; with another version or on a GPU the numbers may differ slightly.

The loss falls quickly in the first epochs and then hovers around 0.07–0.09. The small rise at epoch 20 is normal: mini-batches are random, so the steps are noisy. Summing loss.item() * len(xb) and dividing by N gives the exact mean loss of the epoch (even if the last batch is smaller).

The batch size B is a hyperparameter too. A large batch estimates the gradient more accurately and keeps the GPU busier, but it needs more memory and gives fewer updates per epoch. A small batch is noisy, yet this noise sometimes helps generalisation. In practice people start with values between 32 and 256; when you double the batch, you can usually raise the learning rate somewhat as well.

Real projects add two more elements to the loop. A learning-rate scheduler lowers η over time: for example torch.optim.lr_scheduler.CosineAnnealingLR, with scheduler.step() called at the end of each epoch. Early stopping measures the validation loss after every epoch, keeps the best model and stops training when the loss has not improved for several epochs — the simplest defence against overfitting.

Evaluation: eval mode and accuracy

Two things matter during evaluation. model.eval() changes how some layers behave: Dropout is switched off and BatchNorm uses the statistics collected during training. torch.no_grad() stops graph building. These are different jobs, and you usually need both together. In classification the main metric is accuracy:

accuracy = (1/N) · ∑ᵢ [argmax(zᵢ) = yᵢ]accuracy = (1/N) · ∑ᵢ [argmax(zᵢ) = yᵢ]

Where

argmax(zᵢ)
the class with the largest logit for example i — the prediction
[ … ]
1 if the condition holds, otherwise 0
Python
def accuracy(model, loader):
    model.eval()
    correct = 0
    with torch.no_grad():
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            correct += (model(xb).argmax(dim=1) == yb).sum().item()
    return correct / len(loader.dataset)

print(f'train accuracy: {accuracy(model, train_loader):.3f}')
print(f'test accuracy:  {accuracy(model, test_loader):.3f}')

Expected output

train accuracy: 0.974
test accuracy:  0.975
Training and test accuracy are close — there is no overfitting. The remaining ≈ 2.5% of errors lie where noise makes the two moons overlap.

Example 2: count the steps and the accuracy

Problem

a) With 800 examples and a batch size of 64, how many steps are there per epoch? How many updates in 20 epochs?
b) 195 of 200 test examples are classified correctly. What is the accuracy?

Show solutionHide solution

Solution

a) 800 / 64 = 12.5 ⇒ ⌈12.5⌉ = 13 steps (the last batch has 32 examples). 20 · 13 = 260 updates.
b) 195 / 200 = 0.975, i.e. 97.5% — exactly our model's result.

Saving and loading a model

model.state_dict() is a dictionary “parameter name → tensor”. The recommended practice is to save this dictionary, not the whole object. To load, you create a model with the same architecture again and call load_state_dict; map_location lets you open a file saved on a GPU on a CPU.

Python
torch.save(model.state_dict(), 'moons.pt')
restored = nn.Sequential(
    nn.Linear(2, 16), nn.ReLU(),
    nn.Linear(16, 16), nn.ReLU(),
    nn.Linear(16, 2),
).to(device)
restored.load_state_dict(torch.load('moons.pt', map_location=device))
print(list(restored.state_dict().keys()))
print(f'restored test accuracy: {accuracy(restored, test_loader):.3f}')

Expected output

['0.weight', '0.bias', '2.weight', '2.bias', '4.weight', '4.bias']
restored test accuracy: 0.975

To pause a long training run and resume it later, you save a checkpoint: torch.save({'epoch': epoch, 'model': model.state_dict(), 'optimizer': optimizer.state_dict()}, 'ckpt.pt'). Adam's momentum statistics live in the optimizer's state_dict too — without them, resuming would be a “cold start”.

Remember

  • Dataset provides examples and DataLoader batches and shuffles them; an epoch has ⌈N / B⌉ steps.
  • Regression uses MSELoss; K-class classification uses CrossEntropyLoss with logits and int64 labels.
  • The loop: zero_grad → forward → loss → backward → step; on a new task Adam (lr = 0.001) is a good start.
  • Evaluate with model.eval() and torch.no_grad(); accuracy = correct predictions / N.
  • Save a model with state_dict and load it into the same architecture with load_state_dict.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

10 questions. Every correct answer earns XP.

Question 1 / 10

Question 1 / 10. What is the correct order of a training step?