We now have all the pieces: tensors, autograd and nn.Module. Let us assemble them into the standard recipe that practitioners write every day: data → DataLoader → model → loss → optimizer → training loop → evaluation → saving. Remarkably, this skeleton is almost the same for a tiny classifier and for a language model with billions of parameters — only the size of the data and the model changes.
Dataset and DataLoader
A Dataset is an object that behaves like a list of examples: __len__ returns the number of examples and __getitem__(i) returns the i-th (x, y) pair. If your data is already in tensors, the ready-made TensorDataset is enough; to read from files (images, texts) you write your own class. A DataLoader builds batches from a Dataset, shuffles the examples every epoch if asked (shuffle=True) and can load data in parallel with num_workers.
import torch
from torch.utils.data import Dataset, DataLoader
class SquaresDataset(Dataset):
def __init__(self, n):
self.x = torch.arange(n, dtype=torch.float32)
def __len__(self):
return len(self.x)
def __getitem__(self, i):
return self.x[i], self.x[i] ** 2
ds = SquaresDataset(10)
print(len(ds), ds[3])
loader = DataLoader(ds, batch_size=4)
for xb, yb in loader:
print(xb.tolist(), yb.tolist())Expected output
10 (tensor(3.), tensor(9.)) [0.0, 1.0, 2.0, 3.0] [0.0, 1.0, 4.0, 9.0] [4.0, 5.0, 6.0, 7.0] [16.0, 25.0, 36.0, 49.0] [8.0, 9.0] [64.0, 81.0]
Batch, step (iteration) and epochDefinition
A batch is a group of examples processed together for one update. A step, or iteration, is one parameter update on one batch. An epoch is one pass in which every training example is “seen” once. Training usually lasts tens of epochs, and every epoch consists of many steps.
Where
- N
- number of training examples
- B
- batch size
- ⌈ ⌉
- rounding up: the last incomplete batch is a step too
Loss functions and optimizers
| Task | Last layer | Loss | Targets |
|---|---|---|---|
| Regression | nn.Linear(h, 1) | nn.MSELoss | float32 numbers |
| Binary classification | nn.Linear(h, 1) | nn.BCEWithLogitsLoss | 0.0 or 1.0 (float32) |
| K-class classification | nn.Linear(h, K) | nn.CrossEntropyLoss | class indices 0…K−1 (int64) |
Where
- z
- the model's raw logits
- t
- index of the true class
nn.CrossEntropyLoss combines log-softmax and cross-entropy: −ln softmax(z)ₜ = −zₜ + ln ∑ eᶻʲ. For a batch, the losses are averaged.
Example 1: the loss from logits
Problem
a) Logits z = (2, 0.5, −1), true class 0. What is the CrossEntropyLoss?
b) Predictions (2.5, 0), true values (3, −0.5). What is the MSELoss?
Show solutionHide solution
Solution
L = −2 + 2.2413 = 0.2413.
b) The errors are −0.5 and 0.5, their squares 0.25 and 0.25; the mean is 0.25.
import torch
from torch import nn
logits = torch.tensor([[2.0, 0.5, -1.0]])
target = torch.tensor([0])
print(round(nn.CrossEntropyLoss()(logits, target).item(), 4))
pred = torch.tensor([2.5, 0.0])
true = torch.tensor([3.0, -0.5])
print(round(nn.MSELoss()(pred, true).item(), 4))Expected output
0.2413 0.25
An optimizer updates the parameters using their gradients. The simplest is torch.optim.SGD: its step() applies θ ← θ − η · g. The code below repeats the “one step” example from the first lesson (x = 2, y = 6, w = 1, η = 0.1) and gets the same w = 2.6. zero_grad() resets the gradients to None.
import torch
w = torch.tensor([1.0], requires_grad=True)
optimizer = torch.optim.SGD([w], lr=0.1)
loss = ((2 * w - 6) ** 2).sum()
loss.backward()
print(w.grad)
optimizer.step()
print(w)
optimizer.zero_grad()
print(w.grad)Expected output
tensor([-16.]) tensor([2.6000], requires_grad=True) None
Where
- gₜ
- the gradient at step t
- mₜ, vₜ
- running averages of the gradient and of its square (momentum and scale)
- m̂ₜ, v̂ₜ
- bias-corrected values: mₜ/(1 − β₁ᵗ), vₜ/(1 − β₂ᵗ)
- β₁, β₂, ε, η
- PyTorch defaults 0.9, 0.999, 10⁻⁸, 0.001
The Adam optimizer. Every parameter gets its own step size: a parameter with consistently large gradients is updated cautiously, one with small gradients more boldly. SGD with momentum works as v ← μ·v + g, θ ← θ − η·v.
The training loop
Now to a real task. We generate points shaped like two interleaving half-moons (“two moons”): 500 points per class that cannot be separated by a straight line — so a non-linear model is needed. We split the data into 800 training and 200 test examples.
import math
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader, random_split
torch.manual_seed(42)
def make_moons(n, noise=0.2):
t = torch.rand(n) * math.pi
upper = torch.stack([torch.cos(t), torch.sin(t)], dim=1)
lower = torch.stack([1 - torch.cos(t), 0.5 - torch.sin(t)], dim=1)
X = torch.cat([upper, lower]) + noise * torch.randn(2 * n, 2)
y = torch.cat([torch.zeros(n), torch.ones(n)]).long()
return X, y
X, y = make_moons(500)
train_set, test_set = random_split(TensorDataset(X, y), [800, 200])
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
test_loader = DataLoader(test_set, batch_size=100)
xb, yb = next(iter(train_loader))
print(len(train_set), len(test_set), len(train_loader))
print(xb.shape, yb.shape, yb.dtype)Expected output
800 200 25 torch.Size([32, 2]) torch.Size([32]) torch.int64
int64 because CrossEntropyLoss expects class indices. The test data is not shuffled.Zero the gradients
optimizer.zero_grad()— clear the gradients from the previous step.Forward pass
logits = model(xb)— predictions for the batch.Loss
loss = loss_fn(logits, yb)— a single scalar.Backward pass
loss.backward()— gradients for all parameters.Step
optimizer.step()— update the parameters.
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = nn.Sequential(
nn.Linear(2, 16), nn.ReLU(),
nn.Linear(16, 16), nn.ReLU(),
nn.Linear(16, 2),
).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(1, 21):
model.train()
total = 0.0
for xb, yb in train_loader:
xb, yb = xb.to(device), yb.to(device)
optimizer.zero_grad()
loss = loss_fn(model(xb), yb)
loss.backward()
optimizer.step()
total += loss.item() * len(xb)
if epoch == 1 or epoch % 5 == 0:
print(f'epoch {epoch:2d} loss {total / len(train_set):.4f}')Expected output
epoch 1 loss 0.4741 epoch 5 loss 0.1200 epoch 10 loss 0.0864 epoch 15 loss 0.0715 epoch 20 loss 0.0823
The loss falls quickly in the first epochs and then hovers around 0.07–0.09. The small rise at epoch 20 is normal: mini-batches are random, so the steps are noisy. Summing loss.item() * len(xb) and dividing by N gives the exact mean loss of the epoch (even if the last batch is smaller).
The batch size B is a hyperparameter too. A large batch estimates the gradient more accurately and keeps the GPU busier, but it needs more memory and gives fewer updates per epoch. A small batch is noisy, yet this noise sometimes helps generalisation. In practice people start with values between 32 and 256; when you double the batch, you can usually raise the learning rate somewhat as well.
Real projects add two more elements to the loop. A learning-rate scheduler lowers η over time: for example torch.optim.lr_scheduler.CosineAnnealingLR, with scheduler.step() called at the end of each epoch. Early stopping measures the validation loss after every epoch, keeps the best model and stops training when the loss has not improved for several epochs — the simplest defence against overfitting.
Evaluation: eval mode and accuracy
Two things matter during evaluation. model.eval() changes how some layers behave: Dropout is switched off and BatchNorm uses the statistics collected during training. torch.no_grad() stops graph building. These are different jobs, and you usually need both together. In classification the main metric is accuracy:
Where
- argmax(zᵢ)
- the class with the largest logit for example i — the prediction
- [ … ]
- 1 if the condition holds, otherwise 0
def accuracy(model, loader):
model.eval()
correct = 0
with torch.no_grad():
for xb, yb in loader:
xb, yb = xb.to(device), yb.to(device)
correct += (model(xb).argmax(dim=1) == yb).sum().item()
return correct / len(loader.dataset)
print(f'train accuracy: {accuracy(model, train_loader):.3f}')
print(f'test accuracy: {accuracy(model, test_loader):.3f}')Expected output
train accuracy: 0.974 test accuracy: 0.975
Example 2: count the steps and the accuracy
Problem
a) With 800 examples and a batch size of 64, how many steps are there per epoch? How many updates in 20 epochs?
b) 195 of 200 test examples are classified correctly. What is the accuracy?
Show solutionHide solution
Solution
b) 195 / 200 = 0.975, i.e. 97.5% — exactly our model's result.
Saving and loading a model
model.state_dict() is a dictionary “parameter name → tensor”. The recommended practice is to save this dictionary, not the whole object. To load, you create a model with the same architecture again and call load_state_dict; map_location lets you open a file saved on a GPU on a CPU.
torch.save(model.state_dict(), 'moons.pt')
restored = nn.Sequential(
nn.Linear(2, 16), nn.ReLU(),
nn.Linear(16, 16), nn.ReLU(),
nn.Linear(16, 2),
).to(device)
restored.load_state_dict(torch.load('moons.pt', map_location=device))
print(list(restored.state_dict().keys()))
print(f'restored test accuracy: {accuracy(restored, test_loader):.3f}')Expected output
['0.weight', '0.bias', '2.weight', '2.bias', '4.weight', '4.bias'] restored test accuracy: 0.975
To pause a long training run and resume it later, you save a checkpoint: torch.save({'epoch': epoch, 'model': model.state_dict(), 'optimizer': optimizer.state_dict()}, 'ckpt.pt'). Adam's momentum statistics live in the optimizer's state_dict too — without them, resuming would be a “cold start”.
Remember
Datasetprovides examples andDataLoaderbatches and shuffles them; an epoch has ⌈N / B⌉ steps.- Regression uses
MSELoss; K-class classification usesCrossEntropyLosswith logits andint64labels. - The loop:
zero_grad→ forward → loss →backward→step; on a new task Adam (lr = 0.001) is a good start. - Evaluate with
model.eval()andtorch.no_grad(); accuracy = correct predictions / N. - Save a model with
state_dictand load it into the same architecture withload_state_dict.