PyTorch: autograd and automatic differentiation

In the first lesson we derived the gradients of linear regression by hand — for just two parameters. Modern neural networks have millions of parameters, large language models billions; deriving each derivative on paper is impossible. PyTorch's autograd does this automatically: it records every operation you perform on tensors and then, with a single command, computes exact derivatives with respect to all parameters. This is neither approximate numerical differences nor symbolic algebra — it is automatic differentiation.

The computational graph

Every operation on a tensor with requires_grad=True adds a node to a directed acyclic graph (DAG). The graph's leaves are the inputs and parameters, and its root is the loss. During the forward pass the graph is built and values are computed; during the backward pass autograd walks from the root to the leaves and applies the chain rule at every node. In neural networks this process is called backpropagation.

In PyTorch the graph is dynamic (define-by-run): it is rebuilt on every forward pass. That is why you can freely use ordinary Python if and for statements inside a model — the graph matches whatever path was actually executed.

requires_grad, backward() and .grad

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x + 1
print(y)
y.backward()
print(x.grad)

tensor(16., grad_fn=<AddBackward0>)
tensor(8.)

• requires_grad=True tells PyTorch: “I will need derivatives with respect to this tensor, record the operations”. Parameters (the weights of nn.Linear and so on) do this automatically.
• grad_fn shows which operation created the tensor; it is its node in the graph.
• y.backward() computes dy/dx and stores it in the leaf's .grad: for y = x² + 2x + 1, dy/dx = 2x + 2 = 8 at x = 3.

When a parameter is a vector, .grad is a vector of the same shape: one partial derivative per element. For example, for f = ∑ vᵢ², ∂f/∂vᵢ = 2vᵢ:

import torch

v = torch.tensor([1.0, -2.0, 3.0], requires_grad=True)
f = (v ** 2).sum()
f.backward()
print(v.grad)

tensor([ 2., -4.,  6.])

The requires_grad flag can also be changed later: p.requires_grad_(False) “freezes” a parameter — its gradient is not computed and it does not change during training. This is used in transfer learning: most layers of a pretrained network are frozen and only the last layer is adapted to a new task. You will see this in the lesson on convolutional networks.

The chain rule

import torch

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)
x, y = torch.tensor(3.0), torch.tensor(10.0)
y_hat = w * x + b
loss = (y_hat - y) ** 2
print(type(loss.grad_fn).__name__, type(y_hat.grad_fn).__name__)
loss.backward()
print(loss.item(), w.grad.item(), b.grad.item())

PowBackward0 AddBackward0
9.0 -18.0 -6.0

import torch

w = torch.tensor(0.0, requires_grad=True)
x = torch.tensor(2.0)
y = torch.sigmoid(w * x)
y.backward()
print(y.item(), w.grad.item())

0.5 0.5

The power of the backward pass is its efficiency: one backward() call computes the gradient of the loss with respect to all parameters at a cost of the same order as the forward pass (a small constant factor). If we computed each parameter separately, a network with a million parameters would need a million forward passes.

Gradients accumulate: zeroing and torch.no_grad()

.backward() does not replace the old gradient — it adds to it. Below, the derivative of y = x³ at x = 2 is 3x² = 12, yet after three calls .grad is 36:

import torch

x = torch.tensor(2.0, requires_grad=True)
for i in range(3):
    y = x ** 3
    y.backward()
    print(x.grad)
x.grad.zero_()
print(x.grad)
with torch.no_grad():
    z = x * 2
print(z.requires_grad, x.detach().requires_grad)

tensor(12.)
tensor(24.)
tensor(36.)
tensor(0.)
False False

• x.grad.zero_() resets the gradient in place (a trailing underscore _ means an in-place operation). In the next lessons optimizer.zero_grad() will do this for us.
• Inside with torch.no_grad(): no graph is built: use it when updating parameters and when evaluating a model (inference); it saves memory and time.
• .detach() returns a tensor that shares the same data but is cut off from the graph — for example, to print a value or convert it to NumPy.

Gradient descent with autograd

Now let us put everything together and fit a linear model to 20 noisy points from the line y = 2x + 1. This time we write no derivative formulas — autograd computes them at every step. The five steps of this loop are the core of all deep learning: forward pass → loss → backward → update inside no_grad → zero the gradients.

import torch

torch.manual_seed(42)
X = torch.linspace(0, 1, 20)
Y = 2 * X + 1 + 0.1 * torch.randn(20)
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
lr = 0.5
for step in range(201):
    loss = ((w * X + b - Y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad.zero_()
    b.grad.zero_()
    if step % 50 == 0:
        print(f'step {step:3d}  loss {loss.item():.4f}')
print(f'w = {w.item():.3f}, b = {b.item():.3f}')

step   0  loss 4.4302
step  50  loss 0.0085
step 100  loss 0.0084
step 150  loss 0.0084
step 200  loss 0.0084
w = 1.907, b = 1.068

Remember this loop well: in the next lessons we will shorten it, but its essence will not change. The step() method of the torch.optim.SGD optimizer performs exactly p -= lr * p.grad inside no_grad for every parameter, and zero_grad() clears the gradients. Layers such as nn.Linear create w and b themselves and switch on requires_grad for you. So the high-level code that looks “magical” is just a convenient wrapper around these five lines.