Imagine you must write a program that predicts the price of a flat in Baku from its area, number of rooms and district. Writing all the rules by hand is impossible: conditions like “if the district is Nasimi and the area is over 80 m²...” never end. Machine learning offers another way: we give the computer thousands of examples of flats that were sold, and it learns the rule from the data by itself. This lesson is the mathematical foundation of the whole module: neural networks, CNNs and transformers all rest on it.
Machine learningDefinition
A program learns if its performance on some task, measured by a chosen metric, improves with experience — that is, with data. A model is a function with parameters, and learning means choosing those parameters from data.
Types of learning, features and labels
Each example is described by a vector of features x: for a flat, its area, rooms and floor. The quantity we want to predict is the label y: the price. With n examples and d features, the data is stored as an n × d matrix X and a vector y of length n. If the label is a number, the task is regression; if it is a class (spam / not spam), it is classification.
| Type | Data | Goal | Example |
|---|---|---|---|
| Supervised learning | (x, y) pairs | predict y for a new x | flat prices, spam filters, finding disease on an X-ray |
| Unsupervised learning | only x, no labels | find structure in the data | grouping customers (clustering), dimensionality reduction |
| Reinforcement learning | interaction with an environment and rewards | maximise the total reward | a game-playing agent, a robot learning to walk |
Splitting the data: training, validation and test sets
The real test of a model is data it has never seen. So we shuffle the data randomly and split it into three parts (a typical ratio is 70/15/15 or 80/10/10):
- Training set — the parameters are fitted on this data only.
- Validation set — used to choose hyperparameters (learning rate, model size) and to decide when to stop training.
- Test set — used once, at the very end: an unbiased estimate of real-world performance.
import numpy as np
rng = np.random.default_rng(42)
idx = rng.permutation(10)
train, val, test = idx[:6], idx[6:8], idx[8:]
print('train:', train)
print('val: ', val)
print('test: ', test)Expected output
train: [5 6 0 7 3 2] val: [4 9] test: [1 8]
Loss functions
We need a single number that measures how wrong the model is — this number is given by the loss function L. Learning means searching for the parameters that minimise the loss. Regression usually uses mean squared error; classification uses cross-entropy.
Where
- n
- number of examples
- yᵢ
- true value (label) of example i
- ŷᵢ
- the model's prediction
Mean squared error. Squaring punishes large errors much more; the unit is the square of the label's unit.
Where
- K
- number of classes
- yₖ
- 1 if class k is the correct one, otherwise 0 (one-hot)
- pₖ
- probability the model gives to class k
- pₜ
- probability of the true class
Cross-entropy. For two classes: BCE = −[y · ln p + (1 − y) · ln(1 − p)]. As the probability of the true class approaches 1, the loss approaches 0.
Example 1: computing losses by hand
Problem
a) True values y = (3, 5, 2.5), predictions ŷ = (2.5, 5, 3.5). Find the MSE.
b) An image classifier gives p = (0.7, 0.2, 0.1) for the classes “cat, dog, bird”. Find the cross-entropy if the picture shows a cat, and if it shows a bird.
Show solutionHide solution
Solution
MSE = 1.25 / 3 ≈ 0.4167.
b) Cat: CE = −ln 0.7 ≈ 0.357 — the model was confident and right, so the loss is small.
Bird: CE = −ln 0.1 ≈ 2.303 — only 10% went to the true class, and the loss is more than 6 times larger.
import numpy as np
y = np.array([3.0, 5.0, 2.5])
y_hat = np.array([2.5, 5.0, 3.5])
mse = np.mean((y - y_hat) ** 2)
print(f'MSE = {mse:.4f}')
p = np.array([0.7, 0.2, 0.1])
print(f'CE (true class 0) = {-np.log(p[0]):.4f}')
print(f'CE (true class 2) = {-np.log(p[2]):.4f}')Expected output
MSE = 0.4167 CE (true class 0) = 0.3567 CE (true class 2) = 2.3026
Gradient descent and the learning rate
Imagine walking down a mountain in fog: you cannot see far, but you can feel the slope under your feet. You take a small step in the steepest downhill direction and repeat. The gradient ∇L points in the direction in which the loss grows fastest, so we move against it. The step length is set by the learning rate η.
Where
- θ
- all parameters of the model (weights and biases)
- η
- learning rate, usually 0.0001–0.1
- ∇L(θ)
- gradient: the vector of partial derivatives of the loss with respect to each parameter
The gradient descent update rule. Each repetition is one step; one full pass over the training set is called an epoch.
In linear regression ŷᵢ = w · xᵢ + b and L = MSE. By the chain rule, the derivative of (ŷᵢ − yᵢ)² with respect to w is 2(ŷᵢ − yᵢ) · xᵢ, and with respect to b it is 2(ŷᵢ − yᵢ). Averaging over all examples gives:
Where
- w, b
- slope (weight) and intercept (bias) of the line
- ŷᵢ − yᵢ
- error on example i
Example 2: one step of gradient descent
Problem
Model ŷ = w · x, a single example (x = 2, y = 6), start with w = 1, L = (ŷ − y)². Take one step with η = 0.1 and with η = 0.3 and find the new loss.
Show solutionHide solution
Solution
dL/dw = 2(ŷ − y) · x = 2 · (−4) · 2 = −16.
η = 0.1: w = 1 − 0.1 · (−16) = 2.6; ŷ = 5.2; L = 0.64 — the loss fell 25 times.
η = 0.3: w = 1 + 4.8 = 5.8; ŷ = 11.6; L = 31.36 — we jumped over the minimum (w = 3) and the loss grew!
Here L(w) = 4(w − 3)², whose second derivative is 8; for a quadratic loss, descent converges only if η < 2/8 = 0.25.
Now let us apply the same idea to 50 synthetic points. The data is generated from the line y = 3x + 4 plus noise; the model must find w and b starting from zero, knowing nothing. Run the code — the loss curve is shown automatically.
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(42)
x = rng.uniform(0, 5, size=50)
y = 3 * x + 4 + rng.normal(0, 1, size=50)
w, b, lr = 0.0, 0.0, 0.05
losses = []
for epoch in range(500):
error = (w * x + b) - y
losses.append(np.mean(error ** 2))
dw = 2 * np.mean(error * x)
db = 2 * np.mean(error)
w -= lr * dw
b -= lr * db
if epoch % 100 == 0:
print(f'epoch {epoch:3d} loss {losses[-1]:.4f}')
print(f'w = {w:.3f}, b = {b:.3f}')
plt.plot(losses)
plt.yscale('log')
plt.xlabel('epoch')
plt.ylabel('MSE loss')
plt.show()Expected output
epoch 0 loss 158.9249 epoch 100 loss 0.6019 epoch 200 loss 0.5740 epoch 300 loss 0.5735 epoch 400 loss 0.5735 w = 3.024, b = 3.769
In practice the gradient is not computed on the whole dataset at every step but on a small mini-batch of size B (for example B = 32). This is stochastic gradient descent (SGD): the steps are noisier but much cheaper, and the noise sometimes even helps escape poor local minima.
Overfitting, underfitting and the bias–variance trade-off
A model that is too simple cannot capture the pattern in the data — this is underfitting. A model that is too complex memorises the random noise together with the pattern: its training error is almost zero, but its error on new data is large — this is overfitting. The code below fits polynomials of different degrees to 15 noisy points from sin(2πx):
import numpy as np
from numpy.polynomial import Polynomial
rng = np.random.default_rng(1)
x_train = np.sort(rng.uniform(0, 1, 15))
x_val = np.sort(rng.uniform(0, 1, 15))
f = lambda x: np.sin(2 * np.pi * x)
y_train = f(x_train) + rng.normal(0, 0.2, 15)
y_val = f(x_val) + rng.normal(0, 0.2, 15)
for deg in [1, 3, 9, 12]:
model = Polynomial.fit(x_train, y_train, deg)
train_mse = np.mean((model(x_train) - y_train) ** 2)
val_mse = np.mean((model(x_val) - y_val) ** 2)
print(f'degree {deg:2d}: train {train_mse:.3f}, validation {val_mse:.3f}')Expected output
degree 1: train 0.264, validation 0.235 degree 3: train 0.032, validation 0.037 degree 9: train 0.008, validation 1.894 degree 12: train 0.005, validation 67.532
Where
- f̂(x)
- prediction of a model trained on a random training set
- Bias
- bias: the average error caused by the model's too-simple assumptions
- Var
- variance: how much the prediction depends on the particular training set
- σ²
- irreducible noise in the data
The bias–variance decomposition of the MSE. As a model becomes more complex, bias falls and variance grows; the best model minimises the sum.
| Training loss | Validation loss | Diagnosis | What to do |
|---|---|---|---|
| high | high | underfitting (high bias) | bigger model, more features, train longer |
| low | much higher | overfitting (high variance) | more data, regularisation, early stopping, simpler model |
| low | close to it | good fit | evaluate once on the test set |
Where
- λ
- regularisation strength (a hyperparameter chosen on the validation set)
- ∑ⱼ θⱼ²
- sum of squared weights — penalises large weights
L2 regularisation (weight decay): the model prefers “smoother” functions and is less inclined to memorise noise.
Remember
- Supervised learning learns from (x, y) pairs: regression if the label is a number, classification if it is a class.
- The training set is for parameters, the validation set for hyperparameters; the test set is used only at the very end.
- Regression uses MSE = (1/n)∑(y − ŷ)²; classification uses cross-entropy CE = −ln pₜ.
- Gradient descent: θ ← θ − η · ∇L; a too-large η makes the loss grow, a too-small η makes learning crawl.
- Low training loss with high validation loss means overfitting; both high means underfitting.