In the lesson “Machine learning concepts” we wrote gradient descent by hand. In practice we do not start from scratch every time: the scikit-learn library offers dozens of well-tested models, data preparation tools and metrics — all with the same simple interface. In this lesson you will predict flat prices, classify flowers by species and, most importantly, learn to evaluate a model properly.
One interface: fit, predict, score
The data always has the same form: a feature matrix X (n examples × d features) and a target vector y (length n). Every model (an estimator in scikit-learn) knows three methods: fit(X, y) trains it, predict(X) makes predictions and score(X, y) measures quality. Hyperparameters are passed to the constructor (DecisionTreeClassifier(max_depth=3)), and the names of learned parameters end with an underscore: coef_, intercept_. Objects that change the data (transformers) have fit and transform methods.
| Class | Task | Idea |
|---|---|---|
LinearRegression | regression | ŷ = w · x + b, minimum squared error |
LogisticRegression | classification | a linear score turned into a probability by a sigmoid (softmax) |
DecisionTreeClassifier | classification | a tree of yes / no questions |
KNeighborsClassifier | classification | a vote of the k nearest neighbours |
StandardScaler | transformer | turns every feature into a z-score |
Splitting and linear regression
train_test_split randomly splits the data into a training part and a test part: test_size=0.25 sends a quarter to the test set, random_state makes the split reproducible, and for classification stratify=y keeps the class proportions the same in both parts. Below, synthetic data for 200 flats is generated by the rule y = 1.5 · area + 12 · rooms + 20 + noise (price in thousands of manat); the model must find these coefficients without knowing them.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
rng = np.random.default_rng(0)
area = rng.uniform(40, 140, 200)
rooms = rng.integers(1, 5, 200)
X = np.column_stack([area, rooms])
y = 1.5 * area + 12 * rooms + 20 + rng.normal(0, 10, 200)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
model = LinearRegression().fit(X_train, y_train)
print('coef:', model.coef_.round(2), 'intercept:', round(model.intercept_, 2))
pred = model.predict(X_test)
mse = mean_squared_error(y_test, pred)
print(f'MSE = {mse:.1f}, RMSE = {np.sqrt(mse):.2f}, R2 = {r2_score(y_test, pred):.3f}')
print(X_train.shape, X_test.shape)
print(model.predict([[80, 3]]).round(1))Expected output
coef: [ 1.48 11.06] intercept: 23.97 MSE = 95.4, RMSE = 9.76, R2 = 0.960 (150, 2) (50, 2) [175.4]
Where
- yᵢ, ŷᵢ
- true value and prediction
- RMSE
- typical error in the target's unit (here thousands of manat)
- R²
- share of variation explained: 1 is perfect, 0 is the level of a model that always predicts the mean
The model found the coefficients 1.48 and 11.06 — close to the true 1.5 and 12. RMSE ≈ 9.8 is the level of the noise we added (σ = 10): the model captured the signal completely, and the noise cannot be predicted.
Classification: logistic regression and decision trees
Logistic regression turns a linear score z = w · x + b into a probability between 0 and 1 with the sigmoid function; for more than two classes its generalisation, softmax, is used. The famous iris dataset (a classic example in statistics since 1936) holds 4 measurements of 150 flowers of 3 species and ships with scikit-learn — no internet needed.
Where
- σ(z)
- sigmoid: maps any number into (0, 1), σ(0) = 0.5
- p
- probability that the example belongs to the positive class; “1” is predicted when p ≥ 0.5
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
X, y = load_iris(return_X_y=True)
print(X.shape, np.bincount(y))
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, stratify=y, random_state=9)
clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
pred = clf.predict(X_test)
print('accuracy:', round(accuracy_score(y_test, pred), 3))
print(confusion_matrix(y_test, pred))
print(clf.predict_proba(X_test[:2]).round(2))Expected output
(150, 4) [50 50 50] accuracy: 0.956 [[15 0 0] [ 0 14 1] [ 0 1 14]] [[0. 0.1 0.9] [0. 0.2 0.8]]
predict_proba gives the probability of each class.A decision tree splits the data step by step with questions such as “petal length < 2.45 cm?”. At each step it picks the question that makes the resulting groups as “pure” as possible — purity is measured by Gini impurity. If we do not limit the tree, it keeps growing until it memorises every training example.
Where
- pₖ
- share of class k in the node
- G
- 0 means a pure node (one class); the maximum for two classes is 0.5
- nL, nR
- number of examples in the left and right child nodes
Example 1: how good is a split?
Problem
A node has 10 examples: 6 of class A and 4 of class B. A question splits them into a left group (5 A, 0 B) and a right group (1 A, 4 B). Compute the Gini impurity before and after the split.
Show solutionHide solution
Solution
Left: only A → GL = 0.
Right: GR = 1 − (0.2² + 0.8²) = 1 − (0.04 + 0.64) = 0.32.
After: Gsplit = 5/10 · 0 + 5/10 · 0.32 = 0.16.
The impurity fell by 0.48 − 0.16 = 0.32; the tree picks, among all possible questions, the one with the largest decrease.
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = make_moons(n_samples=400, noise=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
for depth in [1, 3, 5, None]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
train_acc = tree.score(X_train, y_train)
test_acc = tree.score(X_test, y_test)
print(f'max_depth={str(depth):>4}: train {train_acc:.3f}, test {test_acc:.3f}, leaves {tree.get_n_leaves()}')Expected output
max_depth= 1: train 0.814, test 0.817, leaves 2 max_depth= 3: train 0.889, test 0.908, leaves 7 max_depth= 5: train 0.907, test 0.892, leaves 15 max_depth=None: train 1.000, test 0.883, leaves 38
make_moons creates noisy synthetic data shaped like two interleaving “half-moons”. Depth 1 underfits; depth 3 gives the best test result; an unlimited tree memorises 100% of the training set but gets worse on the test set.Metrics beyond accuracy
For binary classification scikit-learn's confusion matrix is laid out like this: the first row is [TN, FP] and the second row is [FN, TP]. Here TP are correctly found positives, FP are false alarms, FN are missed positives and TN are correctly rejected negatives. In a spam filter example, “1” means spam:
Where
- Precision
- what share of the messages we called spam really are spam
- Recall
- what share of all spam we managed to catch
- F1
- the harmonic mean of precision and recall
- N
- total number of examples
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
precision_score, recall_score)
y_true = [1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0]
print(confusion_matrix(y_true, y_pred))
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print('TP, FP, FN, TN =', tp, fp, fn, tn)
print('accuracy :', round(accuracy_score(y_true, y_pred), 3))
print('precision:', round(precision_score(y_true, y_pred), 3))
print('recall :', round(recall_score(y_true, y_pred), 3))
print('F1 :', round(f1_score(y_true, y_pred), 3))Expected output
[[5 2] [1 4]] TP, FP, FN, TN = 4 2 1 5 accuracy : 0.75 precision: 0.667 recall : 0.8 F1 : 0.727
Example 2: the accuracy paradox
Problem
A screening test checks 1000 people, 10 of whom are ill. The model finds 8 of the ill (TP = 8, FN = 2) and gives 40 false alarms (FP = 40, TN = 950). Find the accuracy, precision and recall and compare them with a model that says “everyone is healthy”.
Show solutionHide solution
Solution
Precision = 8 / (8 + 40) ≈ 0.167 — only one alarm in six is real.
Recall = 8 / (8 + 2) = 0.8 — 80% of the ill are found.
The “everyone is healthy” model: accuracy = 990 / 1000 = 0.99 (higher!), but recall = 0 — not a single ill person is found.
Conclusion: with imbalanced data accuracy is misleading; look at recall, precision and F1.
Cross-validation and pipelines
The result of a single test split depends on luck: another random_state can give another number. k-fold cross-validation splits the data into k parts, trains the model k times — each time one part is the test set — and averages the k scores. The spread of the scores shows how reliable the estimate is.
Where
- k
- number of folds, usually 5 or 10
- scoreᵢ
- the score when fold i is the test set
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5)
print(scores.round(3))
print(f'mean = {scores.mean():.3f}, std = {scores.std():.3f}')Expected output
[0.967 1. 0.933 0.967 1. ] mean = 0.973, std = 0.025
A pipeline (Pipeline) combines preparation steps and the model into one estimator. In cross-validation it fits the scaler in each fold on that fold's training part only — no information from the test part leaks into the model. For distance-based KNN scaling is decisive: in the wine dataset one feature ranges from about 11 to 15 and another from 278 to 1680, and the second one completely “swallows” the distance.
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_wine(return_X_y=True)
print(X.shape, X.min(axis=0).round(1)[[0, 12]], X.max(axis=0).round(1)[[0, 12]])
raw = cross_val_score(KNeighborsClassifier(), X, y, cv=5).mean()
pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
scaled = cross_val_score(pipe, X, y, cv=5).mean()
print(f'without scaling: {raw:.3f}')
print(f'with scaling: {scaled:.3f}')
print([name for name, _ in pipe.steps])Expected output
(178, 13) [ 11. 278.] [ 14.8 1680. ] without scaling: 0.691 with scaling: 0.949 ['standardscaler', 'kneighborsclassifier']
Exercise
Split the iris data with test_size=0.3, stratify=y and random_state=1. Train a DecisionTreeClassifier(max_depth=3, random_state=0). Print the test accuracy (2 decimal places) on the first line, then the confusion matrix.
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_iris(return_X_y=True)
# split, train, predict, print accuracy and the confusion matrixExpected output
0.98 [[15 0 0] [ 0 15 0] [ 0 1 14]]
Exercise
For the given true and predicted labels, print accuracy, precision, recall and F1 on separate lines, each rounded to 3 decimal places. Then check by hand: how many TP, FP and FN are there?
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 0]
y_pred = [0, 1, 0, 0, 1, 1, 1, 0, 0, 0]
# print accuracy, precision, recall and F1 (3 decimals each)Expected output
0.7 0.75 0.6 0.667
Remember
- All models share one interface:
fit,predict,score; learned parameters end with_. train_test_split(..., random_state=..., stratify=y)gives a reproducible, balanced split.- Regression: MSE, RMSE, R²; classification: accuracy, precision, recall, F1 and the confusion matrix.
- Accuracy misleads on imbalanced data; a big gap between training and test scores signals overfitting.
- Cross-validation gives a more reliable estimate; a pipeline prevents leakage by fitting preparation only on each fold's training part.