Machine learning with scikit-learn

scikit-learn's unified interface: splitting data, linear and logistic regression, decision trees, MSE, R², accuracy, the confusion matrix, precision and recall, cross-validation and pipelines that protect against data leakage.

  • University
  • 25 min
  • 10 questions
  • Lesson 56 of 66

In this lesson you will learn

  • Train any model through the fit, predict, score interface and split data correctly
  • Evaluate regression with MSE, RMSE and R², and classification with accuracy, precision, recall and F1
  • Recognise overfitting in a decision tree and compute Gini impurity
  • Evaluate a model without leakage using cross-validation and a pipeline

In the lesson “Machine learning concepts” we wrote gradient descent by hand. In practice we do not start from scratch every time: the scikit-learn library offers dozens of well-tested models, data preparation tools and metrics — all with the same simple interface. In this lesson you will predict flat prices, classify flowers by species and, most importantly, learn to evaluate a model properly.

One interface: fit, predict, score

The data always has the same form: a feature matrix X (n examples × d features) and a target vector y (length n). Every model (an estimator in scikit-learn) knows three methods: fit(X, y) trains it, predict(X) makes predictions and score(X, y) measures quality. Hyperparameters are passed to the constructor (DecisionTreeClassifier(max_depth=3)), and the names of learned parameters end with an underscore: coef_, intercept_. Objects that change the data (transformers) have fit and transform methods.

ClassTaskIdea
LinearRegressionregressionŷ = w · x + b, minimum squared error
LogisticRegressionclassificationa linear score turned into a probability by a sigmoid (softmax)
DecisionTreeClassifierclassificationa tree of yes / no questions
KNeighborsClassifierclassificationa vote of the k nearest neighbours
StandardScalertransformerturns every feature into a z-score

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

Splitting and linear regression

train_test_split randomly splits the data into a training part and a test part: test_size=0.25 sends a quarter to the test set, random_state makes the split reproducible, and for classification stratify=y keeps the class proportions the same in both parts. Below, synthetic data for 200 flats is generated by the rule y = 1.5 · area + 12 · rooms + 20 + noise (price in thousands of manat); the model must find these coefficients without knowing them.

Python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

rng = np.random.default_rng(0)
area = rng.uniform(40, 140, 200)
rooms = rng.integers(1, 5, 200)
X = np.column_stack([area, rooms])
y = 1.5 * area + 12 * rooms + 20 + rng.normal(0, 10, 200)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
model = LinearRegression().fit(X_train, y_train)
print('coef:', model.coef_.round(2), 'intercept:', round(model.intercept_, 2))
pred = model.predict(X_test)
mse = mean_squared_error(y_test, pred)
print(f'MSE = {mse:.1f}, RMSE = {np.sqrt(mse):.2f}, R2 = {r2_score(y_test, pred):.3f}')
print(X_train.shape, X_test.shape)
print(model.predict([[80, 3]]).round(1))
Expected output
coef: [ 1.48 11.06] intercept: 23.97
MSE = 95.4, RMSE = 9.76, R2 = 0.960
(150, 2) (50, 2)
[175.4]
MSE = (1/n) · ∑ (yᵢ − ŷᵢ)²RMSE = √MSER² = 1 − ∑ (yᵢ − ŷᵢ)² / ∑ (yᵢ − ȳ)²MSE = (1/n) · ∑ (yᵢ − ŷᵢ)²RMSE = √MSER² = 1 − ∑ (yᵢ − ŷᵢ)² / ∑ (yᵢ − ȳ)²

Where

yᵢ, ŷᵢ
true value and prediction
RMSE
typical error in the target's unit (here thousands of manat)
R²
share of variation explained: 1 is perfect, 0 is the level of a model that always predicts the mean

The model found the coefficients 1.48 and 11.06 — close to the true 1.5 and 12. RMSE ≈ 9.8 is the level of the noise we added (σ = 10): the model captured the signal completely, and the noise cannot be predicted.

Classification: logistic regression and decision trees

Logistic regression turns a linear score z = w · x + b into a probability between 0 and 1 with the sigmoid function; for more than two classes its generalisation, softmax, is used. The famous iris dataset (a classic example in statistics since 1936) holds 4 measurements of 150 flowers of 3 species and ships with scikit-learn — no internet needed.

p = σ(z) = 1 / (1 + e^(−z))z = w · x + bp = σ(z) = 1 / (1 + e^(−z))z = w · x + b

Where

σ(z)
sigmoid: maps any number into (0, 1), σ(0) = 0.5
p
probability that the example belongs to the positive class; “1” is predicted when p ≥ 0.5
Python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

X, y = load_iris(return_X_y=True)
print(X.shape, np.bincount(y))
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=9)
clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
pred = clf.predict(X_test)
print('accuracy:', round(accuracy_score(y_test, pred), 3))
print(confusion_matrix(y_test, pred))
print(clf.predict_proba(X_test[:2]).round(2))
Expected output
(150, 4) [50 50 50]
accuracy: 0.956
[[15  0  0]
 [ 0 14  1]
 [ 0  1 14]]
[[0.  0.1 0.9]
 [0.  0.2 0.8]]
In the confusion matrix rows are the true classes and columns the predicted ones: one versicolor was taken for a virginica and one virginica for a versicolor. predict_proba gives the probability of each class.

A decision tree splits the data step by step with questions such as “petal length < 2.45 cm?”. At each step it picks the question that makes the resulting groups as “pure” as possible — purity is measured by Gini impurity. If we do not limit the tree, it keeps growing until it memorises every training example.

G = 1 − ∑ₖ pₖ²Gsplit = (nL / n) · GL + (nR / n) · GRG = 1 − ∑ₖ pₖ²Gsplit = (nL / n) · GL + (nR / n) · GR

Where

pₖ
share of class k in the node
G
0 means a pure node (one class); the maximum for two classes is 0.5
nL, nR
number of examples in the left and right child nodes

Example 1: how good is a split?

Problem

A node has 10 examples: 6 of class A and 4 of class B. A question splits them into a left group (5 A, 0 B) and a right group (1 A, 4 B). Compute the Gini impurity before and after the split.

Show solutionHide solution

Solution

Before: G = 1 − (0.6² + 0.4²) = 1 − (0.36 + 0.16) = 0.48.
Left: only A → GL = 0.
Right: GR = 1 − (0.2² + 0.8²) = 1 − (0.04 + 0.64) = 0.32.
After: Gsplit = 5/10 · 0 + 5/10 · 0.32 = 0.16.
The impurity fell by 0.48 − 0.16 = 0.32; the tree picks, among all possible questions, the one with the largest decrease.
Python
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = make_moons(n_samples=400, noise=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
for depth in [1, 3, 5, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    train_acc = tree.score(X_train, y_train)
    test_acc = tree.score(X_test, y_test)
    print(f'max_depth={str(depth):>4}: train {train_acc:.3f}, test {test_acc:.3f}, leaves {tree.get_n_leaves()}')
Expected output
max_depth=   1: train 0.814, test 0.817, leaves 2
max_depth=   3: train 0.889, test 0.908, leaves 7
max_depth=   5: train 0.907, test 0.892, leaves 15
max_depth=None: train 1.000, test 0.883, leaves 38
make_moons creates noisy synthetic data shaped like two interleaving “half-moons”. Depth 1 underfits; depth 3 gives the best test result; an unlimited tree memorises 100% of the training set but gets worse on the test set.

Metrics beyond accuracy

For binary classification scikit-learn's confusion matrix is laid out like this: the first row is [TN, FP] and the second row is [FN, TP]. Here TP are correctly found positives, FP are false alarms, FN are missed positives and TN are correctly rejected negatives. In a spam filter example, “1” means spam:

Accuracy = (TP + TN) / NPrecision = TP / (TP + FP)Recall = TP / (TP + FN)F1 = 2 · P · R / (P + R)Accuracy = (TP + TN) / NPrecision = TP / (TP + FP)Recall = TP / (TP + FN)F1 = 2 · P · R / (P + R)

Where

Precision
what share of the messages we called spam really are spam
Recall
what share of all spam we managed to catch
F1
the harmonic mean of precision and recall
N
total number of examples
Python
from sklearn.metrics import (accuracy_score, confusion_matrix, f1_score,
                             precision_score, recall_score)

y_true = [1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0]
print(confusion_matrix(y_true, y_pred))
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print('TP, FP, FN, TN =', tp, fp, fn, tn)
print('accuracy :', round(accuracy_score(y_true, y_pred), 3))
print('precision:', round(precision_score(y_true, y_pred), 3))
print('recall   :', round(recall_score(y_true, y_pred), 3))
print('F1       :', round(f1_score(y_true, y_pred), 3))
Expected output
[[5 2]
 [1 4]]
TP, FP, FN, TN = 4 2 1 5
accuracy : 0.75
precision: 0.667
recall   : 0.8
F1       : 0.727

Example 2: the accuracy paradox

Problem

A screening test checks 1000 people, 10 of whom are ill. The model finds 8 of the ill (TP = 8, FN = 2) and gives 40 false alarms (FP = 40, TN = 950). Find the accuracy, precision and recall and compare them with a model that says “everyone is healthy”.

Show solutionHide solution

Solution

Accuracy = (8 + 950) / 1000 = 0.958.
Precision = 8 / (8 + 40) ≈ 0.167 — only one alarm in six is real.
Recall = 8 / (8 + 2) = 0.8 — 80% of the ill are found.
The “everyone is healthy” model: accuracy = 990 / 1000 = 0.99 (higher!), but recall = 0 — not a single ill person is found.
Conclusion: with imbalanced data accuracy is misleading; look at recall, precision and F1.

Cross-validation and pipelines

The result of a single test split depends on luck: another random_state can give another number. k-fold cross-validation splits the data into k parts, trains the model k times — each time one part is the test set — and averages the k scores. The spread of the scores shows how reliable the estimate is.

CV = (1/k) · ∑ᵢ₌₁ᵏ scoreᵢCV = (1/k) · ∑ᵢ₌₁ᵏ scoreᵢ

Where

k
number of folds, usually 5 or 10
scoreᵢ
the score when fold i is the test set
Python
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5)
print(scores.round(3))
print(f'mean = {scores.mean():.3f}, std = {scores.std():.3f}')
Expected output
[0.967 1.    0.933 0.967 1.   ]
mean = 0.973, std = 0.025

A pipeline (Pipeline) combines preparation steps and the model into one estimator. In cross-validation it fits the scaler in each fold on that fold's training part only — no information from the test part leaks into the model. For distance-based KNN scaling is decisive: in the wine dataset one feature ranges from about 11 to 15 and another from 278 to 1680, and the second one completely “swallows” the distance.

Python
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
print(X.shape, X.min(axis=0).round(1)[[0, 12]], X.max(axis=0).round(1)[[0, 12]])
raw = cross_val_score(KNeighborsClassifier(), X, y, cv=5).mean()
pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
scaled = cross_val_score(pipe, X, y, cv=5).mean()
print(f'without scaling: {raw:.3f}')
print(f'with scaling:    {scaled:.3f}')
print([name for name, _ in pipe.steps])
Expected output
(178, 13) [ 11. 278.] [  14.8 1680. ]
without scaling: 0.691
with scaling:    0.949
['standardscaler', 'kneighborsclassifier']

Exercise

Split the iris data with test_size=0.3, stratify=y and random_state=1. Train a DecisionTreeClassifier(max_depth=3, random_state=0). Print the test accuracy (2 decimal places) on the first line, then the confusion matrix.

Exercise · Python
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
# split, train, predict, print accuracy and the confusion matrix
Expected output
0.98
[[15  0  0]
 [ 0 15  0]
 [ 0  1 14]]

Exercise

For the given true and predicted labels, print accuracy, precision, recall and F1 on separate lines, each rounded to 3 decimal places. Then check by hand: how many TP, FP and FN are there?

Exercise · Python
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

y_true = [0, 1, 1, 0, 1, 0, 1, 1, 0, 0]
y_pred = [0, 1, 0, 0, 1, 1, 1, 0, 0, 0]
# print accuracy, precision, recall and F1 (3 decimals each)
Expected output
0.7
0.75
0.6
0.667

Remember

  • All models share one interface: fit, predict, score; learned parameters end with _.
  • train_test_split(..., random_state=..., stratify=y) gives a reproducible, balanced split.
  • Regression: MSE, RMSE, R²; classification: accuracy, precision, recall, F1 and the confusion matrix.
  • Accuracy misleads on imbalanced data; a big gap between training and test scores signals overfitting.
  • Cross-validation gives a more reliable estimate; a pipeline prevents leakage by fitting preparation only on each fold's training part.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

10 questions. Every correct answer earns XP.

Question 1 / 10

Question 1 / 10. A model has training accuracy 1.00 and test accuracy 0.72. What does this mean?