Murad's class built a helper that sorts messages to the school council into “question” and “complaint”. They tried it three times, it was right three times, and everybody was impressed. Then they ran it on 100 real messages: it got 18 wrong — and almost all of them were complaints that it called questions. Three good-looking answers prove nothing. Evaluation means testing an AI system on purpose, with clear criteria, on many examples chosen in advance. The risks themselves are described in “Limits and risks: hallucinations, bias, privacy, copyright”; this lesson gives you the tools to measure them.
What makes an answer good: rubrics and checklists
“Is this answer good?” is too vague to check. Split it into criteria: correctness (are the facts right?), completeness (is anything important missing?), relevance (does it answer this question?), level and clarity (does it suit the reader?), format (length, structure), safety and fairness (no harmful or biased content) and sources (can the claims be checked?).
RubricDefinition
A table of criteria with a short description of each score level, so that different people — or the same person on different days — score answers the same way. Teachers grade essays with rubrics; you can use one to grade AI answers.
| Criterion | 2 — good | 1 — partly | 0 — poor |
|---|---|---|---|
| Correctness | all facts are right | a minor mistake | a serious mistake |
| Completeness | all key points | one key point missing | most points missing |
| Level | suits the reader | partly too hard or too easy | wrong level |
| Format | as requested | small deviations | ignores the request |
Score an answer with the rubric
Problem
Prompt: “Explain photosynthesis to a grade 7 student in at most 5 sentences.” The AI answer has 7 sentences, uses the words “thylakoid membrane of the chloroplast” without explaining them, says that during photosynthesis plants take in oxygen and release carbon dioxide, and does not mention light.
Show solutionHide solution
Solution
Completeness: 1 — light, the key energy source, is missing.
Level: 1 — unexplained terms are too hard for grade 7.
Format: 1 — 7 sentences instead of at most 5.
Total: 3 / 8. The wrong fact alone makes the answer unusable, however fluent it sounds.
Checking facts, claim by claim
Split
Break the answer into separate claims — one fact per line.
Mark
Mark the claims that need checking: numbers, dates, names, quotations, references, anything surprising.
Find the source
Look for the original source (an official site, a textbook, the original study) — not another AI answer or a copy of a copy.
Label
Label each claim: confirmed, false, or cannot be verified.
Fix or drop
Correct the false claims and remove what cannot be verified.
For important facts, use the two-source rule from “Limits and risks: hallucinations, bias, privacy, copyright”. Remember that an AI that cites a source has not necessarily read it correctly: open the source and find the sentence.
Check three claims
Problem
An AI writes: “1) Baku is the capital of Azerbaijan. 2) The Caspian Sea is the largest lake in the world by area. 3) At sea level, water boils at 90 °C.” Check each claim.
Show solutionHide solution
Solution
2) Confirmed — the Caspian is the largest enclosed inland body of water on Earth by area; geography textbooks often call it the largest lake.
3) False — at normal atmospheric pressure (sea level), water boils at 100 °C. The answer must be corrected before it is used.
Test sets and numbers: accuracy, precision, recall
Test setDefinition
A fixed list of inputs with the correct (expected) answers, prepared before testing. It covers typical, difficult and unusual cases, and it stays the same when you compare two prompts or two models.
With a test set you can count. The simplest measure is accuracy — the share of correct answers. For a system that sorts into two classes (complaint or question, spam or not spam), we also count what kind of mistakes it makes. Call the class we care about “positive” (here: complaint):
| Short name | Meaning |
|---|---|
| TP | true positive: a complaint called a complaint |
| FP | false positive: a question wrongly called a complaint |
| FN | false negative: a complaint missed — called a question |
| TN | true negative: a question called a question |
Where
- A
- accuracy: the share of correct answers among all answers
Counts all mistakes together and does not show what kind they are.
Where
- P
- precision
Of everything the system called a complaint, how much really was one? Low precision means many false alarms.
Where
- R
- recall
Of all the real complaints, how many did the system find? Low recall means many missed cases.
Calculate the metrics
Problem
On 12 test messages the AI got: TP = 4, FP = 2, FN = 1, TN = 5. Calculate accuracy, precision and recall.
Show solutionHide solution
Solution
Precision = 4/(4 + 2) = 4/6 ≈ 0.667 — one in three “complaints” was really a question.
Recall = 4/(4 + 1) = 4/5 = 0.8 — one complaint in five was missed.
The code below gets the same numbers with scikit-learn.
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix
# 12 messages from the test set: 1 = complaint, 0 = question
y_true = [1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1] # what a person decided
y_pred = [1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0] # what the AI said
print(f"accuracy: {accuracy_score(y_true, y_pred):.3f}")
print(f"precision: {precision_score(y_true, y_pred):.3f}")
print(f"recall: {recall_score(y_true, y_pred):.3f}")
print(confusion_matrix(y_true, y_pred))Expected output
accuracy: 0.750 precision: 0.667 recall: 0.800 [[5 2] [1 4]]
confusion_matrix prints [[TN FP] [FN TP]]: 5 questions right, 2 questions called complaints, 1 complaint missed, 4 complaints found.Why accuracy can mislead
Problem
Of 100 messages, only 5 are complaints. A lazy system always answers “question”. Calculate its accuracy and recall.
Show solutionHide solution
Solution
Accuracy = 95/100 = 0.95 — it looks excellent.
Recall = 0/(0 + 5) = 0 — it never finds a single complaint.
When one class is rare, always look at precision and recall, not only at accuracy.
To compare two prompts or two models, run both on the same test set and count. Check consistency too: ask the same question several times and see how often you get the same answer — an assistant that changes its mind half the time cannot be trusted with that question.
expected = ["B", "C", "A", "D", "B", "A", "C", "D", "A", "B"] # answer key
prompt_a = ["B", "C", "A", "B", "B", "A", "D", "D", "A", "C"] # AI with prompt A
prompt_b = ["B", "C", "A", "D", "B", "A", "C", "D", "B", "B"] # AI with prompt B
def accuracy(answers):
return sum(a == e for a, e in zip(answers, expected)) / len(expected)
print(f"prompt A: {accuracy(prompt_a):.0%}")
print(f"prompt B: {accuracy(prompt_b):.0%}")
runs = ["C", "C", "A", "C", "C"] # the same question asked 5 times
most = max(set(runs), key=runs.count)
print(f"consistency: {runs.count(most) / len(runs):.0%} (most common answer: {most})")Expected output
prompt A: 70% prompt B: 90% consistency: 80% (most common answer: C)
Fairness, red-teaming and human review
A system can be accurate on average and still be unfair to a group. There are two simple checks: measure by group — calculate accuracy separately, for example for messages written in Azerbaijani and in Russian, or for different schools; and the swap test — change only a name, gender or dialect in an input and see whether the output changes when it should not. If an essay-feedback tool gives a lower score when only the student's name changes, it is biased.
Find the gap
Problem
An AI marks short answers in a quiz app. On the test set it is right on 45 of 50 answers written in Azerbaijani and on 28 of 40 answers written in Russian. Is there a fairness problem?
Show solutionHide solution
Solution
Overall: 73/90 ≈ 81% — the average hides a 20-point gap.
Yes: students who write in Russian are marked wrongly much more often. Before using the app, find the cause (for example, too few Russian examples in the prompt or the training data) and test again.
Red-teamingDefinition
Testing a system by deliberately trying to make it fail or misbehave — the way an opponent or a curious user would — so that weaknesses are found and fixed before real users meet them.
- Tricky inputs: typos, slang, mixed languages, very long or empty messages.
- False premises: “Why did Nizami Ganjavi write Hamlet?” — a good system corrects the premise instead of inventing an answer.
- Ambiguous questions that need a clarifying question.
- Requests for harmful or private content — the system should refuse.
- Prompt injection hidden in documents or messages (see “AI agents and tool use”).
Numbers do not replace people. In high-stakes uses — health, money, law, grades, safety — a qualified person reviews the AI's output before it is used, and the responsibility stays with that person. Human reviewers also judge what metrics miss: tone, kindness, and whether the answer actually helps.
Trust or verify? Responsible use for students
| Situation | Stakes | What to do |
|---|---|---|
| Ideas for a poster, a title or a joke | low | use freely; your taste is the check |
| An explanation of a school topic | medium | compare with the textbook; ask for an example and test yourself |
| Numbers, dates or quotes for homework or a report | medium to high | check each one in an original source |
| Health, money, law, safety | high | AI is only a starting point; ask a qualified person |
| Facts you cannot check at all | — | do not use them |
Evaluating AI is also part of studying honestly. If you use AI for homework, follow your teacher's rules and say how you used it (see “Studying with AI”). Check before you hand anything in: you are responsible for every fact. And protect privacy when you build test sets: do not paste classmates' names, messages or grades into an AI service — use invented or anonymised examples.
Mini project: test a flashcard generator
Problem
Your class uses an AI prompt that turns notes into flashcards. Evaluate it.
Show solutionHide solution
Solution
2. Rubric: correctness, coverage of the key facts, one fact per card, clear wording (0–2 each, 8 at most).
3. Run: generate cards for all 10 notes with prompt A and prompt B.
4. Score: two classmates score independently, then compare and agree on the differences.
5. Consistency: run 3 notes three times each; do the facts stay the same?
6. Red-team: a note with a wrong fact — does the AI copy it or flag it?
7. Decide: choose the better prompt, write down its known weaknesses and keep the test set for the next version.
Exercise
A spam filter was tested on 100 emails: TP = 18, FP = 6, FN = 2, TN = 74. Calculate precision, recall and accuracy and print each on its own line with 2 decimals, in the format shown in the starter code.
tp, fp, fn, tn = 18, 6, 2, 74
# precision = TP / (TP + FP), recall = TP / (TP + FN),
# accuracy = (TP + TN) / all four
print(f"precision {0:.2f}")
print(f"recall {0:.2f}")
print(f"accuracy {0:.2f}")Expected output
precision 0.75 recall 0.90 accuracy 0.92
Remember
- Judge answers with criteria written in advance: correctness, completeness, level, format, safety, sources.
- Check facts claim by claim in original sources; open a cited source instead of trusting it.
- Accuracy = correct / all; precision = TP/(TP + FP); recall = TP/(TP + FN) — when a class is rare, accuracy alone misleads.
- Compare prompts or models only on the same test set, and check consistency by repeating questions.
- Check fairness by group and with swap tests; red-team only systems you may test; keep a human reviewer when the stakes are high.
- Trust depends on the stakes and on whether you checked; use AI honestly and keep other people's data out of prompts.