Evaluating AI outputs

How to judge AI answers systematically: rubrics and checklists, checking facts claim by claim, test sets, accuracy, precision and recall, comparing prompts, consistency, fairness checks, red-teaming and human review — and when to trust and when to verify.

  • Intermediate
  • 22 min
  • 12 questions
  • Lesson 8 of 16

In this lesson you will learn

  • Judge AI answers with a rubric and check facts claim by claim
  • Build a test set and measure accuracy, precision, recall and consistency
  • Compare two prompts or models fairly and check the results for bias
  • Decide when to trust and when to verify, and use AI honestly and privately as a student

Murad's class built a helper that sorts messages to the school council into “question” and “complaint”. They tried it three times, it was right three times, and everybody was impressed. Then they ran it on 100 real messages: it got 18 wrong — and almost all of them were complaints that it called questions. Three good-looking answers prove nothing. Evaluation means testing an AI system on purpose, with clear criteria, on many examples chosen in advance. The risks themselves are described in “Limits and risks: hallucinations, bias, privacy, copyright”; this lesson gives you the tools to measure them.

What makes an answer good: rubrics and checklists

“Is this answer good?” is too vague to check. Split it into criteria: correctness (are the facts right?), completeness (is anything important missing?), relevance (does it answer this question?), level and clarity (does it suit the reader?), format (length, structure), safety and fairness (no harmful or biased content) and sources (can the claims be checked?).

RubricDefinition

A table of criteria with a short description of each score level, so that different people — or the same person on different days — score answers the same way. Teachers grade essays with rubrics; you can use one to grade AI answers.

Criterion2 — good1 — partly0 — poor
Correctnessall facts are righta minor mistakea serious mistake
Completenessall key pointsone key point missingmost points missing
Levelsuits the readerpartly too hard or too easywrong level
Formatas requestedsmall deviationsignores the request
A rubric with 4 criteria: 8 points at most. Write the criteria before you read the answers, so that the answers do not shape the rules.

Score an answer with the rubric

Problem

Prompt: “Explain photosynthesis to a grade 7 student in at most 5 sentences.” The AI answer has 7 sentences, uses the words “thylakoid membrane of the chloroplast” without explaining them, says that during photosynthesis plants take in oxygen and release carbon dioxide, and does not mention light.

Show solutionHide solution

Solution

Correctness: 0 — it swaps the gases: in photosynthesis plants take in CO₂ and release O₂.
Completeness: 1 — light, the key energy source, is missing.
Level: 1 — unexplained terms are too hard for grade 7.
Format: 1 — 7 sentences instead of at most 5.
Total: 3 / 8. The wrong fact alone makes the answer unusable, however fluent it sounds.

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

Checking facts, claim by claim

  1. Split

    Break the answer into separate claims — one fact per line.

  2. Mark

    Mark the claims that need checking: numbers, dates, names, quotations, references, anything surprising.

  3. Find the source

    Look for the original source (an official site, a textbook, the original study) — not another AI answer or a copy of a copy.

  4. Label

    Label each claim: confirmed, false, or cannot be verified.

  5. Fix or drop

    Correct the false claims and remove what cannot be verified.

For important facts, use the two-source rule from “Limits and risks: hallucinations, bias, privacy, copyright”. Remember that an AI that cites a source has not necessarily read it correctly: open the source and find the sentence.

Check three claims

Problem

An AI writes: “1) Baku is the capital of Azerbaijan. 2) The Caspian Sea is the largest lake in the world by area. 3) At sea level, water boils at 90 °C.” Check each claim.

Show solutionHide solution

Solution

1) Confirmed — in any atlas or on the official state portal.
2) Confirmed — the Caspian is the largest enclosed inland body of water on Earth by area; geography textbooks often call it the largest lake.
3) False — at normal atmospheric pressure (sea level), water boils at 100 °C. The answer must be corrected before it is used.

Test sets and numbers: accuracy, precision, recall

Test setDefinition

A fixed list of inputs with the correct (expected) answers, prepared before testing. It covers typical, difficult and unusual cases, and it stays the same when you compare two prompts or two models.

With a test set you can count. The simplest measure is accuracy — the share of correct answers. For a system that sorts into two classes (complaint or question, spam or not spam), we also count what kind of mistakes it makes. Call the class we care about “positive” (here: complaint):

Short nameMeaning
TPtrue positive: a complaint called a complaint
FPfalse positive: a question wrongly called a complaint
FNfalse negative: a complaint missed — called a question
TNtrue negative: a question called a question
A = (TP + TN)/(TP + TN + FP + FN)A = (TP + TN)/(TP + TN + FP + FN)

Where

A
accuracy: the share of correct answers among all answers

Counts all mistakes together and does not show what kind they are.

P = TP/(TP + FP)P = TP/(TP + FP)

Where

P
precision

Of everything the system called a complaint, how much really was one? Low precision means many false alarms.

R = TP/(TP + FN)R = TP/(TP + FN)

Where

R
recall

Of all the real complaints, how many did the system find? Low recall means many missed cases.

Calculate the metrics

Problem

On 12 test messages the AI got: TP = 4, FP = 2, FN = 1, TN = 5. Calculate accuracy, precision and recall.

Show solutionHide solution

Solution

Accuracy = (4 + 5)/12 = 9/12 = 0.75.
Precision = 4/(4 + 2) = 4/6 ≈ 0.667 — one in three “complaints” was really a question.
Recall = 4/(4 + 1) = 4/5 = 0.8 — one complaint in five was missed.
The code below gets the same numbers with scikit-learn.
Python
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix

# 12 messages from the test set: 1 = complaint, 0 = question
y_true = [1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1]   # what a person decided
y_pred = [1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0]   # what the AI said

print(f"accuracy:  {accuracy_score(y_true, y_pred):.3f}")
print(f"precision: {precision_score(y_true, y_pred):.3f}")
print(f"recall:    {recall_score(y_true, y_pred):.3f}")
print(confusion_matrix(y_true, y_pred))
Expected output
accuracy:  0.750
precision: 0.667
recall:    0.800
[[5 2]
 [1 4]]
confusion_matrix prints [[TN FP] [FN TP]]: 5 questions right, 2 questions called complaints, 1 complaint missed, 4 complaints found.

Why accuracy can mislead

Problem

Of 100 messages, only 5 are complaints. A lazy system always answers “question”. Calculate its accuracy and recall.

Show solutionHide solution

Solution

TP = 0, FN = 5, TN = 95, FP = 0.
Accuracy = 95/100 = 0.95 — it looks excellent.
Recall = 0/(0 + 5) = 0 — it never finds a single complaint.
When one class is rare, always look at precision and recall, not only at accuracy.

To compare two prompts or two models, run both on the same test set and count. Check consistency too: ask the same question several times and see how often you get the same answer — an assistant that changes its mind half the time cannot be trusted with that question.

Python
expected = ["B", "C", "A", "D", "B", "A", "C", "D", "A", "B"]   # answer key
prompt_a = ["B", "C", "A", "B", "B", "A", "D", "D", "A", "C"]   # AI with prompt A
prompt_b = ["B", "C", "A", "D", "B", "A", "C", "D", "B", "B"]   # AI with prompt B

def accuracy(answers):
    return sum(a == e for a, e in zip(answers, expected)) / len(expected)

print(f"prompt A: {accuracy(prompt_a):.0%}")
print(f"prompt B: {accuracy(prompt_b):.0%}")

runs = ["C", "C", "A", "C", "C"]          # the same question asked 5 times
most = max(set(runs), key=runs.count)
print(f"consistency: {runs.count(most) / len(runs):.0%} (most common answer: {most})")
Expected output
prompt A: 70%
prompt B: 90%
consistency: 80% (most common answer: C)
Prompt B is better on this test set (9 of 10 against 7 of 10). The repeated question gets the same answer 4 times out of 5: consistency 80%. With only 10 questions, a difference of one answer can be luck — real test sets are larger.

Fairness, red-teaming and human review

A system can be accurate on average and still be unfair to a group. There are two simple checks: measure by group — calculate accuracy separately, for example for messages written in Azerbaijani and in Russian, or for different schools; and the swap test — change only a name, gender or dialect in an input and see whether the output changes when it should not. If an essay-feedback tool gives a lower score when only the student's name changes, it is biased.

Find the gap

Problem

An AI marks short answers in a quiz app. On the test set it is right on 45 of 50 answers written in Azerbaijani and on 28 of 40 answers written in Russian. Is there a fairness problem?

Show solutionHide solution

Solution

Azerbaijani: 45/50 = 90%. Russian: 28/40 = 70%.
Overall: 73/90 ≈ 81% — the average hides a 20-point gap.
Yes: students who write in Russian are marked wrongly much more often. Before using the app, find the cause (for example, too few Russian examples in the prompt or the training data) and test again.

Red-teamingDefinition

Testing a system by deliberately trying to make it fail or misbehave — the way an opponent or a curious user would — so that weaknesses are found and fixed before real users meet them.

  • Tricky inputs: typos, slang, mixed languages, very long or empty messages.
  • False premises: “Why did Nizami Ganjavi write Hamlet?” — a good system corrects the premise instead of inventing an answer.
  • Ambiguous questions that need a clarifying question.
  • Requests for harmful or private content — the system should refuse.
  • Prompt injection hidden in documents or messages (see “AI agents and tool use”).

Numbers do not replace people. In high-stakes uses — health, money, law, grades, safety — a qualified person reviews the AI's output before it is used, and the responsibility stays with that person. Human reviewers also judge what metrics miss: tone, kindness, and whether the answer actually helps.

Trust or verify? Responsible use for students

SituationStakesWhat to do
Ideas for a poster, a title or a jokelowuse freely; your taste is the check
An explanation of a school topicmediumcompare with the textbook; ask for an example and test yourself
Numbers, dates or quotes for homework or a reportmedium to highcheck each one in an original source
Health, money, law, safetyhighAI is only a starting point; ask a qualified person
Facts you cannot check at all—do not use them

Evaluating AI is also part of studying honestly. If you use AI for homework, follow your teacher's rules and say how you used it (see “Studying with AI”). Check before you hand anything in: you are responsible for every fact. And protect privacy when you build test sets: do not paste classmates' names, messages or grades into an AI service — use invented or anonymised examples.

Mini project: test a flashcard generator

Problem

Your class uses an AI prompt that turns notes into flashcards. Evaluate it.

Show solutionHide solution

Solution

1. Test set: 10 short notes from different subjects, one with a tricky fact and one in Russian; for each, the facts a good set of cards must contain.
2. Rubric: correctness, coverage of the key facts, one fact per card, clear wording (0–2 each, 8 at most).
3. Run: generate cards for all 10 notes with prompt A and prompt B.
4. Score: two classmates score independently, then compare and agree on the differences.
5. Consistency: run 3 notes three times each; do the facts stay the same?
6. Red-team: a note with a wrong fact — does the AI copy it or flag it?
7. Decide: choose the better prompt, write down its known weaknesses and keep the test set for the next version.

Exercise

A spam filter was tested on 100 emails: TP = 18, FP = 6, FN = 2, TN = 74. Calculate precision, recall and accuracy and print each on its own line with 2 decimals, in the format shown in the starter code.

Exercise · Python
tp, fp, fn, tn = 18, 6, 2, 74
# precision = TP / (TP + FP), recall = TP / (TP + FN),
# accuracy = (TP + TN) / all four
print(f"precision {0:.2f}")
print(f"recall {0:.2f}")
print(f"accuracy {0:.2f}")
Expected output
precision 0.75
recall 0.90
accuracy 0.92

Remember

  • Judge answers with criteria written in advance: correctness, completeness, level, format, safety, sources.
  • Check facts claim by claim in original sources; open a cited source instead of trusting it.
  • Accuracy = correct / all; precision = TP/(TP + FP); recall = TP/(TP + FN) — when a class is rare, accuracy alone misleads.
  • Compare prompts or models only on the same test set, and check consistency by repeating questions.
  • Check fairness by group and with swap tests; red-team only systems you may test; keep a human reviewer when the stakes are high.
  • Trust depends on the stakes and on whether you checked; use AI honestly and keep other people's data out of prompts.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

12 questions. Every correct answer earns XP.

Question 1 / 12

Question 1 / 12. What is a rubric?