Embeddings and semantic search

How a model turns words and sentences into vectors of numbers, why the angle between vectors measures similarity of meaning, how semantic search finds nearest neighbours, and where it fails.

  • Advanced
  • 22 min
  • 12 questions
  • Lesson 12 of 16

In this lesson you will learn

  • Explain what an embedding is and why similar meanings get similar vectors
  • Calculate the cosine similarity of two small vectors by hand and with numpy
  • Compare keyword search with semantic search and describe nearest-neighbour search
  • Name the uses and limits of embeddings, including vector databases and multilingual models

Leyla types “how plants make food” into the search box of her school's online library. The search finds nothing useful — or worse, a page of recipes — because the textbook chapter she needs is called “Photosynthesis in green leaves” and does not contain the words she typed. A person would understand her at once. To make a computer search by meaning rather than by letters, we turn texts into lists of numbers called embeddings. They power semantic search, recommendations and the RAG systems of the next lesson.

EmbeddingDefinition

A list of numbers (a vector) that a model produces for a piece of text — a word, a sentence or a paragraph — so that texts with similar meaning get similar vectors. Images and sounds can be embedded too.

Meaning as a list of numbers

Imagine describing every word with a few scores from 0 to 1: how much it is about animals, pets, food and vehicles. “Cat” might get [0.9, 0.8, 0.1, 0.0] and “dog” [0.9, 0.7, 0.1, 0.1] — almost the same list, because the meanings are close. “Bus” gets [0.0, 0.0, 0.0, 0.9] — a completely different direction.

Wordanimalpetfoodvehicle
cat0.90.80.10.0
dog0.90.70.10.1
cow0.90.10.60.0
apple0.00.00.90.0
bus0.00.00.00.9
Hand-made toy vectors: we chose what each dimension means. In the code they appear under their English names.

Real embedding models work the same way on a much larger scale: a neural network trained on huge amounts of text produces vectors with hundreds or thousands of dimensions. Nobody chooses what the dimensions mean, and single dimensions usually cannot be read by people. What matters is the geometry: texts used in similar contexts end up close together. The same model gives a vector for a whole sentence or paragraph, so “how plants make food” and “photosynthesis in green leaves” land near each other although they share no words.

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

Similarity as an angle: cosine similarity

How do we measure “close”? Think of each vector as an arrow starting at the origin. Arrows that point in almost the same direction mean almost the same thing, so we compare the angle between them. The cosine of that angle is easy to calculate from the numbers:

cos θ = (A · B) / (|A| · |B|)cos θ = (A · B) / (|A| · |B|)

Where

A, B
two vectors: (a₁, a₂, …, aₙ) and (b₁, b₂, …, bₙ)
A · B
dot product: a₁b₁ + a₂b₂ + … + aₙbₙ
|A|
length of A: √(a₁² + a₂² + … + aₙ²)
θ
the angle between the vectors

Cosine similarity lies between −1 and 1: 1 means the same direction, 0 unrelated (perpendicular), −1 opposite directions.

Calculate cosine similarity by hand

Problem

1) A = (3, 4), B = (4, 3)
2) A = (1, 0), B = (0, 1)
3) A = (2, 1), B = (−2, −1)

Show solutionHide solution

Solution

1) A · B = 3 · 4 + 4 · 3 = 24; |A| = √(9 + 16) = 5; |B| = 5; cos θ = 24 / 25 = 0.96 — very similar.
2) A · B = 1 · 0 + 0 · 1 = 0, so cos θ = 0 — a 90° angle: unrelated.
3) A · B = −4 − 1 = −5; |A| = |B| = √5; cos θ = −5 / 5 = −1 — opposite directions.

Why the angle and not the distance?

Problem

A = “cat” = (0.9, 0.8, 0.1, 0) and B = 2 · A = (1.8, 1.6, 0.2, 0). Compare the distance between them with their cosine similarity.

Show solutionHide solution

Solution

The distance |B − A| = |A| = √(0.81 + 0.64 + 0.01) = √1.46 ≈ 1.21 — quite large.
But the cosine: A · B = 2 · |A|², |B| = 2 · |A|, so cos θ = 2|A|² / (|A| · 2|A|) = 1 — the same direction.
The length of a vector can depend on things like the length of the text, while the meaning lies in the direction. That is why the angle is the more reliable measure.
Python
import numpy as np

# toy dimensions: animal, pet, food, vehicle (values made up by hand)
words = {
    "cat":   np.array([0.9, 0.8, 0.1, 0.0]),
    "dog":   np.array([0.9, 0.7, 0.1, 0.1]),
    "cow":   np.array([0.9, 0.1, 0.6, 0.0]),
    "apple": np.array([0.0, 0.0, 0.9, 0.0]),
    "bus":   np.array([0.0, 0.0, 0.0, 0.9]),
}

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

for other in ["dog", "cow", "apple", "bus"]:
    print(f"cat ~ {other}: {cosine(words['cat'], words[other]):.3f}")

double = words["cat"] * 2          # same direction, twice as long
print(f"cat ~ 2*cat: {cosine(words['cat'], double):.3f}")
Expected output
cat ~ dog: 0.994
cat ~ cow: 0.724
cat ~ apple: 0.083
cat ~ bus: 0.000
cat ~ 2*cat: 1.000
“cat ~ dog” is almost 1 and “cat ~ bus” exactly 0 (no shared dimension). Doubling a vector does not change the cosine.

Keyword search vs semantic search

Keyword searchSemantic search
Findsdocuments that contain the same wordsdocuments with a similar meaning
Synonyms and paraphrasesmisses them (“car” ≠ “automobile”)finds them
Other languagesonly if the same words are usedpossible with a multilingual model
Exact names, codes, numbersvery precisecan miss or mix them up
Explaining the resulteasy: the matching words are visibleharder: a score from a model
Needsan index of wordsan embedding model and a vector index

In practice many systems use hybrid search: they run both and combine the scores, so that both an exact product code and a question in different words are found.

Nearest-neighbour searchDefinition

Finding the k vectors in a collection that are most similar to a query vector (the k nearest neighbours). Semantic search is nearest-neighbour search over the embeddings of documents.

  1. Embed the documents

    Once, in advance: compute an embedding for every document (or paragraph) and store the vectors.

  2. Embed the query

    With the same model, turn the user's question into a vector.

  3. Compare

    Compute the cosine similarity between the query vector and every document vector.

  4. Rank

    Sort by similarity and keep the top k (for example 3).

  5. Filter and show

    Drop results below a minimum score, then show the titles — or pass the texts to a language model (next lesson).

Python
import numpy as np

# toy dimensions: biology, physics, history, sport, cooking
docs = {
    "Photosynthesis in green leaves": [0.9, 0.2, 0.0, 0.0, 0.1],
    "Healthy food recipes":           [0.2, 0.0, 0.0, 0.1, 0.9],
    "Newton's laws of motion":        [0.0, 0.9, 0.1, 0.2, 0.0],
    "The Great Silk Road":            [0.0, 0.0, 0.9, 0.0, 0.1],
    "Football training plan":         [0.1, 0.2, 0.0, 0.9, 0.0],
}
query = "how plants make food"
query_vec = np.array([0.8, 0.1, 0.0, 0.0, 0.4])   # what a model might return

titles = list(docs)
M = np.array([docs[t] for t in titles])
scores = M @ query_vec / (np.linalg.norm(M, axis=1) * np.linalg.norm(query_vec))

found = [t for t in titles if set(query.split()) & set(t.lower().split())]
print("keyword search:", found)
print("semantic search, top 3:")
for i in np.argsort(-scores)[:3]:
    print(f"  {scores[i]:.3f}  {titles[i]}")
Expected output
keyword search: ['Healthy food recipes']
semantic search, top 3:
  0.935  Photosynthesis in green leaves
  0.623  Healthy food recipes
  0.120  Football training plan
Keyword search finds only the recipe page, because it shares the word “food”. Semantic search puts photosynthesis first. The query vector here is invented; a real system gets it from the embedding model.

Vector databases, many languages, uses and limits

Comparing a query with every vector is fine for a thousand documents but not for a hundred million. Vector databases (or vector extensions of ordinary databases) store embeddings and find neighbours fast with approximate nearest-neighbour (ANN) indexes: they check only promising regions of the space and, in return for a huge speed-up, may occasionally miss a true neighbour.

Multilingual embedding models are trained on many languages at once, so pişik, cat, кошка and kedi get similar vectors. A question in Azerbaijani can then find an English or Russian document. Quality is usually lower for languages with less training data, so test the model on your own language before you rely on it.

  • Search by meaning: help centres, libraries, school notes.
  • Recommendations: “similar articles”, songs or courses close to the ones you liked.
  • Clustering: grouping thousands of survey answers or reviews by topic without reading them all first.
  • Duplicates: finding the same question asked in different words.
  • Retrieval for language models — the RAG method of the next lesson, “Retrieval-augmented generation (RAG)”.

Mini project: a club recommender

The school wants to suggest clubs to students. Every club and every student gets a vector of interests (science, art, sport, technology, languages). Recommending is nearest-neighbour search: normalise all the vectors once, and a single matrix product then gives every cosine similarity at once.

Python
import numpy as np

# toy dimensions: science, art, sport, technology, languages
clubs = {
    "Robotics":      [0.6, 0.1, 0.0, 0.9, 0.0],
    "Chess":         [0.4, 0.1, 0.3, 0.2, 0.0],
    "Drama":         [0.0, 0.9, 0.2, 0.0, 0.5],
    "Football":      [0.0, 0.0, 1.0, 0.0, 0.0],
    "Debate":        [0.1, 0.3, 0.0, 0.0, 0.9],
    "Young chemist": [1.0, 0.1, 0.0, 0.3, 0.0],
}
names = list(clubs)
M = np.array([clubs[n] for n in names], dtype=float)
M = M / np.linalg.norm(M, axis=1, keepdims=True)   # length 1: cosine = dot product

def recommend(likes, k=2):
    v = np.array(likes, dtype=float)
    scores = M @ (v / np.linalg.norm(v))
    best = np.argsort(-scores)[:k]
    return [(names[i], round(float(scores[i]), 2)) for i in best]

print("Aysel:", recommend([0.7, 0.0, 0.1, 0.8, 0.1]))
print("Murad:", recommend([0.0, 0.6, 0.0, 0.0, 0.8]))
print("Elvin:", recommend([0.2, 0.0, 0.9, 0.1, 0.0]))
Expected output
Aysel: [('Robotics', 0.98), ('Young chemist', 0.84)]
Murad: [('Debate', 0.94), ('Drama', 0.9)]
Elvin: [('Football', 0.97), ('Chess', 0.73)]
Aysel likes science and technology → Robotics; Murad likes art and languages → Debate and Drama; Elvin likes sport → Football. With real data an embedding model would create the vectors, for example from the club descriptions and the students' own words.

What could go wrong?

Problem

Name two risks of the club recommender and one fix for each.

Show solutionHide solution

Solution

1) A student only sees clubs similar to what they already like (a “filter bubble”). Fix: always add one random or new club to the list.
2) The vectors can carry stereotypes (for example, suggesting robotics more often to boys). Fix: compare the recommendations for different groups of students — the lesson “Evaluating AI outputs” shows how.

Exercise

Complete the function cosine and print the similarity of a and b and of a and c with 3 decimals. Check the first result by hand: a · b = 1·2 + 2·1 + 2·2 = 8 and |a| = |b| = 3.

Exercise · Python
import numpy as np

a = np.array([1, 2, 2])
b = np.array([2, 1, 2])
c = np.array([2, -1, 0])

def cosine(x, y):
    # dot product divided by the product of the lengths
    return 0

print(f"a ~ b: {cosine(a, b):.3f}")
print(f"a ~ c: {cosine(a, c):.3f}")
Expected output
a ~ b: 0.889
a ~ c: 0.000

Remember

  • An embedding is a vector of numbers for a text; similar meanings get vectors that point in similar directions.
  • Cosine similarity = (A · B) / (|A| · |B|): 1 same direction, 0 unrelated, −1 opposite; for unit vectors it is just the dot product.
  • Semantic search embeds the documents and the query with the same model and returns the nearest neighbours.
  • Keyword search is better for exact names, codes and numbers; hybrid search combines both.
  • Vector databases find neighbours fast with approximate indexes; multilingual models match texts across languages.
  • Similar is not true: embeddings handle negations poorly, mix up exact details and carry the biases of their data.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

12 questions. Every correct answer earns XP.

Question 1 / 12

Question 1 / 12. What is an embedding?