Leyla types “how plants make food” into the search box of her school's online library. The search finds nothing useful — or worse, a page of recipes — because the textbook chapter she needs is called “Photosynthesis in green leaves” and does not contain the words she typed. A person would understand her at once. To make a computer search by meaning rather than by letters, we turn texts into lists of numbers called embeddings. They power semantic search, recommendations and the RAG systems of the next lesson.
EmbeddingDefinition
A list of numbers (a vector) that a model produces for a piece of text — a word, a sentence or a paragraph — so that texts with similar meaning get similar vectors. Images and sounds can be embedded too.
Meaning as a list of numbers
Imagine describing every word with a few scores from 0 to 1: how much it is about animals, pets, food and vehicles. “Cat” might get [0.9, 0.8, 0.1, 0.0] and “dog” [0.9, 0.7, 0.1, 0.1] — almost the same list, because the meanings are close. “Bus” gets [0.0, 0.0, 0.0, 0.9] — a completely different direction.
| Word | animal | pet | food | vehicle |
|---|---|---|---|---|
| cat | 0.9 | 0.8 | 0.1 | 0.0 |
| dog | 0.9 | 0.7 | 0.1 | 0.1 |
| cow | 0.9 | 0.1 | 0.6 | 0.0 |
| apple | 0.0 | 0.0 | 0.9 | 0.0 |
| bus | 0.0 | 0.0 | 0.0 | 0.9 |
Real embedding models work the same way on a much larger scale: a neural network trained on huge amounts of text produces vectors with hundreds or thousands of dimensions. Nobody chooses what the dimensions mean, and single dimensions usually cannot be read by people. What matters is the geometry: texts used in similar contexts end up close together. The same model gives a vector for a whole sentence or paragraph, so “how plants make food” and “photosynthesis in green leaves” land near each other although they share no words.
Similarity as an angle: cosine similarity
How do we measure “close”? Think of each vector as an arrow starting at the origin. Arrows that point in almost the same direction mean almost the same thing, so we compare the angle between them. The cosine of that angle is easy to calculate from the numbers:
Where
- A, B
- two vectors: (a₁, a₂, …, aₙ) and (b₁, b₂, …, bₙ)
- A · B
- dot product: a₁b₁ + a₂b₂ + … + aₙbₙ
- |A|
- length of A: √(a₁² + a₂² + … + aₙ²)
- θ
- the angle between the vectors
Cosine similarity lies between −1 and 1: 1 means the same direction, 0 unrelated (perpendicular), −1 opposite directions.
Calculate cosine similarity by hand
Problem
1) A = (3, 4), B = (4, 3)
2) A = (1, 0), B = (0, 1)
3) A = (2, 1), B = (−2, −1)
Show solutionHide solution
Solution
2) A · B = 1 · 0 + 0 · 1 = 0, so cos θ = 0 — a 90° angle: unrelated.
3) A · B = −4 − 1 = −5; |A| = |B| = √5; cos θ = −5 / 5 = −1 — opposite directions.
Why the angle and not the distance?
Problem
A = “cat” = (0.9, 0.8, 0.1, 0) and B = 2 · A = (1.8, 1.6, 0.2, 0). Compare the distance between them with their cosine similarity.
Show solutionHide solution
Solution
But the cosine: A · B = 2 · |A|², |B| = 2 · |A|, so cos θ = 2|A|² / (|A| · 2|A|) = 1 — the same direction.
The length of a vector can depend on things like the length of the text, while the meaning lies in the direction. That is why the angle is the more reliable measure.
import numpy as np
# toy dimensions: animal, pet, food, vehicle (values made up by hand)
words = {
"cat": np.array([0.9, 0.8, 0.1, 0.0]),
"dog": np.array([0.9, 0.7, 0.1, 0.1]),
"cow": np.array([0.9, 0.1, 0.6, 0.0]),
"apple": np.array([0.0, 0.0, 0.9, 0.0]),
"bus": np.array([0.0, 0.0, 0.0, 0.9]),
}
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
for other in ["dog", "cow", "apple", "bus"]:
print(f"cat ~ {other}: {cosine(words['cat'], words[other]):.3f}")
double = words["cat"] * 2 # same direction, twice as long
print(f"cat ~ 2*cat: {cosine(words['cat'], double):.3f}")Expected output
cat ~ dog: 0.994 cat ~ cow: 0.724 cat ~ apple: 0.083 cat ~ bus: 0.000 cat ~ 2*cat: 1.000
Keyword search vs semantic search
| Keyword search | Semantic search | |
|---|---|---|
| Finds | documents that contain the same words | documents with a similar meaning |
| Synonyms and paraphrases | misses them (“car” ≠ “automobile”) | finds them |
| Other languages | only if the same words are used | possible with a multilingual model |
| Exact names, codes, numbers | very precise | can miss or mix them up |
| Explaining the result | easy: the matching words are visible | harder: a score from a model |
| Needs | an index of words | an embedding model and a vector index |
In practice many systems use hybrid search: they run both and combine the scores, so that both an exact product code and a question in different words are found.
Nearest-neighbour searchDefinition
Finding the k vectors in a collection that are most similar to a query vector (the k nearest neighbours). Semantic search is nearest-neighbour search over the embeddings of documents.
Embed the documents
Once, in advance: compute an embedding for every document (or paragraph) and store the vectors.
Embed the query
With the same model, turn the user's question into a vector.
Compare
Compute the cosine similarity between the query vector and every document vector.
Rank
Sort by similarity and keep the top k (for example 3).
Filter and show
Drop results below a minimum score, then show the titles — or pass the texts to a language model (next lesson).
import numpy as np
# toy dimensions: biology, physics, history, sport, cooking
docs = {
"Photosynthesis in green leaves": [0.9, 0.2, 0.0, 0.0, 0.1],
"Healthy food recipes": [0.2, 0.0, 0.0, 0.1, 0.9],
"Newton's laws of motion": [0.0, 0.9, 0.1, 0.2, 0.0],
"The Great Silk Road": [0.0, 0.0, 0.9, 0.0, 0.1],
"Football training plan": [0.1, 0.2, 0.0, 0.9, 0.0],
}
query = "how plants make food"
query_vec = np.array([0.8, 0.1, 0.0, 0.0, 0.4]) # what a model might return
titles = list(docs)
M = np.array([docs[t] for t in titles])
scores = M @ query_vec / (np.linalg.norm(M, axis=1) * np.linalg.norm(query_vec))
found = [t for t in titles if set(query.split()) & set(t.lower().split())]
print("keyword search:", found)
print("semantic search, top 3:")
for i in np.argsort(-scores)[:3]:
print(f" {scores[i]:.3f} {titles[i]}")Expected output
keyword search: ['Healthy food recipes'] semantic search, top 3: 0.935 Photosynthesis in green leaves 0.623 Healthy food recipes 0.120 Football training plan
Vector databases, many languages, uses and limits
Comparing a query with every vector is fine for a thousand documents but not for a hundred million. Vector databases (or vector extensions of ordinary databases) store embeddings and find neighbours fast with approximate nearest-neighbour (ANN) indexes: they check only promising regions of the space and, in return for a huge speed-up, may occasionally miss a true neighbour.
Multilingual embedding models are trained on many languages at once, so pişik, cat, кошка and kedi get similar vectors. A question in Azerbaijani can then find an English or Russian document. Quality is usually lower for languages with less training data, so test the model on your own language before you rely on it.
- Search by meaning: help centres, libraries, school notes.
- Recommendations: “similar articles”, songs or courses close to the ones you liked.
- Clustering: grouping thousands of survey answers or reviews by topic without reading them all first.
- Duplicates: finding the same question asked in different words.
- Retrieval for language models — the RAG method of the next lesson, “Retrieval-augmented generation (RAG)”.
Mini project: a club recommender
The school wants to suggest clubs to students. Every club and every student gets a vector of interests (science, art, sport, technology, languages). Recommending is nearest-neighbour search: normalise all the vectors once, and a single matrix product then gives every cosine similarity at once.
import numpy as np
# toy dimensions: science, art, sport, technology, languages
clubs = {
"Robotics": [0.6, 0.1, 0.0, 0.9, 0.0],
"Chess": [0.4, 0.1, 0.3, 0.2, 0.0],
"Drama": [0.0, 0.9, 0.2, 0.0, 0.5],
"Football": [0.0, 0.0, 1.0, 0.0, 0.0],
"Debate": [0.1, 0.3, 0.0, 0.0, 0.9],
"Young chemist": [1.0, 0.1, 0.0, 0.3, 0.0],
}
names = list(clubs)
M = np.array([clubs[n] for n in names], dtype=float)
M = M / np.linalg.norm(M, axis=1, keepdims=True) # length 1: cosine = dot product
def recommend(likes, k=2):
v = np.array(likes, dtype=float)
scores = M @ (v / np.linalg.norm(v))
best = np.argsort(-scores)[:k]
return [(names[i], round(float(scores[i]), 2)) for i in best]
print("Aysel:", recommend([0.7, 0.0, 0.1, 0.8, 0.1]))
print("Murad:", recommend([0.0, 0.6, 0.0, 0.0, 0.8]))
print("Elvin:", recommend([0.2, 0.0, 0.9, 0.1, 0.0]))Expected output
Aysel: [('Robotics', 0.98), ('Young chemist', 0.84)]
Murad: [('Debate', 0.94), ('Drama', 0.9)]
Elvin: [('Football', 0.97), ('Chess', 0.73)]What could go wrong?
Problem
Name two risks of the club recommender and one fix for each.
Show solutionHide solution
Solution
2) The vectors can carry stereotypes (for example, suggesting robotics more often to boys). Fix: compare the recommendations for different groups of students — the lesson “Evaluating AI outputs” shows how.
Exercise
Complete the function cosine and print the similarity of a and b and of a and c with 3 decimals. Check the first result by hand: a · b = 1·2 + 2·1 + 2·2 = 8 and |a| = |b| = 3.
import numpy as np
a = np.array([1, 2, 2])
b = np.array([2, 1, 2])
c = np.array([2, -1, 0])
def cosine(x, y):
# dot product divided by the product of the lengths
return 0
print(f"a ~ b: {cosine(a, b):.3f}")
print(f"a ~ c: {cosine(a, c):.3f}")Expected output
a ~ b: 0.889 a ~ c: 0.000
Remember
- An embedding is a vector of numbers for a text; similar meanings get vectors that point in similar directions.
- Cosine similarity = (A · B) / (|A| · |B|): 1 same direction, 0 unrelated, −1 opposite; for unit vectors it is just the dot product.
- Semantic search embeds the documents and the query with the same model and returns the nearest neighbours.
- Keyword search is better for exact names, codes and numbers; hybrid search combines both.
- Vector databases find neighbours fast with approximate indexes; multilingual models match texts across languages.
- Similar is not true: embeddings handle negations poorly, mix up exact details and carry the biases of their data.