Elvin wants to compare headphone prices in several online shops every day, and Leyla wants to collect the announcements from her school’s website into a table. The data is visible in the browser, but there is no API. In such cases a program reads the page’s HTML and extracts the parts it needs — this is called web scraping. In this lesson we learn its main tool, the BeautifulSoup library. The examples read HTML from a string and never touch the network, so they all run right here in the browser. At the end we look at the rules for working with real sites politely and legally.
The HTML tree and the first “soup”
Remember from «How the web works: your first HTML page»: an HTML page is made of nested tags — <head> and <body> inside <html>, headings, paragraphs and lists inside them. A browser and BeautifulSoup turn this text into a tree: every tag is a node with a parent (parent) and children (children), and a tag’s attributes (id, class, href) are kept like a dictionary.
ParserDefinition
A program that reads text and builds the tree. BeautifulSoup(html, 'html.parser') uses Python’s built-in parser — nothing to install. 'lxml' is faster but is a separate package. A parser “forgives” mistakes: it fixes unclosed tags itself, just as a browser does.
from bs4 import BeautifulSoup
html = '''<html><head><title>Educora Shop</title></head>
<body>
<h1 id="top">Stationery</h1>
<p class="intro">Prices are in <b>manat</b>.</p>
</body></html>'''
soup = BeautifulSoup(html, 'html.parser')
print(soup.title)
print(soup.title.string)
print(soup.h1.get_text(), soup.h1['id'])
print(repr(soup.p.string), '|', soup.p.get_text())
print(soup.b.parent.name, soup.b.parent['class'])
print([child.name for child in soup.body.children if child.name])Expected output
<title>Educora Shop</title> Educora Shop Stationery top None | Prices are in manat. p ['intro'] ['h1', 'p']
soup.title is the first <title> tag. .string returns the text only when the tag contains exactly one piece of text; <p> contains text and a <b>, so it gives None. get_text() joins all the inner text. Attributes are read like a dictionary: soup.h1['id']; class is a list, because an element can have several classes.Searching: find, find_all and CSS selectors
find(name, ...) returns the first tag that fits the conditions, or None; find_all(...) returns a list of all matching tags. Conditions: the tag name, class_='item' (with an underscore, because class is a keyword in Python), id='top', attrs={'data-id': '3'} for any attribute, string=... for text and limit=2 for the number of results. A search can start from any tag: li.find('span') looks only inside that li.
from bs4 import BeautifulSoup
html = '''<ul id="products">
<li class="item" data-id="1">Pen <span class="price">1.50</span></li>
<li class="item sale" data-id="2">Notebook <span class="price">3.20</span></li>
<li class="item" data-id="3">Ruler <span class="price">0.80</span></li>
</ul>'''
soup = BeautifulSoup(html, 'html.parser')
items = soup.find_all('li', class_='item')
print(len(items))
for li in items:
name = li.find(string=True).strip()
price = float(li.find('span', class_='price').get_text())
print(li['data-id'], name, price)
sale = soup.find('li', class_='sale')
print(sale['class'], sale.span.string)
print(soup.find('table'), soup.find('li', attrs={'data-id': '3'}).span.string)Expected output
3 1 Pen 1.5 2 Notebook 3.2 3 Ruler 0.8 ['item', 'sale'] 3.20 None 0.80
class_='item' finds every tag whose classes include item — "item sale" too. li.find(string=True) is the tag’s first piece of text ('Pen '), and strip() removes the spaces. soup.find('table') found nothing — None.When chains of find get long, CSS selectors are shorter — the same selectors as in the HTML and CSS course. select(selector) returns all matches as a list, and select_one(selector) the first one or None.
| Selector | What it selects |
|---|---|
li, .price, #catalog | a tag name, a class, an id |
li.item.sale | an li with both classes |
#catalog h2 | an h2 anywhere inside #catalog |
li > a | a direct child |
a[href], a[href^="http"] | has the attribute; the value starts with http |
li:nth-of-type(odd), div:not(.ad) | odd-numbered lis; a div without the ad class |
from bs4 import BeautifulSoup
html = '''<div id="catalog">
<h2>Stationery</h2>
<ul>
<li class="item"><a href="/p/pen">Pen</a> <span class="price">1.50</span></li>
<li class="item sale"><a href="/p/notebook">Notebook</a> <span class="price">3.20</span></li>
<li class="item"><a href="https://other.example/ruler">Ruler</a> <span class="price">0.80</span></li>
</ul>
<a class="next" href="/catalog?page=2">Next page</a>
</div>'''
soup = BeautifulSoup(html, 'html.parser')
print(soup.select_one('#catalog h2').text)
print([a.text for a in soup.select('li.item > a')])
print(soup.select_one('li.sale .price').text)
print([s.text for s in soup.select('li:nth-of-type(odd) span.price')])
print(soup.select_one('a.next')['href'])
print(len(soup.select('a[href^="http"]')), soup.select_one('table'))Expected output
Stationery ['Pen', 'Notebook', 'Ruler'] 3.20 ['1.50', '0.80'] /catalog?page=2 1 None
li:nth-of-type(odd) picks the 1st and 3rd li. a[href^="http"] finds only the link with a full address (the one leading to another site). .text is a short form of get_text().Attributes, links and tables
tag['href'] raises KeyError when the attribute is missing, while tag.get('href') returns None (or the value of its second argument); tag.attrs is the dictionary of all attributes. Links on a page are often relative (/p/pen, notebook.html): combine them with the page’s address using urllib.parse.urljoin — a path starting with / is counted from the site’s root, others from the current folder.
from urllib.parse import urljoin
from bs4 import BeautifulSoup
base = 'https://shop.example/catalog/'
html = '''<a href="/p/pen" title="Blue pen">Pen</a>
<a href="notebook.html">Notebook</a>
<a href="https://other.example/ruler" rel="nofollow">Ruler</a>
<a name="top">Back to top</a>'''
soup = BeautifulSoup(html, 'html.parser')
first = soup.a
print(first.attrs)
print(first.get('title'), first.get('rel'), first.get('rel', []))
for a in soup.find_all('a', href=True):
print(a.text, '->', urljoin(base, a['href']))
try:
soup.find('a', attrs={'name': 'top'})['href']
except KeyError as error:
print('KeyError:', error)Expected output
{'href': '/p/pen', 'title': 'Blue pen'}
Blue pen None []
Pen -> https://shop.example/p/pen
Notebook -> https://shop.example/catalog/notebook.html
Ruler -> https://other.example/ruler
KeyError: 'href'find_all('a', href=True) takes only links that have an href, so “Back to top” is not in the list. Without rel, get('rel') gives None and get('rel', []) an empty list.A table is the most convenient thing to scrape: the first row holds the headers and the rest hold the data. We join the headers and cells with zip and turn every row into a dictionary. Remember: everything that comes from HTML is text — convert it with int() or float() before calculating.
from bs4 import BeautifulSoup
html = '''<table id="scores">
<tr><th>Student</th><th>City</th><th>Score</th></tr>
<tr><td>Aysel</td><td>Bakı</td><td>92</td></tr>
<tr><td>Murad</td><td>Gəncə</td><td>75</td></tr>
<tr><td>Leyla</td><td>Bakı</td><td>88</td></tr>
</table>'''
soup = BeautifulSoup(html, 'html.parser')
rows = soup.select('#scores tr')
headers = [th.get_text(strip=True) for th in rows[0].find_all('th')]
data = []
for tr in rows[1:]:
cells = [td.get_text(strip=True) for td in tr.find_all('td')]
data.append(dict(zip(headers, cells)))
print(headers)
print(data[0])
scores = [int(row['Score']) for row in data]
print(sum(scores) / len(scores))Expected output
['Student', 'City', 'Score']
{'Student': 'Aysel', 'City': 'Bakı', 'Score': '92'}
85.0get_text(strip=True) removes the outer spaces. The average score: (92 + 75 + 88) / 3 = 85.0.For simple tables there is a shorter way: pandas.read_html returns every <table> on the page as a list of DataFrames and converts numbers by itself (see «pandas basics: Series and DataFrame»). We pass the HTML string wrapped in StringIO.
from io import StringIO
import lxml # read_html parses with lxml; importing it lets the browser load it
import pandas as pd
html = '''<table id="scores">
<tr><th>Student</th><th>City</th><th>Score</th></tr>
<tr><td>Aysel</td><td>Bakı</td><td>92</td></tr>
<tr><td>Murad</td><td>Gəncə</td><td>75</td></tr>
<tr><td>Leyla</td><td>Bakı</td><td>88</td></tr>
</table>'''
tables = pd.read_html(StringIO(html))
df = tables[0]
print(len(tables), df.dtypes['Score'])
print(df)
print(df.groupby('City')['Score'].mean())Expected output
1 int64 Student City Score 0 Aysel Bakı 92 1 Murad Gəncə 75 2 Leyla Bakı 88 City Bakı 90.0 Gəncə 75.0 Name: Score, dtype: float64
Score column is already int64. When cells contain merges (colspan) or you need the links, work by hand with BeautifulSoup.Polite and legal scraping
To read a real site you download the HTML with requests (see «Web APIs and the requests library») and pass it to BeautifulSoup. But you are using someone else’s server: a robot sending hundreds of requests a second can slow a site down, enter forbidden sections or collect personal data. The site’s owner writes their wishes in a robots.txt file, which by standard (RFC 9309) lives at the site’s root: https://site/robots.txt. The file is not a technical barrier but an agreement — following it is up to you.
from urllib.robotparser import RobotFileParser
robots_txt = '''User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 5
User-agent: BadBot
Disallow: /
'''
rp = RobotFileParser()
rp.parse(robots_txt.splitlines())
for path in ['/catalog?page=2', '/admin/users', '/search?q=pen', '/searching-tips']:
print(path, rp.can_fetch('EducoraBot', 'https://shop.example' + path))
print(rp.can_fetch('BadBot', 'https://shop.example/catalog'))
print(rp.crawl_delay('EducoraBot'))Expected output
/catalog?page=2 True /admin/users False /search?q=pen False /searching-tips False False 5
urllib.robotparser is in the standard library. Disallow: /search is a prefix: both /search?q=pen and /searching-tips are forbidden. For BadBot the whole site is closed. Crawl-delay is not part of RFC 9309, but many sites use it — wait at least 5 seconds between requests. On a real site: rp.set_url('https://site/robots.txt') and rp.read().- First look for an official API or open data (CSV, JSON) — scraping is the last resort.
- Follow
robots.txtand the site’s terms of use; never try to get around logins, paywalls or CAPTCHAs. - Identify yourself: the bot’s name and a contact address in
User-Agent; pause between requests and do not download the same page again (cache). - If the server answers
429 Too Many Requestsor503, stop and continue later, more slowly. - Do not collect personal data (names, phone numbers, e-mails) and do not republish copyrighted content — national laws regulate both.
Check the source
If there is no API or open data, read
robots.txtand the terms of use.Find the selectors
Use “Inspect” in the browser to learn the tags, classes and attributes of the elements you need.
Download politely
User-Agent,timeout, a pause between requests and a cache.Parse and clean
Extract with BeautifulSoup, check for
None, convert text into numbers and dates.Save
Store the result as CSV, SQLite or a DataFrame, and note the source and the date.
import time
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers['User-Agent'] = 'EducoraBot/1.0 (+mailto:you@example.com)'
def fetch(url, delay=2.0):
response = session.get(url, timeout=10)
response.raise_for_status()
time.sleep(delay)
return response.text
soup = BeautifulSoup(fetch('https://shop.example/catalog.html'), 'html.parser')
print(soup.title.string, [h.text for h in soup.select('.card h3')])run: false — run it on your computer with pip install requests beautifulsoup4 (shop.example is a placeholder; use a site you are allowed to read). Session reuses the connection, timeout prevents hanging, raise_for_status() raises HTTPError on 4xx/5xx, and time.sleep is the polite pause.Mini project: a price table from a catalogue
Let us put it all together. A shop’s catalogue page has product cards; the prices are written in a local format (1,50 ₼), and there is an advertisement card in between. The goal: a clean table, the average price per category and a list of sold-out products. Most real work is exactly this kind of “cleaning”: dropping the currency sign, turning the comma into a point, filtering out the ad.
import pandas as pd
from bs4 import BeautifulSoup
html = '''<div class="card" data-category="Stationery"><h3>Pen</h3>
<span class="price">1,50 ₼</span><span class="stock">In stock</span></div>
<div class="card" data-category="Stationery"><h3>Notebook</h3>
<span class="price">3,20 ₼</span><span class="stock">Sold out</span></div>
<div class="card" data-category="Electronics"><h3>Headphones</h3>
<span class="price">49,90 ₼</span><span class="stock">In stock</span></div>
<div class="card" data-category="Electronics"><h3>USB cable</h3>
<span class="price">7,00 ₼</span><span class="stock">In stock</span></div>
<div class="card ad"><h3>Advertisement</h3></div>'''
def parse_price(text):
return float(text.replace('₼', '').strip().replace(',', '.'))
soup = BeautifulSoup(html, 'html.parser')
rows = []
for card in soup.select('div.card:not(.ad)'):
rows.append({
'name': card.h3.get_text(strip=True),
'category': card['data-category'],
'price': parse_price(card.select_one('.price').text),
'in_stock': card.select_one('.stock').text == 'In stock',
})
df = pd.DataFrame(rows)
print(df)
print(df.groupby('category')['price'].agg(['count', 'mean']).round(2))
print(df.loc[~df['in_stock'], 'name'].tolist())Expected output
name category price in_stock
0 Pen Stationery 1.5 True
1 Notebook Stationery 3.2 False
2 Headphones Electronics 49.9 True
3 USB cable Electronics 7.0 True
count mean
category
Electronics 2 28.45
Stationery 2 2.35
['Notebook']div.card:not(.ad) drops the ad. The average Electronics price is (49.90 + 7.00) / 2 = 28.45, and Stationery (1.50 + 3.20) / 2 = 2.35. ~df['in_stock'] means “not in stock”.Exercise
From a news page take only the articles with the news class (no ads) and print date | title | full link for each. Turn relative links into full addresses with urljoin and base.
from urllib.parse import urljoin
from bs4 import BeautifulSoup
base = 'https://news.example/az/'
html = '''<main>
<article class="news"><h2><a href="/az/sport/1">Sport news</a></h2><time>2026-10-01</time></article>
<article class="news featured"><h2><a href="tech/2">New phone</a></h2><time>2026-10-02</time></article>
<article class="ad"><h2><a href="https://ads.example/x">Buy now!</a></h2></article>
<article class="news"><h2><a href="https://other.example/culture/3">Theatre week</a></h2><time>2026-10-03</time></article>
</main>'''
soup = BeautifulSoup(html, 'html.parser')
# for every article with the class "news": print date | title | full linkExpected output
2026-10-01 | Sport news | https://news.example/az/sport/1 2026-10-02 | New phone | https://news.example/az/tech/2 2026-10-03 | Theatre week | https://other.example/culture/3
Exercise
Read the stock table: print the names of the products that are out of stock and the total value of the stock (the sum of price × quantity, with 2 decimals).
from bs4 import BeautifulSoup
html = '''<table>
<tr><th>Product</th><th>Price</th><th>Stock</th></tr>
<tr><td>Pen</td><td>1.50</td><td>120</td></tr>
<tr><td>Notebook</td><td>3.20</td><td>0</td></tr>
<tr><td>Backpack</td><td>45.00</td><td>7</td></tr>
<tr><td>Ruler</td><td>0.80</td><td>60</td></tr>
</table>'''
soup = BeautifulSoup(html, 'html.parser')
# skip the header row, convert the text to numbers
# print('Out of stock:', ...) and print(f'Total value: {total:.2f}')Expected output
Out of stock: ['Notebook'] Total value: 543.00
Remember
BeautifulSoup(html, 'html.parser')turns HTML into a tree;.stringworks only for a single piece of text, andget_text(strip=True)gives all the text.findreturns the first tag orNone,find_alla list; conditions areclass_,attrs,string,limit; always check forNone.select/select_oneunderstand CSS selectors:li.sale .price,li > a,a[href^="http"],div:not(.ad).tag.get('href')vstag['href']; relative links withurljoin; a table → dictionaries withziporpandas.read_html; convert text to numbers.- Look for an API first; follow
robots.txt(RFC 9309) and the terms, use aUser-Agent, pauses and a cache; do not collect personal data.