Parsing HTML with BeautifulSoup

The HTML tree, `BeautifulSoup(html, 'html.parser')`, `find`/`find_all` and `select` with CSS selectors, attributes and relative links, turning tables into lists and pandas DataFrames, robots.txt and polite, legal scraping — the examples run in the browser.

  • Advanced
  • 22 min
  • 12 questions
  • Lesson 49 of 66

In this lesson you will learn

  • Turn HTML into a tree with BeautifulSoup and work with tags, text and attributes
  • Find elements with find/find_all and CSS selectors (select, select_one)
  • Turn links into full addresses and convert tables into lists of dictionaries and DataFrames
  • Check robots.txt and work with real sites politely and legally

Elvin wants to compare headphone prices in several online shops every day, and Leyla wants to collect the announcements from her school’s website into a table. The data is visible in the browser, but there is no API. In such cases a program reads the page’s HTML and extracts the parts it needs — this is called web scraping. In this lesson we learn its main tool, the BeautifulSoup library. The examples read HTML from a string and never touch the network, so they all run right here in the browser. At the end we look at the rules for working with real sites politely and legally.

The HTML tree and the first “soup”

Remember from «How the web works: your first HTML page»: an HTML page is made of nested tags — <head> and <body> inside <html>, headings, paragraphs and lists inside them. A browser and BeautifulSoup turn this text into a tree: every tag is a node with a parent (parent) and children (children), and a tag’s attributes (id, class, href) are kept like a dictionary.

ParserDefinition

A program that reads text and builds the tree. BeautifulSoup(html, 'html.parser') uses Python’s built-in parser — nothing to install. 'lxml' is faster but is a separate package. A parser “forgives” mistakes: it fixes unclosed tags itself, just as a browser does.

Python
from bs4 import BeautifulSoup

html = '''<html><head><title>Educora Shop</title></head>
<body>
  <h1 id="top">Stationery</h1>
  <p class="intro">Prices are in <b>manat</b>.</p>
</body></html>'''

soup = BeautifulSoup(html, 'html.parser')
print(soup.title)
print(soup.title.string)
print(soup.h1.get_text(), soup.h1['id'])
print(repr(soup.p.string), '|', soup.p.get_text())
print(soup.b.parent.name, soup.b.parent['class'])
print([child.name for child in soup.body.children if child.name])
Expected output
<title>Educora Shop</title>
Educora Shop
Stationery top
None | Prices are in manat.
p ['intro']
['h1', 'p']
soup.title is the first <title> tag. .string returns the text only when the tag contains exactly one piece of text; <p> contains text and a <b>, so it gives None. get_text() joins all the inner text. Attributes are read like a dictionary: soup.h1['id']; class is a list, because an element can have several classes.

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

Searching: find, find_all and CSS selectors

find(name, ...) returns the first tag that fits the conditions, or None; find_all(...) returns a list of all matching tags. Conditions: the tag name, class_='item' (with an underscore, because class is a keyword in Python), id='top', attrs={'data-id': '3'} for any attribute, string=... for text and limit=2 for the number of results. A search can start from any tag: li.find('span') looks only inside that li.

Python
from bs4 import BeautifulSoup

html = '''<ul id="products">
  <li class="item" data-id="1">Pen <span class="price">1.50</span></li>
  <li class="item sale" data-id="2">Notebook <span class="price">3.20</span></li>
  <li class="item" data-id="3">Ruler <span class="price">0.80</span></li>
</ul>'''

soup = BeautifulSoup(html, 'html.parser')
items = soup.find_all('li', class_='item')
print(len(items))
for li in items:
    name = li.find(string=True).strip()
    price = float(li.find('span', class_='price').get_text())
    print(li['data-id'], name, price)
sale = soup.find('li', class_='sale')
print(sale['class'], sale.span.string)
print(soup.find('table'), soup.find('li', attrs={'data-id': '3'}).span.string)
Expected output
3
1 Pen 1.5
2 Notebook 3.2
3 Ruler 0.8
['item', 'sale'] 3.20
None 0.80
class_='item' finds every tag whose classes include item — "item sale" too. li.find(string=True) is the tag’s first piece of text ('Pen '), and strip() removes the spaces. soup.find('table') found nothing — None.

When chains of find get long, CSS selectors are shorter — the same selectors as in the HTML and CSS course. select(selector) returns all matches as a list, and select_one(selector) the first one or None.

SelectorWhat it selects
li, .price, #cataloga tag name, a class, an id
li.item.salean li with both classes
#catalog h2an h2 anywhere inside #catalog
li > aa direct child
a[href], a[href^="http"]has the attribute; the value starts with http
li:nth-of-type(odd), div:not(.ad)odd-numbered lis; a div without the ad class
Python
from bs4 import BeautifulSoup

html = '''<div id="catalog">
  <h2>Stationery</h2>
  <ul>
    <li class="item"><a href="/p/pen">Pen</a> <span class="price">1.50</span></li>
    <li class="item sale"><a href="/p/notebook">Notebook</a> <span class="price">3.20</span></li>
    <li class="item"><a href="https://other.example/ruler">Ruler</a> <span class="price">0.80</span></li>
  </ul>
  <a class="next" href="/catalog?page=2">Next page</a>
</div>'''

soup = BeautifulSoup(html, 'html.parser')
print(soup.select_one('#catalog h2').text)
print([a.text for a in soup.select('li.item > a')])
print(soup.select_one('li.sale .price').text)
print([s.text for s in soup.select('li:nth-of-type(odd) span.price')])
print(soup.select_one('a.next')['href'])
print(len(soup.select('a[href^="http"]')), soup.select_one('table'))
Expected output
Stationery
['Pen', 'Notebook', 'Ruler']
3.20
['1.50', '0.80']
/catalog?page=2
1 None
li:nth-of-type(odd) picks the 1st and 3rd li. a[href^="http"] finds only the link with a full address (the one leading to another site). .text is a short form of get_text().

Attributes, links and tables

tag['href'] raises KeyError when the attribute is missing, while tag.get('href') returns None (or the value of its second argument); tag.attrs is the dictionary of all attributes. Links on a page are often relative (/p/pen, notebook.html): combine them with the page’s address using urllib.parse.urljoin — a path starting with / is counted from the site’s root, others from the current folder.

Python
from urllib.parse import urljoin
from bs4 import BeautifulSoup

base = 'https://shop.example/catalog/'
html = '''<a href="/p/pen" title="Blue pen">Pen</a>
<a href="notebook.html">Notebook</a>
<a href="https://other.example/ruler" rel="nofollow">Ruler</a>
<a name="top">Back to top</a>'''

soup = BeautifulSoup(html, 'html.parser')
first = soup.a
print(first.attrs)
print(first.get('title'), first.get('rel'), first.get('rel', []))
for a in soup.find_all('a', href=True):
    print(a.text, '->', urljoin(base, a['href']))
try:
    soup.find('a', attrs={'name': 'top'})['href']
except KeyError as error:
    print('KeyError:', error)
Expected output
{'href': '/p/pen', 'title': 'Blue pen'}
Blue pen None []
Pen -> https://shop.example/p/pen
Notebook -> https://shop.example/catalog/notebook.html
Ruler -> https://other.example/ruler
KeyError: 'href'
find_all('a', href=True) takes only links that have an href, so “Back to top” is not in the list. Without rel, get('rel') gives None and get('rel', []) an empty list.

A table is the most convenient thing to scrape: the first row holds the headers and the rest hold the data. We join the headers and cells with zip and turn every row into a dictionary. Remember: everything that comes from HTML is text — convert it with int() or float() before calculating.

Python
from bs4 import BeautifulSoup

html = '''<table id="scores">
  <tr><th>Student</th><th>City</th><th>Score</th></tr>
  <tr><td>Aysel</td><td>Bakı</td><td>92</td></tr>
  <tr><td>Murad</td><td>Gəncə</td><td>75</td></tr>
  <tr><td>Leyla</td><td>Bakı</td><td>88</td></tr>
</table>'''

soup = BeautifulSoup(html, 'html.parser')
rows = soup.select('#scores tr')
headers = [th.get_text(strip=True) for th in rows[0].find_all('th')]
data = []
for tr in rows[1:]:
    cells = [td.get_text(strip=True) for td in tr.find_all('td')]
    data.append(dict(zip(headers, cells)))
print(headers)
print(data[0])
scores = [int(row['Score']) for row in data]
print(sum(scores) / len(scores))
Expected output
['Student', 'City', 'Score']
{'Student': 'Aysel', 'City': 'Bakı', 'Score': '92'}
85.0
get_text(strip=True) removes the outer spaces. The average score: (92 + 75 + 88) / 3 = 85.0.

For simple tables there is a shorter way: pandas.read_html returns every <table> on the page as a list of DataFrames and converts numbers by itself (see «pandas basics: Series and DataFrame»). We pass the HTML string wrapped in StringIO.

Python
from io import StringIO
import lxml  # read_html parses with lxml; importing it lets the browser load it
import pandas as pd

html = '''<table id="scores">
  <tr><th>Student</th><th>City</th><th>Score</th></tr>
  <tr><td>Aysel</td><td>Bakı</td><td>92</td></tr>
  <tr><td>Murad</td><td>Gəncə</td><td>75</td></tr>
  <tr><td>Leyla</td><td>Bakı</td><td>88</td></tr>
</table>'''

tables = pd.read_html(StringIO(html))
df = tables[0]
print(len(tables), df.dtypes['Score'])
print(df)
print(df.groupby('City')['Score'].mean())
Expected output
1 int64
  Student   City  Score
0   Aysel   Bakı     92
1   Murad  Gəncə     75
2   Leyla   Bakı     88
City
Bakı     90.0
Gəncə    75.0
Name: Score, dtype: float64
The Score column is already int64. When cells contain merges (colspan) or you need the links, work by hand with BeautifulSoup.

Polite and legal scraping

To read a real site you download the HTML with requests (see «Web APIs and the requests library») and pass it to BeautifulSoup. But you are using someone else’s server: a robot sending hundreds of requests a second can slow a site down, enter forbidden sections or collect personal data. The site’s owner writes their wishes in a robots.txt file, which by standard (RFC 9309) lives at the site’s root: https://site/robots.txt. The file is not a technical barrier but an agreement — following it is up to you.

Python
from urllib.robotparser import RobotFileParser

robots_txt = '''User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 5

User-agent: BadBot
Disallow: /
'''

rp = RobotFileParser()
rp.parse(robots_txt.splitlines())
for path in ['/catalog?page=2', '/admin/users', '/search?q=pen', '/searching-tips']:
    print(path, rp.can_fetch('EducoraBot', 'https://shop.example' + path))
print(rp.can_fetch('BadBot', 'https://shop.example/catalog'))
print(rp.crawl_delay('EducoraBot'))
Expected output
/catalog?page=2 True
/admin/users False
/search?q=pen False
/searching-tips False
False
5
urllib.robotparser is in the standard library. Disallow: /search is a prefix: both /search?q=pen and /searching-tips are forbidden. For BadBot the whole site is closed. Crawl-delay is not part of RFC 9309, but many sites use it — wait at least 5 seconds between requests. On a real site: rp.set_url('https://site/robots.txt') and rp.read().
  • First look for an official API or open data (CSV, JSON) — scraping is the last resort.
  • Follow robots.txt and the site’s terms of use; never try to get around logins, paywalls or CAPTCHAs.
  • Identify yourself: the bot’s name and a contact address in User-Agent; pause between requests and do not download the same page again (cache).
  • If the server answers 429 Too Many Requests or 503, stop and continue later, more slowly.
  • Do not collect personal data (names, phone numbers, e-mails) and do not republish copyrighted content — national laws regulate both.
  1. Check the source

    If there is no API or open data, read robots.txt and the terms of use.

  2. Find the selectors

    Use “Inspect” in the browser to learn the tags, classes and attributes of the elements you need.

  3. Download politely

    User-Agent, timeout, a pause between requests and a cache.

  4. Parse and clean

    Extract with BeautifulSoup, check for None, convert text into numbers and dates.

  5. Save

    Store the result as CSV, SQLite or a DataFrame, and note the source and the date.

Python
import time

import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers['User-Agent'] = 'EducoraBot/1.0 (+mailto:you@example.com)'


def fetch(url, delay=2.0):
    response = session.get(url, timeout=10)
    response.raise_for_status()
    time.sleep(delay)
    return response.text


soup = BeautifulSoup(fetch('https://shop.example/catalog.html'), 'html.parser')
print(soup.title.string, [h.text for h in soup.select('.card h3')])
Real network requests do not work in the browser, so this code is run: false — run it on your computer with pip install requests beautifulsoup4 (shop.example is a placeholder; use a site you are allowed to read). Session reuses the connection, timeout prevents hanging, raise_for_status() raises HTTPError on 4xx/5xx, and time.sleep is the polite pause.

Mini project: a price table from a catalogue

Let us put it all together. A shop’s catalogue page has product cards; the prices are written in a local format (1,50 ₼), and there is an advertisement card in between. The goal: a clean table, the average price per category and a list of sold-out products. Most real work is exactly this kind of “cleaning”: dropping the currency sign, turning the comma into a point, filtering out the ad.

Python
import pandas as pd
from bs4 import BeautifulSoup

html = '''<div class="card" data-category="Stationery"><h3>Pen</h3>
  <span class="price">1,50 ₼</span><span class="stock">In stock</span></div>
<div class="card" data-category="Stationery"><h3>Notebook</h3>
  <span class="price">3,20 ₼</span><span class="stock">Sold out</span></div>
<div class="card" data-category="Electronics"><h3>Headphones</h3>
  <span class="price">49,90 ₼</span><span class="stock">In stock</span></div>
<div class="card" data-category="Electronics"><h3>USB cable</h3>
  <span class="price">7,00 ₼</span><span class="stock">In stock</span></div>
<div class="card ad"><h3>Advertisement</h3></div>'''


def parse_price(text):
    return float(text.replace('₼', '').strip().replace(',', '.'))


soup = BeautifulSoup(html, 'html.parser')
rows = []
for card in soup.select('div.card:not(.ad)'):
    rows.append({
        'name': card.h3.get_text(strip=True),
        'category': card['data-category'],
        'price': parse_price(card.select_one('.price').text),
        'in_stock': card.select_one('.stock').text == 'In stock',
    })

df = pd.DataFrame(rows)
print(df)
print(df.groupby('category')['price'].agg(['count', 'mean']).round(2))
print(df.loc[~df['in_stock'], 'name'].tolist())
Expected output
         name     category  price  in_stock
0         Pen   Stationery    1.5      True
1    Notebook   Stationery    3.2     False
2  Headphones  Electronics   49.9      True
3   USB cable  Electronics    7.0      True
             count   mean
category                 
Electronics      2  28.45
Stationery       2   2.35
['Notebook']
div.card:not(.ad) drops the ad. The average Electronics price is (49.90 + 7.00) / 2 = 28.45, and Stationery (1.50 + 3.20) / 2 = 2.35. ~df['in_stock'] means “not in stock”.

Exercise

From a news page take only the articles with the news class (no ads) and print date | title | full link for each. Turn relative links into full addresses with urljoin and base.

Exercise · Python
from urllib.parse import urljoin
from bs4 import BeautifulSoup

base = 'https://news.example/az/'
html = '''<main>
  <article class="news"><h2><a href="/az/sport/1">Sport news</a></h2><time>2026-10-01</time></article>
  <article class="news featured"><h2><a href="tech/2">New phone</a></h2><time>2026-10-02</time></article>
  <article class="ad"><h2><a href="https://ads.example/x">Buy now!</a></h2></article>
  <article class="news"><h2><a href="https://other.example/culture/3">Theatre week</a></h2><time>2026-10-03</time></article>
</main>'''

soup = BeautifulSoup(html, 'html.parser')
# for every article with the class "news": print date | title | full link
Expected output
2026-10-01 | Sport news | https://news.example/az/sport/1
2026-10-02 | New phone | https://news.example/az/tech/2
2026-10-03 | Theatre week | https://other.example/culture/3

Exercise

Read the stock table: print the names of the products that are out of stock and the total value of the stock (the sum of price × quantity, with 2 decimals).

Exercise · Python
from bs4 import BeautifulSoup

html = '''<table>
  <tr><th>Product</th><th>Price</th><th>Stock</th></tr>
  <tr><td>Pen</td><td>1.50</td><td>120</td></tr>
  <tr><td>Notebook</td><td>3.20</td><td>0</td></tr>
  <tr><td>Backpack</td><td>45.00</td><td>7</td></tr>
  <tr><td>Ruler</td><td>0.80</td><td>60</td></tr>
</table>'''

soup = BeautifulSoup(html, 'html.parser')
# skip the header row, convert the text to numbers
# print('Out of stock:', ...) and print(f'Total value: {total:.2f}')
Expected output
Out of stock: ['Notebook']
Total value: 543.00

Remember

  • BeautifulSoup(html, 'html.parser') turns HTML into a tree; .string works only for a single piece of text, and get_text(strip=True) gives all the text.
  • find returns the first tag or None, find_all a list; conditions are class_, attrs, string, limit; always check for None.
  • select/select_one understand CSS selectors: li.sale .price, li > a, a[href^="http"], div:not(.ad).
  • tag.get('href') vs tag['href']; relative links with urljoin; a table → dictionaries with zip or pandas.read_html; convert text to numbers.
  • Look for an API first; follow robots.txt (RFC 9309) and the terms, use a User-Agent, pauses and a cache; do not collect personal data.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

12 questions. Every correct answer earns XP.

Question 1 / 12

Question 1 / 12. In BeautifulSoup(html, 'html.parser'), what is 'html.parser'?