Packages, modules and virtual environments

One of your projects needs Django 4, another needs Django 5 — install the package for the whole computer, and one of them will break. A teammate runs your code and gets ModuleNotFoundError, because the libraries you installed are not on their machine. Real projects follow three rules: the code is split into modules and packages, every project gets its own virtual environment, and the dependencies are recorded in a file together with their versions.

Modules and packages

A module is any .py file. When you write import pricing, Python searches the folders listed in sys.path, runs the file's top-level code once and stores the module object in sys.modules — later imports reuse that object. A package is a folder of modules; a regular package contains an __init__.py file, which runs when the package is imported for the first time. Names are dotted: shop.pricing is the module pricing inside the package shop. A typical project looks like this:

shop/
├── pyproject.toml
├── requirements.txt
├── .venv/                 (not in git)
├── src/
│   └── shop/
│       ├── __init__.py
│       ├── pricing.py
│       └── cli.py
└── tests/
    └── test_pricing.py

The example below builds a tiny package in a temporary folder and imports it in three different ways (the sys.modules.pop lines are there only so that the example can be run again):

import importlib
import sys
import tempfile
from pathlib import Path

root = Path(tempfile.mkdtemp())
(root / 'shop').mkdir()
(root / 'shop' / '__init__.py').write_text('from .pricing import final_price\n')
(root / 'shop' / 'pricing.py').write_text(
    'VAT = 0.18\n'
    'print("running pricing.py as", __name__)\n'
    'def final_price(net):\n'
    '    return round(net * (1 + VAT), 2)\n'
)
sys.path.insert(0, str(root))
sys.modules.pop('shop', None)
sys.modules.pop('shop.pricing', None)
importlib.invalidate_caches()

import shop
import shop.pricing
from shop import final_price
print(final_price(100), shop.pricing.VAT)
print(__name__)

running pricing.py as shop.pricing
118.0 0.18
__main__

pricing.py ran only once despite three imports. Inside the module, __name__ is its full name (shop.pricing), while in the file that was started directly it is '__main__'. The line from .pricing import final_price in __init__.py is a relative import: the dot means “this package”. Thanks to it, users can write from shop import final_price without knowing the internal layout.

if __name__ == '__main__': script or module

A file can be both an importable module and a runnable script. The condition if __name__ == '__main__': runs the script code only when the file is started directly, not when it is imported. The command python -m shop.cli finds the module as part of its package and runs it as a script, so imports inside the package keep working:

import sys
from shop.pricing import final_price

def main() -> int:
    for arg in sys.argv[1:]:
        print(f'{arg} -> {final_price(float(arg))}')
    return 0

if __name__ == '__main__':
    sys.exit(main())

$ python -m shop.cli 10 25.5
10 -> 11.8
25.5 -> 30.09

Virtual environments: venv and pip

Without a virtual environment, all projects share one site-packages folder and package versions clash. A virtual environment is a folder inside the project (usually .venv) with its own Python interpreter and its own packages. The standard venv module creates it, and pip installs packages into it:

1. 1
   Create

   In the project folder: python -m venv .venv.

2. 2
   Activate

   Windows: .venv\Scripts\activate; macOS and Linux: source .venv/bin/activate. The prompt now starts with (.venv).

3. 3
   Install

   python -m pip install requests — the package goes into this environment only.

4. 4
   Record

   python -m pip freeze > requirements.txt writes the exact versions of the installed packages to a file.

5. 5
   Reproduce

   A teammate creates their own environment and runs python -m pip install -r requirements.txt.

6. 6
   Leave

   The command deactivate returns you to the normal prompt.

$ python -m venv .venv
$ source .venv/bin/activate
(.venv) $ python -m pip list
Package Version
------- -------
pip     25.2
(.venv) $ python -m pip install requests
Collecting requests
...
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2024.8.30 charset-normalizer-3.4.0 idna-3.10 requests-2.32.3 urllib3-2.2.3
(.venv) $ python -m pip freeze
certifi==2024.8.30
charset-normalizer==3.4.0
idna==3.10
requests==2.32.3
urllib3==2.2.3

requirements.txt and pyproject.toml

requirements.txt is a simple list: one package per line, optionally with a version specifier. For applications it is common to pin exact versions (==), so that the program runs with the same libraries on every computer:

# requirements.txt
requests==2.32.3
fastapi>=0.110,<1.0
python-dotenv~=1.0

The main file of a modern project is **pyproject.toml**: it holds the name, the version, the supported Python versions, the dependencies, tools needed only for development, and command-line scripts. Running python -m pip install -e . in the project folder installs the package in “editable” mode: you do not need to reinstall after every change, and the shop command from [project.scripts] works right away.

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "shop"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["requests>=2.31"]

[project.optional-dependencies]
dev = ["pytest>=8"]

[project.scripts]
shop = "shop.cli:main"

import re

requirements = '''
# web
requests==2.32.3
fastapi>=0.110,<1.0

python-dotenv~=1.0
numpy
'''

def parse_requirements(text):
    result = {}
    # go through the lines, skip blanks and comments,
    # split each line into the package name and the specifier
    return result

for name, spec in parse_requirements(requirements).items():
    print(f'{name}: {spec}')

requests: ==2.32.3
fastapi: >=0.110,<1.0
python-dotenv: ~=1.0
numpy: any version

versions = ['0.10.0', '0.9.3', '1.0.0', '0.1.0', '0.9.12']

def version_key(v):
    # turn '0.10.0' into something that compares correctly
    return v

print(sorted(versions))
print(sorted(versions, key=version_key))
print('latest:', max(versions, key=version_key))

['0.1.0', '0.10.0', '0.9.12', '0.9.3', '1.0.0']
['0.1.0', '0.9.3', '0.9.12', '0.10.0', '1.0.0']
latest: 1.0.0