Decorators and closures

You have twenty functions and want each of them to log its calls, retry after a network error or remember results it has already computed. Copying the same lines into twenty places is a bad idea. Python's answer is the decorator: a single line like @lru_cache above a function adds new behaviour without touching its code. Frameworks such as Flask, FastAPI and pytest are built on decorators. To write your own, you need two ideas: functions are objects, and inner functions can remember their surroundings — these are closures.

Functions are objects

In Python a function is an ordinary object, just like a number or a list. You can assign it to another name, store it in a dictionary, pass it to another function and return it from a function. That is why map(square, ...) works: we pass the function itself without calling it (no parentheses).

def square(x):
    return x * x

f = square
print(f(5), f.__name__)

operations = {'double': lambda x: x * 2, 'square': square}
for name, op in operations.items():
    print(name, op(7))

print(list(map(square, [1, 2, 3])))

25 square
double 14
square 49
[1, 4, 9]

Because a function is an object, it also has attributes: __name__, __doc__ and __defaults__, where the default values of its parameters are stored. These values are evaluated once, when def runs, not on every call. This explains one of Python's most famous traps:

def add_item(item, basket=[]):
    basket.append(item)
    return basket

print(add_item('apple'))
print(add_item('pear'))
print(add_item.__defaults__)

def add_item_fixed(item, basket=None):
    if basket is None:
        basket = []
    basket.append(item)
    return basket

print(add_item_fixed('apple'), add_item_fixed('pear'))

['apple']
['apple', 'pear']
(['apple', 'pear'],)
['apple'] ['pear']

Closures and nonlocal

Python looks up a name in four scopes, in the order LEGB: Local (the current function), Enclosing (the outer functions), Global (the module) and Built-in (names like len and print). So an inner function can read the variables of the function around it. Remarkably, it keeps them even after the outer function has finished:

def make_multiplier(factor):
    def multiply(x):
        return x * factor
    return multiply

double = make_multiplier(2)
triple = make_multiplier(3)
print(double(10), triple(10))
print(double.__closure__[0].cell_contents)

20 30
2

Reading an enclosing variable is free, but assigning to it makes Python treat the name as a new local variable, and count += 1 would fail with UnboundLocalError. The keyword **nonlocal** says that the name belongs to the enclosing function. Each call of make_counter() creates a separate, private count:

def make_counter():
    count = 0
    def increment():
        nonlocal count
        count += 1
        return count
    return increment

counter = make_counter()
print(counter(), counter(), counter())
other = make_counter()
print(other())

1 2 3
1

funcs = [lambda: i for i in range(3)]
print([f() for f in funcs])

funcs = [lambda i=i: i for i in range(3)]
print([f() for f in funcs])

[2, 2, 2]
[0, 1, 2]

Decorators

A decorator is a function that takes a function and returns a new one. Usually this is a wrapper that does something before and after calling the original. The line @log_calls above def add is only syntactic sugar for add = log_calls(add). The wrapper accepts *args, **kwargs, so it works with any signature:

def log_calls(func):
    def wrapper(*args, **kwargs):
        print(f'calling {func.__name__}{args}')
        result = func(*args, **kwargs)
        print(f'{func.__name__} returned {result}')
        return result
    return wrapper

@log_calls
def add(a, b):
    return a + b

total = add(2, 3)
print(total, add.__name__)

calling add(2, 3)
add returned 5
5 wrapper

Look at the last line: the function is now called wrapper. The decorator replaced add and lost its name and docstring, which spoils debugging, documentation and tools such as pytest. The fix is the decorator **functools.wraps**: it copies __name__, __doc__ and other attributes of the original function onto the wrapper. A wrapper can also keep its own state in an attribute:

from functools import wraps

def count_calls(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)
    wrapper.calls = 0
    return wrapper

@count_calls
def greet(name):
    """Return a greeting."""
    return f'Salam, {name}!'

print(greet('Leyla'))
print(greet('Elvin'))
print(greet.calls, greet.__name__, greet.__doc__)

Salam, Leyla!
Salam, Elvin!
2 greet Return a greeting.

Decorators with arguments

So how does @retry(times=3) work? Here retry(times=3) is called first, and its result is the real decorator. A decorator with arguments therefore has three levels: a factory that takes the arguments, the decorator that takes the function, and the wrapper that runs on every call. Thanks to closures, the wrapper still sees both times and func:

from functools import wraps

def retry(times):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except ConnectionError as e:
                    print(f'attempt {attempt} failed: {e}')
            raise ConnectionError(f'{func.__name__} failed {times} times')
        return wrapper
    return decorator

calls = 0
@retry(times=3)
def fetch_rates():
    global calls
    calls += 1
    if calls < 3:
        raise ConnectionError('server busy')
    return {'USD': 1.7}

print(fetch_rates())

attempt 1 failed: server busy
attempt 2 failed: server busy
{'USD': 1.7}

Decorators can also be stacked. They are applied from the bottom up — the decorator closest to def wraps the function first:

def bold(func):
    return lambda: '<b>' + func() + '</b>'

def italic(func):
    return lambda: '<i>' + func() + '</i>'

@bold
@italic
def title():
    return 'Educora'

print(title())

<b><i>Educora</i></b>

Ready-made decorators: lru_cache

The standard library has many useful decorators. **functools.lru_cache stores a function's results in a dictionary keyed by its arguments — this is called memoization**. A naive recursive fib(80) would make about 7.6 · 10¹⁶ calls; with the cache every value is computed only once:

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(80))
print(fib.cache_info())

23416728348467685
CacheInfo(hits=78, misses=81, maxsize=None, currsize=81)

cache_info() shows that there were only 81 real calculations (misses); the other 78 requests were answered from the cache (hits). maxsize=None means an unlimited cache; with a number such as maxsize=128, the least recently used (LRU) entries are removed first. Since Python 3.9, @functools.cache is a short form of lru_cache(maxsize=None). In the next lessons you will meet more built-in decorators: @property, @classmethod, @staticmethod, @dataclass and @contextmanager.

from functools import wraps

def uppercase(func):
    # return a wrapper (with @wraps) that returns func's result in upper case
    ...

@uppercase
def greet(name):
    return f'salam, {name}'

print(greet('Aysel'))
print(greet.__name__)

SALAM, AYSEL
greet

def make_accumulator(start=0):
    # return a function add(x) that adds x to a running total and returns it
    ...

acc = make_accumulator(100)
print(acc(10))
print(acc(5))
other = make_accumulator()
print(other(1))

110
115
1