Command-line apps: argparse and click

`sys.argv` and its limits; `argparse`: positional and optional arguments, types, `choices`, defaults, flags, `nargs`/`count`/`append`, help text and errors, subcommands, exit codes and `main(argv)`; `click` with decorators, groups and testing with `CliRunner`.

  • Advanced
  • 22 min
  • 12 questions
  • Lesson 34 of 66

In this lesson you will learn

  • Explain how a program receives arguments through sys.argv and why parsing them by hand is fragile
  • Describe a command line with argparse: positional and optional arguments, types, choices, flags, subcommands
  • Return correct exit codes from main(argv) and test a CLI by passing a list
  • Write the same tool with click and test it with CliRunner

Aysel wrote a script that converts currencies. Every time, her colleagues open the code and change the lines AMOUNT = 100 and RATE = 1.7 — sooner or later someone will break the wrong line. If the script takes its parameters from the command line, typing python convert.py 100 USD --rate 1.7 is enough, and --help explains what to do by itself. In «Automation scripts» you met argparse briefly; now we study it in depth and then write the same tool with the popular click library.

sys.argv: arguments as a list of strings

Every word typed in the terminal reaches the program in the list sys.argv: sys.argv[0] is the script’s name, and the rest are the arguments — always strings. Parsing by hand quickly becomes fragile: a forgotten argument gives IndexError, a value of the wrong type gives a confusing Traceback, there is no help text, and you have to look for named parameters like --rate yourself every time.

Python
import sys

print(sys.argv)
name = sys.argv[1] if len(sys.argv) > 1 else 'world'
times = int(sys.argv[2]) if len(sys.argv) > 2 else 1
for _ in range(times):
    print(f'Hello, {name}!')
greet.py
Terminal
$ python greet.py Aysel 2
['greet.py', 'Aysel', '2']
Hello, Aysel!
Hello, Aysel!
$ python greet.py
['greet.py']
Hello, world!
$ python greet.py Aysel two
['greet.py', 'Aysel', 'two']
Traceback (most recent call last):
  ...
ValueError: invalid literal for int() with base 10: 'two'
'2' arrives as a string and must be converted with int(); when the user types two, they see a Traceback meant for programmers instead of an error message (shortened here).

Sign up free to keep reading

The rest of the lesson, the quiz and your notes open with an Educora account. Signing up takes a minute, and your progress is saved on all your devices.

argparse: positional and optional arguments

Positional argument, option and flagDefinition

A positional argument is recognised by its place and is usually required: convert 100 USD. An option (optional argument) starts with - or --, can be written in any order and has a default: --rate 1.7, short form -r 1.7. A flag is an option without a value: True if --verbose was typed, False if not.

add_argument parameterWhat it does
type=int, type=floatconverts the string to a type; if that fails, gives a clear error
default=...the value when the argument is not given
choices=['USD', 'EUR']allows only these values
action='store_true'a flag: True/False
action='count', action='append'-vvv → 3; collects a repeated option into a list
nargs='?', '*', '+'0 or 1, any number, at least 1 value
help=, metavar=the --help text and the name shown there

In a real script parser.parse_args() is called without arguments and reads sys.argv[1:]. If we pass it a list, the same code also works in the browser and in tests — all examples below do this. The result is a Namespace object: every argument is an attribute, and in names like --dry-run the hyphen turns into an underscore (args.dry_run).

Python
import argparse

parser = argparse.ArgumentParser(prog='convert', description='Convert AZN to another currency.')
parser.add_argument('amount', type=float, help='amount in AZN')
parser.add_argument('currency', choices=['USD', 'EUR', 'TRY'], help='target currency')
parser.add_argument('-r', '--rate', type=float, default=1.7, help='AZN per unit')
parser.add_argument('--digits', type=int, default=2, metavar='N', help='digits after the point')
parser.add_argument('-v', '--verbose', action='store_true', help='also print the formula')

args = parser.parse_args(['100', 'USD', '--rate', '1.7', '-v'])
print(args)
if args.verbose:
    print(f'{args.amount} / {args.rate}')
print(f'{args.amount} AZN = {round(args.amount / args.rate, args.digits)} {args.currency}')
print(parser.parse_args(['50', 'EUR', '-r', '1.95', '--digits', '1']))
Expected output
Namespace(amount=100.0, currency='USD', rate=1.7, digits=2, verbose=True)
100.0 / 1.7
100.0 AZN = 58.82 USD
Namespace(amount=50.0, currency='EUR', rate=1.95, digits=1, verbose=False)
'100' became 100.0 automatically, and the -v flag gave verbose=True. In the second call the options that were not given took their defaults.
Python
import argparse

parser = argparse.ArgumentParser(prog='backup')
parser.add_argument('folders', nargs='+', help='one or more folders')
parser.add_argument('-x', '--exclude', action='append', default=[], help='a pattern to skip (repeatable)')
parser.add_argument('-v', '--verbose', action='count', default=0, help='-v, -vv, -vvv')
group = parser.add_mutually_exclusive_group()
group.add_argument('--zip', action='store_true')
group.add_argument('--tar', action='store_true')

print(parser.parse_args(['docs', 'photos', '-x', '*.tmp', '-x', '*.log', '-vv', '--zip']))
print(parser.parse_args(['docs']))
Expected output
Namespace(folders=['docs', 'photos'], exclude=['*.tmp', '*.log'], verbose=2, zip=True, tar=False)
Namespace(folders=['docs'], exclude=[], verbose=0, zip=False, tar=False)
nargs='+' collected several folders into a list, the repeated -x became a list through append, and -vv counted 2. add_mutually_exclusive_group does not allow --zip and --tar together.

Examples: argparse rules

Problem

1) What name does the option --dry-run get in the Namespace?
2) What does -vvv give with action='count', default=0?
3) add_argument('output', nargs='?', default='out.txt') and parse_args([]) — what is args.output?

Show solutionHide solution

Solution

1) dry_run — the leading hyphens are dropped and the inner hyphen becomes an underscore.
2) 3 — every v adds 1 to the counter.
3) 'out.txt' — nargs='?' makes the positional argument optional, and without a value default is used.

Help text and errors

argparse adds the -h/--help option by itself and builds the help text from description and the help= texts; %(default)s inserts the default value. When the user makes a mistake, argparse writes a short usage line and a clear message to stderr and stops the program with code 2 (it raises SystemExit). In a test or in the browser you can catch it with except SystemExit.

Python
import argparse

parser = argparse.ArgumentParser(prog='convert', description='Convert AZN to another currency.')
parser.add_argument('amount', type=float, help='amount in AZN')
parser.add_argument('currency', choices=['USD', 'EUR', 'TRY'], help='target currency')
parser.add_argument('-r', '--rate', type=float, default=1.7, help='AZN per unit (default: %(default)s)')
parser.add_argument('-v', '--verbose', action='store_true', help='also print the formula')
parser.print_help()
Expected output
usage: convert [-h] [-r RATE] [-v] amount {USD,EUR,TRY}

Convert AZN to another currency.

positional arguments:
  amount           amount in AZN
  {USD,EUR,TRY}    target currency

options:
  -h, --help       show this help message and exit
  -r, --rate RATE  AZN per unit (default: 1.7)
  -v, --verbose    also print the formula
Python
import argparse
import contextlib
import sys

parser = argparse.ArgumentParser(prog='convert')
parser.add_argument('amount', type=float)
parser.add_argument('currency', choices=['USD', 'EUR', 'TRY'])


def run(argv):
    with contextlib.redirect_stderr(sys.stdout):      # show the error text in the output
        try:
            print(parser.parse_args(argv))
        except SystemExit as e:
            print('exit code:', e.code)


run(['100', 'USD'])
run(['100', 'GBP'])
run(['ten', 'USD'])
run(['100'])
Expected output
Namespace(amount=100.0, currency='USD')
usage: convert [-h] amount {USD,EUR,TRY}
convert: error: argument currency: invalid choice: 'GBP' (choose from USD, EUR, TRY)
exit code: 2
usage: convert [-h] amount {USD,EUR,TRY}
convert: error: argument amount: invalid float value: 'ten'
exit code: 2
usage: convert [-h] amount {USD,EUR,TRY}
convert: error: the following arguments are required: currency
exit code: 2
redirect_stderr(sys.stdout) sends the error text to the normal output so that it shows up here. The exact wording of the messages can differ a little between Python versions, but the code is always 2.

Subcommands and exit codes: mini project

Big tools have subcommands: git commit, git push, pip install. In argparse this is done by add_subparsers(dest='command', required=True): every subcommand is a separate parser with its own arguments, and the chosen command’s name is stored in args.command. Let us build a small notes tool: the commands add, list and done, with the notes kept in notes.json.

Exit codeDefinition

The number a program returns to the operating system when it ends: 0 means success, anything else means an error (argparse uses 2 for wrong arguments). Scripts, schedulers and CI systems look exactly at this code: in bash, cmd1 && cmd2 runs the second command only if the first returned 0, and the last code is shown by $? in bash and $LASTEXITCODE in PowerShell. In Python you return it with sys.exit(code).

Python
import argparse
import contextlib
import json
import shlex
import sys
from pathlib import Path

DB = Path('notes.json')


def main(argv=None):
    parser = argparse.ArgumentParser(prog='notes')
    sub = parser.add_subparsers(dest='command', required=True)
    add = sub.add_parser('add', help='add a note')
    add.add_argument('text')
    add.add_argument('-p', '--priority', type=int, choices=[1, 2, 3], default=2)
    sub.add_parser('list', help='show open notes').add_argument('--all', action='store_true')
    sub.add_parser('done', help='mark a note as done').add_argument('number', type=int)
    args = parser.parse_args(argv)

    notes = json.loads(DB.read_text(encoding='utf-8')) if DB.exists() else []
    if args.command == 'add':
        notes.append({'text': args.text, 'priority': args.priority, 'done': False})
        print(f'added #{len(notes)}')
    elif args.command == 'list':
        for i, n in enumerate(notes, start=1):
            if args.all or not n['done']:
                print(f"{i}. [{'x' if n['done'] else ' '}] p{n['priority']} {n['text']}")
    elif not 1 <= args.number <= len(notes):
        print(f'notes: no note #{args.number}', file=sys.stderr)
        return 1
    else:
        notes[args.number - 1]['done'] = True
    DB.write_text(json.dumps(notes), encoding='utf-8')
    return 0


with contextlib.redirect_stderr(sys.stdout):
    for argv in (['add', 'Buy bread'], ['add', 'Python homework', '-p', '1'],
                 ['done', '1'], ['list'], ['list', '--all'], ['done', '7']):
        print('$ notes', shlex.join(argv))
        print('  exit code', main(argv))
Expected output
$ notes add 'Buy bread'
added #1
  exit code 0
$ notes add 'Python homework' -p 1
added #2
  exit code 0
$ notes done 1
  exit code 0
$ notes list
2. [ ] p1 Python homework
  exit code 0
$ notes list --all
1. [x] p2 Buy bread
2. [ ] p1 Python homework
  exit code 0
$ notes done 7
notes: no note #7
  exit code 1
main(argv) returns the exit code instead of calling sys.exit — that is why it can be called several times here or in a test. For a wrong number the message goes to stderr and the code is 1. shlex.join shows the arguments with quotes, as they would be typed in a terminal.
Terminal
$ python notes.py add "Buy bread"
added #1
$ python notes.py done 7
notes: no note #7
$ echo $?
1
$ python notes.py remove 1
usage: notes [-h] {add,list,done} ...
notes: error: argument command: invalid choice: 'remove' (choose from add, list, done)
$ echo $?
2
$ python notes.py list && echo "all good"
1. [ ] p2 Buy bread
all good
The same code in a file, with if __name__ == '__main__': sys.exit(main()) added at the end. There is no remove command, so argparse exited with code 2, and && ran echo because list succeeded.

As the commands grow, the if/elif chain gets long. Then you can write a separate function for every subcommand and attach it to its parser with set_defaults(func=cmd_add), while main simply calls return args.func(args). Notice one more thing: results go to stdout, and error messages go to stderr with print(..., file=sys.stderr). When the user types python notes.py list > notes.txt, the file gets only the notes, and an error stays on the screen.

click: the same with decorators

click is a third-party library (pip install click); for example, Flask’s flask command is built on it. Instead of a parser object you write decorators above a function (see «Decorators and closures»): @click.command(), @click.argument for positional arguments and @click.option for options. The arguments reach the function as ordinary parameters, and the function’s docstring becomes the help text. click.Choice, click.IntRange, is_flag=True, show_default=True and envvar=, which takes a value from an environment variable, come ready; @click.group() builds subcommands.

Python
import click


@click.command()
@click.argument('amount', type=float)
@click.argument('currency', type=click.Choice(['USD', 'EUR', 'TRY']))
@click.option('-r', '--rate', type=float, default=1.7, show_default=True, help='AZN per unit.')
@click.option('-v', '--verbose', is_flag=True, help='Also print the formula.')
def convert(amount, currency, rate, verbose):
    """Convert AZN to another currency."""
    if verbose:
        click.echo(f'{amount} / {rate}')
    click.echo(f'{amount} AZN = {round(amount / rate, 2)} {currency}')


if __name__ == '__main__':
    convert()
convert.py
Terminal
$ python convert.py 100 USD -v
100.0 / 1.7
100.0 AZN = 58.82 USD
$ python convert.py --help
Usage: convert.py [OPTIONS] AMOUNT {USD|EUR|TRY}

  Convert AZN to another currency.

Options:
  -r, --rate FLOAT  AZN per unit.  [default: 1.7]
  -v, --verbose     Also print the formula.
  --help            Show this message and exit.
$ python convert.py 100 GBP
Usage: convert.py [OPTIONS] AMOUNT {USD|EUR|TRY}
Try 'convert.py --help' for help.

Error: Invalid value for '{USD|EUR|TRY}': 'GBP' is not one of 'USD', 'EUR', 'TRY'.
Checked with click 8.5. The help shows the docstring and [default: 1.7], and for a wrong value click exits with code 2.

click’s strong point is testing: click.testing.CliRunner invokes a command without starting a separate process and captures its output and exit code. We write the tests as in «Testing with pytest»:

Python
import click
from click.testing import CliRunner


@click.group()
def cli():
    """A tiny notes tool."""


@cli.command()
@click.argument('text')
@click.option('-p', '--priority', type=click.IntRange(1, 3), default=2, show_default=True)
def add(text, priority):
    """Add a note."""
    click.echo(f'added: {text} (p{priority})')


def test_add():
    result = CliRunner().invoke(cli, ['add', 'Buy bread', '-p', '1'])
    assert result.exit_code == 0
    assert result.output == 'added: Buy bread (p1)\n'


def test_bad_priority():
    result = CliRunner().invoke(cli, ['add', 'Buy bread', '-p', '9'])
    assert result.exit_code == 2
    assert '9 is not in the range 1<=x<=3' in result.output
test_notes_cli.py
Terminal
$ pytest -q test_notes_cli.py
..                                                                       [100%]
2 passed in 0.05s
IntRange(1, 3) rejected 9; result.output holds the error message too. The time (0.05s) differs on every run.
argparseclick
Installationstandard librarypip install click
Stylea parser object, add_argumentdecorators above a function
Subcommandsadd_subparsers@click.group()
Testingcall main([...])CliRunner().invoke(...)
Extras—prompts, colours, envvar
argparse for a small script without dependencies; click is often chosen for a big tool with many commands.

Exercise

Add arguments to the resize parser: the positional image, -w/--width (an integer, default 800), the flag --keep-ratio and --format (jpg, png, webp, default jpg).

Exercise · Python
import argparse

parser = argparse.ArgumentParser(prog='resize')
# add: the positional argument image; -w/--width (int, default 800);
# the flag --keep-ratio; --format with choices jpg, png, webp (default jpg)

args = parser.parse_args(['photo.png', '-w', '1200', '--keep-ratio'])
print(args.image, args.width, args.keep_ratio, args.format)
args = parser.parse_args(['logo.png', '--format', 'webp'])
print(args.image, args.width, args.keep_ratio, args.format)
Expected output
photo.png 1200 True jpg
logo.png 800 False webp

Exercise

Finish the wordcount tool: --top N (default 3) sets how many words are shown, and the flag -i/--ignore-case makes it lowercase the words before counting.

Exercise · Python
import argparse
from collections import Counter


def main(argv):
    parser = argparse.ArgumentParser(prog='wordcount')
    parser.add_argument('text')
    # add --top (int, default 3) and the flag -i/--ignore-case
    args = parser.parse_args(argv)
    words = args.text.split()
    # with --ignore-case, lowercase every word
    for word, n in Counter(words).most_common(3):
        print(word, n)
    return 0


main(['Tea tea coffee Tea milk coffee', '--top', '2'])
print('---')
main(['Tea tea coffee Tea milk coffee', '-i'])
Expected output
Tea 2
coffee 2
---
tea 3
coffee 2
milk 1

Remember

  • sys.argv is a list of strings ([0] is the script name); parsing by hand is fragile, while argparse handles types, help and errors itself.
  • Positional arguments are required, options are written as --name and have defaults; type, choices, store_true, count, append, nargs.
  • On a wrong argument argparse writes to stderr and exits with code 2; in tests and in the browser pass a list to parse_args.
  • Subcommands: add_subparsers; let main(argv=None) return the code and sys.exit(main()) hand it to the system: 0 means success.
  • click does the same job with decorators (command, argument, option, group) and is easy to test with CliRunner.
Spotted a mistake or something inaccurate in this lesson?

Check yourself

12 questions. Every correct answer earns XP.

Question 1 / 12

Question 1 / 12. What is sys.argv after python greet.py Aysel 2?