# Python package guidelines

From ArchWiki
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

This document covers standards and guidelines on writing PKGBUILDs for Python software.

## Package naming

For Python 3 library modules, use `python-modulename`. Also use the prefix if the package provides a program that is strongly coupled to the Python ecosystem (e.g. pip or tox). For other applications, use only the program name.

The same applies to Python 2 except that the prefix (if needed) is `python2-`.

Note: The package name should be entirely lowercase.

## Architecture

See PKGBUILD#arch.

A Python package that contains C extensions using the `ext_modules` keyword in `setup.py`, is architecture-dependent. Otherwise it is most likely architecture-independent.

## Source

Download URLs linked from the PyPI website include an unpredictable hash that needs to be fetched from the PyPI website each time a package must be updated. This makes them unsuitable for use in a PKGBUILD. PyPI provides an alternative stable scheme: PKGBUILD#source `source=()` array should use the following URL templates:

Source package
  `https://files.pythonhosted.org/packages/source/${_name::1}/$_name/$_name-$pkgver.tar.gz`
Pure Python wheel package
  `https://files.pythonhosted.org/packages/py2.py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py2.py3-none-any.whl` (Bilingual – Python 2 and Python 3 compatible)
  `https://files.pythonhosted.org/packages/py3/${_name::1}/$_name/${_name//-/_}-$pkgver-py3-none-any.whl` (Python 3 only)
  Note that the distribution name can contain dashes, while its representation in a wheel filename cannot (they are converted to underscores).
Arch specific wheel package
  in this example for `source_x86_64=('...')`. Also `_py=cp38` can be used to not repeat the python version:
  `https://files.pythonhosted.org/packages/$_py/${_name::1}/$_name/${_name//-/_}-$pkgver-$_py-${_py}m-manylinux1_x86_64.whl`

Note that a custom `_name` variable is used instead of `pkgname` since python packages are generally prefixed with `python-`. This variable can generically be defined as follows:

    _name=${pkgname#python-}

## Installation methods

Python packages are generally installed using language-specific tools, such as pip or easy_install (deprecated), which are comparable to dedicated package managers in that they are designed to fetch the source files from an online repository (usually PyPI, the Python Package Index) and track the relevant files (for a detailed comparison between the two, see pip vs easy_install).

However, for managing Python packages from within PKGBUILDs, the standard-provided distutils proves to be the most convenient solution since it uses the downloaded source package's `setup.py` and easily installs files under `$pkgdir/usr/lib/python<python version>/site-packages/$pkgname` directory.

Note: Dependencies from `setup.py` must be defined in the `depends` array otherwise they will not be installed.

### distutils

A distutils PKGBUILD is usually quite simple:

    build() {
        python setup.py build
    }

    package() {
        python setup.py install --root="$pkgdir" --optimize=1
    }

where:

- python is replaced with the proper binary, `python` or `python2`
- `--root="$pkgdir"` prevents trying to directly install in the host system instead of inside the package file, which would result in a permission error
- `--optimize=1` compiles optimized bytecode files (`.pyo` for Python 2, `opt-1.pyc` for Python 3) so they can be tracked by pacman instead of being created on the host system on demand
- Adding `--skip-build` optimizes away the unnecessary attempt to re-run the build steps already run in the `build()` function, if that is the case.

### setuptools

The Python packaging scene has largely migrated from distutils to setuptools, which is actively developed and functions as a drop-in replacement import in `setup.py`. The main difference for packagers is that setuptools is packaged separately from Python itself, and must be specified as a `makedepends`:

    makedepends=('python-setuptools')

If the resulting package includes executables which import the pkg_resources module, then setuptools must be additionally specified as a `depends` in the split `package_*()` functions; alternatively, if the PKGBUILD only installs the Python package for a single version of Python, setuptools should be moved from `makedepends` to `depends`.

Some packages try to use setuptools and fall back to distutils if setuptools could not be imported. In this case, setuptools should be added as a `makedepends`, so that the resulting python metadata is better.

If a package needs setuptools to build due to including executables (which is not supported by distutils), but only imports distutils, then building will raise a warning, and the resulting package will be broken (it will not contain the executables):

    /usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'entry_points'
      warnings.warn(msg)

An upstream bug should be reported. To work around the problem, an undocumented setuptools feature can be used:

    # fails because of distutils
    python setup.py build

    # works by using a setuptools shim
    python -m setuptools.launch setup.py build

### pip

If you need to use pip (provided by python-pip and python2-pip), e.g. for installing wheel packages, remember to pass the following flags:

    PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps *.whl

- `PIP_CONFIG_FILE=/dev/null` ignores `{/etc,~/.config}/pip.conf` that may be appending arbitrary flags to pip.
- `--isolated` ignores environment variables (and again `{/etc,~/.config}/pip/pip.conf`) that may otherwise also be appending arbitrary flags to pip.
- `--ignore-installed` is necessary until https://github.com/pypa/pip/issues/3063 is resolved (otherwise pip skips the install in the presence of an earlier `--user` install).
- `--no-deps` ensures, that dependencies do not get packaged together with the main package.

pip does not know how to generate `.pyo` (or for 3.5 and later, `.pyc`) files (see https://github.com/pypa/pip/issues/2209). In order to generate them manually after pip has installed the module, run:

    python -O -m compileall "${pkgdir}/path/to/module"

Warning: Use of pip and/or wheel packages is discouraged in favor of setuptools source packages, and should only be used when the latter is not a viable option (for example, packages which only come with wheel sources, and therefore cannot be installed using setuptools).

### Build-time 2to3 translation

Most Python projects target either Python 2 or Python 3, or target both using compatibility layers like six. However, some use the deprecated 2to3 keyword in setuptools to heuristically convert the source code from Python 2 to Python 3 at build time. As a result, the same source directories cannot be used to build both Python 2 and Python 3 split packages.

For packages that do this, we need a prepare() function that copies the source before it is built. Then the Python 2 and Python 3 packages can be converted and built independently without overriding each other.

    makedepends=("python-setuptools" "python2-setuptools")

    prepare() {
        cp -a foo-$pkgver{,-py2}
    }

    build() {
        cd "$srcdir/foo-$pkgver"
        python setup.py build

        cd "$srcdir/foo-$pkgver-py2"
        python2 setup.py build
    }

    package_python-foo() {
        depends=("python")
        cd "$srcdir/foo-$pkgver"
        python setup.py install --root="$pkgdir/" --optimize=1
    }

    package_python2-foo() {
        depends=("python2")
        cd "$srcdir/foo-$pkgver-py2"
        python2 setup.py install --root="$pkgdir/" --optimize=1
    }

### pyproject.toml (PEP 517)

If an upstream only provides a `pyproject.toml` (PEP 517) in their source tarball, most of the time it is possible to convert it to a `setup.py` and use #setuptools as usual.

Add python-dephell to `makedepends` and generate the setuptools integration in `prepare()`:

    makedepends=("python-setuptools" "python-dephell")

    prepare() {
        cd "$pkgname-$pkgver"
        dephell deps convert --from pyproject.toml --to setup.py
    }

#### setuptools without a setup.py

The PEP 517 build-backend `setuptools.build_meta` does not need a `setup.py` file to exist, and if one does not exist, a minimal one is implied (most such projects will, however, include the minimal one in the upstream tarball):

    from setuptools import setup
    setup()

This breaks compatibility with non-PEP 517 installation methods, such as Linux distributions (as of June 2020, there is no option other than pip). Painful hacks are needed to build; either create the minimal setup.py yourself, or use

    python "from setuptools import setup; setup()"

in place of

    python setup.py

An example of this is ansible-lint

This page will eventually be updated once a non-gross way is found.

## Check

Warning: Avoid using `tox` to run testsuites as it is explicitly designed to test repeatable configurations downloaded from PyPI while `tox` is running, and does not test the version that will be installed by the package. This defeats the purpose of having a check function at all.

Most python projects providing a testsuite use nosetests or pytest to run tests with `test` in the name of the file or directory containing the testsuite. In general, simply running `nosetests` or `pytest` is enough to run the testsuite.

    check(){
        cd "$srcdir/foo-$pkgver"

        # For nosetests
        nosetests

        # For pytest
        pytest
    }

If there is a compiled C extension, the tests need to be run using a `$PYTHONPATH`, that reflects the current major and minor version of Python in order to find and load it.

    check(){
      cd "$pkgname-$pkgver"
      local python_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')
      # For nosetests
      PYTHONPATH="$PWD/build/lib.linux-$CARCH-${python_version}" nosetests

      # For pytest
      PYTHONPATH="$PWD/build/lib.linux-$CARCH-${python_version}" pytest
    }

Some projects provide `setup.py` entry points for running the test. This works for both `pytest` and `nosetests`.

    check(){
        cd "$srcdir/foo-$pkgver"

        # For nosetests
        python setup.py nosetests

        # For pytest - needs python-pytest-runner
        python setup.py pytest
    }

## Tips and tricks

### Reproducible bytecode

In the `package()` function bytecode is compiled for the Python source files. This might prove to be non-reproducible (see Reproducible Builds#Verifying packages with repro and finding issues).

By setting the `PYTHONHASHSEED` environment variable it is possible to disable hash randomization (upstream documentation) to make bytecode reproducible.

    package() {
      cd "$pkgname-$pkgver"
      export PYTHONHASHSEED=0
      python setup.py install --optimize=1 --root="${pkgdir}"
    }

### Discovering detached PGP signatures on PyPi

If detached PGP signatures for a given Python sdist tarball exist, they should be used to verify the tarball. However, the signature files do not show up directly in the files download section of any given project on pypi.org. To discover the sdist tarballs and their potential signature files, it's possible to use this service to get an overview per project: https://pypi.debian.net/

For python-requests this would be: https://pypi.debian.net/requests

### Using site-packages

Sometimes during building, testing or installation it is required to refer to the system's `site-packages` directory. To not hardcode the directory, use a call to the system Python version to retrieve the path and store it in a local variable:

    check(){
      cd "$pkgname-$pkgver"
      local site_packages=$(python -c "import site; print(site.getsitepackages()[0])")
      ...
    }

### Test directory in site-package

Make sure to not install a directory named just `tests` into `site-packages` (e.g. `/usr/lib/python2.7/site-packages/tests/`), as it easily conflicts with other Python packages.

Category:

- Arch package guidelines