This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.
Please visit our production wiki at https://wiki.gentoo.org
Project:Python/Dependencies
This article aims to shed some light on which Python packages should be listed as runtime dependencies, build-time dependencies, or both.
Rule of thumb
Build-time dependencies
Build-time dependencies need to include all packages that are required for the package to build correctly and reliably. That is, a package needs to be included in build dependencies if at least one of the following conditions hold:
- A script or a Python module from this package is used (run, loaded) at build time.
- The setup script aborts if the package is not detected to be installed (setuptools).
- The setup script installs incomplete package if the package is not installed.
If the missing package results only in a harmless warning, it can be omitted. However, when in doubt, include all runtime dependencies of the packages in the DEPEND variable to avoid trouble.
Tests usually require all (or most) of the runtime dependencies, so if you do not include them normally in DEPEND, you probably want to use a construct like:
DEPEND=" test? ( ${RDEPEND} )"
If tests have optional dependencies, all of them should be included (preferably unconditionally) in the test dependencies to ensure the best test coverage.
Runtime dependencies
Runtime dependencies need to include all packages that are required for the package to be useful when installed. A package needs to be included in runtime dependencies if the installed Python scripts and/or modules use the package.
If a runtime dependency is optional, common sense should be used to determine the best solution (in order of preference):
- For small and/or commonly used dependencies, the dependency may just be included unconditionally.
- A
pkg_postinstmessage can be printed about missing optional runtime dependencies. - If rebuilding package does not require a significant effort, USE flags might be used to control dependencies.
Please note that a dependency is not optional if it is listed in the installed package's requires.txt file (inside /usr/lib*/python*/site-packages/*.egg-info). While the issue may not be immediately apparent, setuptools may enforce those dependencies when trying to use other packages. You either need to install them unconditionally, or disarm the dependency in the file.
If the package in question installs its test files, it is not strictly required to include the test dependencies in the runtime dependencies. The tests are not considered commonly used components of the package, and therefore their dependencies can be treated as optional.
Finding dependencies
Reading setuptools setup.py files
A common source of dependency information for most of the Python packages are setup.py files using setuptools. Those files pass dependencies using keys to the setup() function, whose values match [1] package names. The following table summarizes different keys used, how to use them in Gentoo and how they are enforced by setuptools.
| setuptools key | Gentoo type | Enforcement | Notes | ||
|---|---|---|---|---|---|
| build-time | test-time | run-time | |||
| setup_requires | DEPEND | Yes | Yes | No | enforced for every setup.py call |
| install_requires | RDEPEND + test-DEPEND | No | Yes | Yes | not required for our setup.py install but enforced for tests and at runtime via pkg_resources (scripts will reject to run unless dependencies are satisfied) |
| tests_require | test-DEPEND | No | Yes | No | enforced for tests |
| extras_require | optional RDEPEND | No | No | No | used to declare optional RDEPs, you can use postinst or USE flags for them; if tests make use of them, recommended to be always-on for tests |
| extras_require w/ platform specifiers | partial RDEPEND | No | Yes | Yes | old syntax for dependencies conditional to platform (i.e. Python version, system), work like install_requires for matching platforms |
Specific dependencies
Use of PYTHON_DEPS (dependency on Python)
Runtime dependencies
PYTHON_DEPS includes the dependency on the Python interpreter and dev-lang/python-exec. If the package installs Python scripts, modules or links to Python libraries, it needs to include PYTHON_DEPS in RDEPEND. In other words, every ebuild using distutils-r1, python-r1, or python-single-r1 needs to use it.
python-any-r1 is intended for build-time dependencies only, so it must not include PYTHON_DEPS in RDEPEND.
Build-time dependencies
The easy rule is: always include PYTHON_DEPS in DEPEND.
However, if you want to be very strict, then you can omit PYTHON_DEPS if the Python interpreter is not called during the build phases in any way. Note that this both includes external calls (e.g. from build system, configure script, starting a local Python script) and calling one of the eclass functions using it. In particular, the following functions require the Python interpreter to be available (and initialized via python_setup/python_foreach_impl):
- python_export and getters for everything except EPYTHON, PYTHON, PYTHON_PKG_DEP, PYTHON_SCRIPTDIR;
- python_optimize (to byte-compile modules);
- python_domodule (implies byte-compiling);
- python_setup (called in pkg_setup) when using python-any-r1;
- Most of distutils-r1 functions (since they interface with setup.py).
Dependency on setuptools
Build-time dependency
Setuptools is an extension to the standard distutils build system commonly used by Python packages. In order to determine whether the package uses setuptools, you should check the setup.py file. If the setuptools module (or any function from it) is imported, then dev-python/setuptools needs to be specified as a build-time dependency.
Note that some packages may have optional setuptools import, with fallback to distutils. In this case, you must unconditionally depend on dev-python/setuptools, in order to enforce consistently building with a single build system only. This is because setuptools install the egg-info metadata as a directory, while distutils install them as a file. The Gentoo package managers can not handle replacing a file with a directory (and the other way around) gracefully, and therefore we must enforce a single format.
Runtime dependency
Although dev-python/setuptools are most commonly a build-time dependency, this package also installs the pkg_resources package that is used by some packages at runtime. If an import to pkg_resources (or setuptools) is found within the installed files, the dependency on dev-python/setuptools should be runtime as well.
The common case for this is the use of entry_points feature. In this case the generated Python scripts do not import the package directly but instead use the pkg_resources module to find and load it. Please also note that this causes setuptools to verify installed dependencies at runtime. If the package declares more dependencies that it actually needs, then the ebuild needs to either enforce them or edit the requirements in setup.py to match reality.
Dependency on Sphinx
When to use PYTHON_USEDEP?
dev-python/sphinx is frequently used to build documentation for Python packages. Since the task is usually done in the common subphase using the sphinx-build executable, it is frequently unclear whether to include PYTHON_USEDEP in the dependency specification or not.
Easy rule: when in doubt, include it. Worst case, the dependency will be overstrict. This will not cause build failures, however.
Complete rule: usually sphinx-build is purely called as an external tool and does not require PYTHON_USEDEP. However, PYTHON_USEDEP is required if one of the following occurs:
- sphinx-build is called via python (e.g. python /usr/bin/sphinx-build ...) — this syntax enforces implementation match between the caller and the script,
- the package uses Sphinx Python API,
- one of the Sphinx extensions interacting with package code is used (e.g. sphinx.ext.autodoc — can be noticed in conf.py).
Furthermore, if external Sphinx plugins are used, the package needs to explicitly enforce implementation match between Sphinx and the plugins. This can be done either using python_gen_any_dep in eclasses supporting this API. In the remaining eclasses, PYTHON_USEDEP needs to be used.
DEPEND="
doc? (
$(python_gen_any_dep '
dev-python/sphinx[${PYTHON_USEDEP}]
dev-python/rst-linker[${PYTHON_USEDEP}]
')
)"