This is Gentoo's testing wiki. It is a non-operational environment and its textual content is outdated.

Please visit our production wiki at https://wiki.gentoo.org

Project:Python/python-r1

From Gentoo Wiki (test)
Jump to:navigation Jump to:search

python-r1 is the python-r1 suite eclass intended for ebuilds that install files that may be used with one or more of installed Python interpreters but not use the distutils build system.

For this reason, the ebuild itself needs to ensure proper install procedure for multiple Python implementations. This often means that it is the hardest to use of all -r1 eclasses. Therefore, if your package doesn't really need to be installed for multiple implementations, you may consider using python-single-r1 instead.

Description

python-r1 eclass is suited for installing Python modules and scripts that do not use the distutils build system. This often involves Python bindings and conditional Python support in applications.

As with other eclasses, you need to set PYTHON_COMPAT (and optionally PYTHON_REQ_USE) before inheriting the eclass. This is necessary so that proper values can be generated for global scope variables.

This eclass affects ebuild metadata only by adding PYTHON_TARGETS flags to IUSE. It also exports PYTHON_DEPS and PYTHON_REQUIRED_USE that need to be explicitly used in RDEPEND, DEPEND and REQUIRED_USE.

No phase functions are exported, and no global environment is set. Instead, local Python build scopes are established within functions run through python_foreach_impl. This function needs to be used to build and install the Python parts of the ebuild.

For information on variables and functions available for building packages, see Python build environment.

Two additional eclass-specific functions are provided. python_setup can set up global Python build environment for the most preferred Python implementation, for example to build documentation. python_replicate_script can be used to convert a plain Python script installed by package to have proper shebang and wrapping.

Concepts

Python multibuild

The core purpose of this eclass is to provide means to install Python code for multiple implementations. This is usually done via repeating all or some of the build steps for every enabled implementation. To implement this, an API based on multibuild.eclass is used.

The main function of this API is python_foreach_impl that runs the command passed as its argument (a custom function, usually) repeatedly for every Python implementation enabled. For the command/function scope, a most complete [[./python-utils-r1#Python_build_environment|Python build environment]] is deployed, along with all the multibuild variables (BUILD_DIR, MULTIBUILD_VARIANT).

A separate build directory is provided for each implementation. If the build system supports out-of-source builds, this directory should be used to store output files. If the build system requires in-source build, BUILD_DIR can be used to hold a copy of sources for each implementation. python_copy_sources is a convenience function that creates those copies.

All of the USE flag API is built around the multibuild concept. The PYTHON_TARGETS flags are used to enable the implementations to build. USE dependencies (PYTHON_USEDEP, python_gen_usedep...) are constructed to require a match for all enabled implementations.

Common code (run only once)

Aside to the code that needs to be repeated for each enabled Python implementation, Python is frequently used for the bits that need to be run only once during the build — e.g. to generate sources, build documentation, etc. The design of python-r1 assumes that this code is run outside of python_foreach_impl, with a single implementation of choice established in the global scope using python_setup.

Depending on the needs, the eclass provides two distinct interfaces for establishing the interpreter for the common code: one based on the USE flag API, and the other on any-of dep API. The USE flag API is simpler to use but it is strictly bound to the enabled implementations. The any-of dep API is preferred if the common bits are restricted to a subset of supported implementations, or they have an external Python dependencies.

Common code with the USE flag API

The simpler API for running the common code bits is the USE flag API. This API is used by default, if the python_check_deps function is not declared.

In the USE flag API, the implementation to run the common code parts is selected among the implementations enabled by the USE flags. This works just fine for most of the simple cases, however it causes a few limitations in more complex cases:

  1. If the common code works only for subset of PYTHON_COMPAT (e.g. docs can be built only with Python 2), you need to restrict the allowed implementations via passing patterns to python_setup, and add an appropriate REQUIRED_USE constraints to ensure that a matching implementation is always enabled. As a side effect, it forces the user to install for a matching implementation even if that is not desired.
  2. If the common code requires any additional Python packages, those packages need PYTHON_USEDEP (or python_gen_usedep, if combined with the above). As a side effect, all implementations enabled for this package are enforced on the dependencies, even though only one of them is actually used.

Those limitations make it feasible to use the any-of dependency API if either of those conditions applies.

Common code with the any-of dependency API

The other API for the common code bits is the any-of dependency API. It is based on the interface provide by the python-any-r1 eclass, and so could be considered a method of combining the strengths of both. It is enabled automatically when the python_check_deps function is defined.

In this API, the implementation for the common code parts is selected using a procedure similar to the one from python-any-r1. USE flags used to select implementations are ignored, making it independent of installed targets. Instead, all supported implementations that match patterns passed to python_setup (if any) and that are installed are considered. For every of those implementations, python_check_deps callback is called to determine whether it satisfies the requirements of the common code (dependencies) and the first implementation for which it returns a successful status is used.

If the common code requires any additional Python dependencies, those dependencies need to be declared using a single call to python_gen_any_dep that generates an appropriate any-of dependency block. The python_check_deps function must perform an equivalent check to determine which of the any-of branches succeeded (i.e. for which the dependencies were installed).

Examples

Plain Python package using autotools

This examples installs a simple pure Python package that uses autotools.

CODE Example ebuild for autotools Python package
PYTHON_COMPAT=( python3_{4,5,6} )

inherit python-r1

RDEPEND="${PYTHON_DEPS}
    dev-python/foo[${PYTHON_USEDEP}]"
DEPEND="${RDEPEND}"

REQUIRED_USE="${PYTHON_REQUIRED_USE}"

src_configure() {
    ECONF_SOURCE=${S} \
    python_foreach_impl run_in_build_dir default
}

src_compile() {
    python_foreach_impl run_in_build_dir default
}

src_test() {
    python_foreach_impl run_in_build_dir default
}

src_install() {
    python_foreach_impl run_in_build_dir default
    # note: this works only if the scripts are identical for all impls
    python_replicate_script "${ED%/}"/usr/bin/{foo,bar}
    einstalldocs
}

API reference

Variables set by ebuilds

PYTHON_COMPAT

Obligatory. Must be set above the inherit line.

A list of all Python implementations supported by ebuild. The possible values are listed in the implementations article.

CODE Example PYTHON_COMPAT
PYTHON_COMPAT=( python{2_7,3_4,3_5} pypy )

PYTHON_REQ_USE

Note
Please do not confuse this with PYTHON_REQUIRED_USE. REQ stands for requested.

Optional, defaults to none. Must be set above the inherit line.

USE-dependency that will be applied to all Python interpreters pulled in as dependencies. Takes the form of EAPI 4 USE dependency string, must apply cleanly to all supported Python implementations. See Project:Python/Implementation USE flags for reference.

CODE Example for PYTHON_REQ_USE
PYTHON_REQ_USE="ssl(+)?,xml(+)"

Variables exported by eclass

Obligatory: It is the ebuild author's responsibility to set DEPEND, RDEPEND, and REQUIRED_USE; the eclass does not set them.

PYTHON_DEPS

Contains the dependency string on Python interpreters and auxiliary tools (dev-lang/python-exec).

It must be used within RDEPEND and/or DEPEND. If the Python dependency is conditional to a USE flag, the reference should be placed in appropriate USE-conditional block.

CODE Example use of PYTHON_DEPS
RDEPEND="python? ( ${PYTHON_DEPS} )"
DEPEND="${RDEPEND}"

PYTHON_REQUIRED_USE

Contains the REQUIRED_USE constraint requiring at least one Python implementation to be selected.

It must be used within REQUIRED_USE. If the Python dependency is conditional to a USE flag, the reference should be placed in appropriate USE-conditional block.

CODE Example use of PYTHON_REQUIRED_USE
REQUIRED_USE="python? ( ${PYTHON_REQUIRED_USE} )"

PYTHON_USEDEP

Contains a USE dependency string that can be used to enforce matching Python implementations on package dependencies.

It must be used on package dependencies which are using the python-r1 eclass.

CODE Example use of PYTHON_USEDEP
RDEPEND="dev-python/jinja[${PYTHON_USEDEP}]"
DEPEND="${RDEPEND}
    dev-python/setuptools[${PYTHON_USEDEP}]"

Metadata needing to be set by ebuilds

RDEPEND, DEPEND

Obligatory. Must be set (somewhere after the inherit line, unless distutils-r1 is used).

RDEPEND and DEPEND need to be defined by the ebuild to provide dependencies on proper Python implementations and packages.

The eclass provides PYTHON_DEPS convenience variable with a proper dependency on Python interpreters. If the Python support in package is unconditional, PYTHON_DEPS needs to be placed directly in RDEPEND and/or DEPEND. If the Python support is conditional to a USE flag, the PYTHON_DEPS reference needs to be placed inside matching USE conditional.

Additionally, the eclass provides PYTHON_USEDEP to provide proper USE dependencies on packages providing Python modules.

CODE *DEPEND example with USE conditional
RDEPEND="
    python? (
        ${PYTHON_DEPS}
        dev-python/foo[${PYTHON_USEDEP}]
    )"
DEPEND=${RDEPEND}
CODE *DEPEND example for unconditional use of Python
RDEPEND="${PYTHON_DEPS}
    dev-python/foo[${PYTHON_USEDEP}]"
DEPEND=${RDEPEND}

REQUIRED_USE

Obligatory. Must be set (somewhere after the inherit line, unless distutils-r1 is used).

REQUIRED_USE needs to be defined by the ebuild so that at least one of the supported Python implementations is enabled at installation time, pulling in the proper Python dependencies.

The eclass provides PYTHON_REQUIRED_USE convenience variable with a proper REQUIRED_USE value. If the Python support in package is unconditional during build- or run-time, PYTHON_REQUIRED_USE needs to be placed directly in REQUIRED_USE. If the Python support is always conditional to a USE flag, the PYTHON_REQUIRED_USE reference needs to be placed inside matching USE conditional.

CODE REQUIRED_USE example with USE conditional
REQUIRED_USE="|| ( alsa pulseaudio ) python? ( ${PYTHON_REQUIRED_USE} )"
CODE REQUIRED_USE example for unconditional use of Python
REQUIRED_USE="|| ( alsa pulseaudio ) ${PYTHON_REQUIRED_USE}"
Note
Failure to add REQUIRED_USE definitions will result in user inconvenience and bug reports.

Metadata writing helpers

Implementation patterns

All of the following functions accept patterns to match implementations. If multiple patterns are passed, at least one of them needs to match the implementation for it to be used. The patterns can be either:

  1. fnmatch-style patterns (matched via bash == operator) matching PYTHON_COMPAT values, e.g. python2*, pypy*;
  2. literal -2 to match all implementations compatible with Python 2 (where python_is_python3 is false),
  3. literal -3 to match all implementations compatible with Python 3 (where python_is_python3 is true).

Please remember that wildcard characters such as *, ? and [...] groups need to be quoted or escaped, or otherwise bash will attempt to perform filename expansion in place. This could cause horrible results depending on the current working directory during ebuild processing.

python_gen_usedep

Usage: python_gen_usedep <pattern>...

Outputs USE dependency string that requires the implementations matching pattern(s) to match. Can be used to output USE-dependencies on dependencies that are available or needed only in some of the supported implementations.

Often, python_gen_usedep needs to be used with proper REQUIRED_USE in order to require any of the implementations to be enabled.

CODE Example use of python_gen_usedep
# epydoc supports only python2*
DEPEND="doc? ( dev-python/epydoc[$(python_gen_usedep 'python2*')] )"
REQUIRED_USE="doc? ( || ( $(python_gen_useflags 'python2*') ) )"

python_gen_useflags

Usage: python_gen_useflags <pattern>...

Outputs space separated list of flags for implementations matching pattern(s). Can be used to create REQUIRED_USE values, usually in conjunction with python_gen_usedep.

python_gen_cond_dep

Usage: python_gen_cond_dep <dependency-string> <pattern>...

Outputs provided dependency strings enclosed in USE-conditional block, making them conditional to one of the implementations matching patterns being enabled.

${PYTHON_USEDEP} may be used inside the dependency string. It will be expanded appropriately.

CODE Example use of python_gen_cond_dep
# from virtual/python-futures
RDEPEND="$(python_gen_cond_dep 'dev-python/futures[${PYTHON_USEDEP}]' 'python2*')

python_gen_impl_dep

Usage: python_gen_impl_dep [<req-use-flags> [<pattern>...]]

Outputs a dependency on Python implementations with the USE dependency string from <req-use-flags>, or no USE dependency string if it is empty or not passed. If any <pattern>s are provided, the dependency will be generated only for matching implementations. Otherwise, all implementations will be used.

This function is intended to be used when more than one form of dependency string is needed. In this case, PYTHON_REQ_USE and PYTHON_DEPS should be used for the common form, and python_gen_impl_dep to generate additional variants.

CODE Example use of python_gen_impl_dep
# USE=foo requires XML support in Python
RDEPEND="foo? ( $(python_gen_impl_dep 'xml(+)') )
    ${PYTHON_DEPS}"

python_gen_any_dep

Usage: python_gen_any_dep <dependency-block> [<impl-pattern>...]

Outputs a dependency string that requires at least one of supported Python implementation matching the patterns to be installed along with packages having support for that implementation enabled.

The dependency-block is a free-form dependency string. It may contain verbatim ${PYTHON_USEDEP} string (use single quotes to avoid its early expansion) that will be replaced by USE dependency string matching particular implementation. The string will be repeated for each of the supported implementations.

This function needs to be used in conjunction with python_check_deps. The dependency strings will enforce a matching implementation being installed but python_check_deps needs to determine which of the supported implementations matched the any-of dep.

This function is part of the any-of dependency API and can be only used to provide build-time dependencies for the common (not impl-specific) part of the build process. The allowed implementations must be a subset of the PYTHON_COMPAT but the implementation used for common phase does not have to be explicitly enabled through USE flags (and therefore installed).

CODE Example use of python_gen_any_dep and python_check_deps
DEPEND="
    doc? ( 
        $(python_gen_any_dep '
            dev-python/sphinx[${PYTHON_USEDEP}]
            dev-python/rst-linker[${PYTHON_USEDEP}]
        ')
    )"

python_check_deps() {
    has_version "dev-python/sphinx[${PYTHON_USEDEP}]" \
        && has_version "dev-python/rst-linker[${PYTHON_USEDEP}]"
}

src_compile() {
    # ...
    if use doc; then
        # (calls python_check_deps to figure out a matching impl)
        python_setup
        emake doc
    fi
}
CODE Resulting dependency string
|| (
    (
        dev-lang/python:3.6
        dev-python/sphinx[python_targets_python3_6(-),python_single_target_python3_6(+)]
        dev-python/rst-linker[python_targets_python3_6(-),python_single_target_python3_6(+)]
    )
    (
        dev-lang/python:2.7
        dev-python/sphinx[python_targets_python2_7(-),python_single_target_python2_7(+)]
        dev-python/rst-linker[python_targets_python2_7(-),python_single_target_python2_7(+)]
    )
)

Build-time functions

python_foreach_impl

Usage: python_foreach_impl [<argv>...]

Runs given bash command for each of enabled Python implementations, starting with least preferred one. The execution will continue throughout all the implementations unless one of the command calls die.

The command is run within properly initiated Python build environment. EPYTHON, PYTHON and BUILD_DIR will be exported, and Python executable & pkg-config wrappers will be set up.

If all commands succeed (return 0 status), the command returns 0. Otherwise, it returns the first non-zero status.

CODE Example use of python_foreach_impl
src_configure() {
    python_configure() {
        mkdir -p "${BUILD_DIR}" || die
        cd "${BUILD_DIR}" || die
        econf
    }
    python_foreach_impl python_configure
}

python_setup

Usage: python_setup [<impl>…]

Find the best supported Python interpreter suitable for running the common code. Set up the build environment for it. EPYTHON, PYTHON and BUILD_DIR are exported, and Python executable & pkg-config wrappers will be set up.

This function has two modes of operation: USE flag mode and any-dep mode. The latter is used if python_check_deps callback function is declared. The former is used otherwise.

In the USE flag mode, the implementation is selected among the implementations enabled by the user (via USE flags), with additional pattern restriction if any patterns are provided. If this mode is desired, the dependencies for common-part build time tools need to be expressed with PYTHON_USEDEP, or python_gen_usedep if patterns are provided. In the latter case, an appropriate REQUIRED_USE needs to be declared as well, to guarantee that one of the implementations needed for the common phase are enabled.

In the any-of mode, the implementation is selected among the supported and installed implementations, with additional pattern restriction if any patterns are provided. For each of the implementations fulfilling those requirements, python_check_deps is called to determine whether the dependencies are installed. The best implementation for which it succeeds is used for the common phase. USE flags are ignored. The dependencies on build-time tools need to be expressed using python_gen_any_dep function.

This function needs to be called only if Python support is required outside of python_foreach_impl calls since the latter sets up a local build environment itself. It needs to be called only once for the ebuild scope.

CODE Example use of python_setup
src_compile() {
    # ...

    if use doc; then
        python_setup
        emake -C docs/ html
    fi
}
CODE Example use of python_setup with implementation restriction
# note: only used in USE flag mode
REQUIRED_USE="doc? ( || ( $(python_gen_useflags 'python2*' pypy) ) )"

src_compile() {
    # ...

    if use doc; then
        python_setup 'python2*' pypy
        emake -C docs/ html
    fi
}
Note
The implementation preference may change in future eclass versions. The ebuilds must not rely on any of the passed implementations being preferred over another.

python_export_best

Usage: python_export_best [<variable>...]

Warning
This function is deprecated in favor of python_setup.

Export the build environment variables for the most preferred of enabled Python implementations. If variable names are passed, the specified variables will be exported. Otherwise, default set of EPYTHON and PYTHON will be exported.

It is recommended to follow the call to this function with python_wrapper_setup to obtain properly wrapped Python executables and pkg-config.

This can be used to set the build environment for running operations that need to be run only once during build-time, e.g. building documentation.

python_replicate_script

Usage: python_replicate_script <path>...

Copy the scripts at specified paths for all enabled implementations, alternating the shebangs and wrapping them properly. Expects absolute path (including ${D}). Dies on failure.

This can be used to clean up scripts installed by non-distutils build systems.

CODE Example use of python_replicate_script
src_install() {
    default

    python_replicate_script "${ED}"/usr/bin/foomatic
}

python_copy_sources

Usage: python_copy_sources

Create a separate copy of package sources for each of enabled Python implementations. The sources will be obtained from initial BUILD_DIR (or S, if it is unset), and they will be copied to implementation-specific BUILD_DIRs as set by python_foreach_impl.

This function is usually used to handle broken build systems. When possible, out-of-source build is preferred.

Ebuild callbacks

python_check_deps

If defined, this function is called by python_setup for each installed and supported Python implementation to determine whether it is properly suited for the build.

The function is run with a minimal build environment, consisting of EPYTHON and PYTHON_USEDEP. The implementation in question is guaranteed to be installed and match PYTHON_REQ_USE.

The function needs to return 0 if the implementation can be used for the build, non-zero code otherwise. The checks done in the function should be accompanied by proper dependency strings (usually obtained using python_gen_any_dep). If all calls to this function return failure, the eclass dies with 'No supported Python implementation installed' message.

Retrieved from "https://wikitest.gentoo.org/index.php?title=Project:Python/python-r1&oldid=727534"