1. Forum
  2. Usenet
  3. LINUX.GENTOO.DEV

  • [gentoo-dev] [PATCH 0/5] distutils-r1.eclass: DISTUTILS_EXT to enable f

    From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:02 2023
    Hi,

    Here's a patch series that adds a DISTUTILS_EXT variable that indicates
    that the package in question builds Python extensions (i.e. loadable
    .so modules) and enables a few related features:

    1. adds DEPEND=${PYTHON_DEPS} to fix cross-compiling

    2. adds IUSE=debug that controls passing -DNDEBUG/-UNDEBUG to builds
    (right now unlike most distros out there we are always building
    with assertions enabled)

    3. limits our parallel build_ext optimization to DISTUTILS_EXT ebuilds
    (i.e. eliminates the few false positives triggering it right now)

    The patches also include logic to automatically detect and report
    (as a QA warning) missing DISTUTILS_EXT if a package is installing .so
    files into site-packages.

    Note that technically none of the above applies to ebuilds building Rust extensions rather than C/Cython. However, there's no trivial way to distinguish one from the other for the purpose of QA check and none
    of the above points do any real harm. Especially that cargo.eclass
    adds IUSE=debug anyway.

    DISTUTILS_EXT should also be set if the extensions are built
    conditionally to USE flags. The only ill effect of doing that is that PYTHON_DEPS are added unconditionally (unless you're using
    DISTUTILS_OPTIONAL) rather than conditionally to the flag responsible
    for the extension but that's a minor point anyway.

    --
    Best regards,
    Michał Górny

    Michał Górny (5):
    distutils-r1.eclass: Add DISTUTILS_EXT control variable
    distutils-r1.eclass: Add PYTHON_DEPS to DEPEND when DISTUTILS_EXT
    distutils-r1.eclass: Support IUSE=debug for DISTUTILS_EXT
    distutils-r1.eclass: Issue build_ext only w/ DISTUTILS_EXT
    distutils-r1.eclass: Detect missing DISTUTILS_EXT

    eclass/distutils-r1.eclass | 50 +++++++++++++++++++++++++++++++++-----
    1 file changed, 44 insertions(+), 6 deletions(-)

    --
    2.40.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:02 2023
    Detect when Python packages install extensions and warn about missing DISTUTILS_EXT appropriately.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 10 ++++++++++
    1 file changed, 10 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index f7510a4640cb..9b31e5db0d2f 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -2069,6 +2069,16 @@ _distutils-r1_post_python_install() {
    eerror "https://projects.gentoo.org/python/guide/qawarn.html#stray-top-level-files-in-site-packages"
    die "Failing install because of stray top-level files in site-packages"
    fi
    +
    + if [[ ! ${DISTUTILS_EXT} && ! ${_DISTUTILS_EXT_WARNED} ]]; then + if [[ $(find "${sitedir}" -name "*$(get_modname)" | head -n 1) ]]
    + then
    + eqawarn "Python extension modules (*$(get_modname)) found installed. Please set:"
    + eqawarn " DISTUTILS_EXT=1"
    + eqawarn "in the ebuild."
    + _DISTUTILS_EXT_WARNED=1
    + fi
    + fi
    fi
    }

    --
    2.40.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:01 2023
    Add IUSE=debug when DISTUTILS_EXT is set, and use it to control
    assertions via appending -DNDEBUG or -UNDEBUG to CPPFLAGS.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 7 +++++++
    1 file changed, 7 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index f56fa5a51efa..257e032a6c83 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -60,6 +60,8 @@ esac
    #
    # - adds PYTHON_DEPS to DEPEND (for cross-compilation support), unless
    # DISTUTILS_OPTIONAL is used
    +#
    +# - adds debug flag to IUSE that controls assertions (i.e. -DNDEBUG)

    # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
    # @DEFAULT_UNSET
    @@ -325,6 +327,7 @@ _distutils_set_globals() {

    if [[ ${DISTUTILS_EXT} ]]; then
    DEPEND="${PYTHON_DEPS}"
    + IUSE="debug"
    fi
    fi
    }
    @@ -1766,6 +1769,10 @@ distutils-r1_run_phase() {
    local -x AR=${AR} CC=${CC} CPP=${CPP} CXX=${CXX}
    tc-export AR CC CPP CXX

    + if [[ ${DISTUTILS_EXT} ]]; then
    + local -x CPPFLAGS="${CPPFLAGS} $(usex debug '-UNDEBUG' '-DNDEBUG')"
    + fi
    +
    # How to build Python modules in different worlds...
    local ldopts
    case "${CHOST}" in
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:02 2023
    Issue the explicit `build_ext` command call only if `DISTUTILS_EXT`
    is set. This should avoid a few false positives where the command
    was called due to irrelevant .c files being present in the source tree.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 17 +++++++++++------
    1 file changed, 11 insertions(+), 6 deletions(-)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index 257e032a6c83..f7510a4640cb 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -62,6 +62,9 @@ esac
    # DISTUTILS_OPTIONAL is used
    #
    # - adds debug flag to IUSE that controls assertions (i.e. -DNDEBUG)
    +#
    +# - calls `build_ext` command if setuptools build backend is used
    +# and there is potential benefit from parallel builds

    # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
    # @DEFAULT_UNSET
    @@ -1452,12 +1455,14 @@ distutils-r1_python_compile() {
    # .pyx is added for Cython
    #
    # esetup.py does not respect SYSROOT, so skip it there
    - if [[ -z ${SYSROOT} && 1 -ne ${jobs} && 2 -eq $(
    - find '(' -name '*.c' -o -name '*.cc' -o -name '*.cpp' \
    - -o -name '*.cxx' -o -name '*.c++' -o -name '*.m' \
    - -o -name '*.mm' -o -name '*.pyx' ')' -printf '\n' |
    - head -n
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:02 2023
    Add `${PYTHON_DEPS}` to `DEPEND` (i.e. `SYSROOT` dependencies) when
    building extensions. This is necessary to support cross-compilations correctly.

    Originally proposed by Raul E Rangel <rrangel@chromium.org>.

    Closes: https://github.com/gentoo/gentoo/pull/30469
    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 9 +++++++++
    1 file changed, 9 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index f283aa90cfaa..f56fa5a51efa 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -55,6 +55,11 @@ esac
    # Set this variable to a non-null value if the package (possibly
    # optionally) builds Python extensions (loadable modules written in C,
    # Cython, Rust, etc.).
    +#
    +# When enabled, the eclass:
    +#
    +# - adds PYTHON_DEPS to DEPEND (for cross-compilation support), unless
    +# DISTUTILS_OPTIONAL is used

    # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
    # @DEFAULT_UNSET
    @@ -317,6 +322,10 @@ _distutils_set_globals() {
    RDEPEND="${PYTHON_DEPS} ${rdep}"
    BDEPEND="${PYTHON_DEPS} ${bdep}"
    REQUIRED_USE=${PYTHON_REQUIRED_USE}
    +
    + if [[ ${DISTUTILS_EXT} ]]; then
    + DEPEND="${PYTHON_DEPS}"
    + fi
    fi
    }
    _distutils_set_globals
    --
    2.40.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Thu Apr 20 16:40:02 2023
    Add a control variable that should be set in ebuilds that build
    C, Cython, Rust extensions, etc.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 7 +++++++
    1 file changed, 7 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index 09684781de2a..f283aa90cfaa 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -49,6 +49,13 @@ case ${EAPI} in
    *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
    esac

    +# @ECLASS_VARIABLE: DISTUTILS_EXT
    +# @DEFAULT_UNSET
    +# @DESCRIPTION:
    +# Set this variable to a non-null value if the package (possibly
    +# optionally) builds Python extensions (loadable modules written in C,
    +# Cython, Rust, etc.).
    +
    # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
    # @DEFAULT_UNSET
    # @DESCRIPTION:
    --
    2.40.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)