1. Forum
  2. Usenet
  3. LINUX.GENTOO.DEV

  • [gentoo-dev] [PATCH 0/7] pypi.eclass: performance optimizations

    From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:01 2023
    Hi,

    Here's a set of patches that improve performance of pypi.eclass
    by eliminating the subshells from the most common code paths. It comes
    with a trivial benchmarking tool that shows roughly 16 times speedup
    from the changes.

    Thanks to Sam for bringing the problem up, and to Eli Schwartz for
    the shopt idea, that is responsible the final big speedup.


    --
    Best regards,
    Michał Górny


    Michał Górny (7):
    pypi.eclass: Move setting globals to a function
    eclass/tests: Add pypi-bench.sh for global scope logic
    pypi.eclass: Translate version once in the default scenario
    pypi.eclass: Normalize names without subshell
    pypi.eclass: Translate version without subshell in common case
    pypi.eclass: Replace pypi_sdist_url in global scope
    pypi.eclass: Avoid subshell for extglob setting

    eclass/pypi.eclass | 128 ++++++++++++++++++++++++++-----------
    eclass/tests/pypi-bench.sh | 23 +++++++
    2 files changed, 113 insertions(+), 38 deletions(-)
    create mode 100755 eclass/tests/pypi-bench.sh

    --
    2.41.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:01 2023
    The benchmark yield roughly 327 ops / s on my machine.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/tests/pypi-bench.sh | 23 +++++++++++++++++++++++
    1 file changed, 23 insertions(+)
    create mode 100755 eclass/tests/pypi-bench.sh

    diff --git a/eclass/tests/pypi-bench.sh b/eclass/tests/pypi-bench.sh
    new file mode 100755
    index 000000000000..6c4696c19fcb
    --- /dev/null
    +++ b/eclass/tests/pypi-bench.sh
    @@ -0,0 +1,23 @@
    +#!/bin/bash
    +# Copyright 2023 Gentoo Authors
    +# Distributed under the terms of the GNU General Public License v2
    +
    +EAPI=8
    +source tests-common.sh || exit
    +
    +PN=foo-bar
    +PYPI_PN=Foo.Bar
    +PV=1.2.3_beta2
    +WORKDIR='<WORKDIR>'
    +
    +inherit pypi
    +
    +doit() {
    + for (( i = 0; i < 1000; i++ )); do
    + _pypi_set_globals
    + done
    +}
    +
    +time doit
    +
    +texit
    --
    2.41.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:01 2023
    Provide an internal helper to normalize names without a subshell.
    This gives 535 ops / s, so a further 44% speedup.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/pypi.eclass | 39 +++++++++++++++++++++++++++------------
    1 file changed, 27 insertions(+), 12 deletions(-)

    diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
    index a8a179d5a3a4..d79e6f06fc1b 100644
    --- a/eclass/pypi.eclass
    +++ b/eclass/pypi.eclass
    @@ -63,6 +63,21 @@ _PYPI_ECLASS=1
    # @CODE
    : "${PYPI_PN:=${PN}}"

    +# @FUNCTION: _pypi_normalize_name
    +# @INTERNAL
    +# @USAGE: <name>
    +# @DESCRIPTION:
    +# Internal normalization function, returns the result
    +# via _PYPI_NORMALIZED_NAME variable.
    +_pypi_normalize_name() {
    + local name=${1}
    + local shopt_save=$(shopt -p extglob)
    + shopt -s extglob
    + name=${name//+([._-])/_}
    + ${shopt_save}
    + _PYPI_NORMALIZED_NAME="${name,,}"
    +}
    +
    # @FUNCTION: pypi_normalize_name
    # @USAGE: <name>
    # @DESCRIPTION:
    @@ -75,12 +90,9 @@ _PYPI_ECLASS=1
    pypi_normalize_name() {
    [[ ${#} -ne 1 ]] && die "Usage: ${FUNCNAME} <name>"

    - local name=${1}
    - local shopt_save=$(shopt -p extglob)
    - shopt -s extglob
    - name=${name//+([._-])/_}
    - ${sh
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:02 2023
    Provide an internal helper to translate versions without a subshell,
    and use it in the common case. Now the benchmark gives 685 ops / s,
    which means it's another 28% speedup.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/pypi.eclass | 31 +++++++++++++++++++++----------
    1 file changed, 21 insertions(+), 10 deletions(-)

    diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
    index d79e6f06fc1b..c8ad2e03d6e8 100644
    --- a/eclass/pypi.eclass
    +++ b/eclass/pypi.eclass
    @@ -95,6 +95,19 @@ pypi_normalize_name() {
    echo "${_PYPI_NORMALIZED_NAME}"
    }

    +# @FUNCTION: _pypi_translate_version
    +# @USAGE: <version>
    +# @DESCRIPTION:
    +# Internal version translation function, returns the result
    +# via _PYPI_TRANSLATED_VERSION variable.
    +_pypi_translate_version() {
    + local version=${1}
    + version=${version/_alpha/a}
    + version=${version/_beta/b}
    + version=${version/_rc/rc}
    + _PYPI_TRANSLATED_VERSION=${version/_p/.post}
    +}
    +
    # @FUNCTION: pypi_translate_version
    # @USAGE: <version>
    # @DESCRIPTION:
    @@ -106,12 +119,9 @@ pypi_normalize_name() {
    pypi_translate_version() {
    [[ ${#} -ne 1 ]] && die "Usage: ${FUNCNAME} <version>"

    - local version=${1}
    - version=${version/_alpha/a}
    - version=${version/_beta/b}
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:02 2023
    Instead of translating version two times, once in pypi_sdist_url
    and then when setting S, do it once and store the result. This gives
    roughly 371 ops / s, i.e. a 13% speedup.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/pypi.eclass | 10 ++++++----
    1 file changed, 6 insertions(+), 4 deletions(-)

    diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
    index 732b0c6184ef..a8a179d5a3a4 100644
    --- a/eclass/pypi.eclass
    +++ b/eclass/pypi.eclass
    @@ -226,12 +226,14 @@ pypi_wheel_url() {
    # @DESCRIPTION:
    # Set global variables, SRC_URI and S.
    _pypi_set_globals() {
    + local version=$(pypi_translate_version "${PV}")
    +
    if [[ ${PYPI_NO_NORMALIZE} ]]; then
    - SRC_URI="$(pypi_sdist_url --no-normalize)"
    - S="${WORKDIR}/${PYPI_PN}-$(pypi_translate_version "${PV}")"
    + SRC_URI="$(pypi_sdist_url --no-normalize "${PYPI_PN}" "${version}")"
    + S="${WORKDIR}/${PYPI_PN}-${version}"
    else
    - SRC_URI="$(pypi_sdist_url)"
    - S="${WORKDIR}/$(pypi_normalize_name "${PYPI_PN}")-$(pypi_translate_version "${PV}")"
    + SRC_URI="$(pypi_sdist_url "${PYPI_PN}" "${version}")"
    + S="${WORKDIR}/$(pypi_normalize_name "${PYPI_PN}")-${version}"
    fi
    }

    --
    2.41.0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (2
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Tue Jun 13 08:50:02 2023
    Introduce an internal helper for _pypi_sdist_url that doesn't require
    subshell, and therefore eliminate all subshells from global scope.
    We're nearing 952 ops / s, further 39% speedup.

    Signed-off-by: Michał Górny <mgorny@gentoo.org>
    ---
    eclass/pypi.eclass | 53 +++++++++++++++++++++++++++++-----------------
    1 file changed, 33 insertions(+), 20 deletions(-)

    diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
    index c8ad2e03d6e8..eade3d955ab5 100644
    --- a/eclass/pypi.eclass
    +++ b/eclass/pypi.eclass
    @@ -124,6 +124,31 @@ pypi_translate_version() {
    echo "${_PYPI_TRANSLATED_VERSION}"
    }

    +# @FUNCTION: _pypi_sdist_url
    +# @INTERNAL
    +# @USAGE: [--no-normalize] [<project> [<version> [<suffix>]]]
    +# @DESCRIPTION:
    +# Internal sdist generated, returns the result via _PYPI_SDIST_URL
    +# variable.
    +_pypi_sdist_url() {
    + local normalize=1
    + if [[ ${1} == --no-normalize ]]; then
    + normalize=
    + shift
    + fi
    +
    + if [[ ${#} -gt 3 ]]; then
    + die "Usage: ${FUNCNAME} [--no-normalize] <project> [<version> [<suffix>]]"
    + fi
    +
    + local project=${1-"${PYPI_PN}"}
    + local version=${2-"$(pypi_translate_version "${PV}")"}
    + local suffix=${3-.tar.gz}
    + local _PYPI_NORMALIZED_NAME=${project}
    + [[ ${normalize} ]] && _pypi_normalize_nam