• [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM u

    From Sam James@21:1/5 to All on Wed Jan 5 00:00:01 2022
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Bug: https://bugs.gentoo.org/570534
    Signed-off-by: Sam James <sam@gentoo.org>
    ---
    eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
    1 file changed, 39 insertions(+), 3 deletions(-)

    diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
    index 2130e2e349141..8c4adc8b4f121 100644
    --- a/eclass/check-reqs.eclass
    +++ b/eclass/check-reqs.eclass
    @@ -43,6 +43,8 @@ case ${EAPI} in
    *) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
    esac

    +inherit multiprocessing
    +
    EXPORT_FUNCTIONS pkg_pretend pkg_setup

    if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
    @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
    # @DESCRIPTION:
    # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M

    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
    +
    # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
    # @DEFAULT_UNSET
    # @DESCRIPTION:
    @@ -346,9 +355,36 @@ _check-reqs_memory() {
    eend 0
    else
  • From Sam James@21:1/5 to All on Wed Jan 5 00:10:01 2022
    --Apple-Mail=_228B33C8-0CE5-48E3-968D-1795BFFE83D9
    Content-Transfer-Encoding: quoted-printable
    Content-Type: text/plain;
    charset=us-ascii



    On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:

    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Bug: https://bugs.gentoo.org/570534
    Signed-off-by: Sam James <sam@gentoo.org>
    ---
    eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
    1 file changed, 39 insertions(+), 3 deletions(-)

    Note that we discussed this on GitHub a bit when I just posted it there
    for some rough feedback: https://github.com/gentoo/gentoo/pull/23311 <https://github.com/gentoo/gentoo/pull/23311>.

    I think this is valuable for reducing invalid bug reports from OOM and
    easing user experience.

    Still kind of a WIP/rough draft, but may be ready in this state. Need
    more testing, so not planning on pushing yet or anything.

    --Apple-Mail=_228B33C8-0CE5-48E3-968D-1795BFFE83D9
    Content-Transfer-Encoding: quoted-printable
    Content-Type: text/html;
    charset=us-ascii

    <html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""
    <div class="">On 4 Jan 2022, at 22:58, Sam James &lt;<a href="mailto:sam@gentoo.org" class="">sam@gentoo.org</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for
    the<br class="">amount of RAM available (uses amount declared as needed<br class="">in the ebuild). Typically should be ~2GB per job.<br class=""><br class="">Bug: <a href="https://bugs.gentoo.org/570534" class="">https://bugs.gentoo.org/570534</a><br
    class="">Signed-off-by: Sam James &lt;<a href="mailto:sam@gentoo.org" class="">sam@gentoo.org</a>&gt;<br class="">---<br class=""> eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---<br class=""> 1 file changed, 39 insertions(+), 3
    deletions(-)<br class=""></div></div></blockquote></div><br class=""><div class="">Note that we discussed this on GitHub a bit when I just posted it there</div><div class="">for some rough feedback:&nbsp;<a href="https://github.com/gentoo/gentoo/pull/
    23311" class="">https://github.com/gentoo/gentoo/pull/23311</a>.</div><div class=""><br class=""></div><div class="">I think this is valuable for reducing invalid bug reports from OOM and</div><div class="">easing user experience.</div><div class=""><br
    class=""></div><div class="">Still kind of a WIP/rough draft, but may be ready in this state. Need</div><div class="">more testing, so not planning on pushing yet or anything.</div></body></html>
    --Apple-Mail=_228B33C8-0CE5-48E3-968D-1795BFFE83D9--

    -----BEGIN PGP SIGNATURE-----

    iQGTBAEBCgB9FiEEYOpPv/uDUzOcqtTy9JIoEO6gSDsFAmHU0k9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYw RUE0RkJGRkI4MzUzMzM5Q0FBRDRGMkY0OTIyODEwRUVBMDQ4M0IACgkQ9JIoEO6g SDtMQgf/dFpjL5xfoxVLqL/H7+2PJ/IFkNl1lfGS/56BZ+9sndvhdbC7vjVGxNMH cg6gmVlMhekbrtjZ3FhYySeGzKqezC0Ng2KcJuG3E2gtDaHyC4Qb0oJAnTxmPkkK NQQskRrNTHzu4S/6xm7Cuq4GglDjAb4xXK33GZuDoW+NIvCFWtyr0D2c2oHWcMyX wwU8O3wTn0Co+y1pIVBMx7ApRy3xWi1I2n3oMIaW7RmVozUoQIL0AFpN/ngq8rDv 4sSfQqaWTrzjvfOPr+g9NAWhkzwGzS07ommhkMxpOzfEe9p5GqpYfjSH4mH+7rst au56Q1cHhiC+p1o0Dl7Dbz8Ja1cEcA==
    =RwjV
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ulrich Mueller@21:1/5 to All on Wed Jan 5 09:30:02 2022
    On Tue, 04 Jan 2022, Sam James wrote:

    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.
    (If only considering C and C++ 2 GB seems to be excessive, i.e. it
    will limit parallel build more than necessary. When linking, the number
    may be _much_ larger.)

    Also not sure if I understand the arithmetic. Shouldn't it use
    CHECKREQS_MEMORY as the basis of the calculation?

    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}

    If anything, the feature should be opt-in rather than opt-out.

    Ulrich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Florian Schmaus@21:1/5 to Ulrich Mueller on Wed Jan 5 17:20:01 2022
    On 05/01/2022 09.28, Ulrich Mueller wrote:
    On Tue, 04 Jan 2022, Sam James wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.

    Surely not, but 2 GiB seems like a good start. I guess it could become
    an eclass variable that ebuilds could modify later (if the need emerges).


    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}

    If anything, the feature should be opt-in rather than opt-out.

    It appears to me that the motivation for this change is to prevent users
    from running into OOM situations when emerging a package. And I believe
    that many users, especially novices, are not directly able to determine
    that portage failed due to OOM, simply because there is no direct hint
    in the build log. Hence opt-out appears to be sensible to me here.

    - Flow

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alexey Sokolov@21:1/5 to All on Wed Jan 5 17:50:01 2022
    I do sometimes run into OOM during emerge, but for a different reason:
    when firefox and webengine are built in parallel. And I'm using tmpfs
    for portage. These packages store lots of data in the build directory. Decreasing -j of a single package may or may not help in this case.

    The filled tmpfs issue becomes more severe if I have tests enabled,
    because ctest has different behavior from make/ninja, and I use -j with combination with -l: make and ninja run at least one job to ensure some progress, while ctest does nothing when the load average is too high,
    and while it's doing nothing, the portage cannot finish the build and
    therefore cleanup the build dir from RAM.

    Neither of these are exactly relevant to this patch, tbh.

    04.01.2022 22:58, Sam James пишет:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Bug: https://bugs.gentoo.org/570534
    Signed-off-by: Sam James <sam@gentoo.org>
    ---
    eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
    1 file changed, 39 insertions(+), 3 deletions(-)

    diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
    index 2130e2e349141..8c4adc8b4f121 100644
    --- a/eclass/check-reqs.eclass
    +++ b/eclass/check-reqs.eclass
    @@ -43,6 +43,8 @@ case ${EAPI} in
    *) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
    esac

    +inherit multiprocessing
    +
    EXPORT_FUNCTIONS pkg_pretend pkg_setup

    if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
    @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
    # @DESCRIPTION:
    # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M

    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
    +
    # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
    # @DEFAULT_UNSET
    # @DESCRIPTION:
    @@ -346,9 +355,36 @@ _check-reqs_memory() {
    eend 0
    else
    eend 1
    - _check-reqs_unsatisfied \
    - ${size} \
    - "RAM"
    +
    + # Has the user allowed us to mangle their MAKEOPTS?
    + if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then
    + local jobs=$(makeopts_jobs)
    +
    + local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G)))
    + if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
    + # Number of jobs exceeds RAM/2GB, so clamp it.
    + local new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20))
    +
    + # This might _still_ be too big on small machines. Give up in such cases.
    + # (Users can still set the do nothing variable which is independent of this.)
    + if [[ $((new_jobs*2)) -gt ${estimated_max_memory} ]] ; then
    + _check-reqs_unsatisfied \
    + ${size} \
    + "RAM"
    + else
    + # The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
    + ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
    + ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
    + ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
    +
    + MAKEOPTS+=" -j${new_jobs}"
    + fi
    + fi
    + else
    + _check-reqs_unsatisfied \
    + ${size} \
    + "RAM"
    + fi
    fi
    else
    eend 1


    --
    Best regards,
    Alexey "DarthGandalf" Sokolov

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Alec Warner@21:1/5 to sam@gentoo.org on Wed Jan 5 19:00:01 2022
    On Tue, Jan 4, 2022 at 3:03 PM Sam James <sam@gentoo.org> wrote:



    On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:

    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Bug: https://bugs.gentoo.org/570534
    Signed-off-by: Sam James <sam@gentoo.org>
    ---
    eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
    1 file changed, 39 insertions(+), 3 deletions(-)


    Note that we discussed this on GitHub a bit when I just posted it there
    for some rough feedback: https://github.com/gentoo/gentoo/pull/23311.

    I think this is valuable for reducing invalid bug reports from OOM and
    easing user experience.

    Still kind of a WIP/rough draft, but may be ready in this state. Need
    more testing, so not planning on pushing yet or anything.

    I'm still not sure I grasp why we cannot make OOMs easier to discover
    from portage.

    Most packages don't even use check-reqs, so your solution is very
    narrow (and I get why, because you get a lot of bug reports from the
    big packages that do use it.)

    Can we write a build log analyzer?

    -A

    PS: If this was a global change I'd downvote it. It's only for
    check-reqs though and most packages don't use check-reqs; I don't
    really care. I'd be concerned about adopting this kind of approach
    wider; its very much a bandaid.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ulrich Mueller@21:1/5 to All on Wed Jan 5 19:30:01 2022
    On Wed, 05 Jan 2022, Florian Schmaus wrote:

    On 05/01/2022 09.28, Ulrich Mueller wrote:
    On Tue, 04 Jan 2022, Sam James wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.
    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.

    Surely not, but 2 GiB seems like a good start. I guess it could become
    an eclass variable that ebuilds could modify later (if the need
    emerges).

    We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
    to introduce another one that is counting memory per job? I doubt that
    would be possible, because the amount of memory greatly varies within a
    single build. For example, it is different for compiler vs linker vs
    building the documentation.

    It appears to me that the motivation for this change is to prevent
    users from running into OOM situations when emerging a package. And I
    believe that many users, especially novices, are not directly able to determine that portage failed due to OOM, simply because there is no
    direct hint in the build log. Hence opt-out appears to be sensible to
    me here.

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.

    Ulrich

    -----BEGIN PGP SIGNATURE-----

    iQFDBAEBCAAtFiEEtDnZ1O9xIP68rzDbUYgzUIhBXi4FAmHV4ecPHHVsbUBnZW50 b28ub3JnAAoJEFGIM1CIQV4u9k8H+wYRX//6grV9pDtIILQobWUIyuSOxQcPExmT ShgbP8H3b4D2oHItNab0tXzl5eeMUgY6nA9F2p4nIDPD0dmq4EC+5rejVRy76DRJ aALZyAGl3BI2J7FTfsXE50ffXqmCrexyd2O0owt3aVOIg9/m67D1isbt6nI7qtBH rUg1zD9YsF9NBcEwHpY4ymEgHdthbbKySoldII9aWP14uPm6TPVw351coYqEp5h3 nRRG3ZHMCcJQSagqfwfbDTLdGNwh5BSFdvwaTvmGtmi2UAXAzvy9VuUr/NTqvemo 70vBWZt/HlxCGTgjc3eKcfMz9odhfIrPxTaeBxAKlEs5o4Jw5Gg=
    =96Ml
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roy Bamford@21:1/5 to Sam James on Wed Jan 5 20:10:01 2022
    On 2022.01.04 22:58, Sam James wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Bug: https://bugs.gentoo.org/570534
    Signed-off-by: Sam James <sam@gentoo.org>
    ---
    eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
    1 file changed, 39 insertions(+), 3 deletions(-)

    diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
    index 2130e2e349141..8c4adc8b4f121 100644
    --- a/eclass/check-reqs.eclass
    +++ b/eclass/check-reqs.eclass
    @@ -43,6 +43,8 @@ case ${EAPI} in
    *) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
    esac

    +inherit multiprocessing
    +
    EXPORT_FUNCTIONS pkg_pretend pkg_setup

    if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
    @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
    # @DESCRIPTION:
    # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M

    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g. make,
    ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
    +
    # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
    # @DEFAULT_UNSET
    # @DESCRIPTION:
    @@ -346,9 +355,36 @@ _check-reqs_memory() {
    eend 0
    else
    eend 1
    - _check-reqs_unsatisfied \
    - ${size} \
    - "RAM"
    +
    + # Has the user allowed us to mangle their
    MAKEOPTS?
    + if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} ==
    "yes" ]] ; then
    + local jobs=$(makeopts_jobs)
    +
    + local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes
    1G)))
    + if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
    + # Number of jobs exceeds RAM/2GB,
    so clamp it.
    + local new_jobs=$(($(_check-reqs_get_number
    ${estimated_max_memory}G)*10/20))
    +
    + # This might _still_ be too big
    on small machines. Give up in such cases.
    + # (Users can still set the do
    nothing variable which is independent of this.)
    + if [[ $((new_jobs*2)) -gt ${estimated_max_memory}
    ]] ; then
    + _check-reqs_unsatisfied \
    + ${size} \
    + "RAM"
    + else
    + # The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
    + ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
    + ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
    + ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
    +
    + MAKEOPTS+=" -j${new_jobs}"
    + fi
    + fi
    + else
    + _check-reqs_unsatisfied \
    + ${size} \
    + "RAM"
    + fi
    fi
    else
    eend 1
    --
    2.34.1





    Sam,

    Do users with FEATURES=distcc still have to opt out of this
    MAKEOPTS clamping?

    --
    Regards,

    Roy Bamford
    (Neddyseagoon) a member of
    elections
    gentoo-ops
    forum-mods
    arm64
    -----BEGIN PGP SIGNATURE-----

    iQEzBAABCAAdFiEEsOrcx0gZrrCMwJzo/xJODTqpeT4FAmHV6zwACgkQ/xJODTqp eT4FrQf+JlRisY15PduLhbiE6eWKRlL/1Xy/0ZXDlurLMmA0kLEee/t24TjJjwbv 5icQBQPPUemgYhGRBBg/KLUXjWzkPllZAyk2wujn8gfKtY89Li0B5sET8Aq+MkHh jjWpSi4U3cTLiTAjuwRYhkMmPIHLaJLgwhF0UlBCKy5cPRDypjCHPFMllk/kPP1J O09ZWcB4sOlfc5T5tDVvunRbWVOe2OJvaeQcoCeq7g+sHNSp84PEfPG7VP5rMqBm qCNDn8p5vDcGqNBwe/Sq452i0bTjz/WAeijQDGtXwpXQC0+K7HI8fArx9R4kKkgZ 8uHeNyEJSxhOeAmQ9Fq4O/9+suKzhw==
    =kkFz
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kai Krakow@21:1/5 to All on Wed Jan 5 20:20:02 2022
    Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org>:

    On Wed, 05 Jan 2022, Florian Schmaus wrote:

    On 05/01/2022 09.28, Ulrich Mueller wrote:
    On Tue, 04 Jan 2022, Sam James wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.
    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.

    Surely not, but 2 GiB seems like a good start. I guess it could become
    an eclass variable that ebuilds could modify later (if the need
    emerges).

    We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
    to introduce another one that is counting memory per job? I doubt that
    would be possible, because the amount of memory greatly varies within a single build. For example, it is different for compiler vs linker vs
    building the documentation.

    It appears to me that the motivation for this change is to prevent
    users from running into OOM situations when emerging a package. And I believe that many users, especially novices, are not directly able to determine that portage failed due to OOM, simply because there is no
    direct hint in the build log. Hence opt-out appears to be sensible to
    me here.

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.

    Well, I'm using a safe combination of jobs and load-average, maybe the documentation should be tweaked instead.

    I'm using

    MAkEOPTS="--jobs=12 --load-average=8"

    on a 8x2+4 core CPU. I choose less then the available 20 cores because
    I'm also running multiple ebuilds in parallel to make better use of
    resources while most packages run their configure phase (which for
    small packages always takes longer than the build itself):

    EMERGE_DEFAULT_OPTS="--jobs=12 --load-average 10"

    The "--jobs" parameter is mostly a safe-guard against "make" or
    "emerge" overshooting the system resources which would happen if
    running unconstrained without "--load-average". The latter parameter
    OTOH tunes the parallel building processes automatically to the
    available resources. If the system starves of memory, thus starts to
    swap, load will increase, and make will reduce the jobs. It works
    pretty well.

    I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
    won't starve emerge from running configure phases of parallel ebuilds.

    Regards,
    Kai

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Florian Schmaus@21:1/5 to Ulrich Mueller on Wed Jan 5 20:20:02 2022
    On 05/01/2022 19.22, Ulrich Mueller wrote:
    On Wed, 05 Jan 2022, Florian Schmaus wrote:

    On 05/01/2022 09.28, Ulrich Mueller wrote:
    On Tue, 04 Jan 2022, Sam James wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.
    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.

    Surely not, but 2 GiB seems like a good start. I guess it could become
    an eclass variable that ebuilds could modify later (if the need
    emerges).

    We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
    to introduce another one that is counting memory per job? I doubt that
    would be possible, because the amount of memory greatly varies within a single build. For example, it is different for compiler vs linker vs
    building the documentation.

    I am not sure what the problem here supposed to be is.


    It appears to me that the motivation for this change is to prevent
    users from running into OOM situations when emerging a package. And I
    believe that many users, especially novices, are not directly able to
    determine that portage failed due to OOM, simply because there is no
    direct hint in the build log. Hence opt-out appears to be sensible to
    me here.

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.

    Yes, exactly. And it is a bandaid solution. But I believe that it will
    do more good than evil. And it is probably only used until portage is
    able to report to the user that the emerge failed due to OOM (which I
    believe to be non-trivial to implement, but I am happy to be proven
    otherwise).

    - Flow

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ulrich Mueller@21:1/5 to All on Wed Jan 5 21:00:01 2022
    On Wed, 05 Jan 2022, Florian Schmaus wrote:

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.

    Yes, exactly. And it is a bandaid solution. But I believe that it will
    do more good than evil. And it is probably only used until portage is
    able to report to the user that the emerge failed due to OOM (which I
    believe to be non-trivial to implement, but I am happy to be proven otherwise).

    Obviously I disagree. Tweaking the value for a subset of packages isn't
    a solution to the problem.

    MAKEOPTS applies to all parallel builds, and users should set it to a
    value suitable for their system (i.e. number of CPUs, available memory,
    etc.). Maybe our documentation needs to be improved? I see that make.conf.example says "The suggested number for parallel makes is
    CPUs+1" which may not be the best possible advice.

    Ulrich

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQFDBAEBCAAtFiEEtDnZ1O9xIP68rzDbUYgzUIhBXi4FAmHV9ysPHHVsbUBnZW50 b28ub3JnAAoJEFGIM1CIQV4uvpMH/3crgMdfCgHI5/xabxGpCNqgfLOb4xpzLb3E U2cShSzse2Yza8w1wpyHUvn2GCa8lIodN7sxKWDlw+e3GO3WH/6dCHg8It6jsXPY 3a0rjuqOWxnza+YC1UOd1Yza6dUBNyXi+e38nEDZkQ9OW+HNdA0XCPhvp5RmfDx8 zCHCo9dW1dyzGXwIhW4Mq0p1k0On8fpMO/LpisoDHlAiQo9YDRJgenTg9XtPuFs8 hOS9vL2AHd2S0+IJHbhYvX38GlXjp7YVL48ejYMzMWxTTO/q5oqX8wD3+Lq4TqII V5tta4eaW6GUwtuSX7PZk6PTBw1x4J6zjMW2K0zzyxsXLNQwsnI|Ym
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam James@21:1/5 to All on Wed Jan 5 21:30:02 2022
    On 5 Jan 2022, at 17:51, Alec Warner <antarus@gentoo.org> wrote:
    On Tue, Jan 4, 2022 at 3:03 PM Sam James <sam@gentoo.org> wrote:

    On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.
    [snip]

    I'm still not sure I grasp why we cannot make OOMs easier to discover
    from portage.

    Most packages don't even use check-reqs, so your solution is very
    narrow (and I get why, because you get a lot of bug reports from the
    big packages that do use it.)


    Yeah, you've got the thinking here :)

    Can we write a build log analyzer?

    I think this is a good idea, even if it's just a few hand-gathered regexes,
    we can try improve the situation a bit, even for non-OOM (like
    some common Perl-related failures).


    -A

    PS: If this was a global change I'd downvote it. It's only for
    check-reqs though and most packages don't use check-reqs; I don't
    really care. I'd be concerned about adopting this kind of approach
    wider; its very much a bandaid.


    Yep, I don't deny that, and I can't really say I'd support this
    if it was global. The specific intent here is that check-reqs
    consumers are generally big beasts and hence it's a direct,
    targeted action to reduce bad bug reports and improve
    the user experience, especially for newcomers who
    end up hitting OOMs early on.

    Best,
    sam


    -----BEGIN PGP SIGNATURE-----

    iQGTBAEBCgB9FiEEYOpPv/uDUzOcqtTy9JIoEO6gSDsFAmHV/mVfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYw RUE0RkJGRkI4MzUzMzM5Q0FBRDRGMkY0OTIyODEwRUVBMDQ4M0IACgkQ9JIoEO6g SDudiQf+ItlvBHNE+p/mp90VFKrj7TMDe7t/sTZiCpgvLou3PBF4z6r3VtXp0Hbw ssYfQu3cEAY9rUa6lfx9NnpbMHQoDIb0K90Mv70eT7WDPLDhmPdtILUnf1MK6Zo3 VlzvQQMYImCxkZQ9sW+Ap9Z/55Lie7qhJy1z5MGWPOe3sLkm/4qNIAwlwt+EWF1x MScwzsif4TwuWLE2NkUGTvKpO9v71iEVMfL6RRsStXuW6qQc0vLlPQJSjmb7kpsg SMXO3caFbC1gMVfksvinLVCwMt7qKW6rUjACyFBlel2gaJ0yDyFqx3yiBdNYjU6h Yf3wamJlaOzI9oprrcziatD6WfP2jw==
    =69G9
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam James@21:1/5 to All on Wed Jan 5 21:20:01 2022
    On 5 Jan 2022, at 19:53, Ulrich Mueller <ulm@gentoo.org> wrote:

    On Wed, 05 Jan 2022, Florian Schmaus wrote:

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.

    Yes, exactly. And it is a bandaid solution. But I believe that it will
    do more good than evil. And it is probably only used until portage is
    able to report to the user that the emerge failed due to OOM (which I
    believe to be non-trivial to implement, but I am happy to be proven
    otherwise).

    Obviously I disagree. Tweaking the value for a subset of packages isn't
    a solution to the problem.

    MAKEOPTS applies to all parallel builds, and users should set it to a
    value suitable for their system (i.e. number of CPUs, available memory, etc.). Maybe our documentation needs to be improved? I see that make.conf.example says "The suggested number for parallel makes is
    CPUs+1" which may not be the best possible advice.


    As you've noted, the memory requirements vary per package,
    and it was somewhat of an oversight/error to not include
    the memory limit from the eclass variable instead here
    (although I recall kind of deciding to go with this approach
    for simplicity).

    It's therefore nontrivial to come up with a good value for their
    whole system :)

    Our documentation has mostly been updated but apparently
    make.conf.example hasn't been.


    -----BEGIN PGP SIGNATURE-----

    iQGTBAEBCgB9FiEEYOpPv/uDUzOcqtTy9JIoEO6gSDsFAmHV/KxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYw RUE0RkJGRkI4MzUzMzM5Q0FBRDRGMkY0OTIyODEwRUVBMDQ4M0IACgkQ9JIoEO6g SDtvswgAqJqdLDJImST3qryYafXthrjdzoeIvF74ISUtQI5wJb5ObYIz0b3qjF3P wdc0/vgetrJ09but+ugoYhBmISb/55esAagFsfYXffonwEYSY3xWgkOvrsKnIotX EDt/pH4HsESBcSIiQnQ0WCsbAlGVKLIMHoaBhvHKfGacDcT+uLpqLD6BT7kl6UhZ CLmEObxrCsI/UfkMqdcoqODXvuUeKpNQoczDF8JHiUy4q/UZMYzLGQV4QGG8G5us 7hXnlG3azE9xC5/974P5XbH6hb0c0ymBy53pCoU3w2YT/mbK+uv/mCfsAM87qJEa OWz8PUdgCpYzo+JYjlemqsJDj+6gSQ==
    =LeVR
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Seifert@21:1/5 to Ulrich Mueller on Wed Jan 5 21:40:01 2022
    On Wed, 2022-01-05 at 09:28 +0100, Ulrich Mueller wrote:
    On Tue, 04 Jan 2022, Sam James wrote:

    Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
    amount of RAM available (uses amount declared as needed
    in the ebuild). Typically should be ~2GB per job.

    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.
    (If only considering C and C++ 2 GB seems to be excessive, i.e. it
    will limit parallel build more than necessary. When linking, the
    number
    may be _much_ larger.)

    https://wiki.gentoo.org/wiki/MAKEOPTS
    The rule of thumb is MAKEOPTS jobs should be the smaller number of: the
    size of RAM divided by 2GB or the number of threads the CPU has.

    You're free to propose a better scheme, but try compiling some template-
    heavy C++ code with 2GB/thread and -O3 and then WONTFIXing all the bug
    reports. OTOH, if you emake -j1, people complain about stuff being
    slower than glacial (https://bugs.gentoo.org/744088). Unless I see a
    better solution, sam's solution is the best way to avoid these annoying,
    rare, edge-case pitfalls.

    Also not sure if I understand the arithmetic. Shouldn't it use CHECKREQS_MEMORY as the basis of the calculation?

    +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# Allow packages to reduce the number of multiprocessing (e.g.
    make, ninja) jobs
    +# to lower memory usage.
    +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}

    If anything, the feature should be opt-in rather than opt-out.

    Ulrich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roy Bamford@21:1/5 to Sam James on Wed Jan 5 22:10:01 2022
    On 2022.01.05 20:22, Sam James wrote:


    On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
    wrote:

    Sam,

    Do users with FEATURES=distcc still have to opt out of this
    MAKEOPTS clamping?


    Great point! I think we could add an exemption for that and make it a
    noop or warning-only.

    Best,
    sam




    Sam,

    You are building a better mousetrap here. That's not a reason to try.

    Do users of I_KNOW_WHAT_I_AM_DOING, who have already
    opted to shoot themselves in both feet, get a free pass here?

    There are users who run emerge --jobs=X with MAKEOPTS='-jY"
    and get firefox, thunderbird and libreoffice all building concurrently
    as they allow X * Y MAKE threads, reduced by this proposed
    throttling, still triggering the OOM.

    I don't think you can head that off beforehand.

    --
    Regards,

    Roy Bamford
    (Neddyseagoon) a member of
    elections
    gentoo-ops
    forum-mods
    arm64
    -----BEGIN PGP SIGNATURE-----

    iQEzBAABCAAdFiEEsOrcx0gZrrCMwJzo/xJODTqpeT4FAmHWCD0ACgkQ/xJODTqp eT6BdwgAv/UhktydJ82g3D8QWN8Vk67Ajt6DEiDp+XbGQRZ/BVzcMPUfaVH8aqcr yFaWTNk2vWDPcy8I5QwUzop52gVqc03xUtsVJt8XOSxTvf+9t0oN+BHAM0P1WfIO xVjVzoTWrqiNl5JrS9/QkVZKeOr/h4GeyhJxpmAo9qNEP0iw28fJAp4gWMg3u4Ar +vOr5TKUupHS4pAIoOEhrO+zDZSCulhJgpmCXda4tMd+SrrxaiF0uWN1dGFgBkp7 mmBrNiRlXzWkzbC0SSI7O1TrlL2zxA1qZGRwuPv2bxEzO4fOKviCoiePTk9glObX TQ2EmRP6u9JJ2QUpcnuZJf7n617pQw==
    =5CBb
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ulrich Mueller@21:1/5 to All on Wed Jan 5 22:20:01 2022
    On Wed, 05 Jan 2022, Sam James wrote:

    On 5 Jan 2022, at 08:28, Ulrich Mueller <ulm@gentoo.org> wrote:

    Where does this number 2 GB come from? The amount of RAM strongly
    depends on the programming language and other factors, so I don't
    believe that there's one number that can be used for everything.
    (If only considering C and C++ 2 GB seems to be excessive, i.e. it
    will limit parallel build more than necessary. When linking, the number
    may be _much_ larger.)

    This is essentially "common law" (or "common lore" if you like!) and
    is well-accepted as a good rule of thumb.

    Well, if the 2 GiB was a universal constant then it would be valid for
    _all_ builds, not just the random subset inheriting check-reqs.eclass.

    The number being larger doesn't really make any difference here;
    the point is to help in cases where we're pretty sure there's going
    to be a problem (only users of check-reqs.eclass, and we can
    Introduce a variable to give ~RAM per job too if needed).

    Yeah, having the ebuild specify an estimate for the average memory usage
    per job would be more correct.

    These are all band-aids however. What we would really need are options
    like GNU parallel's --memfree and --memsuspend.

    This is also about reducing the number of support queries about
    builds which failed for trivial reasons, as someone who has to handle
    quite a lot of those (until such a time as we can implement better
    detection, but this is also a generally nice UX improvement -- as
    per what Florian/flow said).

    But it would defeat the YAFIYGI principle that we commonly apply. :)

    Ulrich

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQFDBAEBCAAtFiEEtDnZ1O9xIP68rzDbUYgzUIhBXi4FAmHWCg0PHHVsbUBnZW50 b28ub3JnAAoJEFGIM1CIQV4uNS0IAJ8GB16BwAcJ54A6P0ZbFLfUK+ZZBC6CndAL FiWEQyfwNzyhsiLV79jL+jzYfm9cBrmU3acAYeyUSMtrRWhQT6k7gAKiTCN+2aR0 vG62aKAPO60ZCfeAqsfXI0C1HN7Qwl+r1u2sxo5bMFU9ubFLViNXP6SYZvnfFuZg t/YPG/L3b09fNvOwALN+29Rb1/1dePzFiutA8mZ8b48/IK31bdqD5srMz0c5Yti3 zdWB/PLQZOmflPn5HOkT6LmwUNXGYwXmTxU/BMDunvUP9Qby54lFlgTZjKUdQOrX xoCllr7xIU0Be7tZBPNKB0ZQAB7veB4gMlge4wBDdHHgQyTaIBc=T2iE
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dale@21:1/5 to Roy Bamford on Wed Jan 5 23:00:01 2022
    Roy Bamford wrote:
    On 2022.01.05 20:22, Sam James wrote:

    On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
    wrote:
    Sam,

    Do users with FEATURES=distcc still have to opt out of this
    MAKEOPTS clamping?

    Great point! I think we could add an exemption for that and make it a
    noop or warning-only.

    Best,
    sam



    Sam,

    You are building a better mousetrap here. That's not a reason to try.

    Do users of I_KNOW_WHAT_I_AM_DOING, who have already
    opted to shoot themselves in both feet, get a free pass here?

    There are users who run emerge --jobs=X with MAKEOPTS='-jY"
    and get firefox, thunderbird and libreoffice all building concurrently
    as they allow X * Y MAKE threads, reduced by this proposed
    throttling, still triggering the OOM.

    I don't think you can head that off beforehand.



    As a user, can I get a large +1 to that.  For me, it is usually
    Seamonkey, Firefox, Libreoffice, that big qt package or some combination
    of two or more of them.  I've had times where all of those just happen
    to upgrade at the same time.  My solution was to put those on spinning
    rust while using tmpfs for everything else and if needed, using
    --exclude to put off one or more of them then do those later. 

    While I like the general idea of this, and would love to see emerge be
    able to handle it without failures, I'm not sure how emerge or it's
    options can prevent it without slowing down other packages.  My
    thinking, have emerge trigger when certain packages are in the upgrade
    list and only allow one package in that list to update at a time.  For example.  If Firefox, Libreoffice and that qt package are in the list of upgrades, whichever hits first causes the others to wait until the first
    one is done.  That will make it so only one large package is being
    upgraded at a time and be able to have fast settings for other smaller
    and less memory hungry packages as well. 

    How to do that, I'm not a coder so no idea but I know there are some
    awesome coders here who may can find a way.  If that idea is even
    something that could be done. 

    Since I rarely post here, keep up the great work.  :-D

    Dale

    :-)  :-) 

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Seifert@21:1/5 to Roy Bamford on Wed Jan 5 22:20:01 2022
    On Wed, 2022-01-05 at 21:06 +0000, Roy Bamford wrote:
    On 2022.01.05 20:22, Sam James wrote:


    On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
    wrote:

    Sam,

    Do users with FEATURES=distcc still have to opt out of this
    MAKEOPTS clamping?


    Great point! I think we could add an exemption for that and make it
    a
    noop or warning-only.

    Best,
    sam




    Sam,

    You are building a better mousetrap here. That's not a reason to try.

    Do users of I_KNOW_WHAT_I_AM_DOING, who have already
    opted to shoot themselves in both feet, get a free pass here? 

    There are users who run emerge --jobs=X with MAKEOPTS='-jY"
    and get firefox, thunderbird and libreoffice all building concurrently
    as they allow X * Y MAKE threads, reduced by this proposed
    throttling, still triggering the OOM.

    I don't think you can head that off beforehand.


    What's your proposed alternative?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roy Bamford@21:1/5 to David Seifert on Wed Jan 5 23:30:01 2022
    On 2022.01.05 21:10, David Seifert wrote:
    On Wed, 2022-01-05 at 21:06 +0000, Roy Bamford wrote:
    On 2022.01.05 20:22, Sam James wrote:


    On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
    wrote:

    Sam,

    Do users with FEATURES=distcc still have to opt out of this
    MAKEOPTS clamping?


    Great point! I think we could add an exemption for that and make
    it
    a
    noop or warning-only.

    Best,
    sam




    Sam,

    You are building a better mousetrap here. That's not a reason to
    try.

    Do users of I_KNOW_WHAT_I_AM_DOING, who have already
    opted to shoot themselves in both feet, get a free pass here? 

    There are users who run emerge --jobs=X with MAKEOPTS='-jY"
    and get firefox, thunderbird and libreoffice all building
    concurrently
    as they allow X * Y MAKE threads, reduced by this proposed
    throttling, still triggering the OOM.

    I don't think you can head that off beforehand.


    What's your proposed alternative?



    I don't really have one ... hence the better mousetrap analogy at
    the start of my post.

    grep -i killed

    on the build log, if portage can do that and tell the user the rules
    of thumb to try to stop it in future.

    --
    Regards,

    Roy Bamford
    (Neddyseagoon) a member of
    elections
    gentoo-ops
    forum-mods
    arm64
    -----BEGIN PGP SIGNATURE-----

    iQEzBAABCAAdFiEEsOrcx0gZrrCMwJzo/xJODTqpeT4FAmHWGgQACgkQ/xJODTqp eT7KgAf+MAu7MMG40Pqix0TdSWXTXMZCLMB9yhV04H2N7v35xdZXCV1xi027/7Ls Jo2b1qZwOdnMd6FAjkEj4bZ3SKQWMZGaPqJeTzwH3xEPnakj+/JuLzwl0oI9kZpu SLhhQX9bTcRPX51LK4/osRDscgPXEq3IYcOdq3Ptt5Np+3+usI6DRqkVtcXQzXPM Evo6GO+AyWDfwodrDNA9xVICzHJGHZ/GMYQKwcNmtZikSQWzMA6rU/S2SgYTtOSd UbVpbCBWqwlravqyHovXIOU3Vmbw0Ly2YsypxNVv3bvWXTrYDxiFDpdqtAtF4Ckq ZLJ64vZZkXmhnQqBaCmXOzwHZrNOeQ==
    =JWkG
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kai Krakow@21:1/5 to All on Thu Jan 6 02:50:02 2022
    Am Mi., 5. Jan. 2022 um 21:21 Uhr schrieb Sam James <sam@gentoo.org>:

    On 5 Jan 2022, at 19:18, Kai Krakow <kai@kaishome.de> wrote:

    Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org>:

    [...]

    That applies to all parallel builds though, not only to ebuilds
    inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
    telling the user that the --jobs setting in their make.conf is wrong,
    in the first place.


    Well, I'm using a safe combination of jobs and load-average, maybe the
    documentation should be tweaked instead.


    I think "safe" is doing some heavy lifting here...

    Well, works "safe" for me at least, but you're right.

    I'm using
    [...]


    The "--jobs" parameter is mostly a safe-guard against "make" or
    "emerge" overshooting the system resources which would happen if
    running unconstrained without "--load-average". The latter parameter
    OTOH tunes the parallel building processes automatically to the
    available resources. If the system starves of memory, thus starts to
    swap, load will increase, and make will reduce the jobs. It works
    pretty well.

    I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
    won't starve emerge from running configure phases of parallel ebuilds.


    ... because it's quite hard for this logic to work correctly enough
    of the time without jobserver integration (https://bugs.gentoo.org/692576).

    Oh there's a bug report about this... I already wondered: Wouldn't it
    be better if it had a global jobserver? OTOH, there are so many build
    systems out there which parallelize building, and many of them won't
    use a make jobserver but roll their own solution. So it looks a bit
    futile on that side. That's why I've chosen the loadavg-based
    approach.

    But indeed, I'd say you're not the target audience for this (but I appreciate the input).

    Maybe not, I'm usually building in tmpfs (except huge source archives
    with huge build artifacts), that means, I usually have plenty of RAM,
    at least enough so it doesn't become the limiting factor.

    But then again, what is the target audience? This proposal looks like
    it tries to predict the future, and that's probably never going to
    work right. Looking at the Github issue linked initially in the
    thread, it looks like I /might/ be the target audience for packages
    like qtwebkit because I'm building in tmpfs. The loadavg limiter does
    quite well here unless a second huge ebuild becomes unpacked and built
    in the tmpfs, at which point the system struggles to keep up and
    starves from IO thrashing just to OOM portage a few moments later.
    That's of course not due to the build jobs itself then, it's purely a
    memory limitation. But for that reason I have configuration to build
    such packages outside of tmpfs: While they usually work fine when
    building just that package alone, it fails the very moment two of such
    packages are built in parallel.

    Maybe portage needs a job server that dynamically bumps the job
    counter up or down based on current memory usage? Or "make" itself
    could be patched to take that into account? But that's probably the
    whole idea of the loadavg limiter. So I'd propose to at least mention
    that in the documentation and examples, it seems to only be little
    known.

    Then again, if we run in a memory constrained system, it may be better
    to parallelize ebuilds instead of build jobs to better make use of
    combining light and heavy ebuild phases into the same time period.

    Also, I'm not sure if 2 GB per job is the full picture - no matter if
    that number is correct or isn't... Because usually the link phase of
    packages like Chrome is the real RAM burner even with sane "jobs"
    parameters. I've seen people failing to install these packages because
    they didn't turn on swap, and then during the link phase, the compiler
    took so much memory that it either froze the system for half an hour,
    or OOMed. And at that stage, there's usually just this single compiler
    process running (and maybe some small ones which almost use no memory
    relative to that). And that doesn't get better with modern compilers
    doing all sorts of global optimization stuff like LTO.

    So maybe something like this could work (excluding the link phase):

    If there's potentially running just one ebuild at a time (i.e. your
    merge list has just one package), the effects of MAKEOPTS is quite
    predictable. But if we potentially run more, we could carefully reduce
    the number of jobs in MAKEOPTS before applying additional RAM
    heuristics. And those heuristics probably should take the combination
    of both emerge jobs and make jobs into account because potentially
    that multiplies (unless 692576 is implemented).

    Compiler and link flags may also be needed to take into account.

    And maybe portage should take care of optionally serializing huge
    packages and never build/unpack them at the same time. This would be a
    huge winner for me so I would not have to manually configure things... Something like PORTAGE_SERIALIZE_CONSTRAINED="1" to build at most one
    package that has some RAM/storage warning vars in the ebuild. But
    that's probably a different topic as it doesn't exactly target the
    problem discussed here - and I'm also aware of this problem unlike the
    target audience.


    Regards,
    Kai

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)