• [gentoo-dev] The inherit-EXPORT_FUNCTIONS ordering problem

    From =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=@21:1/5 to All on Sat Aug 28 18:40:01 2021
    Hi,

    I've been informed of a slight inconsistency in package manager behavior
    that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    xdg.eclass:

    EXPORT_FUNCTIONS src_prepare

    ecm-1.eclass:

    inherit xdg
    EXPORT_FUNCTIONS src_prepare

    ecm-2.eclass:

    EXPORT_FUNCTIONS src_prepare
    inherit xdg


    Now, ecm-1.eclass produces consistent behavior across all PMs -- ecm-1_src_prepare takes precedence. However, ecm-2.eclass is not
    consistent:

    - Portage will take ecm-2_src_prepare, i.e. applies precedence based
    on inherit order and not actual call order

    - PkgCore and Paludis will take xdg_src_prepare since its
    EXPORT_FUNCTIONS are called later (since the inherit is done later).

    Apparently, the Portage behavior was changed in 2009 [1]. PMS is not
    very clear on what should happen.


    Therefore:

    1. I'd like to propose that we explicitly require all inherits to happen
    before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.


    WDYT?


    [1] https://github.com/gentoo/portage/commit/06d4433e8b8be60d606733b9e23f57f8a5869d8f

    --
    Best regards,
    Michał Górny

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ulrich Mueller@21:1/5 to All on Sat Aug 28 19:30:01 2021
    On Sat, 28 Aug 2021, Michał Górny wrote:

    I've been informed of a slight inconsistency in package manager behavior
    that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    xdg.eclass:

    EXPORT_FUNCTIONS src_prepare

    ecm-1.eclass:

    inherit xdg
    EXPORT_FUNCTIONS src_prepare

    ecm-2.eclass:

    EXPORT_FUNCTIONS src_prepare
    inherit xdg


    Now, ecm-1.eclass produces consistent behavior across all PMs -- ecm-1_src_prepare takes precedence. However, ecm-2.eclass is not
    consistent:

    - Portage will take ecm-2_src_prepare, i.e. applies precedence based
    on inherit order and not actual call order

    But it is the reverse of normal inherit order. That is, if an ebuild
    does:

    inherit ecm-2 xdg

    then xdg takes precedence. However, in your second example above, the
    inherit order is the same (ecm-2 first, xdg second), but with Portage
    the _earlier_ eclass takes precedence.

    - PkgCore and Paludis will take xdg_src_prepare since its
    EXPORT_FUNCTIONS are called later (since the inherit is done later).

    Which is consistent with the behaviour when both eclasses are inherited directly from the ebuild.

    Apparently, the Portage behavior was changed in 2009 [1]. PMS is not
    very clear on what should happen.

    Can you provide any pointer to a discussion of that Portage change in gentoo-dev?


    Therefore:

    1. I'd like to propose that we explicitly require all inherits to happen before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.

    WDYT?

    Not sure, but if we follow [2] then this qualifies as a Portage bug.
    I'd really hate to retroactively change the spec to "undefined".

    Ulrich


    [1] https://github.com/gentoo/portage/commit/06d4433e8b8be60d606733b9e23f57f8a5869d8f

    [2] https://projects.gentoo.org/council/meeting-logs/20160313-summary.txt

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQFDBAEBCAAtFiEEtDnZ1O9xIP68rzDbUYgzUIhBXi4FAmEqch8PHHVsbUBnZW50 b28ub3JnAAoJEFGIM1CIQV4uyIEH+gN9m/W2s5Ld6/TlI1OOpF+MXGm6VnBPLOc3 U3Tt5qT1HH0CIlHnPQvev7EMT+axB0aa3/QQXTZdrMD1U52b2tiumRseIODCeZO9 3Bxs6PytaUyPGr0sZipLqnHuDKfNTOOVY5jmNyR0Yt2iv6vC5w96J4FzHjXtt3Ld KG0mqrWgZFDkGLGZcLVB7qy4kzV+K3QmtByTzrS+LGvu9nGo5/2mRNCubu8PsUJK YUI79y8l0k4EzhMC6+qaFZ0X/z4VXQTfMALtryJwItU/iOQfunZOkzcqGeAK7aDJ ITWCAShHO4s+2k7M5XMTNsodLEWyRL9xaNNPSdjVOH8tizewmxg=9sRr
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=@21:1/5 to William Hubbs on Sun Aug 29 19:50:01 2021
    On Sun, 2021-08-29 at 12:23 -0500, William Hubbs wrote:
    On Sat, Aug 28, 2021 at 06:35:06PM +0200, Michał Górny wrote:
    Hi,

    I've been informed of a slight inconsistency in package manager behavior that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    ...

    1. I'd like to propose that we explicitly require all inherits to happen before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.

    I would go with 1 and 2 b, but I also propose another option for the
    next eapi which may be a bit controversial.

    Starting with eapi 9, make export_functions a noop (you can't remove it
    until all eclasses/ebuilds only support eapis that don't require it).

    I understand that this is controversial, because it would require a lot
    of work to convert ebuilds to eapi 9, but it would eliminate this ambiguity/inconsistency in the future because it would require all
    ebuilds to have phase functions unless they can use the default phases
    the eapis provide.

    Thoughts?


    It would not only require work to convert ebuilds but it would also
    require a lot of work when writing new ebuilds. The developers would
    have to explicitly remember to call the correct set of phases for each inherited eclass, and this is a lot of work for the lot of otherwise
    trivial ebuilds.

    Not to mention the quite obvious risk of silent breakage when people
    forget to call a phase function. Just imagine that all python-single-r1 ebuilds would have to call python-single-r1_pkg_setup explicitly or they
    would silently fall back to random Python version that's going to work
    90% of the time, so people won't even notice that the ebuild is broken.

    In the end, this would naturally lead to people hacking this around by reinventing EXPORT_FUNCTIONS inside eclasses. In the best case, we'd
    see functions like 'cmake_export_phases' -- which maybe, just maybe,
    would be better than implicit exports.

    --
    Best regards,
    Michał Górny

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From William Hubbs@21:1/5 to All on Sun Aug 29 19:40:02 2021
    On Sat, Aug 28, 2021 at 06:35:06PM +0200, Michał Górny wrote:
    Hi,

    I've been informed of a slight inconsistency in package manager behavior
    that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    ...

    1. I'd like to propose that we explicitly require all inherits to happen before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.

    I would go with 1 and 2 b, but I also propose another option for the
    next eapi which may be a bit controversial.

    Starting with eapi 9, make export_functions a noop (you can't remove it
    until all eclasses/ebuilds only support eapis that don't require it).

    I understand that this is controversial, because it would require a lot
    of work to convert ebuilds to eapi 9, but it would eliminate this ambiguity/inconsistency in the future because it would require all
    ebuilds to have phase functions unless they can use the default phases
    the eapis provide.

    Thoughts?

    William


    WDYT?


    [1] https://github.com/gentoo/portage/commit/06d4433e8b8be60d606733b9e23f57f8a5869d8f

    --
    Best regards,
    Michał Górny




    -----BEGIN PGP SIGNATURE-----

    iF0EABECAB0WIQTVeuxEZo4uUHOkQAluVBb0MMRlOAUCYSvChQAKCRBuVBb0MMRl OD4mAKCyuSnQKjF+OaSiKnbjYJcjuxfyAACeKbU8JfpOW2jyQzDFejphXv2ykDo=
    =9Bhj
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ionen Wolkens@21:1/5 to William Hubbs on Sun Aug 29 20:50:01 2021
    On Sun, Aug 29, 2021 at 12:23:22PM -0500, William Hubbs wrote:
    On Sat, Aug 28, 2021 at 06:35:06PM +0200, Michał Górny wrote:
    Hi,

    I've been informed of a slight inconsistency in package manager behavior that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    ...

    1. I'd like to propose that we explicitly require all inherits to happen before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.

    I would go with 1 and 2 b, but I also propose another option for the
    next eapi which may be a bit controversial.

    Starting with eapi 9, make export_functions a noop (you can't remove it
    until all eclasses/ebuilds only support eapis that don't require it).

    I understand that this is controversial, because it would require a lot
    of work to convert ebuilds to eapi 9, but it would eliminate this ambiguity/inconsistency in the future because it would require all
    ebuilds to have phase functions unless they can use the default phases
    the eapis provide.

    Thoughts?

    If anything I'd personally prefer the rough opposite of this, in the
    event multiple eclass export the same, have the package manager merge
    them.

    e.g. rather than have `inherit cmake xdg` run xdg_src_prepare over
    cmake's, would export:

    src_prepare() {
    cmake_src_prepare
    xdg_src_prepare
    }

    unless the ebuild redefines it (which I imagine would often be because
    of optional support, e.g. use lua &&, or occasional incompatibilities)

    Some ebuilds have special needs, but then there's all those generic
    ebuilds that should stay nearly empty.
    --
    ionen

    -----BEGIN PGP SIGNATURE-----

    iQEzBAABCAAdFiEEx3SLh1HBoPy/yLVYskQGsLCsQzQFAmEr1L0ACgkQskQGsLCs QzRRWAf/VXrl/00o1Yq5lL+xN5rR0sohGXSTnuy8exqEhESTMX/TKcOogiFfGwun lTNawOOla8zJ7d2twgdR8+uWhYAu8UH4CJILSFIF5t9Ro1PPSgBXbyr9RfE5Heya 1vtvp3Le6fcQfCi9M7JLfhG3RsOF00QvBbdFRCgkG9RvIeUOgtWwDPiVYVNQQSRr YcgZrXOmnOkpR7gEPsP2TUNLcedT19cTiJ8+iRw3ol4OnEEe0QoD5TFLySgLTHd3 aDMVI7H1/OimxGPjP5eoZfSSQzrNW+GcB9liqlO6roVfanjQacz0Ewh4u06Vzjey yx72F6sp2x9+t3f0UleocExvMd/4/g==
    =CQ/O
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=@21:1/5 to Ionen Wolkens on Sun Aug 29 20:50:02 2021
    On Sun, 2021-08-29 at 14:41 -0400, Ionen Wolkens wrote:
    On Sun, Aug 29, 2021 at 12:23:22PM -0500, William Hubbs wrote:
    On Sat, Aug 28, 2021 at 06:35:06PM +0200, Michał Górny wrote:
    Hi,

    I've been informed of a slight inconsistency in package manager behavior that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks for the report!). Please consider the three following snippets:

    ...

    1. I'd like to propose that we explicitly require all inherits to happen before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either solution is correct.

    I would go with 1 and 2 b, but I also propose another option for the
    next eapi which may be a bit controversial.

    Starting with eapi 9, make export_functions a noop (you can't remove it until all eclasses/ebuilds only support eapis that don't require it).

    I understand that this is controversial, because it would require a lot
    of work to convert ebuilds to eapi 9, but it would eliminate this ambiguity/inconsistency in the future because it would require all
    ebuilds to have phase functions unless they can use the default phases
    the eapis provide.

    Thoughts?

    If anything I'd personally prefer the rough opposite of this, in the
    event multiple eclass export the same, have the package manager merge
    them.

    e.g. rather than have `inherit cmake xdg` run xdg_src_prepare over
    cmake's, would export:

    src_prepare() {
    cmake_src_prepare
    xdg_src_prepare
    }

    unless the ebuild redefines it (which I imagine would often be because
    of optional support, e.g. use lua &&, or occasional incompatibilities)

    Some ebuilds have special needs, but then there's all those generic
    ebuilds that should stay nearly empty.

    What about transitive inherits? Say, foo.eclass inherits bar.eclass.
    Does that imply that the ebuild calls foo_src_prepare
    and bar_src_prepare, and the eclass has no way of overriding this? What
    if the eclass needs bar_src_prepare called explicitly before or after
    its own src_prepare?

    --
    Best regards,
    Michał Górny

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam James@21:1/5 to All on Sun Aug 29 21:00:01 2021
    On 29 Aug 2021, at 19:44, Michał Górny <mgorny@gentoo.org> wrote:

    On Sun, 2021-08-29 at 14:41 -0400, Ionen Wolkens wrote:
    [snip]

    If anything I'd personally prefer the rough opposite of this, in the
    event multiple eclass export the same, have the package manager merge
    them.

    e.g. rather than have `inherit cmake xdg` run xdg_src_prepare over
    cmake's, would export:

    src_prepare() {
    cmake_src_prepare
    xdg_src_prepare
    }

    unless the ebuild redefines it (which I imagine would often be because
    of optional support, e.g. use lua &&, or occasional incompatibilities)

    Some ebuilds have special needs, but then there's all those generic
    ebuilds that should stay nearly empty.

    What about transitive inherits? Say, foo.eclass inherits bar.eclass.
    Does that imply that the ebuild calls foo_src_prepare
    and bar_src_prepare, and the eclass has no way of overriding this? What
    if the eclass needs bar_src_prepare called explicitly before or after
    its own src_prepare?

    My expectation would be that the ebuild only called depth=1 exported
    functions and it would be the responsibility of each eclass to call its own inherits. But this becomes a problem with repeated inherits given our
    phases aren't usually idempotent.

    best,
    sam


    -----BEGIN PGP SIGNATURE-----

    iQGTBAEBCgB9FiEEYOpPv/uDUzOcqtTy9JIoEO6gSDsFAmEr2LJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYw RUE0RkJGRkI4MzUzMzM5Q0FBRDRGMkY0OTIyODEwRUVBMDQ4M0IACgkQ9JIoEO6g SDuK4Af9FjXryvpp7ak18xm2gqYxwIuAMpSJzon/8AHK8apcBnw9YknPxExYazzs WeW0X5XXWDg1796SNYQCgHs8pQbrILo8WkD6jgap6rCs4lb7UF7uEJFgghDKWHXk 7V3D7Tvrla8BPSFk3cyS1YgN3v2fAA35PvKx8SyYGkPWUrBPYDeMZE+I4oSodrVl vVGPcD1tD/7ohE0RR9I+qJ7A50d7mUDddw7irPb2BrpZ9hPLPmRl/qc8XxFwWvM+ xsI2hCr2M3wB976MpUalUwGSgs9OBfRvSnF8c2W+n13HcNxo02WtANj2NAXRzibL Vuj8RhpwVjsMom7YXSg7luzwEp2xaw==
    =HtZp
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam James@21:1/5 to All on Sun Aug 29 20:50:02 2021
    On 29 Aug 2021, at 19:41, Ionen Wolkens <ionen@gentoo.org> wrote:

    On Sun, Aug 29, 2021 at 12:23:22PM -0500, William Hubbs wrote:
    On Sat, Aug 28, 2021 at 06:35:06PM +0200, Michał Górny wrote:
    Hi,

    I've been informed of a slight inconsistency in package manager behavior >>> that affects combining EXPORT_FUNCTIONS with inherit (by ionic, thanks
    for the report!). Please consider the three following snippets:

    ...

    1. I'd like to propose that we explicitly require all inherits to happen >>> before EXPORT_FUNCTIONS. This will ensure consistent behavior across
    all package managers.

    2. I'd like to ask your opinion whether we should:

    a. revert the Portage behavior to be consistent with PkgCore/Paludis

    b. update PMS to identify the behavior as 'undefined', i.e. either
    solution is correct.

    I would go with 1 and 2 b, but I also propose another option for the
    next eapi which may be a bit controversial.

    Starting with eapi 9, make export_functions a noop (you can't remove it
    until all eclasses/ebuilds only support eapis that don't require it).

    I understand that this is controversial, because it would require a lot
    of work to convert ebuilds to eapi 9, but it would eliminate this
    ambiguity/inconsistency in the future because it would require all
    ebuilds to have phase functions unless they can use the default phases
    the eapis provide.

    Thoughts?

    If anything I'd personally prefer the rough opposite of this, in the
    event multiple eclass export the same, have the package manager merge
    them.

    This is my preference too, if we're to go this route.

    It's similar to what was proposed in https://bugs.gentoo.org/516014, although this topic tends to lead to lots of different ideas (sometimes solving different
    problems!)


    e.g. rather than have `inherit cmake xdg` run xdg_src_prepare over
    cmake's, would export:

    src_prepare() {
    cmake_src_prepare
    xdg_src_prepare
    }

    unless the ebuild redefines it (which I imagine would often be because
    of optional support, e.g. use lua &&, or occasional incompatibilities)

    Some ebuilds have special needs, but then there's all those generic
    ebuilds that should stay nearly empty.

    best,
    sam


    -----BEGIN PGP SIGNATURE-----

    iQGTBAEBCgB9FiEEYOpPv/uDUzOcqtTy9JIoEO6gSDsFAmEr1XdfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDYw RUE0RkJGRkI4MzUzMzM5Q0FBRDRGMkY0OTIyODEwRUVBMDQ4M0IACgkQ9JIoEO6g SDtFsAf/Vg7RZsctuRBoktYiJCOong5uU4kIwxuJjFEnININHYc3c4qrwjOn7KnV AAQUuJRcIFZeVLuSkSzEeITRIy6bXKIt6c8RVAt2JGZGciExJ34Y87iinOBGF6DX LH3L0Pq4WyYN8kSHL8qdesACAqZ2rI/V7meNMEVxSJ0h3O9Z+7w0c+tDdDkMLG9p MwnuHZ51tj6ivSroftloZJlCo6ZIHklP60DWbyviJzeR208z57A0XtNfaHuA7GWl NMMoDzDRiGpfoQz++ilx5HHN9d0U7oSag+CH0vLQPtnGo3iN3WJZbUD/ApiJ+HFr XNLNDKDFxMi6Vm2IvmDJhyA9NKDogw==
    =vRJ1
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)