Florian Schmaus <flow@gentoo.org> writes:

[[PGP Signed Part:Undecided]]
[in reply to a gentoo-project@ post, but it was asked to continue this
on gentoo-dev@]

On 28/06/2023 16.46, Sam James wrote:

Florian Schmaus <flow@gentoo.org> writes:

On 17/06/2023 10.37, Arthur Zamarin wrote:

I also want to nominate people who I feel contribute a lot to Gentoo and >>>> I have a lot of interaction with (ordered by name, not priority):
[…]
flow

I apologize for the late reply, and thank you for the nomination. I am
honored and accept.

As many of you know, I am spending a lot of time on the EGO_SUM
situation, as it is one of the most critical issues to solve.

I have used the last few days to carefully consider whether a seat on
the council is more harmful or beneficial to my efforts regarding
EGO_SUM. On the one hand, council work means I have less time to
improve the EGO_SUM situation. On the other hand, a seat in the
council increases the probability of positively influencing Gentoo's
future, also regarding EGO_SUM.

That's fine and it's great to see more people running!

Excellent that we share this view. :)

But with regard to EGO_SUM: you didn't appear at the meeting where we discussed
your previous EGO_SUM proposal,

Naively, as I am, I expected that the mailing list would be used for discussion and that the council meeting would be used chiefly for
voting and intra-council discussion. And since the request to the
council to vote on a concrete proposal was preceded by a
multiple-week, if not month-long, mailing list discussion, I assumed
that my presence in the council meeting was optional.

Had I known that my presence was required, or that the absence in the
meeting would be blamed on me afterward, I would have appeared if
possible.

I'm not blaming you for anything. But you didn't speak in
#gentoo-council before the meeting (a few days before IIRC) when we
were discussing the problem, I pinged you during the meeting, and you
didn't appear there afterwards.

You also didn't seem to respond to the council decision (or
non-decision) in that meeting either, unless I've missed it.

It seems self-evident that discussion would happen in the meeting before voting...? What am I misunderstanding?

We regularly discuss things before voting on them. Do you normally
observe council meetings? I don't think what we did in this instance
was at all unusual.

(Also: there's the issue of whether or not the council should really
be voting on overriding an eclass maintainer who would then be forced
to keep something working they don't want to. mgorny raised that.)

and questions remain unanswered on the
ML (why not implement a check in pkgcheck similar to what is in Portage,
for example)?

On 2023-05-30 [1], I proposed a limit in the range of 2 to 1.5 MiB for
the total package-directory size. I only care a little about the tool
that checks this limit, but pkgcheck is an obvious choice. I also
suggested that we review this policy once the number of Go packages
has doubled or two years after this policy was established (whatever
comes first).

But I fear you may be referring to another kind of check. You may be
talking about a check that forbids EGO_SUM in ::gentoo but allows it overlays.

My position on this has been consistent: a check is needed to statically determine when the environment size is too big. Copying the Portage
check into pkgcheck (in terms of the metrics) would satisfy this.

That is, regardless of raw size, I'm asking for a calculation based on
the contents of EGO_SUM where, if exceeded, the package will not be
installable on some systems. You didn't have an issue implementing this
for Portage and I've mentioned this a bunch of times since, so I thought
it was clear what I was hoping to see.

I would also like (which is not what I was referring to here) some
limit on the size, given that we already have a limit on the size of ${FILESDIR}, but this is less of a concern for me given it's bounded
by the aforementioned environment size check.

Intelligibly, EGO_SUM can be considered ugly. Compared to a
traditional Gentoo package, EGO_SUM-based ones are larger. The same is
true for Rust packages. However, looking at the bigger picture,
EGO_SUM's advantages outweigh its disadvantages.

Again, am on record as being fine with the general EGO_SUM approach,
even if I wish we didn't need it, as I see it as inevitable for things
like yarn, .NET, and of course Rust as we already have it.

Just ideally not huge ones, and certainly not huge ones which then
aren't even reliably installable because of environment size.


--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iOUEARYKAI0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCZJ6Skl8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MA8cc2FtQGdlbnRv by5vcmcACgkQc4QJ9SDfkZDfkgEA4yEKRXIJoYGTN4VrDzVdyYBfuMYEahC/oQ6G RDpa+j8A/R+KvoDYQRngXXyJCpcd+PBt9V6EHlCjBSqALRgEUlwP
=uHuA
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-06-30 Fri 02:22, Sam James wrote:

My position on this has been consistent: a check is needed to statically determine when the environment size is too big. Copying the Portage
check into pkgcheck (in terms of the metrics) would satisfy this.

That is, regardless of raw size, I'm asking for a calculation based on
the contents of EGO_SUM where, if exceeded, the package will not be installable on some systems. You didn't have an issue implementing this
for Portage and I've mentioned this a bunch of times since, so I thought
it was clear what I was hoping to see.

I would also like (which is not what I was referring to here) some
limit on the size, given that we already have a limit on the size of ${FILESDIR}, but this is less of a concern for me given it's bounded
by the aforementioned environment size check.

Why do we have to keep exporting the related variables that generally
cause these size issues to the environment? I've asked as much on IRC
multiple times (nearly every time this discussion has been brought up)
and the answers I've gotten are some variation on "it's always been that
way" or "not exporting them would break using commands as external
programs" (e.g. calling via xargs).

The first response isn't a great argument and the second response, while
more valid, also feels less important than having a more minimalistic,
exported environment that causes less issues like this one and others
such as potentially affecting a package's build system in an unexpected fashion. See bug #721088 for the related discussion on environment
variable exports.

From my stance, the spec should state that the only variables to be
exported are ones already "semi-standard" and used externally of package manager internals in the expected fashion, which probably only includes
HOME, TMPDIR, and maybe ROOT. This would of course currently break
packages that use `xargs` while calling internal commands depending on
some of those exported variables, but from a cursory glance at the
gentoo repo, there aren't many ebuilds using that functionality and in
general those that are could be written in an easier to understand
fashion without using xargs. It should also be possible to proxy the
required variables to those commands in various fashions without using
the environment if using commands externally is extremely important to
the few ebuild maintainers who make use of that functionality.

In short, adding checks to portage and pkgcheck feels like a ill-suited workaround that foists hacking around the error onto users or developers
due to a poor decision made decades ago on environment handling.

Tim

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally
cause these size issues to the environment?

I really do not want to make a +1 response but this is an excellent
question that we need to answer before implementing EGO_SUM.

--
Eray

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-07-03 Mon 04:17, Florian Schmaus wrote:

On 30/06/2023 13.33, Eray Aslan wrote:

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally >>>cause these size issues to the environment?

I really do not want to make a +1 response but this is an excellent >>question that we need to answer before implementing EGO_SUM.

Could you please discuss why you make the reintroduction of EGO_SUM
dependent on this question?

Just to be clear, I don't particularly care about EGO_SUM enough to gate
its reintroduction (and don't have any leverage to do so anyway). I'm
just tired of the circular discussions around env issues that all seem
to avoid actual fixes, catering instead to functionality used by a
vanishingly small subset of ebuilds in the main repo that compels a
certain design mostly due to how portage functioned before EAPI 0.

Other than that, supporting EGO_SUM (or any other language ecosystem
trending towards distro-unfriendly releases) is fine as long as devs are cognizant how the related global-scope eclass design affects everyone
running or working on the raw repo. I hope devs continue leveraging the relatively recent benchmark tooling (and perhaps more future support) to improve their work. Along those lines, it could be nice to see sample
benchmark data in commit messages for large, global-scope eclass work
just to reinforce that it was taken into account.

Tim

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Am Dienstag, 4. Juli 2023, 09:13:30 CEST schrieb Tim Harder:

On 2023-07-03 Mon 04:17, Florian Schmaus wrote:

On 30/06/2023 13.33, Eray Aslan wrote:

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally >>>cause these size issues to the environment?

I really do not want to make a +1 response but this is an excellent >>question that we need to answer before implementing EGO_SUM.

Could you please discuss why you make the reintroduction of EGO_SUM >dependent on this question?

Just to be clear, I don't particularly care about EGO_SUM enough to gate
its reintroduction (and don't have any leverage to do so anyway). I'm
just tired of the circular discussions around env issues that all seem
to avoid actual fixes, catering instead to functionality used by a vanishingly small subset of ebuilds in the main repo that compels a
certain design mostly due to how portage functioned before EAPI 0.

Other than that, supporting EGO_SUM (or any other language ecosystem
trending towards distro-unfriendly releases) is fine as long as devs are cognizant how the related global-scope eclass design affects everyone
running or working on the raw repo. I hope devs continue leveraging the relatively recent benchmark tooling (and perhaps more future support) to improve their work. Along those lines, it could be nice to see sample benchmark data in commit messages for large, global-scope eclass work
just to reinforce that it was taken into account.

Tim

Hi,

just to be curious about the whole discussion. I did not follow in the
deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
to be respected. A lot of these Manifest files lead to a extremely
increased Portage tree size. EGO_SUM is just one example (though the
biggest one). Statically linked languages like Rust etc. have the same
problem.
- The current solution is to prepackage all modules, put it somewhere on
a webserver and just manifest that file. This make the Portage tree
small in size again, but requires a webserver/mirror and is thus
unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash
trees / Merkle trees for the manifest file? The idea would be to hash
the standard manifest file a second time if it gets too big and write
down that hash as new manifest file and leave EGO_SUM as is.

When Portage tries to install the package, it can download all modules,
build the "normal" Manifest file like normally, but instead of directly
compare it to the Manifest in the tree it can hash it again and compare
that to the provided Manifest. With this, Portage should have more less
the same guarantees about the validity of the source code, but the
manifest file consists of just two hashes again.
What one would loose is the direct comparison of file names (they are
included in the "meta"-hash, though) or do I miss something?

Gerion

-----BEGIN PGP SIGNATURE-----

iQGzBAABCAAdFiEEM/tVN9WpYYHnPZHxloeAdSYJHeoFAmSj+BcACgkQloeAdSYJ HerobAwAg2CO8kDVsldIAn8ni6RBxgZ71DSNPcBC8V/eZszElcIWPuxylqyYyWf4 50zVbXPzacXCV9uh6W0NJtWITXA/qdtK3LoKnC2Z6d4lS6AWmk7IbAeIltomPEsc LFk/qbWJvYU6OF14ICPgWD188+pJrwtFQen3uRQyye1cZ2i+B1DjYjaYO5ig69Ct W/Rod27hQnL6rP8lXj9eOLuEWC2sY7JvzhmJG4VWt6npymcMu59ypoHpiYGR1VQK W1RDGFc/Bd6+XixE8dkRGlisrKFH67uh5G8l3+O6ButxBo+4tRCuqnbjd50AM+Go hrIDqc3RnM9F9/uPQeY9+X8X7RJlrACTmxvov5sgq943sq7pfovqCkEakjCC499v xACmS2Io7hyYDW9KeZPCBJJ6iaJiktsK6Ue7JMmMnOgSZhNvn/M/QICRgESrhbng AP5nhF3p4CQkyhbFWPrkQejVg6jF4tYMwQiPvB3D02wJp5AOos3o0PynoBYsGeXR
IE6DB15U
=IjWu
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:

just to be curious about the whole discussion. I did not follow in the deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
to be respected. A lot of these Manifest files lead to a extremely
increased Portage tree size. EGO_SUM is just one example (though the
biggest one). Statically linked languages like Rust etc. have the same
problem.
- The current solution is to prepackage all modules, put it somewhere on
a webserver and just manifest that file. This make the Portage tree
small in size again, but requires a webserver/mirror and is thus
unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash
trees / Merkle trees for the manifest file? The idea would be to hash
the standard manifest file a second time if it gets too big and write
down that hash as new manifest file and leave EGO_SUM as is.

This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2 https://marc.info/?l=gentoo-dev&m=165472088822215&w=2

Developing it requires PMS work in addition to package manager
development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run
downloads twice.

The rest of the posts also go into the matter of duplication within
EGO_SUM & the indirect Manifests: limiting the growth requires some form
of content-addressed layout.

It's absolutely something we should get developed, but it's a lot of
work.

The indirect Manifests still provide a hosting challenge for overlays.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it.

iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmSklYhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsTy6A/8DuwD+3qt9aLCme2UZnHRJUuvSY8PDks5HWHFofiYEVjK/vpeKfkjwL8c lUrXibyDv0mqo4qNFMTK2q85D+l6Cllmy2JXUsPAi0BfXhNYca73vkFayTBRl+zi dyPA4ropRXEVp1HSdLqt6qVk1rZWM/6hKLHx3tSuD82buShFdrruGMQBttfPL32/ Y6yee5IjVmxj+87Pa43dayAMrjdMOyfzEtM1R882bgsJeWZb/dtS8xs8jrJZm7Vr KDmidam3HTn+4NJZyrPxa5SPBxJYlDFVWZ1O8sNTLcrNXW0T/ewLV8IgHo/t+P02 slc0bgXzUuKWxAUTiU0gnJVCWIff4gus8u9Vt+jrDQ8PSAwsulEadhA0e1VPHNva +lN96nz6PE8S4l1XRjQ7

On Tue, Jul 04, 2023 at 21:56:26 +0000, Robin H. Johnson wrote:

On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:

just to be curious about the whole discussion. I did not follow in the deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
to be respected. A lot of these Manifest files lead to a extremely
increased Portage tree size. EGO_SUM is just one example (though the
biggest one). Statically linked languages like Rust etc. have the same
problem.
- The current solution is to prepackage all modules, put it somewhere on
a webserver and just manifest that file. This make the Portage tree
small in size again, but requires a webserver/mirror and is thus
unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash trees / Merkle trees for the manifest file? The idea would be to hash
the standard manifest file a second time if it gets too big and write
down that hash as new manifest file and leave EGO_SUM as is.

This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2 https://marc.info/?l=gentoo-dev&m=165472088822215&w=2

Developing it requires PMS work in addition to package manager
development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run downloads twice.

I'm not sure double downloading is required. Consider a flow similar to
this:

1. distfiles are fetched as per the ebuild
2. distfiles are hashed into a temporary Manifest
3. temporary Manifest is hashed and compared with the hashes stored in
the in-tree Manifest for the direct Manifest

A new Manifest format would be required in order to differentiate the
current ones from an indirect one. This may require PMS changes,
although I suspect ammending GLEP 74 may be enough since the PMS seems
to just refer to the GLEP for a description of Manifests.

This would also either rely on a stable ordering of Manifest contents
when generating it or having a separate file listing in the indirect
Manifest which corresponds to the order in the direct Manifest. For the
latter, it should also have separate entries for different package
versions so that every single distfile for every single version of said
package does not need to be fetched in order to build the direct
Manifest.

I'm imagining something along these lines:

INDIRECT true
PACKAGE category/package-version distfile1 distfile2 ... ALGO1 hash1 ALGO2 hash2 ...
PACKAGE ...

Here `ALGO1` and `hash1` correspond to the hash of the direct Manifest containing the distfiles (and potentially other files if a repo does not
have thin-manifests enabled) and their hashes in the order specified previously.

The indirect Manifest as described above would be large-ish for a
package that has lots of distfiles, but likely much smaller than if each distfile had its set of hashes stored directly.

Please correct me if there's some detail I've overlooked.

- Oskari

The rest of the posts also go into the matter of duplication within
EGO_SUM & the indirect Manifests: limiting the growth requires some form
of content-addressed layout.

It's absolutely something we should get developed, but it's a lot of
work.

The indirect Manifests still provide a hosting challenge for overlays.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQfOU+JeXjo4uxN6vCp8he9GGIfEQUCZKSmpQAKCRCp8he9GGIf EbMtAP4uKyqDvNVtx56tu4FLb4yge5D/+95CdaG7KeSTnh+OTQD/YarZpUuGXG3k 7hzf0A0WOE0Mz5rcOe2ndfjkOR6FLgw=
=ML05
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Am Mittwoch, 5. Juli 2023, 01:09:30 CEST schrieb Oskari Pirhonen:

On Tue, Jul 04, 2023 at 21:56:26 +0000, Robin H. Johnson wrote:

On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:

just to be curious about the whole discussion. I did not follow in the deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
to be respected. A lot of these Manifest files lead to a extremely
increased Portage tree size. EGO_SUM is just one example (though the
biggest one). Statically linked languages like Rust etc. have the same
problem.
- The current solution is to prepackage all modules, put it somewhere on
a webserver and just manifest that file. This make the Portage tree
small in size again, but requires a webserver/mirror and is thus
unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash trees / Merkle trees for the manifest file? The idea would be to hash
the standard manifest file a second time if it gets too big and write down that hash as new manifest file and leave EGO_SUM as is.

This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2 https://marc.info/?l=gentoo-dev&m=165472088822215&w=2

Developing it requires PMS work in addition to package manager
development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run downloads twice.

I'm not sure double downloading is required. Consider a flow similar to
this:

1. distfiles are fetched as per the ebuild
2. distfiles are hashed into a temporary Manifest
3. temporary Manifest is hashed and compared with the hashes stored in
the in-tree Manifest for the direct Manifest

This is exactly, what I meant. A webstorage is not needed. A second
download process is also not needed. Just an additional Manifest format
is needed for ebuilds with more than n distfiles.

A new Manifest format would be required in order to differentiate the
current ones from an indirect one. This may require PMS changes,
although I suspect ammending GLEP 74 may be enough since the PMS seems
to just refer to the GLEP for a description of Manifests.

This would also either rely on a stable ordering of Manifest contents
when generating it or having a separate file listing in the indirect
Manifest which corresponds to the order in the direct Manifest. For the latter, it should also have separate entries for different package
versions so that every single distfile for every single version of said package does not need to be fetched in order to build the direct
Manifest.

I'm imagining something along these lines:

INDIRECT true
PACKAGE category/package-version distfile1 distfile2 ... ALGO1 hash1 ALGO2 hash2 ...
PACKAGE ...

Maybe it is reasonable to skip the distfile names at all (or just
provide a hash value of the concatenated file names). Then the manifest
would just contain two/three hashes (for as many distfiles as the ebuild needs). Since these kind of indirect Manifests should be more rare than
the normal ones, a slightly longer processing time does not have much
impact I would say.

Here `ALGO1` and `hash1` correspond to the hash of the direct Manifest containing the distfiles (and potentially other files if a repo does not
have thin-manifests enabled) and their hashes in the order specified previously.

The indirect Manifest as described above would be large-ish for a
package that has lots of distfiles, but likely much smaller than if each distfile had its set of hashes stored directly.

Without storing the filenames, the Manifest file would have the same
small size for any amount of distfiles needed.

Gerion

Please correct me if there's some detail I've overlooked.

- Oskari

The rest of the posts also go into the matter of duplication within
EGO_SUM & the indirect Manifests: limiting the growth requires some form
of content-addressed layout.

It's absolutely something we should get developed, but it's a lot of
work.

The indirect Manifests still provide a hosting challenge for overlays.

-----BEGIN PGP SIGNATURE-----

iQGzBAABCAAdFiEEM/tVN9WpYYHnPZHxloeAdSYJHeoFAmSluSIACgkQloeAdSYJ HepJlQv9G2NveYTWafK/Qxbhdd4Z7/jRhwW7vqt/A4R/hXqlihQLgIWygwwgWDXM iE2IZLHWUOr6e7Cl2OZ6+pz4bQn03wmgMtJQsh9ML+EPb1Gb4et3Wumve0i5uRYQ sRpsV9RK9WIzMXkKCKm2P9q7aEdclL2ZVrxnBtr1DLR94lIq/yrnuboLAV9XubkD o9JEHOC/s8Lv7+9hXAjl5MvSIWcYOGG55W3drr2FTtGbwoDjGDIueWP/UhPVuct2 mL4VkmLjv8/BaG5Au65bduNq4Gh9BfA9xtluG1KJizEYvXkJ7V1HpTTHIEjVDOsu rFQhtEd8KPj9HBYSl2FwhwyE30l3O5uaTU1EtmJzXpH7MQFWhhyySCTlxypz7bH5 Ts/EZKZNppnEgUxj2ri2AX1u5D94qzh26uMrPFIuQ6Otfc0GpIHoelwy/f251Oa0 ImX9dJ1Rp+lLfyTx+ni/0XffPFfUYCtR1ksglNXrG8PkbTxgN6yHVOvOKfgJIQ/z
qZeIsRb4
=8pNd
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jul 5, 2023 at 2:40 PM Gerion Entrup <gerion.entrup@flump.de> wrote:

Am Mittwoch, 5. Juli 2023, 01:09:30 CEST schrieb Oskari Pirhonen:

On Tue, Jul 04, 2023 at 21:56:26 +0000, Robin H. Johnson wrote:

Developing it requires PMS work in addition to package manager development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run downloads twice.

I'm not sure double downloading is required. Consider a flow similar to this:

1. distfiles are fetched as per the ebuild
2. distfiles are hashed into a temporary Manifest
3. temporary Manifest is hashed and compared with the hashes stored in
the in-tree Manifest for the direct Manifest

This is exactly, what I meant. A webstorage is not needed. A second
download process is also not needed. Just an additional Manifest format
is needed for ebuilds with more than n distfiles.

I suspect that Robin was proposing indirect manfests AND src uris, and
not just indirect manifests. In any case, if he wasn't, then I'd
suggest it would make sense to have that so that we don't need giant
lists of src_uris or go sums or whatever in ebuilds. Sure, the
manifests are even larger than the original file references, but those
will still be long. Plus if a file is used by 5 versions of an ebuild
it will be present in the manifests once per hash function, but in the
ebuilds 5 times.

I agree though that if only the manifests are moved to a fetched file
then you could fetch that on the first pass, though you'd still need
the extra logic to parse it. I'm not sure it really is much of a
difference to the effort involved.

Aren't go sums already content hashes? It might make even more sense
to create some kind of modular manifest verification logic in portage
so that the same eclass that handles EGO_SUM could tell the package
manager how to check the integrity of the files that are fetched.
Well, assuming we trust whatever hash function they're using (I'm
afraid to check - maybe this isn't such a great idea...).

--
Rich

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jul 05, 2023 at 20:40:34 +0200, Gerion Entrup wrote:

Am Mittwoch, 5. Juli 2023, 01:09:30 CEST schrieb Oskari Pirhonen:

On Tue, Jul 04, 2023 at 21:56:26 +0000, Robin H. Johnson wrote:

On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:

just to be curious about the whole discussion. I did not follow in the deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
to be respected. A lot of these Manifest files lead to a extremely
increased Portage tree size. EGO_SUM is just one example (though the
biggest one). Statically linked languages like Rust etc. have the same
problem.
- The current solution is to prepackage all modules, put it somewhere on
a webserver and just manifest that file. This make the Portage tree
small in size again, but requires a webserver/mirror and is thus
unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash trees / Merkle trees for the manifest file? The idea would be to hash the standard manifest file a second time if it gets too big and write down that hash as new manifest file and leave EGO_SUM as is.

This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2 https://marc.info/?l=gentoo-dev&m=165472088822215&w=2

Developing it requires PMS work in addition to package manager development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run downloads twice.

I'm not sure double downloading is required. Consider a flow similar to this:

1. distfiles are fetched as per the ebuild
2. distfiles are hashed into a temporary Manifest
3. temporary Manifest is hashed and compared with the hashes stored in
the in-tree Manifest for the direct Manifest

This is exactly, what I meant. A webstorage is not needed. A second
download process is also not needed. Just an additional Manifest format
is needed for ebuilds with more than n distfiles.

A new Manifest format would be required in order to differentiate the current ones from an indirect one. This may require PMS changes,
although I suspect ammending GLEP 74 may be enough since the PMS seems
to just refer to the GLEP for a description of Manifests.

This would also either rely on a stable ordering of Manifest contents
when generating it or having a separate file listing in the indirect Manifest which corresponds to the order in the direct Manifest. For the latter, it should also have separate entries for different package
versions so that every single distfile for every single version of said package does not need to be fetched in order to build the direct
Manifest.

I'm imagining something along these lines:

INDIRECT true
PACKAGE category/package-version distfile1 distfile2 ... ALGO1 hash1 ALGO2 hash2 ...
PACKAGE ...

Maybe it is reasonable to skip the distfile names at all (or just
provide a hash value of the concatenated file names). Then the manifest
would just contain two/three hashes (for as many distfiles as the ebuild needs). Since these kind of indirect Manifests should be more rare than
the normal ones, a slightly longer processing time does not have much
impact I would say.

My reasoning behind having the list of files is so that the
intermediat/direct Manifest can be accurately recreated. Consider the
following (not-so-)hypothetical Manifest:

DIST dist.tar.gz 84703 BLAKE2B ... SHA512 ...
DIST dist.tar.gz.asc 228 BLAKE2B ... SHA512 ...
EBUILD package-r1.ebuild 1535 BLAKE2B ... SHA512 ...
EBUILD package.ebuild 1536 BLAKE2B ... SHA512 ...
MISC metadata.xml 959 BLAKE2B ... SHA512 ...

It is "well behaved" because pkgdev created it. My main concern is if $OTHER_TOOLING generates the Manifest in a different order which would
mean the Manifest may be correct, but you get a false negative since the
hashes don't match what is in the in-tree indirect Manifest. Having the
order specified in the indirect Manifest renders this moot because $OTHER_TOOLING would have to respect this in order to correctly handle
indirect Manifests.

Additionally, in repos without thin-manifests, the SRC_URI is not enough
to build up the Manifest. This may or may not be an issue depending on
if a repo's metadata/layout.conf is parsed as part of the Manifest
verification process.

Here `ALGO1` and `hash1` correspond to the hash of the direct Manifest containing the distfiles (and potentially other files if a repo does not have thin-manifests enabled) and their hashes in the order specified previously.

The indirect Manifest as described above would be large-ish for a
package that has lots of distfiles, but likely much smaller than if each distfile had its set of hashes stored directly.

Without storing the filenames, the Manifest file would have the same
small size for any amount of distfiles needed.

Assuming layout.conf is parsed when the Manifest is verified (thus
handling the thick Maniffest case), the file list can be omitted if GLEP
74 is ammended to specify an ordering on the entries.

Side note: Portage itself does not seem to care about the ordering. I
tested this by copying a package tree, moving some entries around, and
running `ebuild /path/to/ebuild clean unpack`.

- Oskari

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQfOU+JeXjo4uxN6vCp8he9GGIfEQUCZKYrcwAKCRCp8he9GGIf EZEsAQDo+V7V3Galcv+PxjhJ7aIgPbBtBZYVwTsVYJP/x2jXxgD+O4vd+A1Uuq1/ PoF+PXqaMzRRe6PydHmOBq343i3Iiwk=
=QHZ3
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Jul 04, 2023 at 01:13:30AM -0600, Tim Harder wrote:

On 2023-07-03 Mon 04:17, Florian Schmaus wrote:

On 30/06/2023 13.33, Eray Aslan wrote:

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally >>>cause these size issues to the environment?

I really do not want to make a +1 response but this is an excellent >>question that we need to answer before implementing EGO_SUM.

Could you please discuss why you make the reintroduction of EGO_SUM >dependent on this question?

Just to be clear, I don't particularly care about EGO_SUM enough to gate
its reintroduction (and don't have any leverage to do so anyway). I'm
just tired of the circular discussions around env issues that all seem
to avoid actual fixes, catering instead to functionality used by a vanishingly small subset of ebuilds in the main repo that compels a
certain design mostly due to how portage functioned before EAPI 0.

Other than that, supporting EGO_SUM (or any other language ecosystem
trending towards distro-unfriendly releases) is fine as long as devs are cognizant how the related global-scope eclass design affects everyone
running or working on the raw repo. I hope devs continue leveraging the relatively recent benchmark tooling (and perhaps more future support) to improve their work. Along those lines, it could be nice to see sample benchmark data in commit messages for large, global-scope eclass work
just to reinforce that it was taken into account.

Tim

I've been following the EGO_SUM thread for quite some time now. One other thing I did not see mentioned in favour of EGO_SUM so far: reproducibility.

The problem with external tarballs is that they are gone once the ebuild is dropped from the tree. Should a user ever want to roll back to a previous version of an application, either by checking out on older version of the portage tree or copying said ebuild into their local overlay, they still cannot simply run an emerge on the it as they have to somehow recreate the tarball itself too.

While upstream may not host everything forever, it's pretty much guaranteed to be available for much longer than Gentoo's custom tarball bundles of dependencies.

Regarding space we are also likely making trade-off. By deprecating EGO_SUM we are saving space in the portage tree but in exchange inflating distfiles as it will start accumulating the same dependencies potentially multiple times since now the content is hidden in tarballs containing a combination of dependencies. This is essentially the source file version of "statically linking".

Finally a personal opinion: I find dependency tarballs opaque. With EGO_SUM the ebuild defines all the upstream sources it needs to build the package as well as
how to build it, but with the dependency tarball the sources are all hidden and makes verification all that much harder.

Zoltan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Jul 6, 2023 Zoltan Puskas wrote:

I've been following the EGO_SUM thread for quite some time now. One
other thing I did not see mentioned in favour of EGO_SUM so far: reproducibility.

The problem with external tarballs is that they are gone once the
ebuild is dropped from the tree. Should a user ever want to roll back
to a previous version of an application, either by checking out on
older version of the portage tree or copying said ebuild into their
local overlay, they still cannot simply run an emerge on the it as
they have to somehow recreate the tarball itself too.

While upstream may not host everything forever, it's pretty much
guaranteed to be available for much longer than Gentoo's custom
tarball bundles of dependencies.

I see this brought up every once in a while in these EGO_SUM threads,
but I think reproducable tarballs are a solved problem, or at least, the
tools exist and we just need to decide how to best equip people with
them.

thesamesam/sam-gentoo-scripts has maint/bump-go which builds these
tarballs smartly and reproducably:

- use --sort=name to order files inside in a consistent way
- use consistent owner:group (portage:portage)
- use consistent LC and TZ settings
- set a standard timestamp (since 'go mod download' doesn't preserve
upstream timestamps anyway, this loses no useful information)

With that, multiple developers can independently generate a -deps
tarball for a given Go package version with checksums that match. The
main distro tarball's checksums are verified against Manifest, and then
within it are the list and checksums of the individual downloads which
would be verified by go mod download (right?) and the resulting -deps
files should also match Manifest entries.

So a similar approach could be used in the case of expired ::gentoo
versions being installed, or overlays using -deps files without a way to
host them. Set things up so this can be done easily on demand or perhaps automatically as needed (maybe through a variation on pkg_nofetch in a
Go eclass; that part is not obvious to me).

Thanks,

--

Hank Leininger <hlein@korelogic.com>
9606 3BF9 B593 4CBC E31A A384 6200 F6E3 781E 3DD7

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEElgY7+bWTTLzjGqOEYgD243gePdcFAmSnGhoACgkQYgD243ge PdeHyA/9GHp3SRJrDg5nwpBjTF2ixYueHmFTjh3i3RVLvdVMBxKySZR/hcW1jzfo GyxIBn5o+k5V2bSaft+hngMlxrqGgi8n9zGIfJlOonRo7727GFCScfL/f0V1MirC CTqEL73LuWv7p5igL9nsvn/a51Uz5YrJxJBe6nR3faP4MPI6wpE6zOkJ/bF/726k htq52oJm7bdiSs66CaQq+hZ8PQ45shLwieHFK7vZqqh/C7olrR2zMQaR+ZxUQZbu OL9NF9S1oqC3u1d2ONw23qGv7hBeDaBCYOWBjMElZu2SpIgkxRSdk41zPWM8yKqK agMXpNIQZrRBDV9MXMDslXepOe+na6eHQvCMGoO1e5u7NjwS+tASzFBbh3rN/TFQ AvcWO1iRJnQkaPVXD1kimYAUPT0O3IhY3C78smh9v+plWc2cQzzbMIYHj1UhRCIz God70EEv5/U1S0YroHV+3qHAbIG88Y5QdgcbTLg5DKyIxieVJwxSw9bKYls5yVFf X5f6sb+4I9laseM9AdCgb8Wj5NoXHf+8pdXyDAm5OoJYyXK0xzSQ9pm6UySc42ou WevpjDpFft4m+sgz4FEPCM+cllT9g5tOjDkIHD1Dei+6PBlwl3SxUMeBWuoScJoK dqzCjbb2Ku8jvsHErOcMdDNz7i7wWn9slO4MU4irOAYhEUHXtNQ=
=LBf+
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Zoltan Puskas <zoltan@sinustrom.info> writes:

On Tue, Jul 04, 2023 at 01:13:30AM -0600, Tim Harder wrote:

On 2023-07-03 Mon 04:17, Florian Schmaus wrote:

On 30/06/2023 13.33, Eray Aslan wrote:

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally
cause these size issues to the environment?

I really do not want to make a +1 response but this is an excellent
question that we need to answer before implementing EGO_SUM.

Could you please discuss why you make the reintroduction of EGO_SUM
dependent on this question?

Just to be clear, I don't particularly care about EGO_SUM enough to gate
its reintroduction (and don't have any leverage to do so anyway). I'm
just tired of the circular discussions around env issues that all seem
to avoid actual fixes, catering instead to functionality used by a
vanishingly small subset of ebuilds in the main repo that compels a
certain design mostly due to how portage functioned before EAPI 0.

Other than that, supporting EGO_SUM (or any other language ecosystem
trending towards distro-unfriendly releases) is fine as long as devs are
cognizant how the related global-scope eclass design affects everyone
running or working on the raw repo. I hope devs continue leveraging the
relatively recent benchmark tooling (and perhaps more future support) to
improve their work. Along those lines, it could be nice to see sample
benchmark data in commit messages for large, global-scope eclass work
just to reinforce that it was taken into account.

Tim

I've been following the EGO_SUM thread for quite some time now. One other thing
I did not see mentioned in favour of EGO_SUM so far: reproducibility.

The problem with external tarballs is that they are gone once the ebuild is dropped from the tree. Should a user ever want to roll back to a previous version of an application, either by checking out on older version of the portage tree or copying said ebuild into their local overlay, they still cannot
simply run an emerge on the it as they have to somehow recreate the tarball itself too.

I believe Hank's email coves this.


--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iOUEARYKAI0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCZKnMAl8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MA8cc2FtQGdlbnRv by5vcmcACgkQc4QJ9SDfkZD3OAEA7Q/Qs/YW94caiNvfE8YcWF7mnR0jK9ri0Nk4 qaQYaGcA/0o9JGNfvgPUVS1p4lk7/vzetmIX/rx0+mWQLrLpuCcG
=kEGi
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Florian Schmaus <flow@gentoo.org> writes:

[[PGP Signed Part:Undecided]]
On 30/06/2023 10.22, Sam James wrote:

Florian Schmaus <flow@gentoo.org> writes:

[[PGP Signed Part:Undecided]]
[in reply to a gentoo-project@ post, but it was asked to continue this
on gentoo-dev@]
On 28/06/2023 16.46, Sam James wrote:

and questions remain unanswered on the
ML (why not implement a check in pkgcheck similar to what is in Portage, >>>> for example)?

On 2023-05-30 [1], I proposed a limit in the range of 2 to 1.5 MiB for
the total package-directory size. I only care a little about the tool
that checks this limit, but pkgcheck is an obvious choice. I also
suggested that we review this policy once the number of Go packages
has doubled or two years after this policy was established (whatever
comes first).

But I fear you may be referring to another kind of check. You may be
talking about a check that forbids EGO_SUM in ::gentoo but allows it
overlays.

My position on this has been consistent: > a check is needed to
statically
determine when the environment size is too big. Copying the Portage
check into pkgcheck (in terms of the metrics) would satisfy this.

It is not as easy as merely copying existing portage code into
pkgcheck (unless I am missing something).

That's why I said "in terms of the metrics".

I've talked to arthurzam, and there appears to be a .environment file
created by pkgcheck, which we could use to approximate the exported environment.

Another option would be to have pkgcheck count the EGO_SUM
entries. The tree-sitter API for Bash, which pkgcheck already uses,
seems to allow for that. But that would be different from the check in portage. Although, IMHO, counting EGO_SUM entries would be sufficient.

Right.

That is, regardless of raw size, I'm asking for a calculation based on
the contents of EGO_SUM where, if exceeded, the package will not be
installable on some systems. You didn't have an issue implementing this
for Portage and I've mentioned this a bunch of times since, so I thought
it was clear what I was hoping to see.

So pkgcheck counting EGO_SUM entries would be sufficient for the
purpose of having a static check that notices if the ebuild would
likely run into the environment limit?

If you check it actually fires in some of the old broken scenarios
(see Bugzilla), then yes. But I'd be interested in your thoughts on
radhermit's reply (please reply there).

To find a common compromise, I would possibly invest my time in
developing such a test. Even though I do not deem such a check a
strict prerequisite to reintroduce EGO_SUM.

Yes, you've made clear you disagree.

Intelligibly, EGO_SUM can be considered ugly. Compared to a
traditional Gentoo package, EGO_SUM-based ones are larger. The same is
true for Rust packages. However, looking at the bigger picture,
EGO_SUM's advantages outweigh its disadvantages.

Again, am on record as being fine with the general EGO_SUM approach,
even if I wish we didn't need it, as I see it as inevitable for things
like yarn, .NET, and of course Rust as we already have it.
Just ideally not huge ones, and certainly not huge ones which then
aren't even reliably installable because of environment size.

Talking about "reliably installable" makes it sound to me like there
are cases where installing a EGO_SUM-based package sometimes works and sometimes not. But the kernel-limit is fixed and not even
configurable, besides, of course patching the source (and in the
absence of architectures with a page size below 4 KiB) [1].

ulm's reply notes that this is a limitation in the Linux kernel, so I
have no idea why musl tinderboxes seemed to disproportionately hit these
issues and I assume one of us either missing something or it was just
a crazy fluke.

Any developer testing whether or notan ebuild is installable would
become immediately aware if the ebuild runs into the environment
limit, or not.

This clearly didn't happen with the previous examples (see what I said
above too), as there were times when they installed for some people, but
not in CI/tinderboxes. I don't know why and it merits investigation.

-----BEGIN PGP SIGNATURE-----

iOUEARYKAI0WIQQlpruI3Zt2TGtVQcJzhAn1IN+RkAUCZKnUEV8UgAAAAAAuAChp c3N1ZXItZnByQG5vdGF0aW9ucy5vcGVucGdwLmZpZnRoaG9yc2VtYW4ubmV0MjVB NkJCODhERDlCNzY0QzZCNTU0MUMyNzM4NDA5RjUyMERGOTE5MA8cc2FtQGdlbnRv by5vcmcACgkQc4QJ9SDfkZDQEwD+JHAP+faAsqcR4jqDQ+rzrK2SGLNVcFnZJY/o 0rjw9EAA+wfkJcI9sd/rmUjc+4iiG81sNCB0CedkznnL1rHyOUgF
=UckZ
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)