I'm looking to solicit opinions on when it is appropriate for an
ebuild to check for kernel config options using linux-info.eclass. I
don't think we have any guidelines documented, instead leaving it up
to the "common sense" of package maintainers.

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit. It doesn't make sense to check for kernel
options that are very commonly enabled. But what is "very common"?

An obvious example would be CONFIG_INET, which controls IPv4 support
in the kernel. It does not make sense to check for that in every
package that uses AF_INET sockets.

A less obvious example: a user filed a bug against net-misc/dhcpcd
today asking that we check for CONFIG_PACKET [1]. My first thought was
"why would you ever disable that?". The option description even says
"if unsure, say Y". However, I suppose it is technically possible to
run a Linux system with it disabled.

I think a reasonable rule of thumb would be to assume we can rely on
options that are enabled by "make defconfig". If the user chooses to
disable them, they are responsible for anything that breaks.

Thoughts?

[1] https://bugs.gentoo.org/815064

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 9/27/21 12:10 PM, Mike Gilbert wrote:

I'm looking to solicit opinions on when it is appropriate for an
ebuild to check for kernel config options using linux-info.eclass. I
don't think we have any guidelines documented, instead leaving it up
to the "common sense" of package maintainers.

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit. It doesn't make sense to check for kernel
options that are very commonly enabled. But what is "very common"?

An obvious example would be CONFIG_INET, which controls IPv4 support
in the kernel. It does not make sense to check for that in every
package that uses AF_INET sockets.

A less obvious example: a user filed a bug against net-misc/dhcpcd
today asking that we check for CONFIG_PACKET [1]. My first thought was
"why would you ever disable that?". The option description even says
"if unsure, say Y". However, I suppose it is technically possible to
run a Linux system with it disabled.

I think a reasonable rule of thumb would be to assume we can rely on
options that are enabled by "make defconfig". If the user chooses to
disable them, they are responsible for anything that breaks.

Thoughts?

[1] https://bugs.gentoo.org/815064

The challenge I see is that these config checks head off bugs and issues without our intervention.

We as kernel maintainers depend on ebuild maintainers to check these things so they don't become "kernel bugs" to figure out.

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit.

Is this a significant slowdown? Do you have any numbers?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit.

Is this a significant slowdown? Do you have any numbers?

Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
system.

That's ok if a small number of packages are doing it, but it would
become quite annoying if a significant number of them get queued up.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit.

Is this a significant slowdown? Do you have any numbers?

Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
system.

That's ok if a small number of packages are doing it, but it would
become quite annoying if a significant number of them get queued up.

7 seconds is ridiculous.

I think we need to strip out a lot of the crap about trying to detect
things in the stuff being built, and reduce the check to the simplest
possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

real 0m0.032s
user 0m0.021s
sys 0m0.018s

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it.

iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmFSAuRfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsTNkg//UDiBRSrNlnsSD6S+3HjIXpTf/JY5FLAuMZZXmL8qeEv5JD7auf/IBTjm n47ClSpyRtjTOF5n22qidXRRSPhoprlce6YiY+RoMvBs5wpBLD0fS4vkNQtTGbg8 nW0vR4snR8vHFXPfylLFg9A8dxNAdIay9E8bkBf+Ks/Yjd+MOGFC3Tk7VD33tiuh X2QFlfF+H5an2GO1NPciNTDSJ9VLjLfkmt+x8vsNawvKqa7zeXlDhgLVgRNIpKLJ KRf32jSDk9WXwEpytVFUJ0UmJNRCsEvguZdIBC+8SK6U8OgUVzqc2EnldGH8PnAN vIb0UbQfIIiIIaFBqWLcOGCtLY0hB8n2pFGTx1znXGA/vTQ0uv2Anj3hp3qLRjzS jLw8nChb1bU2gPuE6f+T

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:

I think we need to strip out a lot of the crap about trying to detect
things in the stuff being built, and reduce the check to the simplest possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

Sure, just please don't strip out the part that actually does what you
just did - check the running kernel. I don't think we should assume
that the user has the configured+prepared sources for a kernel just
lying around all the time. I don't build in /usr/src (which is
apparently discouraged by upstream anyway), so /proc/config.gz is
really the only reliable indication of how things are setup. Well,
that or a config file in /boot which I'm sure others don't use (but
make install puts it there).

--
Rich

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:

On 9/27/21 12:10 PM, Mike Gilbert wrote:

I'm looking to solicit opinions on when it is appropriate for an
ebuild to check for kernel config options using linux-info.eclass. I
don't think we have any guidelines documented, instead leaving it up
to the "common sense" of package maintainers.

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit. It doesn't make sense to check for kernel
options that are very commonly enabled. But what is "very common"?

An obvious example would be CONFIG_INET, which controls IPv4 support
in the kernel. It does not make sense to check for that in every
package that uses AF_INET sockets.

A less obvious example: a user filed a bug against net-misc/dhcpcd
today asking that we check for CONFIG_PACKET [1]. My first thought was
"why would you ever disable that?". The option description even says
"if unsure, say Y". However, I suppose it is technically possible to
run a Linux system with it disabled.

I think a reasonable rule of thumb would be to assume we can rely on options that are enabled by "make defconfig". If the user chooses to disable them, they are responsible for anything that breaks.

Thoughts?

[1] https://bugs.gentoo.org/815064

The challenge I see is that these config checks head off bugs and issues without our intervention.

We as kernel maintainers depend on ebuild maintainers to check these things so they don't become "kernel bugs" to figure out.

I guess I'm proposing that we set some minimal standard for kernel
configs. It's a waste of time and effort to pepper random ebuilds with
checks for options that everyone should have enabled in the first
place.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:

On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns when running emerge, so we should do so only when there is a compensating benefit.

Is this a significant slowdown? Do you have any numbers?

Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
seconds of delay time to the pkg_setup and/or pkg_pretend phase on my system.

That's ok if a small number of packages are doing it, but it would
become quite annoying if a significant number of them get queued up.

7 seconds is ridiculous.

I think the horribly slow performance is mainly due to this change in
how we detect the kernel version:

https://gitweb.gentoo.org/repo/gentoo.git/commit/eclass/linux-info.eclass?id=d1ea596f034285cd044006b107a60902ea059793

That causes "make" to be called 4 times, and each make invocation adds
a couple of seconds to the get_version call.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 9/27/21 2:15 PM, Mike Gilbert wrote:

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:

On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote: >>>>> Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns >>>>> when running emerge, so we should do so only when there is a

compensating benefit.

Is this a significant slowdown? Do you have any numbers?

Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
system.

That's ok if a small number of packages are doing it, but it would
become quite annoying if a significant number of them get queued up.

7 seconds is ridiculous.

I think the horribly slow performance is mainly due to this change in
how we detect the kernel version:

https://gitweb.gentoo.org/repo/gentoo.git/commit/eclass/linux-info.eclass?id=d1ea596f034285cd044006b107a60902ea059793

That causes "make" to be called 4 times, and each make invocation adds
a couple of seconds to the get_version call.

Changes to these eclasses are like a prison of circular breakage for which there is no escape.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 1:56 PM Rich Freeman <rich0@gentoo.org> wrote:

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:

I think we need to strip out a lot of the crap about trying to detect things in the stuff being built, and reduce the check to the simplest possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

Sure, just please don't strip out the part that actually does what you
just did - check the running kernel. I don't think we should assume
that the user has the configured+prepared sources for a kernel just
lying around all the time. I don't build in /usr/src (which is
apparently discouraged by upstream anyway), so /proc/config.gz is
really the only reliable indication of how things are setup. Well,
that or a config file in /boot which I'm sure others don't use (but
make install puts it there).

I like the idea of a stripped-down implementation that just checks the
running kernel config when packages get installed. I think that
probably means creating a new eclass though, or moving some of the
current complexity into linux-mod.eclass.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Gilbert wrote:

It's a waste of time and effort to pepper random ebuilds with checks
for options that everyone should have enabled in the first place.

It's not for you to say what everyone should have enabled in their kernel.

There's significant value in ebuilds documenting required kernel options in
a structured manner.

So I welcome simplifications in the checking to achieve millisecond
test times. But the benefit of structured requirements is given even
without any runtime checking at all, as long as required options
continue to be codified /somehow/.


Thank you for your work!

//Peter

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 3:11 PM Peter Stuge <peter@stuge.se> wrote:

Mike Gilbert wrote:

It's a waste of time and effort to pepper random ebuilds with checks
for options that everyone should have enabled in the first place.

It's not for you to say what everyone should have enabled in their kernel.

Do you not agree that there are some options that should always be
enabled, or at least that we can assume are enabled?

To use my earlier example, should every package that uses AF_INET
check for CONFIG_INET in the kernel?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 11:44 AM Robin H. Johnson <robbat2@gentoo.org> wrote:

I think we need to strip out a lot of the crap about trying to detect
things in the stuff being built, and reduce the check to the simplest possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

The results from our current method could of course be cached, and
even cached between runs, by just relying on `/proc/version` changing
for different kernels. This seems easy enough to do.

Alternatively, sure, we could require that everyone has /proc/config
in their kernel config. Many kernels don't have this (mine doesn't),
but we could add it to the base requirements.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Sep 27, 2021 at 4:28 PM Jason A. Donenfeld <zx2c4@gentoo.org> wrote:

On Mon, Sep 27, 2021 at 11:44 AM Robin H. Johnson <robbat2@gentoo.org> wrote:

I think we need to strip out a lot of the crap about trying to detect things in the stuff being built, and reduce the check to the simplest possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

The results from our current method could of course be cached, and
even cached between runs, by just relying on `/proc/version` changing
for different kernels. This seems easy enough to do.

Alternatively, sure, we could require that everyone has /proc/config
in their kernel config. Many kernels don't have this (mine doesn't),
but we could add it to the base requirements.

I think it would be reasonable to check a couple fallback locations:

/proc/config.gz
/lib/modules/$(uname -r)/build/.config
/boot/config-$(uname-r)

The slowness really comes from guessing the kernel build location and
invoking the kernel build system via make. Avoiding that is a big
improvement.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Gilbert wrote:

It's a waste of time and effort to pepper random ebuilds with checks
for options that everyone should have enabled in the first place.

It's not for you to say what everyone should have enabled in their kernel.

Do you not agree that there are some options that should always be
enabled, or at least that we can assume are enabled?

"Should be enabled" no, but I agree with the latter - some assumed set
should be fine. I think it would be good if it's (somehow) documented
though.

To use my earlier example, should every package that uses AF_INET
check for CONFIG_INET in the kernel?

CONFIG_INET is a perhaps surprisingly tricky example!

A package could e.g. use getaddrinfo() with no address family hint
but because of USE=-ipv6 exclude all AF_INET6 address results and
so end up using AF_INET based on whether it's available in the
running kernel or even based on third party DNS entries.


I'm not sure about the best approach for very basic options,
CONFIG_NET could be another such candidate.

Thinking towards robbat2's proposal (which I like) it might make sense
to try to map requirements of packages, but there will probably be
cases where it can't really be done successfully.

Ultimately that work should not be the responsibility of distribution
package maintainers but something upstreams deliver, similar to systemd
units, but maybe we'll invent it (if noone else has)..


//Peter

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 2021-09-27 at 15:44 -0400, Mike Gilbert wrote:

On Mon, Sep 27, 2021 at 3:11 PM Peter Stuge <peter@stuge.se> wrote:

Mike Gilbert wrote:

It's a waste of time and effort to pepper random ebuilds with checks
for options that everyone should have enabled in the first place.

It's not for you to say what everyone should have enabled in their kernel.

Do you not agree that there are some options that should always be
enabled, or at least that we can assume are enabled?

There is a distro/Kconfig in gentoo-sources with,

config GENTOO_LINUX
bool "Gentoo Linux support"
...

that could be extended to include a set of assumed features; otherwise
a sub-setting (default enabled) could be added.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Il giorno lun 27 set 2021 alle ore 18:11 Mike Gilbert <floppym@gentoo.org>
ha scritto:

I'm looking to solicit opinions on when it is appropriate for an
ebuild to check for kernel config options using linux-info.eclass. I
don't think we have any guidelines documented, instead leaving it up
to the "common sense" of package maintainers.

<snip>

After so many tentatives to fix the kernel checks in these years (almost
all of which had drawbacks or missed some extreme corner case) it's
probably better to give to the user instruments to do ihs own checks rather than trying to be smart.

An example of how this could work follow:
A file (or directory) is created in /etc/ that contains a list of <ebuild>,<config_option>,<state_required>
Then the user is responsible to check that list against the wanna be
running kernel

This save probably both computational and human time

state_required - should be well thought out since it can be required
present, absent or maybe even suggested

<div dir="ltr"><div dir="ltr">Il giorno lun 27 set 2021 alle ore 18:11 Mike Gilbert &lt;<a href="mailto:floppym@gentoo.org">floppym@gentoo.org</a>&gt; ha scritto:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I&#39;m looking to solicit opinions on when it is appropriate for an<br>
ebuild to check for kernel config options using linux-info.eclass. I<br> don&#39;t think we have any guidelines documented, instead leaving it up<br>
to the &quot;common sense&quot; of package maintainers.<br><br></blockquote><div>&lt;snip&gt; </div><div><br></div><div>After so many tentatives to fix the kernel checks in these years (almost all of which had drawbacks or missed some extreme corner
case) it&#39;s probably better to give to the user instruments to do ihs own checks rather than trying to be smart.<br><br>An example of how this could work follow:</div><div>A file (or directory) is created in /etc/ that contains a list of &lt;ebuild&
gt;,&lt;config_option&gt;,&lt;state_required&gt;<br>Then the user is responsible to check that list against the wanna be running kernel<br><br>This save probably both computational and human time<br><br>state_required - should be well thought out since
it can be required present, absent or maybe even suggested<br><br></div></div></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)