XPost: linux.debian.bugs.dist
Control: reopen 1064976
My apologies for the ping-pong; I do want to keep this open until the discussion has completed. I will set out my thoughts below. I'm afraid this
is fairly long.
A brief history of this issue: in December 2023, the control file for linux-headers-* was altered to include a dependency on linux-image-* (
https://salsa.debian.org/kernel-team/linux/-/merge_requests/903). As far as
I can tell, no bugreport was linked as a problem being addressed with this change; the maintainer's comment was "A lot of problems arise if users use headers of a different version then the associated image. The easiest
solution is to make them depend." Note that this dependency did not exist
in any previous version of linux-headers as far as I can determine; the problems seem to be largely theoretical.
This change worked its way through the release pipeline and eventually
arrived in bookworm-backports around the end of February 2024, with the promotion of the package linux-headers-6.6.13+bpo-amd64 (and others) to backports. I immediately noticed the impact on my build server, and
submitted a bug report (
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064976) requesting that
it be reverted.
The maintainer defended the change, indicating that it was necessary for
people using dkms; when pressed on exactly what failed, he mentioned the
BTF warnings [1] but as far as I can tell, no specific user problem was presented. Several attempts by myself and Luca Boccassi to determine what problem was being addressed were not answered.
The bug was closed as WONTFIX a few days ago, but still there has been no
real explanation as to why the change was introduced in the first place. I would like to go on the record here as saying (especially with the xz-utils exploit still in everyone's memory), that we should be *extremely careful*
with changing things like dependency trees without very well-documented reasons, *especially* for something as critical as the kernel packages. I ordinarily try to be very respectful of maintainers' latitude and
discretion in packaging decisions, but here I am trying to ensure that a serious problem is addressed in BPO before it gets promoted to stable. The change is significant enough that I feel it deserves more discussion and attention than it has so far received.
Having re-read the thread a few times today, I feel that the BTF warnings (which were originally presented as the main reason for this change) are a
red herring and not relevant. The new packaging of vmlinux.h does address
the issue of BPF builds for pretty much all users (it's true that build pipelines will have to be adjusted, but the new system is a significant improvement on the old). The discussion about BPF kernel modules does not
seem to be based on any real user activity, and to be honest it seems
somewhat self-contradictory - why would a kernel module need BPF in the
first place?
Let's consider the possible reasons for having the header package depend on
the image package:
From Debian's policy documentation; "The Depends field should be used if
the depended-on package is required for the depending package to provide a significant amount of functionality." So what functionality is provided by linux-headers-*? I would posit initially that their main function is unspecified apart from "having the header files for the specific kernel
exist under /usr/src", which clearly does not require the image package.
However, a major use case for the header files is to build kernel modules, whether using DKMS or some other mechanism. But this use case *also* does
not require the image package; in fact this is the main reason the header
files were packaged as they are. Hundreds of thousands (at least) of Debian users have been happily building kernel modules using linux-headers
packages without the corresponding image files for decades, and there are
no recent kernel changes which break this ability. The recent introduction
of vmlinux.h additionally addresses an edge case (building BPF programs)
which formerly *did* require a built image for its symbol table. So the important piece of functionality also does not require the kernel image package.
Now, given the maintainer's comments on the original PR and in this bug, I suspect I understand the real reason for the change: in order to *run*
modules built using DKMS etc., obviously the corresponding kernel image
file needs to be present. From the maintainer's most recent comments, I
believe that the problem is something like:
* user has installed linux-headers and linux-image for kernel version X
* user has built additional modules using DKMS which are installed into the running system
* user upgrades linux-headers to version Y, new modules get rebuilt
* user does not upgrade linux-image from X to Y, confusion results
Having linux-image-Y be a dependency of linux-headers-Y does indeed address this problem, but I feel that it is fairly substantial overkill. There are several referenced in the bug thread to DKMS *needing* the image file, but
I honestly don't know what these are - to the best of my knowledge it has always been possible to build kernel modules using only linux-kbuild and linux-headers; linux-image is not needed. I am of course willing to be corrected on this point.
The reasons I feel it is *not* sensible:
* It makes a sudden, large change to the dependency tree of any systems
with linux-headers-ARCH already installed; when this change ends up in
stable, many systems will find sudden new dependencies being installed,
with potentially serious consequences.
* It makes a (largely) source package depend on a substantial binary
package, which in turn can seriously alter the way a system operates - installing a new kernel into /boot might overflow a small partition, or
might trigger a substantial chain of additional installation scripts, which will generally not be expected by someone expecting just to install some
source headers.
* It addresses a largely theoretical issue - a user who is competent enough
to build kernels using DKMS but not competent enough to realize that they
need the corresponding kernel image - which as far as I can see has not
been reported by any Debian users.
* It assumes that the only use case of linux-headers is to build modules
*for the current system* and actively subverts other use cases, such as
having headers present for reference only, or build servers which build
modules for distribution to other systems.
* It seems to go against the spirit of the Debian packaging policy which
says that Depeds implies "significant functionality" is missing absent the depended package. I posit that this is simply not the case in the
relationship between linux-headers and linux-image.
The maintainer has invited proposals for alternative solutions to address
the runtime concern; here are my proposals; roughly in decreasing order of preference:
1) Downgrade the Depends: relationship to Recommends: as proposed in
https://salsa.debian.org/kernel-team/linux/-/merge_requests/1054 - the
default behavior of apt is to install Recommended packages as though they
were hard dependencies, so for most users they will be equivalent. I posit
that users who are likely to fall into the "competence interval" above are
also less likely to have modified the apt installation defaults, so this
should address the issue. I do not understand the maintainer's reasons for rejecting this PR.
2) Introduce a new package called "linux-complete-VERSION" which depends on linux-image-VERSION and linux-headers-VERSION. Users who require that their headers and image files be always installed in synch can install this
package, without suddenly changing the behavior of systems with
linux-headers already installed.
3) Move the linux-headers installation entirely to source packages, and eliminate them from the binary distribution. This would be a substantial
policy change, but could in theory allow for a fairly complete separation
of use cases.
4) Change the dependency of linux-headers as proposed by the maintainer,
but introduce a further package called linux-headers-only or similar which
does not have the dependency. This has several downsides; an inelegant
naming scheme and dependency chain, possible duplication of installation
trees, and a sudden change in the behavior of existing installations.
I welcome the thoughts of the community.
Colm
[1] If the kernel image file is not present, the legacy mechanism for extracting symbols for BTF builds does not work; this is the same issue
which was addressed with vmlinux.h in
https://salsa.debian.org/kernel-team/linux/-/merge_requests/1005
<div dir="ltr"><div>Control: reopen 1064976</div><div><br></div><div>My apologies for the ping-pong; I do want to keep this open until the discussion has completed. I will set out my thoughts below. I'm afraid this is fairly long.</div><div><br></
<div>A brief history of this issue: in December 2023, the control file for linux-headers-* was altered to include a dependency on linux-image-* (<a href="https://salsa.debian.org/kernel-team/linux/-/merge_requests/903">https://salsa.debian.org/
kernel-team/linux/-/merge_requests/903</a>). As far as I can tell, no bugreport was linked as a problem being addressed with this change; the maintainer's comment was "A lot of problems arise if users use headers of a different version then the
associated image. The easiest solution is to make them depend." Note that this dependency did not exist in any previous version of linux-headers as far as I can determine; the problems seem to be largely theoretical.</div><div><br></div><div>This
change worked its way through the release pipeline and eventually arrived in bookworm-backports around the end of February 2024, with the promotion of the package linux-headers-6.6.13+bpo-amd64 (and others) to backports. I immediately noticed the
impact on my build server, and submitted a bug report (<a href="
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064976">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064976</a>) requesting that it be reverted.</div><div><br></div><div>The
maintainer defended the change, indicating that it was necessary for people using dkms; when pressed on exactly what failed, he mentioned the BTF warnings [1] but as far as I can tell, no specific user problem was presented. Several attempts by myself
and Luca Boccassi to determine what problem was being addressed were not answered.</div><div><br></div><div>The bug was closed as WONTFIX a few days ago, but still there has been no real explanation as to why the change was introduced in the first place.
I would like to go on the record here as saying (especially with the xz-utils exploit still in everyone's memory), that we should be *extremely careful* with changing things like dependency trees without very well-documented reasons, *especially* for
something as critical as the kernel packages. I ordinarily try to be very respectful of maintainers' latitude and discretion in packaging decisions, but here I am trying to ensure that a serious problem is addressed in BPO before it gets promoted to
stable. The change is significant enough that I feel it deserves more discussion and attention than it has so far received.</div><div><br></div><div>Having re-read the thread a few times today, I feel that the BTF warnings (which were originally
presented as the main reason for this change) are a red herring and not relevant. The new packaging of vmlinux.h does address the issue of BPF builds for pretty much all users (it's true that build pipelines will have to be adjusted, but the new
system is a significant improvement on the old). The discussion about BPF kernel modules does not seem to be based on any real user activity, and to be honest it seems somewhat self-contradictory - why would a kernel module need BPF in the first place?</
<div><br></div><div>Let's consider the possible reasons for having the header package depend on the image package:</div><div><br></div><div>From Debian's policy documentation; "The Depends field should be used if the depended-on package
is required for the depending package to provide a significant amount of functionality." So what functionality is provided by linux-headers-*? I would posit initially that their main function is unspecified apart from "having the header files
for the specific kernel exist under /usr/src", which clearly does not require the image package.</div><div><br></div><div>However, a major use case for the header files is to build kernel modules, whether using DKMS or some other mechanism. But this
use case *also* does not require the image package; in fact this is the main reason the header files were packaged as they are. Hundreds of thousands (at least) of Debian users have been happily building kernel modules using linux-headers packages
without the corresponding image files for decades, and there are no recent kernel changes which break this ability. The recent introduction of vmlinux.h additionally addresses an edge case (building BPF programs) which formerly *did* require a built
image for its symbol table. So the important piece of functionality also does not require the kernel image package.</div><div><br></div><div>Now, given the maintainer's comments on the original PR and in this bug, I suspect I understand the real
reason for the change: in order to *run* modules built using DKMS etc., obviously the corresponding kernel image file needs to be present. From the maintainer's most recent comments, I believe that the problem is something like:</div><div><br></div><
* user has installed linux-headers and linux-image for kernel version X</div><div>* user has built additional modules using DKMS which are installed into the running system</div><div>* user upgrades linux-headers to version Y, new modules get rebuilt<
/div><div>* user does not upgrade linux-image from X to Y, confusion results</div><div><br></div><div>Having linux-image-Y be a dependency of linux-headers-Y does indeed address this problem, but I feel that it is fairly substantial overkill. There are
several referenced in the bug thread to DKMS *needing* the image file, but I honestly don't know what these are - to the best of my knowledge it has always been possible to build kernel modules using only linux-kbuild and linux-headers; linux-image
is not needed. I am of course willing to be corrected on this point.</div><div><br></div><div>The reasons I feel it is *not* sensible:</div><div><br></div><div>* It makes a sudden, large change to the dependency tree of any systems with linux-headers-
ARCH already installed; when this change ends up in stable, many systems will find sudden new dependencies being installed, with potentially serious consequences.</div><div><br></div><div>* It makes a (largely) source package depend on a substantial
binary package, which in turn can seriously alter the way a system operates - installing a new kernel into /boot might overflow a small partition, or might trigger a substantial chain of additional installation scripts, which will generally not be
expected by someone expecting just to install some source headers.</div><div><br></div><div>* It addresses a largely theoretical issue - a user who is competent enough to build kernels using DKMS but not competent enough to realize that they need the
corresponding kernel image - which as far as I can see has not been reported by any Debian users.</div><div><br></div><div>* It assumes that the only use case of linux-headers is to build modules *for the current system* and actively subverts other use
cases, such as having headers present for reference only, or build servers which build modules for distribution to other systems.</div><div><br></div><div>* It seems to go against the spirit of the Debian packaging policy which says that Depeds implies &
quot;significant functionality" is missing absent the depended package. I posit that this is simply not the case in the relationship between linux-headers and linux-image.</div><div><br></div><div>The maintainer has invited proposals for alternative
solutions to address the runtime concern; here are my proposals; roughly in decreasing order of preference:</div><div><br></div><div>1) Downgrade the Depends: relationship to Recommends: as proposed in <a href="
https://salsa.debian.org/kernel-team/linux/
-/merge_requests/1054">
https://salsa.debian.org/kernel-team/linux/-/merge_requests/1054</a> - the default behavior of apt is to install Recommended packages as though they were hard dependencies, so for most users they will be equivalent. I posit that
users who are likely to fall into the "competence interval" above are also less likely to have modified the apt installation defaults, so this should address the issue. I do not understand the maintainer's reasons for rejecting this PR.</
<div><br></div><div>2) Introduce a new package called "linux-complete-VERSION" which depends on linux-image-VERSION and linux-headers-VERSION. Users who require that their headers and image files be always installed in synch can install
this package, without suddenly changing the behavior of systems with linux-headers already installed.</div><div><br></div><div>3) Move the linux-headers installation entirely to source packages, and eliminate them from the binary distribution. This would
be a substantial policy change, but could in theory allow for a fairly complete separation of use cases.</div><div><br></div><div>4) Change the dependency of linux-headers as proposed by the maintainer, but introduce a further package called linux-
headers-only or similar which does not have the dependency. This has several downsides; an inelegant naming scheme and dependency chain, possible duplication of installation trees, and a sudden change in the behavior of existing installations.</div><div><
</div><div>I welcome the thoughts of the community.</div><div><br></div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>Colm</div><div><br></div></blockquote></div><div><br></div><div><br></div><div><br></div><div>[1] If the
kernel image file is not present, the legacy mechanism for extracting symbols for BTF builds does not work; this is the same issue which was addressed with vmlinux.h in <a href="
https://salsa.debian.org/kernel-team/linux/-/merge_requests/1005">https://
salsa.debian.org/kernel-team/linux/-/merge_requests/1005</a></div></div>
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)