Hi

Our documented, I think, policy is, that we don't support loading new
modules into an old kernel within the same ABI. This forces a reboot
after kernel installation.

However in a lot of cases this just worked. You could update the kernel package and continue loading most modules.

Now we have BTF support enabled, which trashes this compatibility
mostly, as it requires a way more strict match between kernel image and modules.

We need to fix that somehow. Options are as far as I see
- remove BTF from modules,
- allow to load modules even on BTF mismatch, or
- reinforce that a user can't do that.

If we go with the last option we would have also some direct advantages.
We could stop signing modules with the secure boot key, but use a
temporary key. This would for a system with signature checking enabled effectively trash all possibilities to load modules for a different
kernel build.

This would do directly for use:
- A way faster build, as we don't longer require multi-10k signatures,
but only a handfull, from the HSM.
- We might be forced by shim/UEFI to support SBAT if we load secure boot
signed stuff. If we just need to revoke the complete kernel, SBAT on
the kernel itself is enough.
- We can do pre-built initramfs and unified image without two rounds in
the signing service.
- We fix the current signatures without certificate chain, but which are
still chained to the Debian secure boot CA. (sign-file simply can't
handle cert chains, while the kernel itself can check them on loading.)

What should we do?

Regards,
Bastian

--
It would be illogical to kill without reason.
-- Spock, "Journey to Babel", stardate 3842.4

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Bastian,

Thanks for raising the topic.

On Sat, Dec 10, 2022 at 02:27:12PM +0100, Bastian Blank wrote:

Hi

Our documented, I think, policy is, that we don't support loading new
modules into an old kernel within the same ABI. This forces a reboot
after kernel installation.

However in a lot of cases this just worked. You could update the kernel package and continue loading most modules.

Now we have BTF support enabled, which trashes this compatibility
mostly, as it requires a way more strict match between kernel image and modules.

We need to fix that somehow. Options are as far as I see
- remove BTF from modules,
- allow to load modules even on BTF mismatch, or
- reinforce that a user can't do that.

If we go with the last option we would have also some direct advantages.
We could stop signing modules with the secure boot key, but use a
temporary key. This would for a system with signature checking enabled effectively trash all possibilities to load modules for a different
kernel build.

This would do directly for use:
- A way faster build, as we don't longer require multi-10k signatures,
but only a handfull, from the HSM.
- We might be forced by shim/UEFI to support SBAT if we load secure boot
signed stuff. If we just need to revoke the complete kernel, SBAT on
the kernel itself is enough.
- We can do pre-built initramfs and unified image without two rounds in
the signing service.
- We fix the current signatures without certificate chain, but which are
still chained to the Debian secure boot CA. (sign-file simply can't
handle cert chains, while the kernel itself can check them on loading.)

What should we do?

I do agree we have to handle this before the bookworm release. Most of
the time it was a non-issue but only due to the fact that an ABI bump
was the most sensible next step for the upload (recent stable updates
have enough changes per release which makes as well easier rollback
for users in case of issues). So at least in my case I almost every
time bumped ABI, but every time we do not a report about the BTF
mismatch problem get reported.

That said, I'm not sure what we should do. Ben?

Salvatore

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Dec 10, 2022 at 02:27:12PM +0100, Bastian Blank wrote:

Hi

Our documented, I think, policy is, that we don't support loading new
modules into an old kernel within the same ABI. This forces a reboot
after kernel installation.

However in a lot of cases this just worked. You could update the kernel package and continue loading most modules.

Now we have BTF support enabled, which trashes this compatibility
mostly, as it requires a way more strict match between kernel image and modules.

We need to fix that somehow. Options are as far as I see
- remove BTF from modules,
- allow to load modules even on BTF mismatch, or
- reinforce that a user can't do that.

If we go with the last option we would have also some direct advantages.
We could stop signing modules with the secure boot key, but use a
temporary key. This would for a system with signature checking enabled effectively trash all possibilities to load modules for a different
kernel build.

This would do directly for use:
- A way faster build, as we don't longer require multi-10k signatures,
but only a handfull, from the HSM.
- We might be forced by shim/UEFI to support SBAT if we load secure boot
signed stuff. If we just need to revoke the complete kernel, SBAT on
the kernel itself is enough.
- We can do pre-built initramfs and unified image without two rounds in
the signing service.
- We fix the current signatures without certificate chain, but which are
still chained to the Debian secure boot CA. (sign-file simply can't
handle cert chains, while the kernel itself can check them on loading.)

What should we do?

+1 on using the ephemeral key from me, those advantages seem to
outweight the drawbacks. It should be possible, in theory, to teach
diffoscope to ignore the embedded ephemeral public key in the kernel
image when comparing builds?

--
Kind regards,
Luca Boccassi

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmObALwACgkQKGv37813 JB5ltw/+IJu5Nqb8z7m47x2/ZM78eHO3cpewlJzX6AyUafSJZ2SH4pggfHt2cTxJ R9UOxbXTjQXIFzPKXgMu0Ybyzuzfx3rX1RwJSq6Y6TMqfyXPWx5gEiYcJtZa73h0 8D9rcBsCaUlyBp0Qa8XPQS9Opml4AJkfuor4GJBAYi665fPMI/u3H4QMNGfweQcc 5Fp/fOzILCToQF/mszTBj33mcUBDgxQSUdi6iUe6uesyq8tDAiEKk9hVMR183TcM Z/aYlVoRSbr7l6/6MFJ502AiKzSdaDTYif6K78gJbkDERrMEM4nON8PMxlQbwwSJ lznUxQYWv6qu4/pF8wtXIfC6St/OmLcZ1bF65pkQucIL+XBkwHyuYRfNEavY7eLV MUOjRbLrf0+WfgK7sgZbOv7AI+ZAOiDEAhonuFeNc+CeVUUMEs6bvuI8m36luyIq HET/eF4Q3bMk6h5WAfvvZp623Rrpf1keuP8kLdlARGPmSnXSa8pHnFigw6szDZze vpsMv2a7HDTjNZDbudBY329KrsFmRAxd40zMXrpQpBJmtgmJJ+0Ga+okUMLk6euK jDaN42DfEhuHwrqwDCXXoNwzAy9OsGm6Eg2juT+AVlzoaXJsedUSme8nObLglPME T0DMEbU6GplPL3dJq1tN/zU8TPpTIYnOnEPvFuBJnxxWtaw6PJk=
=/Jbb
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

(Bcc:'ed to debian-boot@)

On 10/12/2022 16:27, Bastian Blank wrote:

Our documented, I think, policy is, that we don't support loading new
modules into an old kernel within the same ABI. This forces a reboot
after kernel installation.

However in a lot of cases this just worked. You could update the kernel package and continue loading most modules.

Now we have BTF support enabled, which trashes this compatibility
mostly, as it requires a way more strict match between kernel image and modules.

One thing to note here is Debian Installer netboot images. Those built
with a certain kernel-image-$ABI-di can download *-modules-$ABI-di udebs
for that ABI at installation-time, but don't check the actual version.
They do display an error if the archive doesn't have modules for that
ABI name.

I recently encountered this incompatibility with a slightly out-of-date
custom netboot image I was testing, and it's not really visible to a
user that this has happened. It complained about kernel not supporting
RAID and so on, then inexplicably failed after partitioning (looked like
it couldn't mount root as ext4). I needed to build a new image.

We need to fix that somehow. Options are as far as I see
- remove BTF from modules,
- allow to load modules even on BTF mismatch, or
- reinforce that a user can't do that.

I fear I'm being ignorant here, but 'Bump ABI on every build' sounds to
me like it would be technically valid. Although, that might defeat the
purpose of tracking ABIs, is that the reason it's not an option?

If we go with the last option we would have also some direct advantages.
We could stop signing modules with the secure boot key, but use a
temporary key. This would for a system with signature checking enabled effectively trash all possibilities to load modules for a different
kernel build.

Anyway, I guess it would be possible to modify d-i to check modules more strictly, by using package version instead of ABI name (in src:anna ?).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 15 Dec 2022 11:10:52AM +0000, Luca Boccassi wrote:

On Sat, Dec 10, 2022 at 02:27:12PM +0100, Bastian Blank wrote:

If we go with the last option we would have also some direct advantages.
We could stop signing modules with the secure boot key, but use a
temporary key. This would for a system with signature checking enabled
effectively trash all possibilities to load modules for a different
kernel build.

What should we do?

+1 on using the ephemeral key from me, those advantages seem to
outweight the drawbacks. It should be possible, in theory, to teach diffoscope to ignore the embedded ephemeral public key in the kernel
image when comparing builds?

Would using a multi-stage module-signing approach[1] help? (if I
understand correctly, the embedded certificate material should be
static and thus reproducible)

[1] - https://www.kernel.org/doc/html/v6.1/kbuild/reproducible-builds.html#module-signing

(note: apologies for the lack of an in-reply-to email header on this
message. I'm not subscribed to the list but wanted to add a reply,
and couldn't figure out how to set that header manually in the email
client I'm using)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)