• Ensuring initramfs is compressed with zstd

    From Ben Hutchings@21:1/5 to All on Sun Mar 5 03:30:01 2023
    I did a test installation with d-i bookworm alpha 2, and I noticed a
    warning message from update-initramfs during base installation that it
    was falling back from zstd to gzip compression. zstd has been the
    default compressor in initramfs-tools for a while, but since the
    compressor is configurable it only Recommends zstd.

    It looks like later package upgrades lead to a new initramfs being
    generated with zstd. However, it is possible that in some conditions
    an installation would not include zstd, or would leave the initramfs
    compressed with gzip even though zstd is installed.

    Does the patch below for base-installer seem like a reasonable change?
    (I haven't tested it yet, but may do so tomorrow.)

    Ben.

    diff --git a/library.sh b/library.sh
    index e7fb8f60..120a39ae 100644
    --- a/library.sh
    +++ b/library.sh
    @@ -542,12 +542,16 @@ EOF
    if [ "$do_initrd" = yes ]; then
    rd_generator=initramfs-tools

    - # initramfs-tools needs busybox pre-installed (and only
    - # recommends it)
    + # initramfs-tools may need busybox, and prefers to use
    + # zstd over gzip, but only recommends them
    if ! log-output -t base-installer apt-install busybox; then
    db_subst base-installer/kernel/failed-package-install PACKAGE busybox
    exit_error base-installer/kernel/failed-package-install
    fi
    + if ! log-output -t base-installer apt-install zstd; then
    + db_subst base-installer/kernel/failed-package-install PACKAGE zstd
    + exit_error base-installer/kernel/failed-package-install + fi

    # Make sure the ramdisk creation tool is installed before we
    # change its configuration

    --
    Ben Hutchings
    Q. Which is the greater problem in the world today,
    ignorance or apathy?
    A. I don't know and I couldn't care less.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiE
  • From Cyril Brulebois@21:1/5 to All on Sun Mar 5 04:20:01 2023
    Hi Ben,

    Ben Hutchings <ben@decadent.org.uk> (2023-03-05):
    I did a test installation with d-i bookworm alpha 2, and I noticed a
    warning message from update-initramfs during base installation that it
    was falling back from zstd to gzip compression. zstd has been the
    default compressor in initramfs-tools for a while, but since the
    compressor is configurable it only Recommends zstd.

    Right, I've been seeing this message (esp. while trying to fight some
    u-i runs, see [1]), but didn't dive into it; thanks for doing so!

    1. https://salsa.debian.org/installer-team/hw-detect/-/commit/ead9c6a980ad179b7c6f43889ab3512345fde389

    It looks like later package upgrades lead to a new initramfs being
    generated with zstd. However, it is possible that in some conditions
    an installation would not include zstd, or would leave the initramfs compressed with gzip even though zstd is installed.

    It makes sense to address that upfront, yes.

    Does the patch below for base-installer seem like a reasonable change?
    (I haven't tested it yet, but may do so tomorrow.)

    Ben.

    diff --git a/library.sh b/library.sh
    index e7fb8f60..120a39ae 100644
    --- a/library.sh
    +++ b/library.sh
    @@ -542,12 +542,16 @@ EOF
    if [ "$do_initrd" = yes ]; then
    rd_generator=initramfs-tools

    - # initramfs-tools needs busybox pre-installed (and only
    - # recommends it)
    + # initramfs-tools may need busybox, and prefers to use
    + # zstd over gzip, but only recommends them
    if ! log-output -t base-installer apt-install busybox; then
    db_subst base-installer/kernel/failed-package-install PACKAGE busybox
    exit_error base-installer/kernel/failed-package-install
    fi
    + if ! log-output -t base-installer apt-install zstd; then
    + db_subst base-installer/kernel/failed-package-install PACKAGE zstd
    + exit_error base-installer/kernel/failed-package-install + fi

    # Make sure the ramdisk creation tool is installed before we
    # change its configuration

    Given the existing code for busybox, your proposal looks good to me.
    For just two packages, duplicating and looping are probably equivalent,
    I'll leave the choice up to you.


    On a tangential note, in the “asked and answered” category: I've been adding pigz/pixz support to various d-i and debian-cd parts lately, but
    didn't check what's happening at the initramfs level. Are we getting
    parallel compression by making sure zstd is used? (The manpage mentions
    a zstdmt variant.)

    Getting 15 seconds per (gzip) run on a powerful machine seemed a little embarrassing (but thankfully nudged me into optimizing the number of
    runs)…

    From a quick look at current mkinitramfs, d-i is currently using gzip at
    first since we're not explicitly pulling pigz, and switching to zstd
    will give us zstd -T0 (aka. zstdmt) for free, so installing that package (earlier or at all) will give us a speed boost in addition to improved consistency and correctness.

    Doubly great!


    Cheers,
    --
    Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
    D-I release manager -- Release team member -- Freelance Consultant

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEtg6/KYRFPHDXTPR4/5FK8MKzVSAFAmQECVsACgkQ/5FK8MKz VSBfCw/9HgQLlORSg7puxevUIIyv6sGZKMiNnG7/CeK8UtsUPyZul6oOE8j7eDIM liPH5TLmwlbzgl2Ztvz5CyKRNsOeV0a1Y6t+y6cb6th9U9svtcVyf5qIOCX461YU ZW8dweioCbcMgoW78v0serSirAfmuHU68dPze95AeBQoEG9TIcGMvSVRDWloGgMm 56GeKoJ41DLqNx1aCDxFZ684YbZaJ109tiWBCGDpVmtS+B4SznvQSQ4fGh2U1QoC 4nzuSpiYoevTAIGxzDQIBa2Lh9zISf3f9PRchJIYgBw51lQ6GqgKrohuj940T9k3 zd70fsQEiKWe5e1c3PEMCwOm27RKgr9Rn3T8wcSl15AD8izHQgQeD3HoDtHbEmf2 bWLe2Am14IvrqeWfD5aDfemfH74q1KsAc1M0aaYouehLU8Y1sOoAjb1RxBqXHoX4 Qt16S3/fYgaY323vg8phmiezs6LAkEJUZw+297w07LwW5T+MYflDg8gJP5/Mxp4S XnSKQaxqzTCinZuQo6Q1QgzYKP9u4b6FzIv7JPGB6n9OacvcsA9ch3BEyLuEpQWV Ot4giq8Mx7rgtTz3FmeR9i4Yb9CbeZkT21eSeRDqS1l2cvS3u2vfOtw5UiYVnEBU AggTMjW9xhhRoZ7bt46NT/eiF77d5TdCWKOzysrOLerCGnNQoAw=
    =aZX7
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    *
  • From Emanuele Rocca@21:1/5 to Ben Hutchings on Tue Apr 4 15:40:01 2023
    Hi,

    On Sun, Mar 05, 2023 at 02:56:59AM +0100, Ben Hutchings wrote:
    I did a test installation with d-i bookworm alpha 2, and I noticed a
    warning message from update-initramfs during base installation that it
    was falling back from zstd to gzip compression.

    Just FTR this is still the case with d-i RC1, I suppose Ben's patch never made it to base-installer.

    ema

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Cyril Brulebois@21:1/5 to All on Wed Apr 12 07:10:02 2023
    Emanuele Rocca <ema@debian.org> (2023-04-04):
    On Sun, Mar 05, 2023 at 02:56:59AM +0100, Ben Hutchings wrote:
    I did a test installation with d-i bookworm alpha 2, and I noticed a warning message from update-initramfs during base installation that
    it was falling back from zstd to gzip compression.

    Just FTR this is still the case with d-i RC1, I suppose Ben's patch
    never made it to base-installer.

    Merged in 1.212, just uploaded. (I haven't checked the runtime yet, but
    it looks straightforward enough to upload as is.)

    Thanks to both of you!


    Cheers,
    --
    Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
    D-I release manager -- Release team member -- Freelance Consultant

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEtg6/KYRFPHDXTPR4/5FK8MKzVSAFAmQ2OzYACgkQ/5FK8MKz VSBtOg//aij0KJyphXwJ1uuVLGUHAoZKOvlowbGJ3WrurTpWiSAffT1NXE4fAcrB ohu+CJ7NtqiDx8cnjJTk2dePR02UqgXXwQvAP1YK/G771dHUhDt4wt/R9lW9zyQ5 D8Qu6ZkC/aiwJZrfNJqVEbNIsXnT8Sr9cYlj+YpdQXAsCVnFpIdUJzKcUA7dt7Hr 4bCmLS6jIx8bDu1GwauCRPQJZ5l9y8ftFmoNEwWEvgLZ1f/QDuFBn7PdmVAPaBGo jn6FcZlVE8Jf7jJZqvgUMfLvQpx8JWF4TTHMffmZAczcwSm7pBQa+Sg43eAgRyqR 5yJpJjJyKZmrqIIXXbLfMxyeUlm3gAMB0jynSiZztGkCQwmpRNgnBgh3UYhGwiby zl//XLmPi9biWbw4VGwBDKliRuWQrImG4lCaDrBcNKI5rXHGi0lHmM9YmalJGHPG l+3MKi7oVGUHqMeRNeu+dxwutuwQNqhZa9CKlW69VfMsdeAnrugP1Xjz7SRY/pAi gc17K6ka7lMUGEYa7bBaN2g/Oka9XPVduwK5zCiIxYImWQheCUnFsxq5vCVp3MZ8 fVu95D4UcVi9Y7Rpkr8zTV/zlB4QUFdsVXrgo+GbfVHpGMRsAxhMiG8lVFho5qTz HwaGnXk8gv9Nzf3TxUfFnRDlL78DUWSESaNMGfv3YY8Sm6STaak=
    =ihe5
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    *