Hi,

I just created https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
which removes the "grep -v mini.iso" from the "get-images.sh" script.
The idea being that it can be useful to have a mini.iso available
locally in case you want to install a VM with libvirt. While it's
possible to PXE boot a VM and install that way, this involves some infrastructure that needs to be set up, and pointing to a .iso file that
exists locally seems easier.

I usually have a netboot mini.iso file in my home directory for that
purpose, but it needs to be kept up to date with every point release,
and that's a bit of an annoyance. I think it would be easier to just
have it in the debian-installer-netboot-images package.

I haven't been involved with d-i work for several years now, so I made
it a merge request rather than just committing to master, even though I
would theoretically have been able to. Please feel free to reject the MR
in case there are reasons (that I'm not aware of) for which the mini.iso
file should not be in this package.

Thanks,

--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Wouter,

Wouter Verhelst <wouter@debian.org> (2023-12-10):

I just created https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
which removes the "grep -v mini.iso" from the "get-images.sh" script.
The idea being that it can be useful to have a mini.iso available
locally in case you want to install a VM with libvirt. While it's
possible to PXE boot a VM and install that way, this involves some infrastructure that needs to be set up, and pointing to a .iso file that exists locally seems easier.

I usually have a netboot mini.iso file in my home directory for that
purpose, but it needs to be kept up to date with every point release,
and that's a bit of an annoyance. I think it would be easier to just
have it in the debian-installer-netboot-images package.

Do you have stats before/after? Looks like it could easily double or
triple the size of each binary package?


Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEtg6/KYRFPHDXTPR4/5FK8MKzVSAFAmV1xJcACgkQ/5FK8MKz VSC33RAAmnGIUwLMub1vWaQOd7gJ8Fw+cnqOacEfbwP4ZYLgkbaBoB807mo6l4Yi 6X9hCgWkrhmipe48yxxlw6p+EqfIJysD1yUtNz6Z6Qw8CHBMs+Q0r6kBL9qTGvSi vBUNnTtFlJ3+QL6VDPJ4LVrfwojqDBqzhxWQZF0B2C0h0W/SXYta9Th+8vqSMb9s iDqGC6ssK2pPYQ56/FczHg+OZnyYHxuBYd/Cn0EfPKYimAvAblBgOyVdXsZZLL4p CT6l8nG3CPVPTQE6JnBwcm0RXz8WLlZQibXQRvSApRMneIgeporTTRE7sqJ+M+9Z ukRohiVhg6r3Sy8odHuecI991qRsnJE/wvAeV8RxqxhA8lEQZlKkBKQdIDloYBVH Fd8yvACQeahshl2eGfC5fcjsry6eeqILsZeaOqtvvugxgbZ5VBUFWeUDE75lw+2K vIlU3OAd0gVYVJa9jgtL8ar692XVYBpHr9zKs58kWapxemhJtJfykiKOyaBwvAV7 I1HiEKr1XpuTw8GOYEYEfV4yOEcmeabWBXQSDsPoz9OM9zf1srWDKlPX3xo3ZtL+ EVLNLmjUqzsddImgq60i7KKtLQTr3fIf/4hrTE6FaKWP++xRud11liA2UM7iMUBB Pi0fCn21GroWR8tt+/6DI+n96KWUWvphfXDvM/669mLD1GdUF6Y=
=gaLZ
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
*

Hi kibi,

On Sun, Dec 10, 2023 at 03:00:56PM +0100, Cyril Brulebois wrote:

Hi Wouter,

Wouter Verhelst <wouter@debian.org> (2023-12-10):

I just created https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
which removes the "grep -v mini.iso" from the "get-images.sh" script.
The idea being that it can be useful to have a mini.iso available
locally in case you want to install a VM with libvirt. While it's
possible to PXE boot a VM and install that way, this involves some infrastructure that needs to be set up, and pointing to a .iso file that exists locally seems easier.

I usually have a netboot mini.iso file in my home directory for that purpose, but it needs to be kept up to date with every point release,
and that's a bit of an annoyance. I think it would be easier to just
have it in the debian-installer-netboot-images package.

Do you have stats before/after? Looks like it could easily double or
triple the size of each binary package?

I don't. Creating those stats currently is somewhat difficult, because
of the version flux resulting from last weekend's two releases. But
yeah, presumably it would at least double the size of each binary
package, because the mini.iso files contain the same thing as the other
files do. I'll give it a few days and will try again afterwards.

If size is a concern, we could perhaps create
"debian-installer-*-mini-iso-*" packages instead (e.g., "debian-installer-12-mini-iso-amd64") which would contain the mini.iso
files, rather than the actual netboot files? This obviously in addition
to current packages.

--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Wouter Verhelst <wouter@debian.org> writes:

Hi,

I just created https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
which removes the "grep -v mini.iso" from the "get-images.sh" script.
The idea being that it can be useful to have a mini.iso available
locally in case you want to install a VM with libvirt. While it's
possible to PXE boot a VM and install that way, this involves some infrastructure that needs to be set up, and pointing to a .iso file that exists locally seems easier.

I usually have a netboot mini.iso file in my home directory for that
purpose, but it needs to be kept up to date with every point release,
and that's a bit of an annoyance. I think it would be easier to just
have it in the debian-installer-netboot-images package.

I presume you're currently downloading a published mini.iso, rather than building them locally, is that right?

Rather than packaging up the mini.iso's, how about having a package that
acts as an installer, and downloads the published image that matches the relevant kerrnel version.

The reason I think that might be worth the effort is that I suspect that
quite a lot of the mini.iso's would have almost no users (depending on architecture etc), so copying them across the whole mirror network seems
like a significant waste of resources.

Personally, I tend to use the netinst for the purpose you describe, and
think that it might be quite useful to have a package that would
maintain a copy of the current image on my systems, so such a package
could be more widely useful than just for mini.iso downloads.

The 'di-netboot-assistant' package does something a bit like this (and
much more, for TFTP booting), so perhaps it could be extended for this
use case, although I think it currently only downloads stuff in response
to user requests, rather than automatically on upgrade.

Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE3/FBWs4yJ/zyBwfW0EujoAEl1cAFAmV4MxMACgkQ0EujoAEl 1cCw4BAAw+G1GyCRflIKtIeVm7lpJ286rU6bQziUZ+xMAMVIfihAeWqj5LsIgXeo I7A7eCX1/ryP69p4aS4z/MQuD6ll4BI7u5fCPrKoO3Mi+BusPUIp+tt1MEfd6MsO OlcjwfOUrA0d2q2eaoXVvvhoQQcOsj6AOqAZlP/A3t55uMUvSTMB2I5ttydb3Gtg tTkmewsGP7i1cY6Kk2OE7iIioUNR3irPpG/cTKWIStI1RZ61Yh1AJJ2m+w9QIU8k EMCzCBTSSTn+P8BjMftqwALWNx736bUrgqEWYPi6e0ErnoMKx1frZgxTgNvzJToa zSr2sWzaScp/hL6hrxbpkwZT2d6F3HEYTSy/XRmYCiEVM1mjzEeRYnG1Kta3S/Nl +ibbYJCZcaQtBLsY2Dd3SoBWXuM3DkAa6ImiWWD1uRlor8U2oSdXoyP2Z1Bh8yZz kmGsPjzwb0XrmTT4b0iygkKg8CEUOZaeXFWFADkIRDSk1ndJEDP2EI60mVm1+g6F zYgiro3LlcPneGz8j7sk/gXqDP3taUwPcKw9NpV5fFb9fW+mtb50mFqeV9KE/nJO Db7+WWYS1qOHvtBvSVgsUTCpRBsQnKbU6UERV2hRzwEnNfwKzhkImAWJ/WgL+WQa cR7qFdXYEhfErt4p3Sx/+5W8Tk0HUHK24QjpCaZPf0M6EPn2Knk=2v1K
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gatewa

On Tue, Dec 12, 2023 at 11:16:51AM +0100, Philip Hands wrote:

Wouter Verhelst <wouter@debian.org> writes:

Hi,

I just created https://salsa.debian.org/installer-team/debian-installer-netboot-images/-/merge_requests/3,
which removes the "grep -v mini.iso" from the "get-images.sh" script.
The idea being that it can be useful to have a mini.iso available
locally in case you want to install a VM with libvirt. While it's
possible to PXE boot a VM and install that way, this involves some infrastructure that needs to be set up, and pointing to a .iso file that exists locally seems easier.

I usually have a netboot mini.iso file in my home directory for that purpose, but it needs to be kept up to date with every point release,
and that's a bit of an annoyance. I think it would be easier to just
have it in the debian-installer-netboot-images package.

I presume you're currently downloading a published mini.iso, rather than building them locally, is that right?

Correct.

Rather than packaging up the mini.iso's, how about having a package that
acts as an installer, and downloads the published image that matches the relevant kerrnel version.

... that could very much work too! I hadn't thought of that.

The reason I think that might be worth the effort is that I suspect that quite a lot of the mini.iso's would have almost no users (depending on architecture etc), so copying them across the whole mirror network seems
like a significant waste of resources.

Personally, I tend to use the netinst for the purpose you describe, and
think that it might be quite useful to have a package that would
maintain a copy of the current image on my systems, so such a package
could be more widely useful than just for mini.iso downloads.

I had a quick look, and it turns out that both the d-i directory in the
mirror network and the CD mirror network have a SHA256SUMS file with
image names and relative paths. Both have other checksum files too, but
the two are not compatible there (d-i has MD5SUMS (yuck), cdimage has SHA512SUMS).

I could imagine writing a tool that, when given a URI for a SHA256SUMS
file and a local path, and updates the file if the checksum does not
match (or the file is absent)

They are both signed, but unfortunately not in the same way -- the d-i directory is signed through (In)Release(.gpg), the cd network through a SHA256SUMS.sign file right next to the SHA256SUMS one. But that can be
worked around.

That sounds like a much more reasonable way forward, although I'm not
keen on extending di-netboot-assistant (it's massive already, and does
very different things). At any rate, I'll have a look at implementing
this. Thanks for the suggestion!

--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

So.

On Wed, Dec 13, 2023 at 10:45:36AM +0200, Wouter Verhelst wrote:

That sounds like a much more reasonable way forward, although I'm not
keen on extending di-netboot-assistant (it's massive already, and does
very different things). At any rate, I'll have a look at implementing
this. Thanks for the suggestion!

This turned out to be fairly simple in the end. First, create a apt.conf
file like so:

Acquire::IndexTargets::deb::SHA256SUMS {
MetaKey "$(COMPONENT)/installer-$(ARCHITECTURE)/current/images/SHA256SUMS";
ShortDescription "SHA256SUMS";
Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) d-i SHA256SUMS (deb)";
};

Next, create a config file like this:

images:
mini.iso:
mirror_type: deb
limit:
Suite: stable
Architecture: amd64
basedir: main/installer-amd64/current/images
relative_name: ./netboot/gtk/mini.iso
target_filename: /var/lib/libvirt/images/bookworm-mini.iso
netboot.iso:
mirror_type: cd
basedir: current/amd64/iso-cd
filename_regex: debian-[0-9.]+-amd64-netinst.iso
target_filename: /var/lib/libvirt/images/bookworm-netinst.iso

And then once you have that, the following perl script should do the
job.

(config file stored either as /etc/debian-isosync/config.yaml or
passed as the first argument to the script)

---
#!/usr/bin/perl -w

use v5.36;

use Crypt::Digest::SHA256 qw/sha256_file_hex/;
use YAML::XS qw/LoadFile Dump/;
use Dpkg::Control::HashCore;
use LWP::UserAgent;
use File::Temp qw/tempdir/;

my $ua = LWP::UserAgent->new(show_progress => 1);
$ua->env_proxy;

my $cfilename = shift;
if(!defined($cfilename)) {
$cfilename = "/etc/debian-isosync/config.yaml";
}

my $cfile = LoadFile($cfilename) or die $!;

die "need list of images in config file" unless exists($cfile->{images});
die "images setting in config file is not a hash" unless ref($cfile->{images}) eq "HASH";

sub get_apt_sha256sum {
my $rv = [];
my $found;
open my $indextargets, "-|", "apt-get", "indextargets";
do {
my $indexes = Dpkg::Control::HashCore->new;
$found = $indexes->parse($indextargets, "index targets");
push @$rv, $indexes if $found;
} while $found;

close $indextargets;

return $rv;
}

sub ensure_file($url, $sha256_expected, $filename) {
if(-f $filename) {
my $sha256sum_found = sha25

On Wed, Dec 13, 2023 at 01:21:13PM +0200, Wouter Verhelst wrote:

So.

On Wed, Dec 13, 2023 at 10:45:36AM +0200, Wouter Verhelst wrote:

That sounds like a much more reasonable way forward, although I'm not
keen on extending di-netboot-assistant (it's massive already, and does
very different things). At any rate, I'll have a look at implementing
this. Thanks for the suggestion!

This turned out to be fairly simple in the end. First, create a apt.conf
file like so:

Acquire::IndexTargets::deb::SHA256SUMS {
MetaKey "$(COMPONENT)/installer-$(ARCHITECTURE)/current/images/SHA256SUMS";
ShortDescription "SHA256SUMS";
Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) d-i SHA256SUMS (deb)";
};

Next, create a config file like this:

images:
mini.iso:
mirror_type: deb
limit:
Suite: stable
Architecture: amd64
basedir: main/installer-amd64/current/images
relative_name: ./netboot/gtk/mini.iso
target_filename: /var/lib/libvirt/images/bookworm-mini.iso
netboot.iso:
mirror_type: cd
basedir: current/amd64/iso-cd
filename_regex: debian-[0-9.]+-amd64-netinst.iso
target_filename: /var/lib/libvirt/images/bookworm-netinst.iso

And then once you have that, the following perl script should do the
job.

(config file stored either as /etc/debian-isosync/config.yaml or
passed as the first argument to the script)

I've pushed this to https://salsa.debian.org/installer-team/d-i/isosync.
Happy to move it elsewhere if necessary, but I think it fits within the
d-i team.

Haven't added it to the mr configuration (yet) nor uploaded to the
archive; feedback on whether people think this a good idea is welcome.

--
w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)