On Sat, May 21, 2022 at 01:57:59PM +0200, Thomas Schmitt wrote:

Hi Thomas,

Hi,

some technical nitpicking.

Andrew M.A. Cater wrote:

Also, keeping large files around on disk for a long
time - there's some likelihood of data corruption.

The .jigdo and .template files of the DLBD ISOs are together smaller than
a netinst CD ISO. (Less than 90 MiB, see https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dlbd/ )

Building an ALL-IN-ONE would cost the computing time of another ISO set (*) and 50 % percent more virtual memory than building DLBD1 (**).

That's if you assume that building from jigdo and a mirror is always fine.
If the original poster wants one huge .iso as one file to download from cdimage.debian.org - then 2 x double layer Blu-Ray (say) as one file
would be 100GB or so. Even on a good quality link, that's quite a time.

[If you've actually got a physical mirror near to you, the jigit scripts
work even better than jigdo-file - but they produce the .iso on the same machine, which is not ideal for everyone.]


Then there's storage - generating multiples of those per architecture
per point release adds up to space on cdimage.debian.org.

I assume that the main workload with an additional set of jigdoized ISOs
is the need to once more shovel 75 GB of package files through libisofs
and libjte so that they can create .template and .jigdo files. The ISOs themselves get directly piped into /dev/null, i guess.

I don't actually know how long it take to create the .jigdo and .template
files - I don't think it's as long as generating the full .iso files and copying them around between machines as they're generated on each point
release day, for example, and they don't get generated multiple times.

I'd hate a couple of
bit flips three quarters of the way through a 6TB file, say, to mean that the whole thing isuseless.

This is the case for the hypothetical "all in one" .iso to contain all architectures - be a "one source for Debian 10.12" .iso - which is feasible
but probably not sensible.

The vast majority of data is stored as packages on the worldwide mirrors.
I'd expect some quality of filesystem and backup which keeps damage
confined to a few packages.

There are a lot of outdated mirrors around where one could dig for those which really got lost from all active mirrors. I remember the hunt for
a package which once was overwritten by a newer version with the same
.deb file name. It was needed for building an old ISO from .jigdo which
was created when the new version did not yet exist.
But even without finding the older version, the emerging ISO would have
been usable for all purposes which don't touch that one rogue package.

------------------------------------------------------------------------

Every time this wish pops up, i begin to ponder what is theoretically
needed to unite several pool trees from multiple ISOs into one, so that
it works like an official ALL-IN-ONE ISO.

Putting all files together and making the ISO bootable would be no
problem. But what does a neat pool have to offer as merged lists or other meta-data so that it properly announces its content ?

------------------------------------------------------------------------

Footnotes:
(*) Computing time:
Maybe it needs a bit more than DLBD[12] together, because the insertion
algorithm of libisofs has a quadratic aspect in its personality. But
the branchy pool tree helps a lot to keep this problem under the cover. (**) Virtual memory consumption:
The memory consumption of libisofs mainly depends on the number of
files and the length of their names. *11.3.0-amd64-DLBD-[12].jigdo
together list ~ 59,000 files with ~ 2.2 MiB of basename length.
My daily BD backups have ~ 70,000 files with ~ 1.4 MiB of basename
length and do not demand gigabytes of RAM.

Have a nice day :)

Thomas

And you too - it's always good to have otehr people to think round a problem with.

All best, as ever,

Andy Cater

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)