Package: release.debian.org
Severity: normal
Tags: bookworm
User: release.debian.org@packages.debian.org
Usertags: pu
X-Debbugs-Cc: debootstrap@packages.debian.org, helmut@subdivi.de
Control: affects -1 + src:debootstrap
Control: block 1025708 by -1

[ Reason ]
Part of the transition to merged-/usr, and more specifically, allowing
us to stop shipping files in trixie whose physical path on disk does
not match their path in the dpkg database due to directory aliasing.

This change needs to be in bookworm (and bullseye, and maybe buster)
before that process can continue, because official buildds run debootstrap
from stable (or older).

I also took the opportunity to backport changes that make the autopkgtests pass.

[ Impact ]
If not accepted, trixie will continue to be stuck in a
mostly-but-not-entirely merged-/usr limbo, with the moratorium from #1035831 remaining in place.

[ Tests ]
More details of testing on <https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/102>.
A prerelease (differing only in the changelog) is available from <https://people.debian.org/~smcv/12.2/pool/main/d/debootstrap/>.

I used this version of debootstrap to install sid, trixie, bookworm,
bullseye and buster on amd64, in the default, minbase and buildd
variants, and compared the results to corresponding pairs of reference
chroots. The reference chroots were installed with the Debian 12.1
version of debootstrap, explicitly forcing --[no-]merged-usr.

All default and minbase chroots continue to be merged-/usr by default.

The sid and trixie buildd chroots are now merged-/usr by default (this
is an intentional change).

The bookworm, bullseye and buster buildd chroots continue to be
non-merged-/usr by default.

When I used diffoscope to compare each chroot tarball to the reference
chroot tarball with the same suite, variant and (non-)merged-/usr status,
all differences were expected or ignorable:

- /lib32, /libx32 symlinks not created (an intentional change)
- empty /usr/lib32/, /usr/libx32/ not created (an intentional change)
- non-reproducible timestamps (ignorable)
- non-reproducible machine ID (ignorable)
- non-reproducible ldconfig cache (ignorable)
- non-reproducible systemd-journald message catalog in buster (ignorable)
- non-reproducible /var/log (ignorable)

Philip Hands built a d-i mini.iso with the proposed version, and it seems
to have installed GNOME successfully under openQA.

There is also an autopkgtest which bootstraps Debian testing and
inspects various subtleties of the resulting chroot. It now passes under autopkgtest-virt-qemu (which previously failed), autopkgtest-virt-lxc
and Salsa-CI.

The changes were backported from testing/unstable, where there were
no regression reports that I've seen. The last of them migrated to
trixie today.

[ Risks ]
Packages that were relying on sid and trixie buildds to be non-merged-/usr could break or misbehave. This is intentional: only merged-/usr is
supported, and this change is mainly to get the buildds into a supported
state for the future.

Packages that were relying on the existence of compat symlinks for
non-default multilib flavours (for example /lib32 and /libx32 on amd64)
will no longer find that they exist in all cases. I would say this is only
a minor risk. In principle it could be mitigated by creating the compat symlinks unconditionally when bootstrapping older suites (<= bookworm)
but if that's wanted, we should do it in unstable first.

[ Checklist ]
[x] *all* changes are documented in the d/changelog
[x] I reviewed all changes and I approve them
[x] attach debdiff against the package in (old)stable
[x] the issue is verified as fixed in unstable (and trixie)

[ Changes ]

* debootstrap:
- Add --merged-usr to the --help (#1031828). Minor documentation fix,
no functional change.

* functions:
- can_usrmerge_symlink(), merge_usr_entry(), merge_usr():
Helmut Grohne's implementation of a new bootstrap protocol for
merged-/usr, which unpacks Essential packages and then does the
equivalent of the usrmerge package's convert-usrmerge before
proceeding, instead of creating the compat symlinks and then unpacking
Essential packages over the top of them.
This is a prerequisite for lifting the moratorium imposed by #1035831.
(#1049898)

Unlike the old setup_merged_usr(), this does not create compat symlinks
for non-default multilib libQUAL directories unless they are mentioned
in an Essential package: the practical effect is that on for example
amd64, the /lib32 and /libx32 symlinks are no longer created (but /lib64
still is, because libc6:amd64 needs it).

- In merge_usr() (new) and setup_merged_usr() (no longer used by
debootstrap, but some versions of mmdebstrap rely on it),
apply the /usr merge to trixie, sid and future buildd chroots.
Technical Committee consensus is that we want this, and it is a
prerequisite for lifting the moratorium imposed by #1035831.

* scripts/*:
- As above, use functions' merge_usr() after unpacking Essential
packages, instead of using setup_merged_usr() before. This change is
smaller than it looks from the debdiff, because all affected Ubuntu
and PureOS scripts are symlinks to each other, but debdiff follows
symlinks.
(#1049898)

* d/tests/debian-testing: Tests only.
- Don't assert that the mockup of schroot behaves perfectly if
debootstrap was run under a container manager: in practice it will not,
but this is a schroot bug, not a debootstrap bug.
This was making the autopkgtest fail when I run it in
autopkgtest-virt-qemu. (#983197)

* d/tests/fake/: Tests only.
- Make a mockup of schroot emulate the real thing more accurately, so
that we are testing what we claimed to have been testing. (#983311)
This also turns out to be necessary to make the autopkgtest
pass under autopkgtest-virt-qemu. (#983197)

[ Other info ]
A bullseye equivalent will follow on #1025708 when I've finished testing it.
To keep bookworm >= bullseye at all times, this bookworm update should be accepted before the bullseye equivalent is.

Review and testing took place on <https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/102>.

smcv

diffstat for debootstrap-1.0.128+nmu2 debootstrap-1.0.128+nmu2+deb12u1

debian/.gitignore | 6 +
debian/changelog | 30 +++++++++
debian/gbp.conf | 1
debian/salsa-ci.yml | 1
debian/tests/debian-testing | 27 +++++++-
debian/tests/fake/schroot-1.6.10-3 | 2
debootstrap | 3
debootstrap.8 | 5 -
functions | 117 ++++++++++++++++++++++++++++++++++---
scripts/amber | 2
scripts/artful | 2
scripts/bionic | 2
scripts/byzantium | 2
scripts/cosmic | 2
scripts/crimson | 2
scripts/debian-common | 8 +-
scripts/disco | 2
scripts/eoan | 2
scripts/focal | 2
scripts/groovy | 2
scripts/gutsy | 2
scripts/hardy | 2
scripts/hirsute | 2
scripts/impish | 2
scripts/intrepid | 2
scripts/jammy | 2
scripts/jaunty | 2
scripts/karmic | 2
scripts/kinetic | 2
scripts/lucid | 2
scripts/maverick | 2
scripts/natty | 2
scripts/oneiric | 2
scripts/precise | 2
scripts/quantal | 2
scripts/raring | 2
scripts/saucy | 2
scripts/trusty | 2
scripts/utopic | 2
scripts/vivid | 2
scripts/wily | 2
scripts/xenial | 2
scripts/yakkety | 2
scripts/zesty | 2
44 files changed, 214 insertions(+), 54 deletions(-)

diff -Nru debootstrap-1.0.128+nmu2/debian/changelog debootstrap-1.0.128+nmu2+deb12u1/debian/changelog
--- debootstrap-1.0.128+nmu2/debian/changelog 2022-10-18 23:49:31.000000000 +0100
+++ debootstrap-1.0.128+nmu2+deb12u1/debian/changelog 2023-08-30 15:21:17.000000000 +0100
@@ -1,3 +1,33 @@
+debootstrap (1.0.128+nmu2+deb12u1) bookworm; urgency=medium
+
+ * Non-maintainer upload targeting Debian 12.2, with maintainer approval
+ from Luca Boccassi.
+ * Backport merged-/usr support changes from trixie:
+ - Implement merged-/usr by post-merging.
+ This changes the bootstrap order so that it will be possible for a
+ future version of base-files in trixie/sid to take responsibility for
+ creating the /bin, /sbin, /lib* symlinks, and is a prerequisite for
+ lifting the moratorium imposed by #1035831.
+ (Closes: #1049898; implementation by Helmut Grohne)
+ - functions: Reinstate setup_merged_usr() as it existed before
+ fixing #1049898, for backwards-compatibility with older versions of
+ mmdebstrap.
+ - functions: Default to merged-/usr for suites newe