XPost: linux.debian.maint.glibc

Hi Helmut,

On 2024-02-05 07:44, Helmut Grohne wrote:

Hi Aurelien,

So this confirms your initial suspicion on the actually affected case.
Thank you.

c-t-b has repacking code with arch-specific mangling (of slibdir)
already. https://sources.debian.org/src/cross-toolchain-base/68/debian/rules/#L563 Adding to that wouldn't be the worst. A relatively easy measure would be running the libc-alt.postinst manually with DPKG_ROOT set to have it
create the symlink that way. Do you think this is too much of a hack?

That's indeed an option, with the hope that we do not add more
incompatible things to that file at a later point.

However it's probably the best to ask Matthias as the maintainer of
c-t-b. If doing that way the uploads need to be synchronized to not
break c-t-b.

Cheers
Aurelien

--
Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://aurel32.net

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.maint.glibc

Hi Helmut,

I finally got time to look at your patch and do very basic testing of
it. Overall it looks good, I have a few points or questions.

On 2024-02-04 21:20, Helmut Grohne wrote:

So here is an updated patch with a few notes:
* This patch moves all the files including the runtime dynamic linker
in the main multiarch library. Therefore, this patch needs to be
synced with the corresponding base-files change to add the aliasing
symlinks or debootstrap breaks. In other words: Don't upload this
yet.

Ok.

* As mentioned earlier, only the multiarch packages install the runtime
dynamic linker via data.tar. All the multilib packages install it via
maintainer scripts now.
* When installing libc6-x32, /libx32 will be created because partial
upgrades might otherwise be broken. Removing libc6-x32 will not
remove /libx32 though. I suggest fixing this in base-files by
installing a trigger interest in /usr/libx32 and having
base-files.postinst create/remove /libx32 as needed. This way, we
centralize the management of aliasing links into base-files. libx32
only serves as an example here and it works the same way for any
other non-essential multilib directory. Do you agree with the
approach?

It sounds good yes. I never liked the fact the fact that the top level
symlinks like libx32 have to be handled by libc6. I explained that
clearly in #926699, and even suggested to do that in base-files, however
I didn't get the clever idea to use triggers. I got pressure from a
member of the TC to be constructive and given I didn't have a patch to
provide, I had to accept getting it managed by glibc...

* The multilib packages install a trigger interest on the runtime
dynamic linker such that they notice when a multiarch package deletes
it and can then recreate it as needed. Thus the multiarch packages do
not have to pay attention to a possibly installed multilib package
when they are removed.

Does it means we can just remove the Replaces: in the multiarch and
multilib libc? It should not be necessary anymore, even if without file conflicts, they should not be an issue. However not sure what could
happen during the upgrade between the old and new packages.

* I've tried quite a few multiarch upgrades involving amd64 and i386
using dpkg --auto-deconfigure --unpack with various packages and even
cross grading libc6-x32 from :i386 to :amd64 during the upgrade.
While dpkg --verify was occasionally unhappy during a partial upgrade
(where half-unpacked packages were around), once no package was
half-installed, dpkg --verify was also happy again in all attempts. I
did not come up with a systematic enumeration of possible upgrade
scenarios though.

Great.

* The patch works will deboostrap/cdebootstrap/mmdebstrap (in
combination with more patches including base-files, bash, dash and
util-linux).

Ok

* The change to install ldconfig to /usr/sbin can be uploaded right
away. It's limited to debian/debhelper.in/libc-bin.install and
debian/debhelper.in/libc-bin.lintian-overrides and doesn't contribute
much diff, so doing it later is fine as well.

Ok noted. The idea was to get the 2.38 into testing, but the glibc
transition is currently blocked by the time_t transition, so the
development is currently stalled, and I am not sure when we'll do an
upload.

I also noticed that you clearly marked the code that can be removed only
after "released:forky", thanks for doing so. Do you really mean after
forky is released, so in the development cycle of forky+1? Or do you
mean for the release of forky, so in the development cycle of forky?

Cheers
Aurelien

--
Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://aurel32.net

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.maint.glibc

On Sun, Feb 04, 2024 at 09:20:09PM +0100, Helmut Grohne wrote:

Hi Aurelien and Sven,

On Wed, Jan 24, 2024 at 09:19:12PM +0100, Aurelien Jarno wrote:

On 2024-01-23 17:40, Helmut Grohne wrote:

Conflicting runtime dynamic linkers between multiarch packages ==============================================================

Some architecture combinations such as s390, powerpc, hppa, m68k, mipsn32, and hurd-i386 or alpha, i386, sh4, and sparc have conflicting runtime dynamic loaders. In theory this violates Multi-Arch: same, but there is basically nothing we can do about this and dropping Multi-Arch: same from the packages would completely break any kind of multiarch setup. There is little we can do here and this is also unrelated to DEP17.

We tried to add conflicts for those, like it's done for the multilib packages, but at the time the infrastructure exploded (see #745552). I don't remember the details, but I guess it was either dak or dose-builddebcheck.

Yeah, I also remember something like that. It's not the first time I
trip into this. "There is little we can do here" still counts.

I guess the symlink in the maintainer script could indeed work. I am not sure why it has been done like that, it was part of the multiarch
patchset from Steve Langasek more than 10 years ago.

Practically speaking, it appears to work rather well. On the flip side,
I also thought that way of my first patch. ;)

Note however that the those packages are used by cross-toolchain-base (which builds them from glibc-source) and mangled to install them in the cross path. See for instance libc6-amd64-x32-cross. For such cases, we probably do want to keep the dynamic linker symlink, as those packages
do not have maintainer scripts.

I don't think those -$arch-cross packages need the runtime dynamic
linker at all. Unlike the libc6-$multilib packages, we don't use
-$arch-cross packages to actaully run any binary. Simply not installing
it might just work in practice.

So here is an updated patch with a few notes:
* This patch moves all the files including the runtime dynamic linker
in the main multiarch library. Therefore, this patch needs to be
synced with the corresponding base-files change to add the aliasing
symlinks or debootstrap breaks. In other words: Don't upload this
yet.
* As mentioned earlier, only the multiarch packages install the runtime
dynamic linker via data.tar. All the multilib packages install it via
maintainer scripts now.
* When installing libc6-x32, /libx32 will be created because partial
upgrades might otherwise be broken. Removing libc6-x32 will not
remove /libx32 though. I suggest fixing this in base-files by
installing a trigger interest in /usr/libx32 and having
base-files.postinst create/remove /libx32 as needed. This way, we
centralize the management of aliasing links into base-files. libx32
only serves as an example here and it works the same way for any
other non-essential multilib directory. Do you agree with the
approach?
* The multilib packages install a trigger interest on the runtime
dynamic linker such that they notice when a multiarch package deletes
it and can then recreate it as needed. Thus the multiarch packages do
not have to pay attention to a possibly installed multilib package
when they are removed.
* I've tried quite a few multiarch upgrades involving amd64 and i386
using dpkg --auto-deconfigure --unpack with various packages and even
cross grading libc6-x32 from :i386 to :amd64 during the upgrade.
While dpkg --verify was occasionally unhappy during a partial upgrade
(where half-unpacked packages were around), once no package was
half-installed, dpkg --verify was also happy again in all attempts. I
did not come up with a systematic enumeration of possible upgrade
scenarios though.
* The patch works will deboostrap/cdebootstrap/mmdebstrap (in
combination with more patches including base-files, bash, dash and
util-linux).
* The change to install ldconfig to /usr/sbin can be uploaded right
away. It's limited to debian/debhelper.in/libc-bin.install and
debian/debhelper.in/libc-bin.lintian-overrides and doesn't contribute
much diff, so doing it later is fine as well.

I hope you don't manage to punch holes into my theory this time around.

Ubuntu testing revealed that due to /lib64 going away in the package we
also are going to need Depends or PreDepends on base-files, such that base-files trigger can then create the symlink.

Or we temporarily handle the trigger ourselves or so.

I am not sure if Depends are enough, I guess if you do a release
upgrade we could end up with

unpack base-files
unpack libc6
preinst for something we want to unpack -> poof, no /lib64 here, can't find linker.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.maint.glibc

Hi Aurelien and Santiago,

my DEP17 work on moving essential files has been uploaded to Ubuntu
noble by Julian Andres Klode and we gain some insights there. As a
result, I've been refining the patches and published my testing
infrastructure. You can find it all (though maybe not in an easily
consumable way) at
https://salsa.debian.org/helmutg/bootstrap-usrmerge-demo.

One thing we noticed on the Ubuntu side is that as long as there are
packages installing into aliased locations, the protective diversions
(DEP17 M4) that I propose for base-files cause surprises for users. We
divert e.g. /bin to /bin.usr-is-merged and this becomes a real
directory. I've reconsidered this and concluded that there is no benefit
in diverting /bin, /lib and /sbin. All that we have to do to avoid their accidental deletion is to ensure that they are installed by some package
at all times. In bookworm and older, these are directories installed by base-files. In trixie, these will be symlinks provided by base-files.
dpkg will resolve symlink vs directory conflicts in our favour.

For /lib64 (on amd64 and some others), the story is more nuanced. This
is not installed by base-files in bookworm. Therefore upgrading
libc6:amd64 (whose bookworm version contains /lib64) before upgrading base-files is prone to loosing this symlink. Therefore, we need the
protective diversion from libc6.preinst until base-files.postinst. On
the final installation, /lib64 will not be diverted on amd64.

For other multilib links, I discussed the matter with some members of #debian-devel and there were some conclusions. In general, we want to
avoid the presence of unnecessary multilib symlinks (which is what the
trigger approach in base-files is supposed to ensure). However, that
means that there can be no package containing them and hence we really
need to divert them for longer (in a finished trixie installation). This
is what the previous patches already did, but I'm reconfirming that we
really need this.

The trigger-interest I previously added for base-files was broken and
not actually activating. This is also fixed.

As a further measure to reduce annoyed user reports, I'm changing the
diversion targets for aliasing links from /$orig.usr-is-merged to /.$orig.usr-is-merged hoping that the leading dot will prevent the empty directories from being listed to users. Note that we cannot delete them, because that would cause dpkg --verify to report them as missing.
They'll automatically disappear once no installed package installs any
files into an aliased location.

Niels Thykier made me aware of dh_installdeb -D. Using it avoids the
need for a pile of .in files in base-files. I hope you like this
refactoring. Let me know if not.

I hope the coding style now fully meets base-files.

Regarding the repository mentioned above, most of the changes are on the
main branch, but the avoidance of diversions for /bin, /lib and /sbin as
well as the change in diversion target are currently on a separate
branch fewer-m4-diversions to ease comparing the approaches.

This is mostly a report of what I've been doing and not something
actionable to you except that I appreciate reviews of this work (e.g. commenting on the choices and rationale given above or actually looking
into the changes).

I still don't have a proposed upload date as the time64 transition
unfolds and also breaks debootstrap currently. I hope we get this done
by the end of March.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 28/2/24 a las 21:19, Helmut Grohne escribió:

Niels Thykier made me aware of dh_installdeb -D. Using it avoids the
need for a pile of .in files in base-files. I hope you like this
refactoring. Let me know if not.

Nice feature indeed. To simplify the usr-merge patch, I've adopted
the -D change in debian/postinst *before* we apply the usr-merge patch.

Please take a look at branch "wip-202404-usrmerge" here:

https://salsa.debian.org/sanvila/base-files-not-yet/-/tree/wip-202404-usrmerge?ref_type=heads

(The repository name speaks for itself... I'm still not comfortable enough
with salsa, but I'd like to experiment and see how it goes).

I've rebased the patch relative to version 13.1 which I have just uploaded.

If the rebase is ok, I'd like to make some minor editorial changes there as well.

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Santiago,

On Wed, Apr 17, 2024 at 01:40:21PM +0200, Santiago Vila wrote:

Nice feature indeed. To simplify the usr-merge patch, I've adopted
the -D change in debian/postinst *before* we apply the usr-merge patch.

Thank you.

Please take a look at branch "wip-202404-usrmerge" here:

https://salsa.debian.org/sanvila/base-files-not-yet/-/tree/wip-202404-usrmerge?ref_type=heads

(The repository name speaks for itself... I'm still not comfortable enough with salsa, but I'd like to experiment and see how it goes).

I've rebased the patch relative to version 13.1 which I have just uploaded.

Yeah, looks reasonable.

If the rebase is ok, I'd like to make some minor editorial changes there as well.

I noticed your upload before your mail and already updated the
integration branch:

https://salsa.debian.org/helmutg/bootstrap-usrmerge-demo/-/blob/main/patches/base-files

Please let me know when you did your editorial changes such that I can
replace my patch with yours and retest.

At this time, we're down to the 5 planned packages base-files, bash,
dash, glibc and util-linux and could upload in principle, but we really
want time64 to migrate first and it's not clear when that'll happen. I'm negotiating a transition slot.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

El 17/4/24 a las 20:07, Helmut Grohne escribió:

Please let me know when you did your editorial changes such that I can replace my patch with yours and retest.

I did those minor changes in the branch wip-202404-usrmerge, just after your patch.
(If you adopt those changes, I'll rewrite the branch to be a single commit again)

Additionally (but did not have time to look at it yet), I'd like this line:

$(foreach d,$(USR_MERGE),ln -s usr/$(d) debian/base-files/$(d) &&) :

to be a "shell" for instead (similar to the way debian/triggers is created). The foreach feature is perfect in the dh_installdirs line, but in this case,
I would prefer shell code, for readability.

BTW: I am completely ok that we wait until the t64 transition is finished.

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi.

I've added two commits, one to create symlinks with a "shell" for, and the last one for a sample changelog (because this is really a "team upload" or a "guest upload"
more than a NMU). I think for simplicity it's ok if you skip the changelog part in your repo.

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Santiago,

On Fri, Apr 19, 2024 at 07:24:15PM +0200, Santiago Vila wrote:

I've added two commits, one to create symlinks with a "shell" for, and the last
one for a sample changelog (because this is really a "team upload" or a "guest upload"
more than a NMU). I think for simplicity it's ok if you skip the changelog part
in your repo.

I've updated my demo repository with your patch.

https://salsa.debian.org/helmutg/bootstrap-usrmerge-demo/-/commit/6425c8cde53596199cd37bb1625d1dfb2a4b74d0

I'm happy to call it guest upload while I find team upload slightly
misleading.

I avoid patching changelogs in the demo to avoid the need for
unnecessary rebasing. It's a functional demo.

For util-linux, I think Chris will send me an encrypted version of a
signed .changes file such that I only perform the upload not even the
signing. Please let me know if you prefer that approach.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I've updated my demo repository with your patch.

https://salsa.debian.org/helmutg/bootstrap-usrmerge-demo/-/commit/6425c8cde53596199cd37bb1625d1dfb2a4b74d0

Great. I'll take a look.

I'm happy to call it guest upload while I find team upload slightly misleading.

I avoid patching changelogs in the demo to avoid the need for
unnecessary rebasing. It's a functional demo.

For util-linux, I think Chris will send me an encrypted version of a
signed .changes file such that I only perform the upload not even the signing. Please let me know if you prefer that approach.

I'd like to use that approach too. At least until I make
the package reproducible-from-git (which currently it's not).

BTW: There is a minor glitch in postinst which I introduced
a long time ago: /var/run and /var/lock are created
as real directories, but then, at debootstrap time, the
function migrate_directory() function converts them to symlinks.

(I do not remember why it's done that way, maybe shipping
a real symlink would confuse dpkg?)

Do you think I should try to do this in a more orthodox way?
Or maybe it does not worth the effort?

(I ask because, as in the usr-merge patch, this is also
about "avoiding dpkg to follow symlinks")

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi.

Since the t64 transition has not finished completely yet (and also
I'd like the dust to settle a little bit after it finish), I've made
another release with minor changes.

The rebased branch is now wip-20240504-usrmerge.

I've also created a rebased branch with changelog called wip-20240504-usrmerge-rc.

If you use such rc branch and just change "unreleased" to "unstable"
then I'm ok with you making the upload (no need to send you signed changes),
as I've introduced a little hack to preserve the timestamps of the common-licenses (this was a prerequisite which I set for myself before
I would switch to git).

Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)