Guillem Jover <guillem@debian.org> writes:

The fact that the supporters of a *filesystem layout* have been happy to dismiss and ignore this and have been pushing for what I think can be
easily described as the worst ever "transition" done in Debian, very
sadly, for me this whole topic marks a before and after in Debian, and
has put my trust in the technical side of the project into question.

I agree that you've been pointing out potential problems for years and
that your warnings have been at least partly vindicated by the problems we
are indeed seeing.

My understanding (which may be entirely incorrect, in which case please
let me know what your preferred solution actually is!) is that your
preferred solution was to migrate individual packages. What I'm hearing
from the frequent threads in debian-devel is that a sufficient number of
Debian contributors have rejected that approach (for various reasons) as
to make that solution not viable. In other words, my reading of the
consensus is that this option has effectively been vetoed by the rest of
the project (noting that this veto was certainly not unanimous).

Please note that I'm not taking a position on the merits of that decision, simply noting that, based on my reading of debian-devel, this is has what
has happened, and I do not believe it will be possible at this point to convince the project as a whole to unwind usrmerge and go back to doing individual package migrations.

Given that as a design constraint (we will not be doing this transition
via one-by-one changes to each package), what would you support as a good architectural solution to this transition? Even ruling out that approach,
the design space seems large and flexible; surely there must be some
coherent way of doing this transition that does not require individual
action for each affected package and preserves some of the "be done with
it" design goals of the current usrmerge approach while avoiding the
problems you have pointed out.

I think it's obvious that any such design will require support from dpkg.

You're very understandably upset that people are insisting on a solution
that you feel is clearly incorrect, but I think that's also how the people
who are disagreeing with you are feeling, and as long as that's true on
both sides it's hard to see how we're going to arrive at a solution that
isn't going to make you even more unhappy. In order to break this
deadlock, I think we have to have a design discussion in the shared space
of mutually agreeable solutions and not (on all sides) retreat back to a
single preferred architectural decision and only point out the problems
with any other approach.

--
Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Sun, 2021-08-22 at 09:18:25 +0200, Andreas Metzler wrote:

Afaict we have still no idea on how to move on.

1 I think you agree that there is a significant number of usrmerged Debian
installations out there. It does not really matter whether there are 7% or
40%. They exist and should (keep) working.

By my definition, these have never been working correctly, but
semantics I guess. My wish would be to indeed salvage those systems,
that's why I implemented dpkg-fsys-usrunmess. But if people refuse,
then at some point I'd personally consider them lost, TBH. :/

2 As you have stated there are known issues with dpkg and usrmerged
systems. Some of them are are triggered by moving files from / to /usr.

Starting from here it is hard to see a way forward.

Well, in my mind the first and most immediate action that would be
done, is to stop the bleeding, by:

- reverting the changes in deboostrap in sid, bullseye (and ideally
in buster too),
- reverting the notion that split-/usr is unsupported (which includes
the extremely confusing interpretation about this applying to
sid/testing too), and update documentation such as release-notes,
etc.,
- adding a Conflicts against usrmerge somewhere (dpkg, base-files or
similar) for extra caution,
- (optionally) removing usrmerge from buster, bullseye and sid,

I'd be ready to prepare patches or file reports for any of these.

But then, my expectations regarding Debian have plummeted so…

Is it a realistic option to require something like
-----
dpkg --purge usrmerge
dpkg-fsys-usrunmess
-----
pre dist-upgrade to bookworm to get back all systems to a sane (from
dpkg POV) state and start fresh?

If by requiring you mean documenting the requirement, then I don't
think that would be enough.

But I could see introducing a preinst somewhere (perhaps base-files,
to permit upgrading dpkg itself) which would check for those unsupported systems and emit a big fat message explaining the situation and
instructions to follow to be able to proceed, then abort. Similar in vein
with how other preventions (AFAIR) have been implemented for example on
glibc or udev for environment, multi-arch or kernel requirements.

Sadly, even introducing that into bullseye would not guarantee that the bookworm upgrade would already be in a sane state, as there's no
guarantee people do not skip point releases. But I assume it would cover
most of the installations. So the problem would be greatly minimized.
Only after the bookworm upgrade has been completed this would be
guaranteed.

Is this robust (except for crash as stated in the manpage), or would
you consider it beta-quality?

There's a known problematic interaction with a systemd's emergency mode misbehavior/bug, which I've worked around, plus some other robustness improvements I've prepared that will be included in 1.21.0, and targeted
for 1.20.10. I should probably also add a temporary policy-rc.d to
batch and defer all service actions for after the mass reconfiguration,
which I'll try to code up soonish to further improve robustness.

Otherwise I've taken great care and consideration on trying to make it
as robust as possible, but I guess out of cautiousness and given the
nature of what the script is doing (which is still a hack, although a self-contained one at least), I'd be hard pressed to probably ever
declare it 100% safe. :/ Of course more testing is always helpful!

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Today's update, Debian test can't read my windows partition. I fix it
inside the bios configuration.

El mié, 25 de ago. de 2021 a la(s) 15:27, Aurelien Jarno ( aurelien@aurel32.net) escribió:

On 2021-08-20 23:15, Simon Richter wrote:

I think that one of the release goals should be that any freshly

installed

or upgraded system should have a dpkg database that is consistent with reality, and I'd prioritize that higher than actually finishing the transition, because as long as we can have files vanish from under us, we can't expect that the transition will be reliable for bullseye->bookworm updates.

Basically, we've been lucky so far.

I am not sure we have been so lucky. Problems have been found during the release cycles, some of them got fixed (#953562), some other are still
there (#926699).

--
Aurelien Jarno GPG: 4096R/1DDD8C9B aurelien@aurel32.net http://www.aurel32.net

--
*Grupo Santos Frias S.R.L*
*Danilo Alejandro Santos Frias*
*Cel: (809) 708-7460*

<div dir="ltr">Today&#39;s update, Debian test can&#39;t read my windows partition. I fix it inside the bios configuration.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El mié, 25 de ago. de 2021 a la(s) 15:27, Aurelien Jarno (<a
href="mailto:aurelien@aurel32.net">aurelien@aurel32.net</a>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2021-08-20 23:15, Simon Richter wrote:<br>
&gt; I think that one of the release goals should be that any freshly installed<br>
&gt; or upgraded system should have a dpkg database that is consistent with<br> &gt; reality, and I&#39;d prioritize that higher than actually finishing the<br>
&gt; transition, because as long as we can have files vanish from under us, we<br>
&gt; can&#39;t expect that the transition will be reliable for bullseye-&gt;bookworm<br>
&gt; u

On Aug 26, Guillem Jover <guillem@debian.org> wrote:

By my definition, these have never been working correctly, but
semantics I guess.

It is not semantics. You keep saying that countless Debian and Ubuntu
systems are not working correctly, but since this obviously does not
reflect the experience of the owners of these systems then just about everybody will believe that you are wrong about merged-/usr.

--
ciao,
Marco

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYSc2+wAKCRDLPsM64d7X gSQMAQDJ3sJnebxu6f4gycaZ19PT6cr06eUg7cSLfPFxUK43twD/f1USPdyxYjiC q+oDFvrhHFAuoCqjm1xOFiJzLV7h4go=
=2FmY
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ty80FA9gPJC5YMb5orXwgtLx0LH3t2Qau
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:

By my definition, these have never been working correctly, but
semantics I guess.

It is not semantics. You keep saying that countless Debian and Ubuntu
systems are not working correctly, but since this obviously does not
reflect the experience of the owners of these systems then just about everybody will believe that you are wrong about merged-/usr.

Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.

Simon


--ty80FA9gPJC5YMb5orXwgtLx0LH3t2Qau--

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEtjuqOJSXmNjSiX3Tfr04e7CZCBEFAmEnagsACgkQfr04e7CZ CBE8lgf8CZYDgf2hLJJfiXtaSPeTPUDg32dfEqQ5dvnMFyS3WERDxWA4De7RTX5C hvcHmBKB7MntTR8XNy9JkpiwiEDZ64fWPm5kZsaBd/KB4kTCl6Y4/PSOIxlfPp0S WnB+pFbnefpxbcZ+6tpjs2tGJ6v/nkyrq6l53neVsKQxsAsN0lCSuYGoM7bzYElk lIZ29WOEb4vIYvfdEdRGvnIB/KwhkyE7fZW2gY2f9L4D/NbFYMR3hc9zr44YmD/o 2KXX/cZpNQ7ydrQo0gnaI7eoiF2nqY4GxYcthYcbT3x0VVgaXBZoSDSQ97I61qfx znlrPwZRhzXmisF80fnkZRKNS8UI5w==
=xWl2
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:

By my definition, these have never been working correctly, but
semantics I guess.

It is not semantics. You keep saying that countless Debian and Ubuntu systems are not working correctly, but since this obviously does not reflect the experience of the owners of these systems then just about everybody will believe that you are wrong about merged-/usr.

Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.

    Simon

Precisely - and the correct technical question is, how many bug reports
were opened by users? Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are. Conversely this
doesn't mean the dpkg database bug shouldn't be fixed, so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

--
Kind regards,
Luca Boccassi

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmEneDcACgkQKGv37813 JB6fGw/+PKjb3I8fffDsJqVGW6+QcAw0y3IQhz2+mE09PyoQ3qWIuMxrBWgamnTu WugbA5wU491LyTpJ123iRmYMh1ftaVOtObMqu5MLrg2LoJyHZ05XtesFg4tOq+la +OgMnR+6+MuouY9w6ANsCwhImw1XLI5RJVIuyvShUbqXL7IMxeaHED4NAPF7AETm eKEFy3/M0Z0qn9ASG6+ch0aEXQKivMqYLPrX34ypQI6d5uXiiq25MsYgGJQMeza1 CcVKH7jj7Owr5fQ6eHeETNrEGA9frm1qOMzxVUlhM2cBfetSEZXLT1oitm50853O kcSU1YsTa8O8/tG0cac3sr7E9dPdjTbeMYBMv1aF9uey94eReMqj81O4k2a4z9OD NDziJHnC6TGRV6elJaW9a1huL5wgmWl5dhYLaoZzbitbYPoKvWTsQjhtOXpfN60x cCrBRzqqSij8BvrneFHQJ7UZnRfkqZXnoGIgEPrK35X0Uu19PV20pxbKWadQkdqo YDCrXIEHIM7FoUjus9eu8AEZ6pi+nUnXPYcriA3VjlBiBFqKYWZ/2tkFLlM4eIPB Zp8haA7b2Hi5W+QdAaJSDpKwglHbCYxGMbe2McxUNeQn4zblb4Q1aLqMSnYqmZCx wfewok/N4qxPBqMbCv+RpXBe6eUG4I9+rqi39nNACT/YBrcYk9A=
=wMxa
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Luca Boccassi <bluca@debian.org> writes:

On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:

By my definition, these have never been working correctly, but
semantics I guess.

It is not semantics. You keep saying that countless Debian and Ubuntu
systems are not working correctly, but since this obviously does not
reflect the experience of the owners of these systems then just about
everybody will believe that you are wrong about merged-/usr.

Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.

    Simon

Precisely - and the correct technical question is, how many bug reports
were opened by users?

While I'm sympathetic with the argument that a lack of bug reports
suggests that the theoretical problems (that I hope we all agree, do
exist) seem not to be exhibiting themselves in the wild, it is very far
from proof.

If some random file disappears on your system one day, what happens?

Most likely, nothing, and you never notice.

Possibly, your system starts to misbehave. What do you do then?

If you're a naive user, such as a recent arrival from Windows, then
misbehaviour is something that you've been trained to expect and you
install from scratch. The idea of reporting a bug never enters your
head.

If you're aware that this sort of thing really should not happen, then
what happens next is probably down to how busy you are. If you're
busy, you probably check your SMART stats, and having convinced
yourself that the disks are OK, either restore from backup and check
for what ealse is missing, or use debsums to see the extent of the
damage and reinstall a package or two. Again, since you're only
trying to get back to work, and didn't track down the cause, no bug
report.

The only bug reports you're going to get about this are either going to
be the useless "Something didn't work" bugs, that I doubt would ever get
tied to this cause, or the ones submitted by experienced, diligent, and time-rich bug reporters (which is a rather rare combination).

The fact that we don't see bug reports should surprise no one.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE3/FBWs4yJ/zyBwfW0EujoAEl1cAFAmEnf+0ACgkQ0EujoAEl 1cC8qQ//bnsyFVGzR+/GWHLRIJ+6NTrAiCVd8UvWLUfmlEzK+9EnPRuzCpeZINKc QoI3PYJIkaxH0MaiiNkvlBR5opv5CuYJHWtwpal7IP2ktx1CPyREIP6/p+rqDTDB VSYS9UsJ1X9Me/40Q4bSGG1yhVl+yNw2PCSFFFsZ9Pamen5HcSnam8WyE3EG7hQr f21osvDM2ZSmL3D0zSQL9MvQ4GRRSr1gm3UjGwlcI7FyJvBz40keTsgURKS8CqKe SiE9x25D9AoQv7KnU3VD/o+NhkH18JI7JqTtZ4dOgWvdGMe6WwFJmZfJOpy1cuDM yI6H2nshYRDqp7SpV9qFPvvoW2Il2j9gAxVwa5a4xYFQLMwd5Xf40a+eP5gJSH5i 20jDrGMG8D5VAgnpDDeT9UtVl+NBXPXpmlPmdi19Ro7X3tuyIrBSCPXbmT/7Tlng jrEUszdELuA8EoQSeQTR2bMP9ZMzMbEUlyHRXEmA7muDXuPOQu0Hf3QTTb0ccRu9 fuEwUy6Oc38HSux7ZcgPgdbXyJhFTOcdY2qz9BskxDQbzjbRjUGLGHDLf1gTL8bL oT5gnTlin4DFPzHi6ve9lrh8pHJdxN2daBJC/P/3bSl9X23Qqselx6GoPJmHjEgo Osqc411aqu8Qm/C

On Thu, 2021-08-26 at 13:50 +0200, Philip Hands wrote:

Luca Boccassi <bluca@debian.org> writes:

On Thu, 2021-08-26 at 12:16 +0200, Simon Richter wrote:

Hi,

On 8/26/21 8:38 AM, Marco d'Itri wrote:

By my definition, these have never been working correctly, but semantics I guess.

It is not semantics. You keep saying that countless Debian and Ubuntu systems are not working correctly, but since this obviously does not reflect the experience of the owners of these systems then just about everybody will believe that you are wrong about merged-/usr.

Ideally the question whether a system works correctly would be a technical, not a political one that is based on a majority vote of people who do not look behind the facade.

    Simon

Precisely - and the correct technical question is, how many bug reports were opened by users?

While I'm sympathetic with the argument that a lack of bug reports
suggests that the theoretical problems (that I hope we all agree, do
exist) seem not to be exhibiting themselves in the wild, it is very far
from proof.

If some random file disappears on your system one day, what happens?

Most likely, nothing, and you never notice.

Possibly, your system starts to misbehave. What do you do then?

  If you're a naive user, such as a recent arrival from Windows, then   misbehaviour is something that you've been trained to expect and you   install from scratch. The idea of reporting a bug never enters your   head.

  If you're aware that this sort of thing really should not happen, then   what happens next is probably down to how busy you are. If you're   busy, you probably check your SMART stats, and having convinced   yourself that the disks are OK, either restore from backup and check   for what ealse is missing, or use debsums to see the extent of the   damage and reinstall a package or two. Again, since you're only   trying to get back to work, and didn't track down the cause, no bug   report.

The only bug reports you're going to get about this are either going to
be the useless "Something didn't work" bugs, that I doubt would ever get
tied to this cause, or the ones submitted by experienced, diligent, and time-rich bug reporters (which is a rather rare combination).

The fact that we don't see bug reports should surprise no one.

Cheers, Phil.

Actually in unstable (not in a release) this bug was observed, caught,
properly reported and fixed: https://bugs.debian.org/953562 which would suggest these instances can and have been indeed detected when they
happened - but anyway, that's why the full quote without the cut is:

Precisely - and the correct technical question is, how many bug
reports
were opened by users? Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are. Conversely this
doesn't mean the dpkg database bug shouldn't be fixed, so that the replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken,
revert
everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.

IE, it's fine to think about all the possible ways things can go wrong,
but the solutions need to be proportionate. So proposing to trash
everything and going against the TC decision in absence of any user
report and any knowledge from anybody of an actual breakage in buster/bullseye/19.10/20.04/20.10/21.04 is as far from proportionate as
one can possibly be, just as suggesting to revert multiarch because
'apt install bash:ppc64el' breaks an amd64 machine beyond repair would
be completely unreasonable. Am I suggesting the replaces+move issue
should be ignored? No, hence the second part of the quote that was cut:

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other
day,
namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

--
Kind regards,
Luca Boccassi

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmEnhP8ACgkQKGv37813 JB4KsRAA0YyirpPOiWRHp93asQLsOVYNZzadr9TK5wVqaphQEI3nJ+7Jvujk1b9U blMGp6yGMXOXi0eV2Bxj4upAiqHF5ccPD3hqV+6pPWOv1b5GBFl/JrVhceIgPzqc h4jaNIDoLrFzerjxR9P4xhcZ9acTxfxB9Y1MSMElTCIfGu09JxwmhUaHxgyR/5SO T2quJjLf9RFDg9wLIuDqBKQApziq5axIi1VsVnioaJTMNRhM0af3DuMfMOeD8w8J cSBCzzCIELIxP4vjHoSqOjANuwzS68htWt4bB0vfM4m2HNiDDhgDvVr+MP20fpFv DfvTV+BhKbj/8wHa7GWI92tKbPIY+Zjhs32AHTqfM0cnEysInxvBCqAYb3awsfeI B0RFsqPhDXpJjpwp4BTZMLi7w4mXUbHi/MVcDxHAUNXSVAu0zC3fErLISUdxphFa eamjMP4Y02iyids8JMky5a5+kb9bR6p3Etp7t9hDbKKFG90Ebc2PMemdh5XxNvU2 lQ3TgkKAC89JoEMkCC3/0B6ZCPIK6oB6/5vxSLaXf/5W0dHpNFBI9CNY9V5T9EsI zphryyHGwe+3jiStxhuCx5C3yTYz92sUArYbY84UZDyq/NDju/pciw2Rt5Wx9XtR 3OS0cQs+ftagR0HzS/CQvje+CHJxQwByiLQVMcOafaJjxwQnpUo=
=/1tt
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --fDw2tkVwJM65arxJUZjXfG1OC3AfQLOFB
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

Hi,

On 8/26/21 1:17 PM, Luca Boccassi wrote:

Ideally the question whether a system works correctly would be a
technical, not a political one that is based on a majority vote of
people who do not look behind the facade.

Precisely - and the correct technical question is, how many bug reports
were opened by users?

That is the majority vote by people who do not look behind the facade.

Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are.

This is precisely the debate we would be having if there was a package
that enables armhf as a foreign architecture whenever it is installed:
it would not break systems on its own, but we'd have to be a lot more
careful in other packages about expressing dependencies.

I have armhf enabled on my laptop, and the resolver has suggested
replacing bash:amd64 with bash:armhf a few times -- but I know that my
setup is unusual.

Conversely this
doesn't mean the dpkg database bug shouldn't be fixed,

Keep in mind that this isn't a dpkg bug, because dpkg never created a filesystem layout that was inconsistent with its database; usrmerge did.

What we are talking about is adding a workaround for the usrmerge
package to dpkg, and how to make sure that it works, regardless of
whether the system has already been converted or not, if it has, which
version of usrmerge was used, whether the package is currently installed
or not, whether it will be installed as part of an upgrade and if so,
which state the dpkg package is in at the time it is installed.

For example, a possible scenario could be that the system was converted
with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge
package was deinstalled as a later version declared a conflict that the
user resolved in favour of the other package, usrmerge hasn't been
installed since, then during the upgrade to bookworm, a new version of
dpkg is unpacked (but not configured yet), then a new version of
usrmerge is unpacked, and then dpkg --configure -a is run.

So we need a list of things that need to happen to get back into a
consistent state and a way of sequencing that inside the package
dependency DAG.

so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic
solution.

The TC decision does not specify technical details, which was a grave
mistake because the implementation was so shoddy that I doubt the TC
would have signed off on that, had they been consulted on the details.

Pretty much no one is interested in rolling this back, but a lot of
people, me included, are indifferent and cannot see the benefits,
especially as they consist mainly of things that used to work in the
past, were broken during the systemd rollout and subsequently declared
out of scope when people complained, so all users of these features have already left anyway.

An actual, realistic solution will work for all users, including those
who have now installed or upgraded to buster from DVD, run offline and
will upgrade to bookworm from DVD when it comes out.

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day, namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

No. We couldn't sequence that correctly, we'd have to somehow make installation of the usrmerge package mandatory, and that tool would need
to touch the dpkg database at a point where dpkg is active in the
background.

It makes a lot more sense to have a dpkg-internal tool that can
investigate the differences between the file system and the dpkg
database, resolve conflicts (possibly in an interactive process when
required by a corner case like the one I mentioned earlier -- luckily,
these should be really rare) and then leave a consistent state again
that allows package maintainers to use all dpkg features again after the bookworm release.

That will be a lot of work, and we cannot speed it up anyway because
we're tied to the release cycle here, so I'd rather do this properly
than deploy another paper mâché thing that then turns out to have
another weird bug that will delay the transition to bookworm+2.

Simon


--fDw2tkVwJM65arxJUZjXfG1OC3AfQLOFB--

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEtjuqOJSXmNjSiX3Tfr04e7CZCBEFAmEnjlEACgkQfr04e7CZ CBEShwf/S5qLGyl/BmzIqnv6cGVmsbGYW1fQxqh2HL/qzPC/QnJ5ySEAL2Y+TT5K iuJNiUn+2xPQ/aUzQh5zewgR/uCMYUYkLhgJFIbc7FJWiRxmVoEXZbg4tdwRpdSF P5PjZDHy5n4N/HzNVotUhrEvK2wuuux+bo8JXCrEXpVCwIlnRMGmgcWWl8MIkfHK PVq1M7OZS8M+OhR6kmYhgb//JgM/oOAkKKZi0b3OWpzavP3SleytLQHphWvjQe7E lou+mgZkuF2IVDpWcEoLOUuyv/cIuLJki0x35T/K1Vv73ypDWAOYtyU07t3X+Jm9 fpfYvCsw/86IzcKKWe30J1Gc9gv8OQ==
=fIRh
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 2021-08-26 at 14:51 +0200, Simon Richter wrote:

Hi,

On 8/26/21 1:17 PM, Luca Boccassi wrote:

Ideally the question whether a system works correctly would be a technical, not a political one that is based on a majority vote of
people who do not look behind the facade.

Precisely - and the correct technical question is, how many bug reports were opened by users?

That is the majority vote by people who do not look behind the facade.

A vote is a conscious decision and arbitrary act. Your system breaking
is not.

Strawmen can always be constructed, for example
you can completely brick a system by running 'apt install bash:armhf'
but that doesn't mean multiarch should be scrapped and reverted, it
would be silly to suggest that - and yet here we are.

This is precisely the debate we would be having if there was a package
that enables armhf as a foreign architecture whenever it is installed:
it would not break systems on its own, but we'd have to be a lot more careful in other packages about expressing dependencies.

I have armhf enabled on my laptop, and the resolver has suggested
replacing bash:amd64 with bash:armhf a few times -- but I know that my
setup is unusual.

It's exactly the same. This dpkg bug manifests itself via a carefully
crafted sequence of package upgrades/downgrades with a specific set of
metadata between them. If I uploaded a new version of iproute2 which
enabled armhf and installed bash:armhf the suggestion wouldn't be to
revert the debian multiarch implementation.

Conversely this
doesn't mean the dpkg database bug shouldn't be fixed,

Keep in mind that this isn't a dpkg bug, because dpkg never created a filesystem layout that was inconsistent with its database; usrmerge did.

What we are talking about is adding a workaround for the usrmerge
package to dpkg, and how to make sure that it works, regardless of
whether the system has already been converted or not, if it has, which version of usrmerge was used, whether the package is currently installed
or not, whether it will be installed as part of an upgrade and if so,
which state the dpkg package is in at the time it is installed.

For example, a possible scenario could be that the system was converted
with usrmerge version 9 (which modified dpkg.cfg), then the usrmerge
package was deinstalled as a later version declared a conflict that the
user resolved in favour of the other package, usrmerge hasn't been
installed since, then during the upgrade to bookworm, a new version of
dpkg is unpacked (but not configured yet), then a new version of
usrmerge is unpacked, and then dpkg --configure -a is run.

So we need a list of things that need to happen to get back into a consistent state and a way of sequencing that inside the package
dependency DAG.

This is 100% a dpkg bug. rpm doesn't have it. Other package managers
don't have it. And it's fine - all software has bugs.

so that the
replace+move bug can't happen in the future, but it simply makes
overblown and hyperbolic ideas such as "all systems are broken, revert everything" and "let's cancel the TC decision" appear counter-
productive and deeply unhelpful toward finding an actual, realistic solution.

The TC decision does not specify technical details, which was a grave mistake because the implementation was so shoddy that I doubt the TC
would have signed off on that, had they been consulted on the details.

You are missing a gigantic 'IMHO'. For me the implementation was great
- simple, to the point, matching other distros so that we don't get yet-another-incompatibility for the sake of being 'special', and
working just fine across millions of installations for 2+ years and
counting with hardly a hiccup. Of course it has bugs, because it's
software, or more precisely yet it makes old bugs in other packages
(dpkg) come to light, which happens and it's not the end of the world -
we'll fix them or work around them, like the tens of thousands of other
bugs affecting debian. Also, from what we have heard "unofficially"
here and elsewhere, it appears that the TC has always indeed preferred
the current approach of matching other distros and symlinking the top
dirs, rather than the objectively failed symlinks farm approach.

Pretty much no one is interested in rolling this back, but a lot of
people, me included, are indifferent and cannot see the benefits,
especially as they consist mainly of things that used to work in the
past, were broken during the systemd rollout and subsequently declared
out of scope when people complained, so all users of these features have already left anyway.

An actual, realistic solution will work for all users, including those
who have now installed or upgraded to buster from DVD, run offline and
will upgrade to bookworm from DVD when it comes out.

You say that, and yet just a few hours ago this was posted: https://lists.debian.org/debian-devel/2021/08/msg00490.html
What is that if not a request to revert the TC decision and rollback? I
mean it could hardly be more explicit than that.

Given this and the last few mails, it seems to me at this point the
only possible way forward is what Timo and Ted suggested the other day, namely to have an external tool, possibly part of src:usrmerge, scan
the dpkg database and fix it? Possibly on a trigger?

No. We couldn't sequence that correctly, we'd have to somehow make installation of the usrmerge package mandatory, and that tool would need
to touch the dpkg database at a point where dpkg is active in the background.

It makes a lot more sense to have a dpkg-internal tool that can
investigate the differences between the file system and the dpkg
database, resolve conflicts (possibly in an interactive process when required by a corner case like the one I mentioned earlier -- luckily,
these should be really rare) and then leave a consistent state again
that allows package maintainers to use all dpkg features again after the bookworm release.

It might make more sense from the technical perspective, but it's
glaringly obvious that from the social perspective that it's not going
to happen, so I don't see how it's worth anyone's time to speculate
over it, rather than how to make the realistic alternative work
reasonably well.

--
Kind regards,
Luca Boccassi

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmEnneoACgkQKGv37813 JB5dIg/7BCTr8prZ2bpajbUWMudraSRjy5o0Q2qHtfdR5zoEp+KLxjRDdI1ZgKCl 35nDjZuQ7Tn8PIkQn4MAHwRxlO6MSLR0OkloCbbRBPvfVsmJSMRXan+oBTE4CvTO AwCL+Blj5QkdgevG9gM/ZTjHbSt0SF9i5FCPTMLSNj93KGWOv7td2NHblJTjKNPm e3NW9xZiORlk3KyTvDsxEXEj0GtTOP0y3Elc+UXXLe6Ee+OWW5b+IPLt4m+o40C4 WOitSLria6gagDiM2M2NPy5WA+EL1ifRC1OWkr4J2BFoZLIA1QBweanvZy3439wJ cdhtcKiJM9DnEQyhX5DS5aNXsxofBGkslPREPRaE+gn48uakIWrWE4G+mDcuLU/v YbVSnj+lZ4m/90qjyNmzKwHMpI//PJMQywFm33E1qpBtfjKRO7p89Logh+hE1pyv NYmbO968OavZA22vJSGJPjqLc9UTzylrKyW4XpLrPCSRdia1qkP63uLEeuDhEE+0 mkJdXpOHLVBlR7Nw7NF4c95k/qYbVjXPyz0dSTQR0PdbbAZy9SimHF8vOzSJT4Mx RZAFe7ZHSb9EFGpBN6Kh2mhsG05v/mfkLDkLMlatkolPhz4J2k28dz+RLBS4bgbr 1nLZwrzblTJSWUVOzbIMqyKHVpYDnxCnZrd/rvRp4nqpn/7daAI=
=HyMu
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* Simon Richter <sjr@debian.org> [2021-08-26 14:51]:

It makes a lot more sense to have a dpkg-internal tool that can
investigate the differences between the file system and the dpkg
database, resolve conflicts (possibly in an interactive process when >required by a corner case like the one I mentioned earlier -- luckily,
these should be really rare) and then leave a consistent state again
that allows package maintainers to use all dpkg features again after
the bookworm release.

Agreed. Having yet another tool messing with dpkg "behind its back"
will only exacerbate the problems we're trying to solve.

Also, I wouldn't say it is the *only* possible way to move forward,
but it is *a* possible way. I am also still trying to understand
Guillem position and his objections better.

This proposal addresses one half on Guillem's objections by making
dpkg's point of view consistent with the actual filesystem reality.

However, Guillem also seems to think that dpkg can manage file
symlinks in a real directory better than an directory symlinks in /,
which is why he proposed symlink farms in the first place. Assuming
dpkg implements the proposed canonicalization, is there a scenario
where the aliased dirs break, but would not break with a symlink
farm?

My apologies if this has been answered already. Just point me to the
relevant emails/wiki pages then.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │
⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯

-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEEJvtDgpxjkjCIVtam+C8H+466LVkFAmEnlYoACgkQ+C8H+466 LVnVYAv/RB+zCuPPxzwtAGpW5ETMIApIgeOMOlR4ynIxX5EBNjUjqopJ4+pErAnL zdrXWSZeH14lN7/AlJyvmgU+fJr4N7Uc+Tbh6o7dBEuPhhCvEkt4rnt/+4hh7lRP 5dGxrZThEJyItYuWwderf4gTlZWKnDTgfTgY2yoH0Um5dLElyXZpKjoTC48XZwDC KhbTjp5CGgChvnS1ca18idIxK+rTbYRTc+zM2aVf7v8T6e1pwZ6NLX5E0tuBrqj8 ZnwUb+NZKc7Qqhrp2LHMWcnAbEE8F4fAr+Wn6T12znU/SIuYuct20Es1MgQNkAhF KgtqASJjDOOgyVv4TICn6fbM9rcfqxrjVRoR2qFfJPY

"Timo" == Timo Röhling <roehling@debian.org> writes:

Timo> However, Guillem also seems to think that dpkg can manage file
Timo> symlinks in a real directory better than an directory symlinks
Timo> in /, which is why he proposed symlink farms in the first
Timo> place. Assuming dpkg implements the proposed
Timo> canonicalization, is there a scenario where the aliased dirs
Timo> break, but would not break with a symlink farm?

Quite possibly.
However, I haven't been examining that part of the design space because
the symlink farms approach misses key features that are important to
some usrmerge use cases.

In particular, if you're sharing a /usr between containers or similar,
and moving toward the empty /etc at boot ostree style approach, symlink
farms don't work and base level symlinks do work.

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this instance that I consider the issue moot.

Symlink farms are a different thing, and that's not what we're trying to
get here.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

* Sam Hartman <hartmans@debian.org> [2021-08-26 08:56]:

That may not matter so much for Debian (at least in some people's minds)
but being compatible with the rest of the world has enough value in this >instance that I consider the issue moot.

I agree that this is a very convincing argument. A considerably
weaker supportive argument would also be that these symlinks are
here to stay for years, possibly even decades, which makes four or
five static symlinks much simpler to maintain long-term.

Still, I would prefer to know in advance if there are additional
technical concerns which need to be fixed or worked around.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │
⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯

-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEEJvtDgpxjkjCIVtam+C8H+466LVkFAmEntHcACgkQ+C8H+466 LVldmgwA8YH3j9F9L8UDcC6dKc0e3dAxxs4gB5alyPGBquRnJEcaDS/sO/baX6/f atWKgicf614O+i7XsGB7XehwvT04av1uUmJawp3HSzLO1IyUYn24DEWeOCH58w6q dexDHohX7b92kYOSFI+vYXLrNGaWs3fYMFxawXmvNxbFH+YN2WxlHOGjUFCnhnUH JmLEX7M81fTI1uF/TYlbbgMxRh5bPh7cjsUxc+qS2n1t+UE50iyWn3Uafi9vyWhT TvJ6mkJje6198PQkH41bLfwq3duYZbPZFdTdODMsatE/EBdejZ/brXL+0339+ec+ YKJ7GzYUCU5YfWhW8qbdZXV87wE+9J8JKNJ4WwPcziX

On 2021-08-26 Timo R�hling <roehling@debian.org> wrote:
[...]

However, Guillem also seems to think that dpkg can manage file
symlinks in a real directory better than an directory symlinks in /,
which is why he proposed symlink farms in the first place.

Hello,

Afaiui, the symlink farm would just work from dpkg's point of view, even
with dpkg from oldoldoldoldstable. It can handle symlinks to files just
fine.

usrmerge is conceptually a lot harder because dpkg needs to get to handle
the fact that /bin/foo and /usr/bin/foo conflict.

cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 2021-11-12 at 04:57 +0100, Guillem Jover wrote:

On Sun, 2021-08-22 at 11:21:38 +0100, Luca Boccassi wrote:

On Sat, 2021-08-21 at 22:57 +0200, Guillem Jover wrote:

On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:

The bug is real, nobody doubts that - it has been filed on dpkg 20 years ago.

You keep repeating this, but I have no idea what bug you refer to.

There's #148258 (from 2002), which is conffile related, and not actionable and should probably just be closed.

There's #182747 (from 2003), which while apparently similar is
something else completely. This is about the (IMO) misfeature of supporting a local admin to redirect (not alias) a directory using a local symlink (mainly for space management reasons). For an
explanation
see <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779060#10>.

There's #406715 (from 2007) which is related to the above misfeature.

I am referring to #134758 since it's linked as the root cause from usrmerge's #848622.

Well, that's a bogus block then, because that's obviously not the root
cause. I see I was CCed when that block was set, I guess I missed it.
:/ Fixed it now…

"dpkg-query: Make -S handle unowned symlinks resolving to owned
pathnames" filed in February 2002 - 19 years and a half ago. I refer to that because it's linked as the root cause in the BTS of the relevant
issue with usrmerge we are discussing.

Even if the wishlist from that report got implemented, it would still
not fully solve all the problems, where among them «dpkg-query -S»
is probably the lesser one, which would not work in the other direction anyway (querying a path under /usr/ known to dpkg as being under /).

And then I'm not convinced this should even be implemented at all,
as it would introduce behavior differences between literal pathnames
and patterns, and making them slower (for the first case) or potentially extremely slower (for the second case), in addition to making the queries dependent on the on-disk layout (so unreliable from the packaging PoV,
as it would invent on the spot, pathnames not truly coming from any
package nor otherwise known to dpkg).

This for a misfeature in dpkg (supporting redirecting symlinks) that
allowed the current mess anyway. So I'm inclined to wontfix and close
that one.

Not looking forward to further interactions…
Guillem

Thanks for following up! There are currently 469 open bugs against
src:dpkg, it's of course entirely up to you which ones you choose to
fix and which one you close+wontfix. As far as workarounds go, this one
is really really trivial to deal with, so I personally wouldn't mind at
all.

Thank you for your work!

--
Kind regards,
Luca Boccassi

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmGOSPAACgkQKGv37813 JB75zBAAtakNHXQcXWwhH6GVuUBtK81LReGfHHA2i3omUbyG+xcansM7OBocXK+e IPXzbOz3doJInla3BZ/dmwIehygkoTiQ+3gco3552+l/OgsX7ZeNZbB+9KapzW8T +CLmpBkduFo7LsSg58qwck3FbAyIWQmFNHo+mUsTwxzANRajIIIFfDWLZMUHW0gX N7ywASl2l9VaPJFX+fJfNFDXn+gJWGi/lPzliPul3eDDncVP7TiDGjR4dGnFn2ZK KNvDboPVVn/BoHmoTry+wueQ79AOtpgBOpSF3eSK/h/y+w0Kt2kXNBl/6uFPUMIT 4B4MQ1KWlX5XOotpJX8/AVXUkEaDIUZ85avvtyp2c+lLVwEw5OxWQcVo9okqMDeK ve/RXUV56lC2vVuvRxJjWY+reYjqT6X6WIrEJ15bjbSXx6sEB3rlB38cj8SXrlfm d6L++bhrj9A9Qx9RXnaggUWCwVXRcKlMdwGrXMF9stONvYmpH60eT+ABPwMwarVh 8626l0+XmaPQbYE+RESGILvDhphsyetjs5DmqETPcvHA/Dr6pvnn9BjLH2rCp5L0 pNjG32pU/y4kFI8YlSMma0sgFHy1vGUCCL9zudwAbTV9suVB+TGl64jX1rid92xX xluPOS+ffkyle8PRVBWT4zrWLW6vqAoZJ0No361g5Oix1K0Gdw0=
=YlTv
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)