Hi,

On 25.08.21 21:45, Sam Hartman wrote:

The dpkg maintainer hasn't been happy with the discussions here, and
I think facilitating to a level where Guillem is part of the
consensus is beyond my skill.

The discussion so far has been around the question whether there is
actually a problem and whether it is actually required for the dpkg
database to be consistent with the file system. It is unsurprising that
the dpkg maintainer has an opinion about that.

So I don't actually know how to get to something actionable. I do
believe the chance of breakage if we move around paths inside
packages is high enough that we should block path canonicalization on
a dpkg that can handle that, even if that takes a long time.

We have a few half-baked solution proposals.

Combining the parts from Ted Ts'o (for usrmerged systems) and mine (for
not-yet usrmerged systems) would be the complex and generic approach.

I think I've also seen some ideas along the lines of "have the usrmerge
package patch the dpkg database", which would be simpler.

Would it make sense to start a wiki page?

Simon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Sam Hartman:

"Niels" == Niels Thykier <niels@thykier.net> writes:

Niels> As I understand it, the issue does not depend on whether
Niels> "usrmerge" is run before or after installing the "/lib"
Niels> version of "foo". On that assumption, running "usrmerge" as
Niels> a part of the upgrade and "cleaning up" in bookworm+1 is
Niels> liable to exactly the same risk as before.

I don't think so.
My assumption is that we will eventually produce a dpkg that handles
this situation better.

I can appreciate the allure of assuming that "a fixed dpkg" appears and
solves the entire problem. It would certainly make all the problem go
away *if* it appears.

I am not ready to believe in that dpkg as a solution to the bookworm
release because the dpkg development flow has never been "fast" due to a
strong focus on "a generic solution that gets it right in every case" -
and I expect it to be even slower than usually given how demotivated
Guillem feels and the complexity involved in such a change to dpkg.

And until the "fixed" dpkg materializes, every package shipping
something has to keep files in /lib - even if that is a decade after the bookworm release - or risk breaking if they later need to split the
package in the same release cycle.
To me, that sounds like we will drag the transition on "until the
fixed dpkg comes along" no matter how long it takes. Finishing the
transition is a key element for me in the transition.

[...]
So my hope is that debhelper and maintainers refrain from moving files
from / to /usr prior to being able to depend on an as yet unwritten
dpkg.

For me, this reads as "until = eventually", which I stated I would find unconvincing.

For the record, I did not immediately expect a better answer which is
also why I am waiting with moving forward in case a better answer
materializes "soon".

I think that if debhelper moves systemd units and later a maintainer
moves units between packages, we can run into trouble with today's
dpkg. So we could potentially run into trouble as soon as someone
installs such a package from testing onto a bullseye system.

If debhelper were to hold off until after a fixed dpkg exists and we can guarantee it is available,
I think that we avoid the risk of files disappearing.
So, based on my understanding, I think the risk is worse today than it
would be if we rolled back the debhelper change.

[...]

It's possible I'm missing something .
If so, I'd appreciate help understanding what it is.

On the assumption that a "fixed" dpkg will appear in bookworm, I would
agree with you. However, I do not believe in that assumption/timeline -
to explain why I believe we fundamentally disagree on the priority/solution.

This strongly depends on:

* Who volunteered to be the >> "we" that provide this plan? >> * When is "until" we have a >> defined plan?

So, I think that the discussions here have been converging on things
that would work.
I'm happy to volunteer to assist in trying to find what consensus there
is if that helps.

I appreciate you volunteering to a part of it. My interest in a
detailed transition plan with a fixed end date where we are *done* with
the "clean up" no later than bookworm+1. I do not see that directly in
these discussions - but certainly, a consensus is probably the first
step there. Maybe the consensus will motivate someone to volunteer to
create the plan.

(Side-note my desired timeline implies that a "fixed" dpkg lands in
bookworm.)

[...]

So I don't actually know how to get to something actionable. I do
believe the chance of breakage if we move around paths inside packages
is high enough that we should block path canonicalization on a dpkg that
can handle that, even if that takes a long time.

I would strongly prefer a timely actionable transition plan that does
not involve assuming a "fixed" dpkg will show up and magically fix
everything when the volunteer working dpkg is strongly demotivated by
this transition. In the absence of such a transition plan ...

If the project consensus of this discussion is aligned with the belief
that we should block decentralized volunteer work on the transition, I
will respect the decision.

~Niels

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Simon Richter:

Hi,

On 25.08.21 21:45, Sam Hartman wrote:

The dpkg maintainer hasn't been happy with the discussions here, and
I think facilitating to a level where Guillem is part of the
consensus is beyond my skill.

The discussion so far has been around the question whether there is
actually a problem and whether it is actually required for the dpkg
database to be consistent with the file system. It is unsurprising that
the dpkg maintainer has an opinion about that.

So I don't actually know how to get to something actionable.  I do
believe the chance of breakage if we move around paths inside
packages is high enough that we should block path canonicalization on
a dpkg that can handle that, even if that takes a long time.

We have a few half-baked solution proposals.

Combining the parts from Ted Ts'o (for usrmerged systems) and mine (for not-yet usrmerged systems) would be the complex and generic approach.

I think I've also seen some ideas along the lines of "have the usrmerge package patch the dpkg database", which would be simpler.

Would it make sense to start a wiki page?

   Simon

As I understand it, the "have usrmerge package patch the dpkg database" approach will only work if we ensure that each and every package stop
using / in bookworm+1. Else we are back to the same problem that Sam
listed with package splits (just with the paths inverted).

That is, a solution based on that plan should also involve a plan for
getting each and every package affected by the usrmerge updated in
bookworm+1.

Thanks,
~Niels

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Niels" == Niels Thykier <niels@thykier.net> writes:

Niels> If the project consensus of this discussion is aligned with
Niels> the belief that we should block decentralized volunteer work
Niels> on the transition, I will respect the decision.

I was really frustrated reading that, and I hope that my reading is more
loaded than you meant.
If what you're saying is that you'll respect it if the project consensus
is that individual package maintainers should not move paths around at
this time, then I think that's the key question.

I'll point out that we get a lot of value even if we don't move paths
around in packages.
In particular, we get a uniform environment where we can depend on a
single directory layout.
That removes classes of bugs even if we don't get to update canonical
paths.



What I originally heard in your statement was a consensus that volunteers are not needed,
and I don't think anyone support that.

I think there are several ways in which volunteers are needed:

* Working on figuring out how to trigger the transition.
Ideas included so far are to make usrmerge transitively essential, to
include such code in dpkg, or to detect non usrmerged systems and force
the administrator to do something manually.

* Figure out whether we'll require build chroots to remain
non-usr-merged in bookworm (thus requiring some way to generate such a
system) or whether we'll somehow guarantee that usrmerge transition
happens at the beginning of the upgrade.

* Finish out the discussion Simon Richter and Ted Ts'o are having
exploring changes in dpkg behavior.

* Write patches for dpkg.
I appreciate that getting those patches merged may be a challenge, but
the situation is different with patches in hand.

Yes, a lot more of these volunteer tasks are about talking to people
than normal package maintenance.
And some of them are kind of tricky. I don't even know who makes the
decision about what the upgrade procedure is between releases in Debian.
I mean I can guess to some extent the release team is involved and to
some extent the release notes authors are involved.
And I'd probably talk to the apt maintainers too.
But I'm honestly not sure how we'd evaluate a proposed change to the
upgrade procedure.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Sam Hartman:

"Niels" == Niels Thykier <niels@thykier.net> writes:

Niels> If the project consensus of this discussion is aligned with
Niels> the belief that we should block decentralized volunteer work
Niels> on the transition, I will respect the decision.

I was really frustrated reading that, and I hope that my reading is more loaded than you meant.

Hi Sam,

I am sorry that my email caused you frustration. That part was loaded
with my own frustration over the situation and how we - as a project -
are handling the transition, which I failed to weed out in my
self-review of my outgoing email.

Since I do not know to what extend you took it personally, I want you
know that none of that frustration was aimed at you as an individual.
Once again, if you in any way felt that, then I apologies for that part.

If what you're saying is that you'll respect it if the project consensus
is that individual package maintainers should not move paths around at
this time, then I think that's the key question.

That is what I wanted to say.

I'll point out that we get a lot of value even if we don't move paths
around in packages.
In particular, we get a uniform environment where we can depend on a
single directory layout.
That removes classes of bugs even if we don't get to update canonical
paths.

I believe we both agree on those statements being true (like many of the previous ones). Where we seem to disagree is what should have priority
over other things. I sense that the timeliness of completion is of less importance to you compared to other values and I respect that.


However, I will be considerably more demotivated by what I feel is a never-ending transition than I am motivated by all of the points you
listed above. Which makes it a net-loss for me in years to come even if
it is a net-win for many others if the transition is not resolved in a
timely fashion.

What I originally heard in your statement was a consensus that volunteers are not needed,
and I don't think anyone support that.

My frustration had a different direction than the one what you seemed to
have understood it as, which is why I will not answer your extended
follow up to that part in detail - nor do I intend to expand on my
original words because I doubt it will make any of us happy. Once
again, my sincerest apologies for frustration.



Finally, I will retract myself from this debate for the time being. I do
not feel I have anything additional of constructive value to add to it
nor have enough spoons to invest to become a constructive participant.
I will await the evaluation of the consensus. I kindly ask that you CC
that to debhelper@packges.debian.org (or, at your choosing, report it as
a bug if it involves reverting the change) as I am not sure I will keep
track of this thread any more.

~Niels

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2021-08-25 Niels Thykier <niels@thykier.net> wrote:
[...]

As I understand it, the "have usrmerge package patch the dpkg database" approach will only work if we ensure that each and every package stop
using / in bookworm+1.

Hello,

you missed the second part of the "plan". Editing dpkg database syncs
the db with reality. In addition to that we need:
| if dpkg sees the top-level symlink, canonicalizes
| any files referenced in the packages to /usr/{bin,lib,sbin}/$1, with a
| fallback searching for /{bin,lib,sbin}/$1 in the file system, this
| would solve the problem.

A one-time rewrite does not solve the issue. We cannot guarantee that
dpkg never sees a file with /bin/foo because of local or third party
packages.

cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)