• merged-/usr transition: debconf or not?

    From Marco d'Itri@21:1/5 to All on Mon Nov 8 00:10:01 2021
    I have been asked to remove from the usrmerge package the debconf
    question which asks to confirm the conversion.
    Does anybody REALLY believe that it should not be removed?

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYYhbLQAKCRDLPsM64d7X gVCMAQDZ4M2Vqv67Ls7XWEJSPoJqEEEsZACLTc8U0/mnQmf4iwEAlmcrqMo+HKvN fRhggoamifA+1Z0BUOy6Vyybc8MU3Ak=
    =fL59
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Simon Richter on Mon Nov 8 13:00:02 2021
    On Nov 08, Simon Richter <sjr@debian.org> wrote:

    Right now, it is sufficient to preseed debconf to disallow the usrmerge package messing with the filesystem tree outside dpkg. Managed installations usually have a ready-made method to do so.
    This is not really relevant, since the conversion is mandatory.
    The CTTE stated that the only exception is "Testing and QA systems
    should be able to avoid this transition, but if they do, they cannot be upgraded beyond Debian 12", and my plan is to arrange for this with
    a flag file.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYYkQgQAKCRDLPsM64d7X gUsrAQDeo2vz31GKfFVg33emGVkULR6vxs+LghOrjyLPGq2b3AD8C5a3NF98ZAXx gTc2K1sqhQgpX0FSLXmE/6uHvxwnpAk=
    =7Kvc
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Richter@21:1/5 to Marco d'Itri on Mon Nov 8 12:30:02 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --beJO6MgdJ1d7ghPLGHOgGmBLSEq82Tv1H
    Content-Type: text/plain; charset=windows-1252; format=flowed
    Content-Language: en-US
    Content-Transfer-Encoding: quoted-printable

    Hi,

    On 11/8/21 12:03 AM, Marco d'Itri wrote:

    I have been asked to remove from the usrmerge package the debconf
    question which asks to confirm the conversion.
    Does anybody REALLY believe that it should not be removed?

    While it may be more convenient for home users to not be bothered with questions they don't understand, I believe predictable behaviour would
    be preferable in a professional environment.

    Right now, it is sufficient to preseed debconf to disallow the usrmerge package messing with the filesystem tree outside dpkg. Managed
    installations usually have a ready-made method to do so.

    With that change, the only way to do so would be to blacklist usrmerge
    via APT preferences, and while that is certainly possible for management frameworks to handle, the resulting ripple effects from packages
    becoming uninstallable through this might cause additional problems.

    Simon


    --beJO6MgdJ1d7ghPLGHOgGmBLSEq82Tv1H--

    -----BEGIN PGP SIGNATURE-----

    iQEzBAEBCgAdFiEEtjuqOJSXmNjSiX3Tfr04e7CZCBEFAmGJCOgACgkQfr04e7CZ CBHuEQgAhgfykMikTFLToTq4c/QZeAfPxCdvuxgJHOeq62kmuCiTiuRiukQ85XxJ qV1M+v1V9X+B6zA2zOIgni9T71FmIfQxZ55Xh0AFb3ocI+sClHxaeVZ4yC1QX/cA dvFDQHV0G2OgHistpUw2oSvkBDX0HQXQwiRwo2qp3FaSuMuFWI8EI4BG/T9THSBK 2Lkp+YcLtj8vWNul6W4qq5lrKyZiCe+djgmUQbSuRsnHx4uzpIAstqlq1w8v8/r9 3ayYUfpklNp1NmBaazUlcpE7X2p+1JGSxUCrA6p/DNoMkOgwBuEylxzl6WQMKaVl PrYalhN3W9Ql3JMj8fgvLkboL5IEXA==
    =/Np9
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Richter@21:1/5 to Marco d'Itri on Mon Nov 8 15:10:01 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XLJy77itKuXODqg13ZCwn4egYvccAk593
    Content-Type: text/plain; charset=windows-1252; format=flowed
    Content-Language: en-US
    Content-Transfer-Encoding: quoted-printable

    Hi,

    On 11/8/21 12:56 PM, Marco d'Itri wrote:

    Right now, it is sufficient to preseed debconf to disallow the usrmerge
    package messing with the filesystem tree outside dpkg. Managed installations >> usually have a ready-made method to do so.

    This is not really relevant, since the conversion is mandatory.

    Yes, but the conversion needs to be performed by dpkg, because the
    usrmerge package is not in a position to complete the transition, it
    will always leave a filesystem that is inconsistent with the dpkg
    database, and it can not fix up the database because there is neither a
    time during package installation where the dpkg database is unlocked,
    nor is it desirable that another package besides dpkg changes the dpkg database, nor is it safe to do so (in fact, that would almost certainly
    cause files to be lost).

    Ergo, the usrmerge package needs to be phased out before the bookworm
    release, it only creates complications for finishing the transition, and
    while I find that useful with my Devuan user hat on (as it means that
    Debian packages will continue to ship essential files in /bin until at
    least bookworm+1, and we don't need to fork these), my Debian hat says
    that we should complete this with bookworm.

    Last but not least, it is disingenuous to remove control from users (especially those who have previously objected) and pretend they have consented. I would, at the very least, expect the package to honor
    previous debconf answers (and these would remain fixed if the question
    were removed).

    Simon


    --XLJy77itKuXODqg13ZCwn4egYvccAk593--

    -----BEGIN PGP SIGNATURE-----

    iQEzBAEBCgAdFiEEtjuqOJSXmNjSiX3Tfr04e7CZCBEFAmGJLWIACgkQfr04e7CZ CBFXHggAhFZu0OAzNhJque8JsUG6ql8tbkhGGxS3z6xhvk0zYbaLbDLKVf5KPiEb 6Usj9oxDJkSpL40YrhiIR3B0sF3lq7WqFCkUjKmpIT5U0e0DXBun8OHtOG97l2Ou NdIQw160WCHEfvgw7ZggO8Pc3evzVuy3EkBsdOxQU0CpT+VaD3swhVD47w0vWbKM xGvbFCGAk0V68cXXMUxAXHPYUyudRs2BlLKpgog1OYDeendqIHtXx2ExVEcRL1ce QnDKW0UnwgRgp93edLZFcdcJ02kSd/88iE+MpYBYTd9Eds4oDLhhFanCP9ksClEU +1Ss3QYb/uzRcwS2n1Rq8HNgk8Oqlw==
    =LxDq
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Simon Richter on Mon Nov 8 16:00:03 2021
    On Nov 08, Simon Richter <sjr@debian.org> wrote:

    Yes, but the conversion needs to be performed by dpkg, because the usrmerge
    Please kindly stop beating this long-dead horse.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYYk6MAAKCRDLPsM64d7X gcRgAQCr/55YhEGj8ZBqtCGNZGsT0ZYR7eIXyss8NW1XqwMcfgD/dUerhF/KjJnw fcby3jN+TY3SG6o5E9SNvNinKD6qegQ=
    =scE+
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Kalnischkies@21:1/5 to Marco d'Itri on Tue Nov 9 15:30:02 2021
    On Mon, Nov 08, 2021 at 12:56:49PM +0100, Marco d'Itri wrote:
    On Nov 08, Simon Richter <sjr@debian.org> wrote:
    Right now, it is sufficient to preseed debconf to disallow the usrmerge package messing with the filesystem tree outside dpkg. Managed installations
    usually have a ready-made method to do so.
    This is not really relevant, since the conversion is mandatory.
    The CTTE stated that the only exception is "Testing and QA systems
    should be able to avoid this transition, but if they do, they cannot be upgraded beyond Debian 12", and my plan is to arrange for this with
    a flag file.

    As I see it the CTTE decision effectively allows the transition to be
    deferred until the moment you want to upgrade to 13. Ideally the
    transition is performed already in the 11→12 upgrade automatically for
    you, but you could prevent that automatism and do it manually someday
    while you have 12 installed (as no 12 package can depend on merged /usr
    as it would not be installable on upgrade from 11 and/or executable on buildds/testing/qa systems at the least).


    So, wouldn't it make sense to go with an (extreme) low priority debconf question defaulting to 'yes, convert now' which [I think] non-experts
    aren't bothered with and users/systems wanting to opt-out for the moment
    (like buildds) have a standard way of preseeding rather than inventing a homegrown flag-file and associated machinery?

    As a bonus, if I had previously decided to forgo the automatic
    transition for whatever reason (lets say to test build packages on that
    box) I also have a standard way of triggering the conversion by calling dpkg-reconfigure on usrmerge at leisure before the 13 upgrade.


    Best regards

    David Kalnischkies

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmGKg4MACgkQMRvlz3HQ eIMwtQ/+MpymfpdlXO3eaTYtAVVbdF9wU9W9The5jAdkTIYp3nLh6edG2Z5JeymV NUUDt0jugFmFYzYAVV2Hw1ATm07juxepU+pqaCY1+tGCnVeUHPWOUixVQvD4aXGM lDCjPvZ+TAzK3yUBTP2GhIl17T7Y8Np7VX2lqRKjKWOy5eaYDVBFRQLmcrc42h0f Rg2rxaMEZIAGxOU7LwzOSegiFgs9b67a1xgvauxLQatHKfe54AW44ORziMQ072L4 sJG9bvk04gfew9sS1o5/V350WVbcHoLX/QA8X4KurnUA1x5101Vx7JEzydhNMiyk yu+/OgBdrtysa+HDkTwbsquUL6s8gdR8rLJjmGYIwIElIdrau7V3JHVGp8YQjlMP UYpF+W+X4spPDNhFNFu/wLxMSIunID6FSit9k2Mjc2AAfSUpQKiKEHphQi/pQNjs YFEIrnhuz2dR87K3y3Y1CjTmqWCQtMKr8So1VENLvqraGjkrJhtv6QUfE6KT2dNU LtGgOxVuOdvdfGRWNHQjCdodtwSYfEVZZtmAm/2XoDl2R92NjV4Rrpwz5X8VYAvE FHpxE6+eBBFiLIyqy69lF5yrsNtT9/sB8e9Ty7riEN81q30Uj92h6HktOmxqYYn7 awffyreK+l3rD3XsAR8ihJn9ML5c28eu4invUAA/q+kcsnPqrSs=
    =Z+Ta
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to David Kalnischkies on Tue Nov 9 16:30:01 2021
    (Speaking only for myself, not the TC as a whole - but I wrote the first
    draft of what became the TC resolution we're discussing.)

    On Tue, 09 Nov 2021 at 15:19:53 +0100, David Kalnischkies wrote:
    On Mon, Nov 08, 2021 at 12:56:49PM +0100, Marco d'Itri wrote:
    On Nov 08, Simon Richter <sjr@debian.org> wrote:
    Right now, it is sufficient to preseed debconf to disallow the usrmerge package messing with the filesystem tree outside dpkg. Managed installations
    usually have a ready-made method to do so.
    This is not really relevant, since the conversion is mandatory.
    The CTTE stated that the only exception is "Testing and QA systems
    should be able to avoid this transition, but if they do, they cannot be upgraded beyond Debian 12", and my plan is to arrange for this with
    a flag file.

    For reference, the wording in the TC resolution is:

    We anticipate that during the development cycle that will lead to
    Debian 12, it is likely to be useful for testing and QA infrastructure
    (such as autopkgtest, piuparts and/or reproducible-builds) to be able
    to produce an installation of Debian testing/unstable that is not
    merged-/usr, in order to be able to verify that packages targeted
    for inclusion in Debian 12 can still be installed and/or built
    successfully in a non-merged-/usr environment during partial upgrades.

    As a result, we recommend that if there is an automatic migration from
    non-merged-/usr to merged-/usr, it should be possible to prevent
    that migration. **However, systems where that migration has been
    prevented are not required to be fully-supported**, and in particular,
    upgrading them to Debian 13 or to the state of testing/unstable
    during development of Debian 13 should be considered unsupported.

    (emphasis added in this mail, not present in the TC resolution)

    As Marco implies, it was not my intention to have this as a
    general-purpose way for change-averse sysadmins to avoid or defer the
    automatic transition. I had intended this to be specifically there as a
    way to do the testing and QA that will ensure that the transition can go
    as smoothly as possible (for example, reproducible-builds does one build
    with merged-/usr and one with unmerged /usr, and this should continue, otherwise we'll lose the ability to detect packages that build differently
    in those two scenarios, which in practice is strongly correlated with
    packages that won't work on non-merged-/usr systems if they have been
    built on merged-/usr).

    I can see that if people are developing dpkg enhancements that allow it to reconcile its view of the filesystem with merged-/usr by having a record
    of the directories that "should be" merged (perhaps a --paths-aliased
    analogous to --path-include?), then they would also benefit from the
    ability to construct up-to-date systems that have and haven't undergone
    this transition, so they can compare the two.

    As I see it the CTTE decision effectively allows the transition to be deferred until the moment you want to upgrade to 13.

    I think you mean: until the moment you want to upgrade to testing after
    Debian 12 release day. That's not Debian 13 *yet*, although you could
    think of it as some sort of Debian 13~alpha, and the TC resolution allows packages in testing/unstable to start requiring merged-/usr immediately
    after Debian 12 is released.

    I tried to make sure the TC resolution distinguished carefully between
    Debian stable releases, and the states of testing/unstable that will
    exist at particular points in the stable release cycle.

    So, wouldn't it make sense to go with an (extreme) low priority debconf question defaulting to 'yes, convert now' which [I think] non-experts
    aren't bothered with and users/systems wanting to opt-out for the moment (like buildds) have a standard way of preseeding rather than inventing a homegrown flag-file and associated machinery?

    Speaking only for myself and not for the TC, I think a debconf question
    would be OK as an implementation of this, but the debconf question should indicate that the result of opting out is an unsupported system.

    I would personally see this as a bit like using dpkg --path-exclude, or downgrading packages: it's allowed, but it might break things immediately,
    or it might have weird side-effects later on (even after the configuration change has been reverted).

    As a bonus, if I had previously decided to forgo the automatic
    transition for whatever reason (lets say to test build packages on that
    box) I also have a standard way of triggering the conversion by calling dpkg-reconfigure on usrmerge at leisure before the 13 upgrade.

    I had intended this to be for the class of systems that you would expect
    to discard and re-bootstrap rather than upgrading (chroots, lxc/Docker containers, virtual machines, etc. used for autopkgtest, piuparts, reproducible-builds, etc.), where a way to undo the opt-out isn't really necessary because the system is treated as disposable.

    However, you're right that if a way to undo the opt-out is desired,
    using debconf provides an implementation "for free".

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Kalnischkies@21:1/5 to Simon McVittie on Tue Nov 9 19:10:01 2021
    On Tue, Nov 09, 2021 at 03:21:25PM +0000, Simon McVittie wrote:
    As I see it the CTTE decision effectively allows the transition to be deferred until the moment you want to upgrade to 13.

    I think you mean: until the moment you want to upgrade to testing after Debian 12 release day. That's not Debian 13 *yet*, although you could

    Yes, I meant that indeed… should have used codenames after all.


    So, wouldn't it make sense to go with an (extreme) low priority debconf question defaulting to 'yes, convert now' which [I think] non-experts aren't bothered with and users/systems wanting to opt-out for the moment (like buildds) have a standard way of preseeding rather than inventing a homegrown flag-file and associated machinery?

    Speaking only for myself and not for the TC, I think a debconf question
    would be OK as an implementation of this, but the debconf question should indicate that the result of opting out is an unsupported system.

    Sure.

    (Minus that for 12 it is technically still supported as long as it
    remains 12, but those who have to know will know that and everyone else
    is better of following the default anyhow)


    I had intended this to be for the class of systems that you would expect
    to discard and re-bootstrap rather than upgrading (chroots, lxc/Docker containers, virtual machines, etc. used for autopkgtest, piuparts, reproducible-builds, etc.), where a way to undo the opt-out isn't really necessary because the system is treated as disposable.

    That is likely what happens to most of them, but as we support running
    the merge somewhere between a few years ago and first unpack of a trixie package anyhow I don't see the harm of having an official opt-out of the opt-out as long as it happens in time.

    (Perhaps it comes with the job as apt maintainer, but I don't "discard
    and redo" systems or even chroots… upgrades until hardware failure…
    my current build chroots are from 2013. So I can totally see me opt-out
    first and later in… although probably more with apt_preferences)


    Best regards

    David Kalnischkies

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmGKt2cACgkQMRvlz3HQ eIPcDw//Sw3lG0W/gCFATcWZkXQTXstX6G1Ycv2CPairm76WuFinuXSFhaZCj5YK 7LiedOWUjTHrnzqyTqXIy/XT38c+jfzKO4JqmE7O+C/mlFQvtuLHEDwfHsQhpczA 167mtwrFVaCZxoLP6zf4im/DvFyqNHvUNLrRNBa9eUUAwOrNj9ie1l5/3/0jGGcH 9Djveno3nIkZQcbJRK/hg3YdffHiuJv20slIGQItyR4f/KZS1JbzPReJXcanALee fSudLUEYsBaZnfFR8wLaDEyzDntsUwvcBSx8LRtsHz99enH/oCJsoiuYKaFy0xfF mJarxzJ3VjGUy5fQ5eGV6LXFtbLMbuEyN7fAhIQRGHCsJ1m+T+Fav1FfEQtuL7wX w6SA0CCOryVC4Lri723x02NnKeZ/Mr3IyQkGVjPMNOeF0u3isKqjiH06BbSSbdfy RK7I63rgzsQ/JLGOeDZCyPCIIzzL9MF2RVIfgDJYx/t5ioZv1iy2lF0UOjMwE2B7 tr/ItfuK1OgTdM4Y5udO8U/VGAaFwtTeryzxcKCCFFvsUFJEkPeGmzuR3BpUObUy XgU8hLTyf0q/P9g3AE3v91xeBIFcQ/G/DF5fI0U4LUQ3iSoNwm4Conj52OH/0a+B 58r7WjI/5FQlNPGofKPDj47O/Xa886VK1A54x99rZaVS3lAAouo=
    =hzvs
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Biebl@21:1/5 to All on Tue Nov 9 20:30:02 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------jV3s80rhH0x4U0FNyIeS0e5Q
    Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

    T24gMDkuMTEuMjEgMTk6MDEsIERhdmlkIEthbG5pc2Noa2llcyB3cm90ZToNCj4gKE1pbnVz IHRoYXQgZm9yIDEyIGl0IGlzIHRlY2huaWNhbGx5IHN0aWxsIHN1cHBvcnRlZCBhcyBsb25n IGFzIGl0DQo+ICAgcmVtYWlucyAxMiwgYnV0IHRob3NlIHdobyBoYXZlIHRvIGtub3cgd2ls bCBrbm93IHRoYXQgYW5kIGV2ZXJ5b25lIGVsc2UNCj4gICBpcyBiZXR0ZXIgb2YgZm9sbG93 aW5nIHRoZSBkZWZhdWx0IGFueWhvdykNCg0KSSdtIHdvcnJpZWQgdGhhdCBieSBzYXlpbmcg dGhhdCB1bm1lcmdlZCBpcyBzdGlsbCBzdXBwb3J0ZWQgaW4gMTIsIHdlIA0Kb3BlbiBhIGNh biBvZiB3b3JtcyBhbmQganVzdCBwdW50IHRoaXMgZG93biB0byB5ZXQgYW5vdGhlciByZWxl YXNlIGN5Y2xlLg0KRS5nLiwgd2hhdCBleGFjdGx5IGRvZXMgdGhpcyBtZWFuIGZvciBiYWNr cG9ydHM/DQpJZiBJIHByb3ZpZGUgYSBiYWNrcG9ydCBmcm9tIGJvb2t3b3JtKzEsIEkgZG9u J3QgdGhpbmsgaXQncyBmZWFzaWJsZSANCihhbmQgaW50ZW5kZWQpIHRvIHJvbGwgYmFjayBh bnkgY2hhbmdlcyByZWxhdGVkIHRvIHVzcm1lcmdlLCBzbyBJIHRoaW5rIA0KYmFja3BvcnRz IHNob3VsZCBiZSBhYmxlIHRvIGFzc3VtZSBhIGJvb2t3b3JtIHN5c3RlbSBpcyB1c3JtZXJn ZWQ/DQoNClNpbW9uLCBjb3VsZCB5b3UgY2xhcmlmeSB5b3VyIHRob3VnaHRzIHJlZ2FyZGlu ZyB0aGlzIGFzcGVjdD8NCg0K

    --------------jV3s80rhH0x4U0FNyIeS0e5Q--

    -----BEGIN PGP SIGNATURE-----

    wsF5BAABCAAjFiEECbOsLssWnJBDRcxUauHfDWCPItwFAmGKycwFAwAAAAAACgkQauHfDWCPItxr kg/9HtY1I9VEOFzDEsc6Cs4Wlud3omGGLd914HDsIkO/0YHZVC2bUdJoc75u3ca6R1NT7aiKr1fH KSS7h91MBcZJUdRtGlzrUYuX3CnwnRMUyKpvLxnvLJNiCGyNrChg4bZ/LCuf7KQMeYw0aaW5klpB DVALOPh14GR8uPxv5LlqxxfhEaQ6aIfc55dIFwuBzlokdMSgWgkDho3/tAxPj3w0RPmnhEVZ5WFu 29iQQwA13RMaBwDotC4p2xLNi49oLgdPCxr7wAuf/RLfI2i1Hg592bQHnnu0WxwoYNbdrXFtF1E6 mRj5MvGLyfkg/m16w/YeM87TDeAi94Tx14CPz+mVVd6gdXJh3dP9/Y8UjfoYEtlo3zwbCvMgcmjw l7iNYnXSXeuRcZZHxxPkahmhUq1nt9VRMtNIMGRJn7LIQnkh2GGnUp2xzZf1YLeG5UBCarLz44RF RcpicCW5yeYS3nXCnVRMAWSKDNknXsD9TzWoacjRzNxq4BVMDSTxorbPDx3B2OSK9lu6CwjCQz4H AcA+9Z+QLfBqObawbLjAz79eVurVJAQyskZb0TuMbbtNBHIspVVF+CU0G2CrXilBQ+lQvz5LWs8Y 6JEH72RlJE0OdDpORz6KM5LYuck32UTgIstf1hdw9yr+QTCRKJSGJb9RV0N1W+z31btSOBQLTdqh pVg=
    =I51Z
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Bastian Blank@21:1/5 to Michael Biebl on Tue Nov 9 20:50:01 2021
    On Tue, Nov 09, 2021 at 08:19:40PM +0100, Michael Biebl wrote:
    I'm worried that by saying that unmerged is still supported in 12, we open a can of worms and just punt this down to yet another release cycle.

    No, unmerged will not be supported in 12. Having the ability to create something does not make it supported.

    E.g., what exactly does this mean for backports?

    Stuff from backports is post-release, so requires a merged system.

    Bastian

    --
    Star Trek Lives!

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Bastian Blank on Tue Nov 9 21:30:01 2021
    On Tue, 09 Nov 2021 at 20:27:24 +0100, Bastian Blank wrote:
    On Tue, Nov 09, 2021 at 08:19:40PM +0100, Michael Biebl wrote:
    I'm worried that by saying that unmerged is still supported in 12, we open a
    can of worms and just punt this down to yet another release cycle.

    No, unmerged will not be supported in 12. Having the ability to create something does not make it supported.

    That was my intention, yes. My intention when drafting the TC resolution
    was that unmerged-/usr Debian 12 'bookworm' systems should be technically possible but unsupported - similar to how downgrading packages is possible
    but unsupported, and configuring dpkg --path-exclude is possible but unsupported. If you contrive to create an unmerged bookworm system
    and then try to upgrade it, that is likely to fail, and that's on you
    to resolve.

    E.g., what exactly does this mean for backports?

    Stuff from backports is post-release, so requires a merged system.

    Yes, I think this is right. IMO you can't validly enable bookworm-backports without first upgrading to bookworm, by which time the automatic transition mechanism called for by the TC resolution should have taken effect.

    However, I think bookworm-updates and bookworm-security still need
    to cope with unmerged-/usr systems to the same extent that bookworm
    itself does, because it's valid to upgrade from bullseye to bookworm + bookworm-updates + bookworm-security in a single step.

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to David Kalnischkies on Tue Nov 9 21:50:01 2021
    On Tue, 09 Nov 2021 at 19:01:18 +0100, David Kalnischkies wrote:
    On Tue, Nov 09, 2021 at 03:21:25PM +0000, Simon McVittie wrote:
    As I see it the CTTE decision effectively allows the transition to be deferred until the moment you want to upgrade to 13.

    I think you mean: until the moment you want to upgrade to testing after Debian 12 release day. That's not Debian 13 *yet*

    Yes, I meant that indeed… should have used codenames after all.

    To be clear about this: when drafting the TC resolution, I did not intend
    for this to "allow the transition to be deferred" and still leave you with
    a supported system.

    All I was aiming to do there was to make it possible to create an
    unsupported, not-quite-Debian system that our QA infra can compare with
    the supported Debian system that closely resembles it, so that for example
    if a package builds differently on unmerged-/usr and merged-/usr systems,
    we can flag it as "this is likely to break upgrades" and look more closely.

    Speaking only for myself and not for the TC, I think a debconf question would be OK as an implementation of this, but the debconf question should indicate that the result of opting out is an unsupported system.

    Sure.

    (Minus that for 12 it is technically still supported as long as it
    remains 12

    No, it doesn't have to be supported, and the TC resolution explicitly
    said that it doesn't have to be supported.

    What *does* need to be supported is the upgrade path from 11 to 12,
    or from current testing (11-and-a-bit) to 12, with any ordering of apt transactions that doesn't violate the packages' dependency conditions -
    and the TC's reasoning was that the simplest, most conservative, most
    robust way to make sure that continues to work was to mandate that all
    Debian 12 packages, individually, are installable onto unmerged-/usr
    Debian 11 (assuming that "installing a package" implies installing its dependencies, in any order that apt/dpkg consider to be valid and not
    breaking any dependency relationships).

    (Perhaps it comes with the job as apt maintainer, but I don't "discard
    and redo" systems or even chroots… upgrades until hardware failure…
    my current build chroots are from 2013. So I can totally see me opt-out
    first and later in… although probably more with apt_preferences)

    For full systems that are managed as full systems ("pets" in the cattle
    vs. pets terminology), sure, do that; the Debian installation I'm typing
    this into has been copied from several older machines. However, deferring
    or avoiding the merged-/usr transition on these systems is not intended
    to be something that is considered valid for bookworm.

    For installations that are meant to be predictable, like build chroots
    and the chroots/containers used for automated QA systems, keeping a 2013 installation and successively upgrading it comes with a significant risk
    of ending up with an installation that nobody else (including you, if
    the underlying hardware fails!) can replicate without starting from a 1:1 backup of that installation. I would not recommend this approach:
    I think installations that are meant to be predictable should be "cattle".

    I had intended that preventing the /usr merge (with debconf preseeding or Marco's flag file or whatever) would be something that people should only
    be doing in those "cattle": the steps to reproduce those systems would
    include applying the debconf preseeding or creating the flag file.

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Kalnischkies@21:1/5 to Simon McVittie on Tue Nov 9 23:20:02 2021
    On Tue, Nov 09, 2021 at 08:44:52PM +0000, Simon McVittie wrote:
    On Tue, 09 Nov 2021 at 19:01:18 +0100, David Kalnischkies wrote:
    On Tue, Nov 09, 2021 at 03:21:25PM +0000, Simon McVittie wrote:
    (Minus that for 12 it is technically still supported as long as it
    remains 12

    No, it doesn't have to be supported, and the TC resolution explicitly
    said that it doesn't have to be supported.

    What *does* need to be supported is the upgrade path from 11 to 12,
    or from current testing (11-and-a-bit) to 12, with any ordering of apt transactions that doesn't violate the packages' dependency conditions -
    and the TC's reasoning was that the simplest, most conservative, most
    robust way to make sure that continues to work was to mandate that all
    Debian 12 packages, individually, are installable onto unmerged-/usr
    Debian 11 (assuming that "installing a package" implies installing its dependencies, in any order that apt/dpkg consider to be valid and not breaking any dependency relationships).

    Yes, any Debian 12.x package (even the very last security fix build for
    it) needs to be installable on a Debian 11.y system as it could be part
    of the upgrade from 11 to 12 and as it has no idea if its the first or
    last package in that upgrade (within reason) it has to work on 11 as
    well as on very-very-close-to-12.

    As such, if you promise 11.x to 12.y upgrades I would expect 12.x to
    12.y to work just as well as 12.x is very-very-close-to-12(.y).

    If you say 12.x to 12.y isn't supported on unmerged it means effectively
    that all "cattle" have to be constantly recreated as you can't have
    a single package be considered an 'upgrade', they all need to be 'new
    install' while e.g. installing build dependencies (as ironically a fully upgraded 12 system is indistinguishable from an upgrade-in-progress-from-
    11 system which just happens to install a bunch of new packages in the
    end).

    Its also quite a disaster for all systems already technically bookworm
    like testing and sid as any upgrade, including to the release 12.0, will
    be unsupported in your logic. Unsupported machines (aka our buildds)
    building supported packages seems sad and I thought we had talked about
    this before.


    (Perhaps it comes with the job as apt maintainer, but I don't "discard
    and redo" systems or even chroots… upgrades until hardware failure…
    my current build chroots are from 2013. So I can totally see me opt-out
    first and later in… although probably more with apt_preferences)

    For full systems that are managed as full systems ("pets" in the cattle
    vs. pets terminology), sure, do that; the Debian installation I'm typing
    this into has been copied from several older machines. However, deferring
    or avoiding the merged-/usr transition on these systems is not intended
    to be something that is considered valid for bookworm.

    As the transition hasn't started everyone not already merged is currently deferring it. That is true for those who upgrade daily as well as for
    those people who seemingly only upgrade their sid systems once in a blue
    moon. So, at which point have all those systems stopped deferring?

    I would say that the first time you can say with absolute certainty that
    a given system is no longer deferring the transition is the moment an
    unpack of a trixie pkg is attempted as skipping releases is not
    supported. All unpacks before that could happened on an unmerged
    system as that system might very well be in the process of upgrading
    from 11 to 12 at the moment.

    (and btw, what I meant with me opting out for a while was delaying the
    upgrade of my sid "beasts" to a more exciting problem space than the
    first possible moment as that wouldn't be much of a test for apt, /usr
    merge and all those other packages installed around here. If I am
    asking for an upgrade path its only fair to not take the easiest road
    of transitioning to merged before anything could implicitly require
    it and hence fail for less lucky people not equipped to deal with it.
    My "eldritch horrors" are fine and behave, thanks for asking. 😉)


    Best regards

    David Kalnischkies

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmGK840ACgkQMRvlz3HQ eIOlbw/+JtM60lIIo9QIXIERwmymrhyYce5UWJmdszpXkGegE/WHo/Mhz5n57rnT KI7QuKf9rMQXIMCTvV7ilHbB78t7/crmCExBCrNiTinlHEM1YXY0+rpn4WCJdQ5s RgRXW3vZlVRUX93osjTFwLLSb4SET/kCyexL0H88yn02KrzF1eqvQJPSSpjuFQB4 03WnqGX0dtK2OlUzMgyrBU+P8mxbRj3J338nrnJdTxALvGd9Ffuz2tv1eP11AamW qgsakbgQYyv3h45Yw0EeyqUobG3/qAkOdZP7Yqk9KT69xPaz16qyTK5d6ZuiF0sp mMt6+OFn94NPm+dwLtbSaENKpUVIAz/KIb+weLvS69IK49BJPrLDK3l1aW8y1gCu UxbGTnWXTLAUgky/dZVR06U1wGtWTIVlJiyg8ZkWc2ndNaDNq908Ze+u9nk6o+NT ghGbvFX/Pt8kBLutF6XvHmkbnMgoQWVC4nTdbZl2WEkEZHteCiWO2wjBGn4NKGcq GYgnrJHgh/PjYy8cWXS5tnXpvZTCT9f5JU+exHDXKXygP1Sy09Lcdrx8SbL5oKGu z5kqaNKQBeZp3kR8X2X644kYP+eFbitJpn5s40WyZBP94J7ksb/mGWJogLpVs6dA g68uZBnsu8XHaZb/gTbkGBgonjujwcrivn+Elj01ZsAqDHwlq0s=
    =UNek
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Wed Nov 10 19:10:02 2021
    "Marco" == Marco d'Itri <md@Linux.IT> writes:

    Marco> On Nov 08, Simon Richter <sjr@debian.org> wrote:
    >> Yes, but the conversion needs to be performed by dpkg, because
    >> the usrmerge
    Marco> Please kindly stop beating this long-dead horse.

    I'm sorry, but I think the only way in which that horse is dead is that
    no one has proposed patches to dpkg.

    Simon has raised a valid issue with doing the transition in something
    other than dpkg.
    I don't quite agree with everything he says--for example I think it
    might be possible for a package to cooperate with dpkg--but I do agree
    with the base issue that eventually dpkg needs to learn about aliasing,
    and that likely means that dpkg needs to be involved with usrmerge
    transition either during the transition or after the fact.
    (And if it is after, it's going to be more clunky).


    Moreover, the TC has effectively agreed with the issue by talking about
    how we don't want to move files in packages.

    I don't think you need to block waiting for dpkg work, and I think it's
    fine for usrmerge to become transitively essential and to go transition systems.
    But I don't think it is fair to say Simon's horse is dead or to try and
    shut down discussion about how we're going to eventually solve the dpkg aliasing problem.
    --Sam

    -----BEGIN PGP SIGNATURE-----

    iHUEARYIAB0WIQSj2jRwbAdKzGY/4uAsbEw8qDeGdAUCYYwH2gAKCRAsbEw8qDeG dJCxAQCpYB74HAExX+6m/43DX8/80MrKaxmMCbjCgBYVztgxEAEA3MBD9OUAbu7A +9ZfY9rtVOeSJtcLUUTuXOhYIcUHWAU=
    =lQzP
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Sam Hartman on Wed Nov 10 19:10:02 2021
    On Nov 10, Sam Hartman <hartmans@suchdamage.org> wrote:

    I'm sorry, but I think the only way in which that horse is dead is that
    no one has proposed patches to dpkg.
    Indeed, because the sides of this argument are like three people (one of
    them being the dpkg maintainer) versus everybody else.
    Since some significant work on dpkg is reasonably not forthcoming then
    this is clearly not a viable transition method.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYYwI6wAKCRDLPsM64d7X gaW+AQDH3kHJztFajQGateYQtsosZZcXS1sxLAubUawDN5B4HAD+K4yEDHxCg+Fx 5hq1+p/YWWo0dmC87zeMV0sAbRSp4wk=
    =vEky
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Adam Borowski@21:1/5 to Marco d'Itri on Wed Nov 10 19:30:02 2021
    On Wed, Nov 10, 2021 at 07:01:15PM +0100, Marco d'Itri wrote:
    On Nov 10, Sam Hartman <hartmans@suchdamage.org> wrote:

    I'm sorry, but I think the only way in which that horse is dead is that
    no one has proposed patches to dpkg.
    Indeed, because the sides of this argument are like three people (one of
    them being the dpkg maintainer) versus everybody else.
    Since some significant work on dpkg is reasonably not forthcoming then
    this is clearly not a viable transition method.

    Until we have that in place, going any further with usrmerge is not viable.

    All the TC brought us so far is a wishlist, not a working implementation.
    Thus, I'd say it's time to consider whether the trouble we have
    (substantial, as evidenced by discussions still going and going) is worth
    the gains (are there even any?).

    So either you do the work to revive the horse or it stays put.


    Meow.
    --
    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁ Polexit is brewing? Let's skip that smelly Polsha and reactivate
    ⢿⡄⠘⠷⠚⠋⠀ the Free City of Danzig and/or reapply to the Hansa. ⠈⠳⣄⠀⠀⠀⠀

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Kalnischkies@21:1/5 to Uoti Urpala on Thu Nov 11 14:00:02 2021
    On Wed, Nov 10, 2021 at 01:48:07AM +0200, Uoti Urpala wrote:
    David Kalnischkies wrote:
    As the transition hasn't started everyone not already merged is currently deferring it. That is true for those who upgrade daily as well as for
    those people who seemingly only upgrade their sid systems once in a blue moon. So, at which point have all those systems stopped deferring?

    I think the logical answer is that you're "deferring" in this sense if
    you are using the suggested flag file or whatever other mechanism to
    prevent the merge. Until you do an upgrade which would perform the
    merge without use of such a mechanism, your system is just out of date,
    not deferring.

    A distribution upgrade is not atomic. Between an unpack of package foo
    and the configure of foo a million other packages can pass through
    various stages. Ideally, that window will be pretty small for usrmerge
    the package (or whatever the transition mechanism will be in the end),
    but that depends on various factors and easily balloons out of hand.
    In a previous thread I mentioned how not too long ago the entire KDE
    desktop environment had to be at least unpacked before dpkg could be
    upgraded due to one tiny Conflicts (which was correct). If you hadn't
    KDE installed dpkg was one of the first things upgraded even without
    users going out of their way to explicitly request it (as it should
    be, as its essential and apt does special dances for those).

    So the easiest way to check if an upgrade on a "quantum state merge"
    system is going to work is to keep it at unmerged for the entire time
    and manually trigger the merge at the end as that is what could
    theoretically happen, but is likely not for most testers.
    If it works with merged is already checked by already merged systems.


    So presumably it is valid for packages to gain dependencies which force
    merge or "deferring" state on installation.

    Valid perhaps, but I would hope that it isn't lightheartedly plastered
    all around just in case as the guarantees it provides for the package
    depending on the transition mechanism package are slim (as in, the
    system might or might not be merged, regardless of deferred or not¹,
    while depending package itself passes through various stages) to
    non-existent² depending on the specific implementation of the transition
    while it puts potentially enormous problems on the shoulders of dpkg and
    apt to produce an acceptable ordering:

    The package usrmerge is e.g. currently implemented in perl (the big one,
    not -base) and so any other package implemented in perl is effectively forbidden from forming dependencies on usrmerge as we otherwise run into
    loops of the form app -> usrmerge -> perl -> app which might or might
    not be breakable based on the dependency type (and version) of each ->.
    Oh, and if you happen to have a dependency on something written in perl, congrats, you are part of this elusive group as well as everything else depending on you…

    It will be hard enough to have one essential package trigger the
    mechanism without running into issues, the last we need is a couple
    other packages inserting themselves needlessly into the loop just
    because "it is valid".


    Best regards

    David Kalnischkies

    ¹ Spoiler alert: Even a Pre-Depends technically only makes guarantees
    for the preinst scripts, not for the unpack itself, but that is fine
    print usually only encountered in the deeper horrors of loops… you
    need explicit approval for Pre-Depends anyhow.

    ² Spoiler alert: You can e.g. Pre-Depend all you want on dpkg, but that
    doesn't mean that the version you are pre-depending on is actually
    used to work on your package instead of just lying around on disk.
    That is true for a few other packages, the most obvious perhaps apt
    and the kernel.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmGNEkEACgkQMRvlz3HQ eIPJWg/8DuJkrRYHMOLsO4P6dmiS+E+HuIhq8Us01VpLPwTJRKl+LP6CXP/9B09u yFMbVdpQL8aw8rQHxnJZa3AqHt5PDo17oOh/5yQnT2C8OVU1OzesJuIPLeJwsRZW mx7psyeOrrqY1JblaU/Ri+nE5L0vNqDmcNMab+yYeFWUnxjTMvicN74u5atkTG/K qL81zaQpcc1Nj4LOenyXKjUHnrOdKNH1Ci4Qztw1m/SVnirgPccgQhkwlxtPEx0T CXNHqSa27Z+Rgbk6HRDm5HEwdk5V/7gsmitNsDtcKcXW663JK2whmpWobshjSMOu HAziIYjkhjn4+nR8fPJMm58DsHhUv5QTKjdD496AmtF6npVyJjTq5DQMur7JiBlF zd2TxnR0X1sxTGbJLzxD9p/rZShTemRzndjhWAXb775zm7PjeBoAuUCT1oHViy43 48BH34LAl8CQTPyhVpUzq2HCde6/7zuPQty+7edKEYvZsiXZWHn+SzfuyePyZ5DJ dMgn0Vz+YqgLiRI2lnQvTo6i6jRpWrzTu33Mk6T/jOzNqsYkoKL78cJQcIErgnN3 iFW4G2IkpG95e6X/eodPbXeGfqEgeyUCnKubHM7+LXNjrDd9oCMZeRM4ts6zOvJz 4/ASjWkHXrvuWyKYJPFToA+oPzg0wxGG8SZmzAT6qK9SLxdJxbQ=
    =j7KY
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Marco d'Itri on Thu Nov 11 22:30:02 2021
    Marco d'Itri <md@linux.it> wrote:
    On Nov 10, Sam Hartman <hartmans@suchdamage.org> wrote:
    I'm sorry, but I think the only way in which that horse is dead is that
    no one has proposed patches to dpkg.
    Indeed, because the sides of this argument are like three people (one of
    them being the dpkg maintainer) versus everybody else.

    It's not a subject of debate. The dpkg maintainer says that dpkg
    does not support what usrmerge does, and that it can lead to package corruption. In the previous debian-devel thread on this, it was proven empirically that he is correct.

    Since some significant work on dpkg is reasonably not forthcoming

    Yeah, because _you,_ Marco, prefer to spend your time trying to
    gaslight the project into thinking there isn't a critical-severity bug
    in usrmerge. This could have all been over by now if you had rolled
    up your sleeves and written the necessary patches for dpkg when
    Guillem originally notified you of the problem (in 2016; #848622; the
    bug log does not reflect the actual severity, but again that appears
    to be all on you).

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Svante Signell@21:1/5 to Zack Weinberg on Thu Nov 11 23:30:01 2021
    On Thu, 2021-11-11 at 16:07 -0500, Zack Weinberg wrote:
    Marco d'Itri <md@linux.it> wrote:
    Since some significant work on dpkg is reasonably not forthcoming

    Yeah, because _you,_ Marco, prefer to spend your time trying to
    gaslight the project into thinking there isn't a critical-severity
    bug in usrmerge.  This could have all been over by now if you had
    rolled up your sleeves and written the necessary patches for dpkg
    when Guillem originally notified you of the problem (in 2016;
    #848622; the bug log does not reflect the actual severity, but again
    that appears to be all on you).

    I'm not sure he has the skill or experience enough to submit a patch to
    dpkg. Complaining is much easier than proposing something constructive.

    Just my $5c.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ansgar@21:1/5 to Svante Signell on Fri Nov 12 08:40:10 2021
    Hi Svante,

    On Thu, 2021-11-11 at 23:22 +0100, Svante Signell wrote:
    I'm not sure he has the skill or experience enough to submit a patch to
    dpkg. Complaining is much easier than proposing something constructive.

    I would like to remind you that Debian expects somewhat decent behavior
    of contributors. Please see https://www.debian.org/code_of_conduct

    Ansgar

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to Ansgar on Fri Nov 12 12:10:01 2021
    On Fri, 2021-11-12 at 08:38 +0100, Ansgar wrote:
    Hi Svante,

    On Thu, 2021-11-11 at 23:22 +0100, Svante Signell wrote:
    I'm not sure he has the skill or experience enough to submit a patch to dpkg. Complaining is much easier than proposing something constructive.

    I would like to remind you that Debian expects somewhat decent behavior
    of contributors. Please see https://www.debian.org/code_of_conduct

    Ansgar


    And I want to also add that even if the same request was expressed in
    an acceptable form (it was not), may I also remind participants in this
    thread that according to our Constitution (2.1), no project member is
    obliged to do work on anything they don't want to, and hence it follows
    that it is not appropriate to berate a fellow project member for this
    reason, even if it was done in a respectful tone (which again it was
    not, in this occasion).

    So if any project member feels particularly strongly about this issue,
    which to the best of my knowledge has not been actually observed in the
    wild despite this setup being the default for 100% of Ubuntu users who install/upgrade to 21.10, 100% of new Ubuntu installs since ~2018 and
    an unspecified number of Debian installs being the default in our
    installer too for the past two stable releases (and despite other
    unrelated Replaces: bugs actually existing and being reported over
    time), the constructive way forward would seem to me to contribute the
    work themselves to solve it. I dare say it would help their cause a
    great deal more, instead of rekindling flame wars on a mailing list, to
    come here and say "Hey, here's an alternative
    tool/patchset/software/mechanism that has the exact same result but
    does not suffer from the issue we are worried about. Could you help us
    test it, please?". I for one would be happy to help testing such
    alternatives.

    --
    Kind regards,
    Luca Boccassi

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmGOSegACgkQKGv37813 JB4raQ/8DtX39oeAoRFzRfNs3MdjpV2G9Cb6RnTlsj9pqbQj5WfxF4t2mnPBVaMs zL10nfNFIjpXweYd9oIALb7wYWV/mDzT/vgkwPRCG7gb8ITQ9B6HZwy7iMnxuS6s RPeLaJg55OezBDcgpdgnjLTLGit49iu8ZXdlLowkIBWmUwV4QkatTTGULku5WQin YWGECDkCcM8cO4DzbtuaJ+o6vVvqDvGQwQJacPo8dca5Ky4bhlTrhQDTOU/SlxDI pdnyO/fDRrdk9rwdY0KuBEKVXzrei/nnee5pxII79jcqFzLdrznuEI9Agr2x8wU0 rUJASt3R6vHAmHZnaK0zFADKHpeeL9sidqAMiF+GfxuulW0imTCoD3uYZZXyXeAI Yhm6aUHk6T97U1IuL8eNEILDUj+AObL04A3VRwwElOQkH8VXmNqOJ0OCA5rKGU0T OdwnzI7j79ftHoly+YIfWXovNaqUBz001nSyALnZ2Opt0THOxqbvegBEGiHusckO lh+nPKMLTzjUnSfJ+GTWtgocwcVao1p6btRMKwaetbojmMdaprHr/VCBLIQJ2Jll 1lFUrfTvWINK+KkeswL+52UlUlYpDndpuTlCk7pXwpGhiFzGqh500AqW+4KPitn+ FwiRuJe1DJsOJR3TBlKjmAZIspKIIEIAV8h3tKqlzEb8OXoLJnA=
    =Qp9G
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Luca Bonassi on Wed Nov 17 00:10:01 2021
    Luca Bonassi wrote:
    may I also remind participants in this thread that according to our Constitution (2.1), no project member is obliged to do work on
    anything they don't want to

    Yes, and it follows that the people who want a change to happen are
    the people who will have to do the work to make that change happen,
    including fixing any bugs that are exposed by that change. If they
    don't want to do that work, and nobody else does either, then maybe
    the change isn't going to get done.

    As I said in the previous thread about this, I personally don't care
    whether merged-/usr ever actually happens. It is not relevant to what
    I use Debian for. So I am not motivated to do any work to make it
    happen.

    But I do very much care that the transition isn't botched, and right
    now it looks like the greatest risk of the transition being botched
    comes from proponents who are trying to rush to the end state without
    fixing all the bugs exposed in the process. Since one of the exposed
    bugs involves files going missing from Required and Essential packages
    upon seemingly-unrelated upgrades, during some indefinite period
    *after* the transition, you can, I hope, see why I feel it necessary
    to make a stink.

    [The bug where files disappear from packages upon being moved from
    /bin to /usr/bin or vice versa, after / is a symlink to /usr] to the
    best of my knowledge has not been actually observed in the wild
    despite this setup being the default for 100% of Ubuntu users who install/upgrade to 21.10, 100% of new Ubuntu installs since ~2018
    and an unspecified number of Debian installs being the default in
    our installer too for the past two stable releases

    There's a very good reason for that: people aren't doing the package
    changes that trigger the bug, yet. They can't, because that would
    break systems where /usr hasn't been merged. If the bug is not fixed
    I expect it will start causing problems in unstable *after* bookworm,
    since (as I understand the current transition plan) bookworm+1
    development is the earliest that package maintainers may assume /bin
    is a symlink.

    The existence of the bug is not in question. A step-by-step
    reproduction recipe was posted in the previous thread. Losing files
    from Essential packages when they are upgraded is severity:critical.
    Therefore: either someone fixes the bug, or the transition will have
    to be canceled. It appears to me that the tech ctte agrees with all
    of this.

    I dare say it would help their cause a great deal more, instead of
    rekindling flame wars on a mailing list,

    Marco is the person who restarted the argument. I will cheerfully
    drop the subject again as soon as Marco acknowledges that (a) the bug
    exists, and (b) it is a hard blocker for the transition.

    zw

    p.s. apologies for breaking threading, i'm not subscribed to d-d

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Wed Nov 17 19:40:01 2021
    "Zack" == Zack Weinberg <zack@owlfolio.org> writes:
    Zack> The existence of the bug is not in question. A step-by-step
    Zack> reproduction recipe was posted in the previous thread. Losing
    Zack> files from Essential packages when they are upgraded is
    Zack> severity:critical

    Agreed so far.

    Zack> . Therefore: either someone fixes the bug,
    Zack> or the transition will have to be canceled. It appears to me
    Zack> that the tech ctte agrees with all of this.

    There's a third option.
    We sit around in the state where /bin is a symlink, but where you cannot
    move files to /usr paths in the package system until the bug is fixed.

    I.E. I don't think the transition is going to get canceled; we're too
    far down that path.
    I do think that it might be a critical bug to move packages from /bin to /usr/bin in the packaging system in bookworm+1 if this bug doesn't get
    fixed.

    In other words, there are people who are motivated to do the work of
    creating /bin->/usr/bin symlinks.
    There may not be people motivated to make it possible for the later
    packaging changes to happen.
    That may be the work that doesn't get done until motivated people show
    up to do it.

    -----BEGIN PGP SIGNATURE-----

    iHUEARYIAB0WIQSj2jRwbAdKzGY/4uAsbEw8qDeGdAUCYZVLzQAKCRAsbEw8qDeG dFRbAQCc5541GXt8udkgN4QYhKI4CT3odPEv2A1uCVP+E2+qcwEA82I5JCAz9t1z hKe4P+rsHHVcDFhrxP8IvSgh/xFxfwk=
    =sRuu
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Wed Nov 17 20:00:03 2021
    "Marco" == Marco d'Itri <md@linux.it> writes:

    Marco> On Nov 10, Sam Hartman <hartmans@suchdamage.org> wrote:
    >> I'm sorry, but I think the only way in which that horse is dead
    >> is that no one has proposed patches to dpkg.
    Marco> Indeed, because the sides of this argument are like three
    Marco> people (one of them being the dpkg maintainer) versus
    Marco> everybody else. Since some significant work on dpkg is
    Marco> reasonably not forthcoming then this is clearly not a viable
    Marco> transition method.

    Um, no one still involved in this thread is talking about transition
    methods other than what the TC proposed.
    We're all talking about transition methods where /bin becomes a symlink
    to /usr/bin.

    The question is whether we ever get to a place where people can update
    files in a package currently installed to /bin/foo and instead install
    them to /usr/bin/foo.
    We have a consensus that dpkg bugs make that a bad idea.
    That consensus is validated by the TC.
    It's not clear to me that bug is likely to get fixed in the bookworm +1
    time.

    I don't think that means we halt the transition: the TC has made a
    decision that we're going forward.
    I do think it means probably even after the release of bookworm we
    cannot rearrange where packages claim files get installed.
    That's the horse I think is not dead.

    -----BEGIN PGP SIGNATURE-----

    iHUEARYIAB0WIQSj2jRwbAdKzGY/4uAsbEw8qDeGdAUCYZVQgQAKCRAsbEw8qDeG dKbaAP4z4PkLTWEl2Z/EWmuIMVAclUUb5dxxH1W2f+aVD8x6ewD8Do97+/Z3lp89 e6UE3XmspVAl13bLBrKiVY4TPH8llgs=
    =9+6+
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Biebl@21:1/5 to All on Wed Nov 17 21:10:02 2021
    Am 17.11.2021 um 19:57 schrieb Sam Hartman:
    The question is whether we ever get to a place where people can update
    files in a package currently installed to /bin/foo and instead install
    them to /usr/bin/foo.
    We have a consensus that dpkg bugs make that a bad idea.

    Is that really so? If I'm not mistaken, the problematic part was moving
    files from / to /usr and at the same time moving files between packages.
    So doing that at the same time can be problematic. If that would affect
    many packages is another question.

    AIUI simply moving files from / to /usr within the same package is not problematic.

    Sam, am I missing something here?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Sam Hartman on Wed Nov 17 21:40:01 2021
    On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
    "Zack" == Zack Weinberg <zack@owlfolio.org> writes:
    Zack> Therefore: either someone fixes the bug,
    Zack> or the transition will have to be canceled. It appears to me
    Zack> that the tech ctte agrees with all of this.

    There's a third option.
    We sit around in the state where /bin is a symlink, but where you cannot
    move files to /usr paths in the package system until the bug is fixed.

    I guess that is a potential outcome. In a sense we are already in that state, given the installed base of systems where /bin is already a symlink.

    I don't *like* that outcome; I think it's asking for lots and lots of accidental breakage in unstable post-bookworm, as packages are rebuilt on systems that now have /bin a symlink. But I can't personally offer to fix dpkg, either (way oversubscribed on
    other projects, and this doesn't seem like a job to be tackled by someone new to dpkg, tbh).

    I.E. I don't think the transition is going to get canceled; we're too
    far down that path.

    *Are* we? It seems to me that we could still, at this point, pull usrmerge from testing and stable, push point updates to the installers for all supported releases to flip the default back to non-merged /usr, and advise the installed base to run dpkg-
    fsys-usrunmess before their next apt upgrade. It'd be ugly but it might well be *less* ugly than being stuck in the state you describe. I understood the tech-ctte to be explicitly holding that option open.

    The proponents of merged /usr would be unhappy, but that does not bother me.

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Wed Nov 17 22:20:01 2021
    "Zack" == Zack Weinberg <zack@owlfolio.org> writes:

    Zack> On Wed, Nov 17, 2021, at 1:37 PM, Sam Hartman wrote:
    >>>>>>> "Zack" == Zack Weinberg <zack@owlfolio.org> writes:
    Zack> Therefore: either someone fixes the bug, or the transition
    Zack> will have to be canceled. It appears to me that the tech ctte
    Zack> agrees with all of this.
    >>
    >> There's a third option. We sit around in the state where /bin is
    >> a symlink, but where you cannot move files to /usr paths in the
    >> package system until the bug is fixed.

    Zack> I guess that is a potential outcome. In a sense we are
    Zack> already in that state, given the installed base of systems
    Zack> where /bin is already a symlink.

    Zack> I don't *like* that outcome; I think it's asking for lots and
    Zack> lots of accidental breakage in unstable post-bookworm, as
    Zack> packages are rebuilt on systems that now have /bin a symlink.

    I don't like that outcome either.
    I think that we have reasonable migigations for the specific problem you describe.
    In particular, we do as I understand it have QA plans to build both with
    merged /usr and without prior to bookworm to catch problems like the
    ones you describe.
    After bookworm releases it's totally fine if programs reference
    filesystem paths like /usr/bin/bash, so long as they don't move where
    things get installed.

    So, while I agree with you getting stuck in the middle is not a good
    place,
    I don't think it happens to suffer from the problems you describe.

    >> I.E. I don't think the transition is going to get canceled; we're
    >> too far down that path.

    Zack> *Are* we? It seems to me that we could still, at this point,
    Zack> pull usrmerge from testing and stable, push point updates to
    Zack> the installers for all supported releases to flip the default
    Zack> back to non-merged /usr, and advise the installed base to run
    Zack> dpkg-fsys-usrunmess before their next apt upgrade.

    That seems crazy to me: dpkg-fsys-usrunmess has almost certainly
    involved less testing than usrmerge. Talk about not having people to do
    the work: I don't think we would find people to do enough testing to
    validate that was a reasonable thing to do. But also, the usrmerge
    proponents get most of what they wanted even if we're stuck in the
    middle. If you want a minimal root because you care about containers,
    ostrees, and the like, you don't care that much where the package
    database thinks files are being installed.

    You've been working on this for years and you've gotten most of what you
    hoped for.
    You're going to fight really hard if someone tries to take that away.
    You can easily fight hard enough to make that cost much higher than the
    cost of fixing dpkg.
    And you can present a reasonably looking political front: hey, we've
    been working on this for years, the TC made a decision, the TC later
    gave advice, we're basically just done except for this one dpkg bug.
    Backing out is insane; just fix dpkg.

    I want to reiterate that the initial process surrounding usrmerge was
    horrible.
    The dpkg maintainer had what turned out to be legitimate concerns that
    were not reasonably addressed.
    I think that's in part because they were presented poorly and in part
    because people didn't want to hear them.
    That's a real process failure.
    We should learn from that mistake and do better in the future.

    We don't have the political will to unwind years of work because we made
    that process error, and I don't think it would be a good idea to do so
    even if we did.
    I do think we should be careful to be better about listening to each
    other in the future.
    We don't want a community where you can get your way by ignoring
    legitimate objections.
    I think people on both sides were unwilling to listen to each other.

    And yes, if the dpkg maintainer had asked for usrmerge to block on dpkg
    gaining support for aliasing, and if that had been done around the time
    it was proposed to change the default for debootstrap, we as a project
    should have listened.
    That's not what happened though.
    The dpkg maintainer refused to consider the usrmerge approach that the
    usrmerge proponents wanted.
    He proposed a different scheme that didn't actually meet the needs of
    the usrmerge proponents.
    He was unwilling to take the time to understand why his proposal didn't
    meet other people's needs.

    And yes, the usrmerge proponents (and especially the TC) should have
    worked harder to understand the actual underlying objections.


    So, there's a huge chunk of ways we could improve to go around--enough
    for everyone involved.


    But reliving past battles as anythying other than a way to do better in
    the future will drag us down.

    --Sam

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iHUEARYIAB0WIQSj2jRwbAdKzGY/4uAsbEw8qDeGdAUCYZVwdwAKCRAsbEw8qDeG dKWqAP0XK3TG5aht7swLXN6yQVGZrGKjmHZDBNRvYVcTyM/awAEAlaOySduxcO9N Tg0QZf502UGRHObKNqkH/lUbcmVAyws=VH0u
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to All on Thu Nov 18 03:40:02 2021
    "Sam" == Sam Hartman <hartmans@debian.org> wrote:
    "Zack" == Zack Weinberg <zack@owlfolio.org> writes:

    There's a third option. We sit around in the state where /bin is
    a symlink, but where you cannot move files to /usr paths in the
    package system until the bug is fixed.

    Zack> I guess that is a potential outcome. In a sense we are
    Zack> already in that state, given the installed base of systems
    Zack> where /bin is already a symlink.
    Zack> I don't *like* that outcome; I think it's asking for lots and
    Zack> lots of accidental breakage in unstable post-bookworm.

    I don't like that outcome either.
    I think that we have reasonable mitigations for the specific problem you describe.
    In particular, we do as I understand it have QA plans to build both with merged /usr and without prior to bookworm to catch problems like the
    ones you describe.
    After bookworm releases it's totally fine if programs reference
    filesystem paths like /usr/bin/bash, so long as they don't move where things get installed.

    I don't think that will mitigate the problem. I think that post-bookworm, packages like coreutils and libc6 are going to notice, in their upstream configure scripts, that /bin etc are symlinks into /usr, and start installing stuff that used to be in / into /usr. Maintainers are going to need to take positive action to _prevent_ the move, and there will no longer be automation to detect that files have moved.

    I.E. I don't think the transition is going to get canceled; we're
    too far down that path.

    Zack> *Are* we? It seems to me that we could still, at this point,
    Zack> [roll back the transition]

    I don't think we would find people to do enough testing to
    validate that was a reasonable thing to do. But also, the usrmerge proponents get most of what they wanted even if we're stuck in the
    middle.

    That is exactly why I think a rollback should not be taken off the table:
    it is very, very bad if the usrmerge proponents get most of what they want while the rest of the project is left to clean up their mess. The project cannot afford to reward such misbehavior. I honestly think the social
    fallout of allowing that to happen would be *worse* than the social and technical fallout of forcing a rollback through.

    ...
    I want to reiterate that the initial process surrounding usrmerge was horrible. The dpkg maintainer had what turned out to be legitimate concerns that were not reasonably addressed. I think that's in part because they were presented poorly and in part because people didn't
    want to hear them. That's a real process failure.
    ...
    I think people on both sides were unwilling to listen to each other.

    I'm an outside observer and I have not followed this argument closely from
    the beginning, but my perception of it is much more one-sided than you are making it out to be. This is what _I_ saw:

    - usrmerge was deployed to unstable in late 2014. It's not clear to me
    how much testing it got in 2015.
    - Reports of it causing problems for dpkg started to appear in early 2016
    (e.g. #810499). These seem to have been addressed civilly, but even then
    a "well it works for _me_ so :shrug:" attitude from the merged-/usr
    proponents is evident in the bug logs.
    - The dpkg maintainer filed a severity:serious bug on usrmerge in late
    2016 (#848622), saying that it breaks dpkg -S. This is the earliest
    instance I can find of Marco in particular overtly blowing off a bug
    report that he didn't think was significant ("Over one year of
    experience with merged-/usr systems has not exposed any failures",
    which is an invalid argument). This bug is also significant in
    hindsight because, although not described as such, it appears to me
    to have the same root cause as the file lossage on upgrade that you
    and I, at least, agree is severity:critical.
    - Over the next, um, *five years*, Marco continued to ignore or reject
    Guillem's attempts to get him to take problems seriously that were
    caused by usrmerge rendering the dpkg database inconsistent with the
    file system.
    - Marco also reacted hostilely to concerns raised by others, e.g.
    Adrian Bunk (#863269).

    - Guillem, for his part, has displayed far more patience than I would
    have in his situation, repeatedly attempting to explain why there is
    a serious problem, offering concrete solutions - that they may not be
    practical is not the point; he's done more work toward that end than
    anyone else - and never, that I have seen, losing his temper.
    At the same time, he has said in so many words that this has caused
    him to become burnt out on Debian in general and dpkg maintenance in
    particular.

    There's no "both sides were unwilling to listen" when all one side is
    bringing to the table is "there's no problem, your concerns are bullshit"
    (the word "bullshit" is actually used in #863269).

    Maybe it was not obvious to anyone in 2016 that the package database
    being inconsistent with the filesystem could cause actual data loss.
    However, I think it was the responsibility of the developers of usrmerge
    to find out how bad that problem actually was, rather than dismissing it.
    And, as it has proven to be a genuinely critical problem, it is also their responsibility to fix it, per the 'no one can be forced to do anything' rule.

    We don't want a community where you can get your way by ignoring legitimate objections.

    Exactly! Taking a rollback off the table at present, *rewards* the bad behavior of the merged-/usr proponents, and demonstrates to the community
    that making life miserable for one's fellow DDs is a *successful* strategy
    for getting one's way. That can't be allowed to happen. There need to be consequences for burning people out - such as "either fix the mess you created or see your work to date undone".

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard Laager@21:1/5 to Zack Weinberg on Thu Nov 18 06:30:02 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ZFXLgK8dpZNQqnpwE4NpBjj2tFhMNSXtK
    Content-Type: text/plain; charset=utf-8; format=flowed
    Content-Language: en-US
    Content-Transfer-Encoding: quoted-printable

    On 11/17/21 8:18 PM, Zack Weinberg wrote:
    However, I think it was the responsibility of the developers of usrmerge
    to find out how bad that problem actually was, rather than dismissing it. And, as it has proven to be a genuinely critical problem, it is also their responsibility to fix it, per the 'no one can be forced to do anything' rule.
    At least for this email, I'll stipulate to everything above.

    Is this particularly hard to fix, though?

    Can't we just do something like the following pseudo-code:

    ----

    /* At dpkg start... */
    usrmerged = FALSE;
    if (the_flag_is_set) {
    usrmerged = TRUE;
    } elif (bin symlinked to /usr/bin) {
    usrmerged = TRUE;
    /* Update e.g. /bin/foo to /usr/bin/foo in the database. */
    update_database();
    set_the_flag();
    }


    /* As dpkg is deciding to write a file path into the database. */
    /* e.g. the package is trying to install to /bin/foo, put
    * /usr/bin/foo into the database. */
    if (usrmerged) {
    path = canonicalize(path);
    }

    ----

    (I've used /bin -> /usr/bin as an example, but the canonicalization must
    deal with the other merged paths too.)

    The second bit ensures that all new operations write canonicalized paths
    (e.g. /usr/bin/foo) into the database regardless of whether the .deb
    shipped with /bin/foo or /usr/bin/foo. This ensures the database stays
    in sync with the filesystem moving forward.

    The first bit performs a one-time canonicalization of paths in the
    existing database. An on-disk flag ensures that this is run only once
    (for performance reasons only, because it's already idempotent). This
    corrects the existing database-filesystem mismatch.

    The one-time canonicalization can be removed once non-usrmerged systems
    are no longer supported. The second bit needs to live (probably a lot)
    longer, until it's no longer reasonable to install a .deb containing non-usrmerged paths (which might be old or from a third party).

    Am I missing something here?

    --
    Richard


    --ZFXLgK8dpZNQqnpwE4NpBjj2tFhMNSXtK--

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEE1Ot9lOeOTujs4H+U+HlhmcBFhs4FAmGV41sACgkQ+HlhmcBF hs4WFg/+L+nvZmDc5SlPSq9XJQn1eDxuk/+xA+BukpHCadJ2ZA1/GvyQ3QbYwjOU y05rsaKopzjIuHX3PqTg7XQqNH7ChfHvCYNQ1U1Y29m2JtwY37ctyYMjuwnqHsjw RYusKiqWenN7NZrsNLnkzgiKS7eGIw85eDgxHNfLn01U1L41bFzs2Le7NVnzl250 8pQxAnSQABf/rQ5X2YPMactPitZQK16b2keTpX8F/9eT6FTi6X9+tvfzZndWCBID 3cm4DLVFzTvnBIw+3L0w8UvG9I72Qesz+zJle56PmXwqNllVX8SsIUmwNXfQq7ZJ I40t21wC/Q+iMldn3EfzuBAQ1ZKR19hgiQKasdblYn9XuOYWYcN7qOkD7eXMN74P LmMn0iOwbo9gMps6D1gVnOdD5Nq5tC4aykXvjINkwd9OwXMrnRcyfSBa9ozjgI+J U3B+7lPBgV4ZagRI+j7WgeFWUYY4xVSqkiLBCYsH24z6LA5zJCsYwMxzXs2TuBa4 MtAzrJegjkgTEzBkI58lK10/E5s3NLui/EtV8Hp20rYWRkc5/Ff36xiWubI+uN6S EvsRlOlo2tP9SXV5lHhqB0kCZB+hz58zqGUeZ76hO1EMVlYb/ijdZdwNvI/Uuszv PkqJW5XbNHxRaACdpM5Krp3llXVZMeA6BMJ/U7sQrsEcwGcTzms=
    =7szh
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?utf-8?Q?Bj=C3=B8rn_Mork?=@21:1/5 to Michael Biebl on Thu Nov 18 11:40:02 2021
    Michael Biebl <biebl@debian.org> writes:
    Am 17.11.2021 um 19:57 schrieb Sam Hartman:
    The question is whether we ever get to a place where people can update
    files in a package currently installed to /bin/foo and instead install
    them to /usr/bin/foo.
    We have a consensus that dpkg bugs make that a bad idea.

    Is that really so? If I'm not mistaken, the problematic part was
    moving files from / to /usr and at the same time moving files between packages. So doing that at the same time can be problematic. If that
    would affect many packages is another question.

    AIUI simply moving files from / to /usr within the same package is not problematic.

    Won't you then end up with a package which cannot be split for the next
    two releases? Maybe not a big problem, but I think it will be so hard
    to keep track of that it's better to not move the files.


    Bjørn

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Zack Weinberg on Thu Nov 18 13:20:01 2021
    On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:

    And, as it has proven to be a genuinely critical problem, it is also their
    No, it has not.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYZZChAAKCRDLPsM64d7X gZYHAP0ab3Oqww/Ux/3YIRMKJfY06GSRdtvmPzY7j/8QbfeIIAEAqYDcwztyeTHV S7nLJrV5q/L5zSOEo+x3PyfFt7H6tg0=
    =lD+B
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to Marco d'Itri on Thu Nov 18 14:50:01 2021
    On Thu, 2021-11-18 at 13:09 +0100, Marco d'Itri wrote:
    On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:

    And, as it has proven to be a genuinely critical problem, it is also their
    No, it has not.

    Indeed - it seems to me there's a convenient tendency to forget that
    this is not something new that has never been seen before, to be
    introduced without any testing. This is the default. It has been the
    default for multiple releases of multiple distributions. All Ubuntu installations that were not using this default are now forcibly
    converted upon upgrade to 21.10.

    And yet nobody has actually seen this happen, to the best of my
    knowledge. So either this combination that would allegedly not work
    hasn't been used once in 4+ years across multiple distros, or the
    required alignment of coincidences to make the issue happen is so rare
    that it just never happens in practice.

    All we have is a proof of concept. By all means, if anybody cares
    enough, go and fix it, no problem with that. Propose an actual, working alternative that gives the same result too - will happily spend my time
    to test it. And put in place QA checks and whatnot to be sure - that
    seems a great idea.

    But talks of blocking everything and reverting things with a hacky
    script that has never actually been used before, in the face of
    multiple decisions by the tech committee and no evidence whatsoever of real-world impact, and despite tens of thousands of actual, real bugs
    affecting Debian that don't get even a fraction of the same treatment
    (even the Replaces: feature has been affected by unrelated, actual,
    reported bugs, and might very well still be, haven't checked) seems to
    me a tiny bit hyperbolic and exagerated.

    --
    Kind regards,
    Luca Boccassi

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmGWWEcACgkQKGv37813 JB4HKw//Zp+kESkBtKWyfWr4O7dcP/qwot9DEQduMQeyq5bU8W0Dj9m5t16G7bzY ZXtVsQuiqjq7sfeDjdywceekJy7qlxQpHx2MhYd/9ALpzXwosjMXOaHfGzAZcWqL 9HncGPJqhF55OQ2dbG2EaVTdwHv6U1/Zs0kWlfJ5k/Y1j3lFMDUvLj9UVgV0qsnq GkfYZK0QTm5VdwK5w3TeW8HkArh8ZVARnqPcC+hJFN+u+CUhBAg7PitMKiR/Dqu+ BYRu9t1VJAThTOCULkHzcBlO7PQIhDcxDqIWed9hVL2nHWD9L+v7wmadAtnqi8jE XLM8iwvuBEHXp3CM0ppv+OVakCn/yD6aMtXsOGER2kxYjw3fCqlOqZw52F/3msJ+ DWVgyT0R9QJ3zk/8elCIbEkO8LcY1Q01rDup2XVZkVeIs50CnzhiqH/0vIgFKpjz wW3IaqD4Xa+uZF//pD4vGU+FftJ6JnKM5PhoKRMAtC4k0Ce+xIdDae8bPej2m+RI zQvJYhxQcuXLysxLg4MjBMO8efN0yEyRZ0OknufEpdPwt19ZxsH1orQYS/yMcLKV Bxcby3vsoUMkwpWzpRVQhustHWm0YV/8v7q9BxZwriUiGnqRj3e3OeagjSujps2E tTL0kzivQcmj3dhyrCX1JhbL4ArStL8IWc+2eiGs+jSm0cPnbok=
    =enaG
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Luca Bocassi on Thu Nov 18 22:50:01 2021
    Luca Bocassi wrote:
    ...
    [merged /usr] is the default. It has been the
    default for multiple releases of multiple distributions. All Ubuntu installations that were not using this default are now forcibly
    converted upon upgrade to 21.10.

    And yet nobody has actually seen [the file disappearance bug]
    happen, to the best of my knowledge.

    I already explained why that doesn't prove the bug is a non-issue. To the contrary; it means there is an enormous installed base of systems where the bug is latent, waiting to cause problems under conditions which we can reasonably expect to occur
    shortly after the release of bookworm. Please do not make me repeat myself.

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Marco d'Itri on Thu Nov 18 22:40:01 2021
    Marco d'Itri wrote:
    On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:
    as it has proven to be a genuinely critical problem
    No, it has not.

    In the previous long thread on debian-devel on this subject, someone posted a step-by-step recipe to reproduce a phenomenon where a file that has been moved (in its package) from an installation path of /bin (for example) to /usr/bin, possibly in
    conjunction with a change to which package owns it, while /bin is a symlink to /usr/bin, will disappear from the file system when the updated set of packages is installed. (I regret I do not have time today to dig up the exact email in question.)

    Are you seriously claiming that that phenomenon is not a severity:critical bug?

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to Luca Bocassi on Fri Nov 19 02:10:02 2021
    On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
    Luca Bocassi wrote:
    ...
    [merged /usr] is the default. It has been the
    default for multiple releases of multiple distributions. All Ubuntu installations that were not using this default are now forcibly
    converted upon upgrade to 21.10.

    And yet nobody has actually seen [the file disappearance bug]
    happen, to the best of my knowledge.

    I already explained why that doesn't prove the bug is a non-issue. To the contrary; it means there is an enormous installed base of systems where the bug is latent, waiting to cause problems under conditions which we can reasonably expect to occur
    shortly after the release of bookworm. Please do not make me repeat myself.

    zw

    I'm afraid you have not. Why would the release of bookworm make any
    difference? There will be nothing new that hasn't already been
    happening for years.

    --
    Kind regards,
    Luca Boccassi

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEErCSqx93EIPGOymuRKGv37813JB4FAmGW+J4ACgkQKGv37813 JB7+qg//UJEYry89Z0N/0jzKd6PJGewgQ0kL0D0ewS3l2+mfNFgE/ZyQUIUrefao x3mi89vcRtXJLJuiNGfPTR264cLUUdL3hw56y0rgBlkkrg6ZVYlohntTnSxr9pi3 qWU/KjZSFnoAS9CzWL+zLFIw7tbL/h5hEu4/sl3F2yoGWl0x+fQ+3+ihfWvw3vJ7 SjeVAXKRlz6pXBnGkOa08lUCBI/Sh3tj9LJ1HKt8YVCJl8RsfiRvhxL8KYdxCJQ4 EvKpNgb2649CvMK2BXati1PC2saQ/OCaXjpx8uR3+/t2NdsktFcHVF6lz99fO4ka QBgCQEvfLKnD8FjLYZ4cjVQEjJVWR08wpS+x6/x4w2198SsQImPKkx4FZULdwBdl MbahmkkTW8lr21My4JSVoHyOMk5bDO/7QVDVFf3vHZPX0hJK+tqjv56wnUZzFtwN HAp7pEBcP7NlREUIMvHzipOpmeGsOIHORVZiNsUI4FVuSyFPD6t2aoneBGtVo8nr ZqxK+KzpGg8VmBiFJ37kUSU/qDIkmbIj9iO3W7E/QuNy6g8Q/2K8puJRBvlifTbH aXrqvLH4nlBppPGLnNutXcZ6qiGPDNBoJqDyaS2pu4m9uryII7zGTJBXQfDI6b4N J8jGXp9UCtFJnbTiUvqJkd1L+8XPElqi8YMPhecLQhuz0vhdA0w=
    =qcME
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From The Wanderer@21:1/5 to Luca Boccassi on Fri Nov 19 05:10:01 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
    On 2021-11-18 at 20:06, Luca Boccassi wrote:

    On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:

    Luca Bocassi wrote:

    [merged /usr] is the default. It has been the default for
    multiple releases of multiple distributions. All Ubuntu
    installations that were not using this default are now forcibly
    converted upon upgrade to 21.10.

    And yet nobody has actually seen [the file disappearance bug]
    happen, to the best of my knowledge.

    I already explained why that doesn't prove the bug is a non-issue.
    To the contrary; it means there is an enormous installed base of
    systems where the bug is latent, waiting to cause problems under
    conditions which we can reasonably expect to occur shortly after
    the release of bookworm. Please do not make me repeat myself.

    zw

    I'm afraid you have not. Why would the release of bookworm make any difference? There will be nothing new that hasn't already been
    happening for years.

    I interpret Zack's comment as referring to this, which he said in https://lists.debian.org/debian-devel/2021/11/msg00205.html:

    [P]eople aren't doing the package changes that trigger the bug, yet.
    They can't, because that would break systems where /usr hasn't been
    merged. If the bug is not fixed I expect it will start causing
    problems in unstable *after* bookworm, since (as I understand the
    current transition plan) bookworm+1 development is the earliest that
    package maintainers may assume /bin is a symlink.

    IOW, I interpret him as disagreeing with you that "there will be nothing
    new that hasn't already been happening for years". Specifically, I parse
    him as arguing that:

    * to date, package maintainers have not yet begun moving
    already-packaged files from / to /usr/ (specifically because doing so
    would break systems that have not yet been migrated to merged-/usr, and
    Debian has not yet declared that such systems are unsupported),

    * after bookworm, package maintainers will start moving already-packaged
    files from / to /usr/, and

    * doing this will, in a non-negligible number of cases, trigger the bug
    to manifest on systems where that package is upgraded from a version
    where the move had not taken place to one where it has.

    (Zack, if I've gotten any of those wrong, please don't hesitate to
    correct me; I'll either apologize, or drop right back out of the
    discussion to go hide in a metaphorical hole, if not both.)

    Do you dispute any of those three points? If so, I'd be interested to
    know which one(s), and why.

    --
    The Wanderer

    The reasonable man adapts himself to the world; the unreasonable one
    persists in trying to adapt the world to himself. Therefore all
    progress depends on the unreasonable man. -- George Bernard Shaw


    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEEJCOqsZEc2qVC44pUBKk1jTQoMmsFAmGXIaIACgkQBKk1jTQo MmsJ8RAAiFKfi2ztxB1FnQGEPnM8dr190Hxta5KwHxCQkiwF2PfN8K5Wk8eIxigq uJV9i7a4g8O7QPaItSA7suDZvoR8ABd8YV1s+8W2+UDVcOMlrN9k9PGLAP5JzJ6l 2MDEfQvkUbARrJij3xnqq38/NQB4qExjkq88rPeHvKoVHiTkOM+klu5IcVm02+TX yLZQMoOAOip1O+JU3nO2GqyckGa6de82HDFZCpwilaoDpSGJMPPUNotZh7AVrB8R l7DVlyxCazKk+kk93vnv9OaK84AvpzY8IqHThCTpA0fgI0OJlz0f/DB1xOaCP8fN M/o470RzPXJ2RH+sj5O1FSqpNM5woe2CnnH/E5z+xcAR4vctngX1w3d9q0UTUC/7 Iz1sfvy6TV2oheYbH4p/jOU1TaE3N17TGkGOb33e5ChhTUFSjnSzW+lmqPgMZVhP TRqAiop0FaFgnZzpNdvBW+lvM2Zn8mfFypO2mAbJIP1G+2GaEIdYmwDmbBXTsus2 MNK40oSzlamqfhH72s0UlpfYGyT+3Zh1Pmba6qNm1cmsuQXjV6DOvQYU7T6cXWyd y1KK+kOZ865TVjR1bp0nHaJ8glg2trCB3PRCRg7O0AfAmC4Q25VYCXluj2rFUKzN 1yZgBVDSQ/y0x14qi0UT7zwXbWgF
  • From Marco d'Itri@21:1/5 to Zack Weinberg on Fri Nov 19 06:20:01 2021
    On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:

    Are you seriously claiming that that phenomenon is not a severity:critical bug?
    I am seriously claming that whatever you are referring to, if true, is
    such a contrived example that does not actually happen in real life
    (or at least, it does not happen frequently enough).

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCYZbJOgAKCRDLPsM64d7X gXobAQD0VYQLFaE5zaY4oA0Fuag73n3aQ6HhERHlFdHSZNPgqAD+O/hFooWhBoIM gFUIJfR9GQUq0xie1hRZeOfjxh1Y6AY=
    =3ZwG
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ansgar@21:1/5 to The Wanderer on Fri Nov 19 10:10:02 2021
    On Thu, 2021-11-18 at 23:01 -0500, The Wanderer wrote:
    * to date, package maintainers have not yet begun moving
    already-packaged files from / to /usr/ (specifically because doing so
    would break systems that have not yet been migrated to merged-/usr,
    and Debian has not yet declared that such systems are unsupported),

    That claim is incorrect:

    - Some packages have moved files from /(s)bin to /usr/(s)bin. (This
    sometimes requires a compat symlink.)

    - More packages have moved files from /lib to /usr/lib. (This often
    doesn't require any extra care.)

    * after bookworm, package maintainers will start moving already-
    packaged files from / to /usr/, and

    s/start/continue/

    * doing this will, in a non-negligible number of cases, trigger the
    bug to manifest on systems where that package is upgraded from a
    version where the move had not taken place to one where it has.

    Why do you claim that?

    Given packages already did such moves in the last years and you claim
    this happens in a non-negligible number of cases, could you please
    point to some examples where this already happens in practice?

    Ansgar

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Philip Hands@21:1/5 to Ansgar on Fri Nov 19 12:10:01 2021
    Ansgar <ansgar@43-1.org> writes:

    * doing this will, in a non-negligible number of cases, trigger the
    bug to manifest on systems where that package is upgraded from a
    version where the move had not taken place to one where it has.

    Why do you claim that?

    Given packages already did such moves in the last years and you claim
    this happens in a non-negligible number of cases, could you please
    point to some examples where this already happens in practice?

    My understanding is that in order to trigger this bug you need at least
    to both move a file from one place to the other, and also to rename the
    package that contains that file or move ownership to another package.

    I suspect that you might also need to be unlucky with the order that
    apt/dpkg decides to do the installation and, depending upon how far
    apart the move and the rename happens, also unlucky with your choice of
    from and to versions of the packages in question.

    Given that these bugs are going to be utter bastards to reproduce, and
    you can be sure that we'll have enough diversity in installed systems
    that some people are going to manage to be sufficiently unlucky, it
    would be nice to know the sort of damage we might expect.

    It strikes me that we ought to be able to screen our own repos for
    packages that could be able to tickle this bug. That would give us the
    chance to look at what sorts of files we might realistically expect to
    be clobbered, it should give some indication of how many packages we
    should expect to be able to trigger this, and knowing this might suggest plausible work-arounds.

    Of course, that doesn't help with packages from third-party repos,
    including our downstreams, but at present we seem to be discussing this
    with very little hard data.

    It occurs to me that one could lose quite a few files on the average
    Debian install (if they were selected at random) without even noticing,
    whereas a very few files would render systems unbootable, so knowing a
    bit more about which files are realistically at risk would be very
    helpful in understanding the severity of the problem.

    If anyone's got good ideas about how to gather this information, I'm
    very happy to help with the effort to do so.

    Cheers, Phil.
    --
    |)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
    |-| http://www.hands.com/ http://ftp.uk.debian.org/
    |(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEE3/FBWs4yJ/zyBwfW0EujoAEl1cAFAmGXg18ACgkQ0EujoAEl 1cCntw//aZ/22i2++ItQOcwxW/W6VrDhyN37VDbRLobFtLq34MY/j/ZH8Ym2YvyN N29cNJ9c1JRG21cDJa4hXjbJPFTDqEjVNP9kOUw0BMClObqF9dxJEIFqBiE2Mn+w YWt8GUgZYtJlDnluKyquLy0NQSA8qEysSMkTCzypxVrYDZJq2wFRvraaI6lllzJW AdidBi8w6kFaYaF8YNTAyU3wnN/VjinlY/aXwLZ/hx6x4fd+SWn1+onnF2XQYvNy tjfM6hRXFMi0fKAE00HGqtSDM6Xnjpbw8X53immcL3FyrfGJnhpPKK5+Ye/940MU uvlQEBLMp3CnGkwNcoQcT3GSFR2d+IeunXCdWwuOmqna8VI3hJ0kgm6gg2Ew5quJ YOPegPDn0JoewbIvFA6anECCeVy4Uw4r63KQPrO43utCTVgP9fK54p6UHO3qpNcy o9WY1w3PJoUQ1WtE9mxkRE4OqjyRXEw+TNrYpTCAXsxUrIYKD5FyHWGKFyjt3g1/ AXgI9ITxdlO/VZ7iKgRq+SCyc3Q1QGDMN7aqK4/vvIRQoo6TRxojPxFew07HPNbq ql4uwoZTmRjJRlrTSxxDmgWRIVl4jdosJA+AIcRmuUFnSG3Y2HPAlBsM7c8pjqLW 06CvcecuNHb/EQE
  • From David Kalnischkies@21:1/5 to Ansgar on Fri Nov 19 15:40:03 2021
    --gofkz6cs2mlwhhem
    Content-Type: text/plain; charset=utf-8
    Content-Disposition: inline
    Content-Transfer-Encoding: quoted-printable

    On Fri, Nov 19, 2021 at 10:06:13AM +0100, Ansgar wrote:
    Given packages already did such moves in the last years and you claim
    this happens in a non-negligible number of cases, could you please
    point to some examples where this already happens in practice?

    You need a / → /usr¹ and a pkg-a → pkg-b move in the same release.
    Also, you need to have (the new) pkg-b be unpacked before pkg-a.

    An example³ would be coreutils/util-linux/… moving everything from /bin
    to /usr/bin and in the same Debian release splitting out one (or more)
    of their tools into their own package (as they usually do).
    As those are essential they will Pre-Depend² on the split out package
    which guarantees that pkg-b will be unpacked before pkg-a.

    The result is that the split out tool will be gone from /usr-merged
    systems – which at that point should be all systems.


    Another example would be systemd files debhelper moved for some time
    already while the package does a foo and foo-data split in the same
    Debian release. Just need to "solve" the unpack order now, but I will
    leave that as an exercise for the reader.



    The move and reorganisation is both forbidden by the CTTE for Debian 12
    in "Moratorium on moving files' logical locations into /usr" which even describes this problem as one of the reasons for it, but hopes to have
    it resolved by 13 (without mentioning how).

    Are you suggesting that Debian will use 13 to move each and every
    file in / to its /usr counter-path while forbidding that packages
    including moves are reorganised before 14 ?

    Good thing 'which' isn't in /bin I guess. (SCNR)

    Disclaimer: I am as usual not arguing for switching into full speed
    reverse mode. I would "just" prefer if we acknowledge that problems
    exist we have to deal with. Its gonna be hard enough to actually resolve
    them given all bridges have been burned down years ago by pretending its
    not a problem that dpkg has no idea what is done behind its back to the
    files its supposed to manage.

    (The problem itself isn't unique⁴ to /usr-merge, so ideally it would be
    resolved regardless, but /usr-merge undoubtedly makes heavy use of it
    so in an ideal world those interested in it would not only acknowledge
    the problems but actually work together to resolve them.
    Sadly, that isn't the world we seem to be stuck in at all.)


    Best regards

    David Kalnischkies

    ¹ You could of course also move the other way around.
    ² You can achieve the same with other dependency types, it is just
    harder to trigger in simple testcases as apt & dpkg try to avoid
    the auto-deconfiguration of pkg-a if there is an easy way out.
    ³ For fun I wrote an apt testcase with coreutils splitting out ln⁴,
    that might be a bit unrealistic, but you get the idea (attached).
    ⁴ as, you know, /usr-merge being the last symlink we will ever need

    --gofkz6cs2mlwhhem
    Content-Type: text/plain; charset=us-ascii
    Content-Disposition: attachment; filename=skip-this-is-not-a-problem

    #!/bin/sh
    set -e

    TESTDIR="$(readlink -f "$(dirname "$0")")"
    . "$TESTDIR/framework"

    setupenvironment
    configarchitecture 'native'

    #mkdir -p rootdir/bin
    ln -s usr/bin rootdir/bin

    touch ln

    mkdir -p tree/bin
    cp -a ln tree/bin
    buildsimplenativepackage 'coreutils' 'native' '1' 'stable' '' '' '' '' 'tree/bin'
    rm -rf tree

    buildsimplenativepackage 'coreutils' 'native' '2' 'unstable' 'Pre-Depends: unneeded-ln'

    mkdir -p tree/usr/bin
    cp -a ln tree/usr/bin
    buildsimplenativepackage 'unneeded-ln' 'native' '2' 'unstable' 'Breaks: coreutils (<< 2)
    Replaces: coreutils (<< 2)' '' '' '' 'tree/usr'
    rm -rf tree

    setupaptarchive

    testfailure test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln
    testsuccess apt install coreutils=1 -y
    testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln

    testsuccess apt full-upgrade -y
    testsuccess test -e rootdir/bin/ln -o -e rootdir/usr/bin/ln

    --gofkz6cs2mlwhhem--

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmGXtlwACgkQMRvlz3HQ eIMWng/+ICMlwozdcFU04fddUwpzhliD9RDsOE4Z4AQEex+PNMHXftEitxk5Q1pV VemmjuZiDJlp+o3o2gYCfbzHphDaGUi/MetZa2ZPRS1af5ibm7mDbKqGP97SgwH1 nTpVKSVbu1RGeg1ZErunSBIfInh4r+T7R2DjazMtSkb/DwbbTgIxOeaovmAQEs0C M33yT/VwagffGlCityfZ1fkJjLiB8Nlebjn3qLxlgruhsZ4jJZniLayrBLxWXiKJ nVDjglEJYodBfrQ8KWp4i3p1JGCHagYlqdzDFbety3SDDMA3zx2AKyt7PJlh/zQa s6eGYDZ4wSPgb2vsYh7NpSsjyfPCGXdIO7BEHHqaFmzvipwL+injqipBp57ceyr9 hZpbcjl4rsKQkpbiVcGchJ7ci5I2/PmfKG6kGrRycPdUHJvJ1PGFaTm1zXlT21aa Dj/eCPhMgx9Bk4nnS22hHWjukVB5BsIOD0E3oPeiVIs8TOwUy2+ZXYAcD2nW/2XI 2B1G2rg22E/BYZJeLMGOdqBwrPLJEX5BunCWGmYnts/nGLdYIe/GYybuzq1canY4 8kX9TyfRrPoFuPOoL+1A7YrfTQi3ot9BFybvXjyFskXyeJtolygECNa10Z2azfK4 YKrfFhOl4TawyvoFB5sn/LFHGz8+vyXWsPI5c6vKFlTlzEAPrq4=
    =AJkW
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Biebl@21:1/5 to All on Fri Nov 19 15:40:03 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------BOKxOjZV2kYE9dtFLhiHR0cP
    Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

    T24gMTkuMTEuMjEgMTE6NTgsIFBoaWxpcCBIYW5kcyB3cm90ZToNCj4gQW5zZ2FyIDxhbnNn YXJANDMtMS5vcmc+IHdyaXRlczoNCj4gDQo+Pj4gKiBkb2luZyB0aGlzIHdpbGwsIGluIGEg bm9uLW5lZ2xpZ2libGUgbnVtYmVyIG9mIGNhc2VzLCB0cmlnZ2VyIHRoZQ0KPj4+IGJ1ZyB0 byBtYW5pZmVzdCBvbiBzeXN0ZW1zIHdoZXJlIHRoYXQgcGFja2FnZSBpcyB1cGdyYWRlZCBm cm9tIGENCj4+PiB2ZXJzaW9uIHdoZXJlIHRoZSBtb3ZlIGhhZCBub3QgdGFrZW4gcGxhY2Ug dG8gb25lIHdoZXJlIGl0IGhhcy4NCj4+DQo+PiBXaHkgZG8geW91IGNsYWltIHRoYXQ/DQo+ Pg0KPj4gR2l2ZW4gcGFja2FnZXMgYWxyZWFkeSBkaWQgc3VjaCBtb3ZlcyBpbiB0aGUgbGFz dCB5ZWFycyBhbmQgeW91IGNsYWltDQo+PiB0aGlzIGhhcHBlbnMgaW4gYSBub24tbmVnbGln aWJsZSBudW1iZXIgb2YgY2FzZXMsIGNvdWxkIHlvdSBwbGVhc2UNCj4+IHBvaW50IHRvIHNv bWUgZXhhbXBsZXMgd2hlcmUgdGhpcyBhbHJlYWR5IGhhcHBlbnMgaW4gcHJhY3RpY2U/DQo+ IA0KPiBNeSB1bmRlcnN0YW5kaW5nIGlzIHRoYXQgaW4gb3JkZXIgdG8gdHJpZ2dlciB0aGlz IGJ1ZyB5b3UgbmVlZCBhdCBsZWFzdA0KPiB0byBib3RoIG1vdmUgYSBmaWxlIGZyb20gb25l IHBsYWNlIHRvIHRoZSBvdGhlciwgYW5kIGFsc28gdG8gcmVuYW1lIHRoZQ0KPiBwYWNrYWdl IHRoYXQgY29udGFpbnMgdGhhdCBmaWxlIG9yIG1vdmUgb3duZXJzaGlwIHRvIGFub3RoZXIg cGFja2FnZS4NCj4gDQo+IEkgc3VzcGVjdCB0aGF0IHlvdSBtaWdodCBhbHNvIG5lZWQgdG8g YmUgdW5sdWNreSB3aXRoIHRoZSBvcmRlciB0aGF0DQo+IGFwdC9kcGtnIGRlY2lkZXMgdG8g ZG8gdGhlIGluc3RhbGxhdGlvbiBhbmQsIGRlcGVuZGluZyB1cG9uIGhvdyBmYXINCj4gYXBh cnQgdGhlIG1vdmUgYW5kIHRoZSByZW5hbWUgaGFwcGVucywgYWxzbyB1bmx1Y2t5IHdpdGgg eW91ciBjaG9pY2Ugb2YNCj4gZnJvbSBhbmQgdG8gdmVyc2lvbnMgb2YgdGhlIHBhY2thZ2Vz IGluIHF1ZXN0aW9uLg0KDQpSaWdodC4gSSB0aGluayBpdCB3b3VsZCBiZSBpbW1lbnNlbHkg dXNlZnVsIHRvIGhhdmUgYW4gYWN0dWFsIHJlcHJvZHVjZXIgDQpzbyBvbmUgY291bGQgc3R1 ZHkgdGhlIGlzc3VlIG1vcmUgY2xvc2VseSBvciBhdCBsZWFzdCBhIGJ1ZyByZXBvcnQsIA0K d2hpY2ggZGVzY3JpYmVzIHRoZSBpc3N1ZSBpbiBtb3JlIGRldGFpbCwgbGlrZSB0aGUgZXhh Y3QgY2lyY3Vtc3RhbmNlcyANCndoZW4gdGhpcyBjYW4gaGFwcGVuLg0KU28gZmFyIHRoaXMg aXMgbWVyZWx5IHRoZW9yZXRpY2FsLCByaWdodD8NCk9yIGRvIHdlIGhhdmUgYSBkb2N1bWVu dGVkIGluc3RhbmNlIG9mIHRoaXMgaGFwcGVuaW5nPw0KDQoNCj4gR2l2ZW4gdGhhdCB0aGVz ZSBidWdzIGFyZSBnb2luZyB0byBiZSB1dHRlciBiYXN0YXJkcyB0byByZXByb2R1Y2UsIGFu ZA0KPiB5b3UgY2FuIGJlIHN1cmUgdGhhdCB3ZSdsbCBoYXZlIGVub3VnaCBkaXZlcnNpdHkg aW4gaW5zdGFsbGVkIHN5c3RlbXMNCj4gdGhhdCBzb21lIHBlb3BsZSBhcmUgZ29pbmcgdG8g bWFuYWdlIHRvIGJlIHN1ZmZpY2llbnRseSB1bmx1Y2t5LCBpdA0KPiB3b3VsZCBiZSBuaWNl IHRvIGtub3cgdGhlIHNvcnQgb2YgZGFtYWdlIHdlIG1pZ2h0IGV4cGVjdC4NCj4gDQo+IEl0 IHN0cmlrZXMgbWUgdGhhdCB3ZSBvdWdodCB0byBiZSBhYmxlIHRvIHNjcmVlbiBvdXIgb3du IHJlcG9zIGZvcg0KPiBwYWNrYWdlcyB0aGF0IGNvdWxkIGJlIGFibGUgdG8gdGlja2xlIHRo aXMgYnVnLiBUaGF0IHdvdWxkIGdpdmUgdXMgdGhlDQo+IGNoYW5jZSB0byBsb29rIGF0IHdo YXQgc29ydHMgb2YgZmlsZXMgd2UgbWlnaHQgcmVhbGlzdGljYWxseSBleHBlY3QgdG8NCj4g YmUgY2xvYmJlcmVkLCBpdCBzaG91bGQgZ2l2ZSBzb21lIGluZGljYXRpb24gb2YgaG93IG1h bnkgcGFja2FnZXMgd2UNCj4gc2hvdWxkIGV4cGVjdCB0byBiZSBhYmxlIHRvIHRyaWdnZXIg dGhpcywgYW5kIGtub3dpbmcgdGhpcyBtaWdodCBzdWdnZXN0DQo+IHBsYXVzaWJsZSB3b3Jr LWFyb3VuZHMuDQo+IA0KPiBPZiBjb3Vyc2UsIHRoYXQgZG9lc24ndCBoZWxwIHdpdGggcGFj a2FnZXMgZnJvbSB0aGlyZC1wYXJ0eSByZXBvcywNCj4gaW5jbHVkaW5nIG91ciBkb3duc3Ry ZWFtcywgYnV0IGF0IHByZXNlbnQgd2Ugc2VlbSB0byBiZSBkaXNjdXNzaW5nIHRoaXMNCj4g d2l0aCB2ZXJ5IGxpdHRsZSBoYXJkIGRhdGEuDQo+IA0KPiBJdCBvY2N1cnMgdG8gbWUgdGhh dCBvbmUgY291bGQgbG9zZSBxdWl0ZSBhIGZldyBmaWxlcyBvbiB0aGUgYXZlcmFnZQ0KPiBE ZWJpYW4gaW5zdGFsbCAoaWYgdGhleSB3ZXJlIHNlbGVjdGVkIGF0IHJhbmRvbSkgd2l0aG91 dCBldmVuIG5vdGljaW5nLA0KDQpTaG91bGRuJ3QgZGVic3VtcyBiZSBhYmxlIHRvIGRldGVj dCBzdWNoIG1pc3NpbmcgZmlsZXMgKGF0IGxlYXN0IGZvciB0aGUgDQp2YXN0IG1ham9yaXR5 IG9mIHBhY2thZ2VzIHdoaWNoIHNoaXAgYSBtZDVzdW1zIGZpbGUpLg0KDQpJIHJ1biB0aGF0 IHNlbWktcmVndWxhcmx5IG9uIGFsbCBvZiBteSBzeXN0ZW1zLCBtb3N0IG9mIHRoZW0gYXJl IA0KL3Vzci1tZXJnZWQgYW5kIEkgaGF2ZW4ndCBub3RpY2VkIGFueSBtaXNzaW5nIGZpbGVz IHlldCB3aGljaCBJIGNvdWxkIHRyYWNlDQoNCj4gd2hlcmVhcyBhIHZlcnkgZmV3IGZpbGVz IHdvdWxkIHJlbmRlciBzeXN0ZW1zIHVuYm9vdGFibGUsIHNvIGtub3dpbmcgYQ0KPiBiaXQg bW9yZSBhYm91dCB3aGljaCBmaWxlcyBhcmUgcmVhbGlzdGljYWxseSBhdCByaXNrIHdvdWxk IGJlIHZlcnkNCj4gaGVscGZ1bCBpbiB1bmRlcnN0YW5kaW5nIHRoZSBzZXZlcml0eSBvZiB0 aGUgcHJvYmxlbS4NCj4gDQo+IElmIGFueW9uZSdzIGdvdCBnb29kIGlkZWFzIGFib3V0IGhv dyB0byBnYXRoZXIgdGhpcyBpbmZvcm1hdGlvbiwgSSdtDQo+IHZlcnkgaGFwcHkgdG8gaGVs cCB3aXRoIHRoZSBlZmZvcnQgdG8gZG8gc28uDQoNCkknZCBiZSBtb3JlIHRoZW4gaGFwcHkg dG8gaGVscCBoZXJlIGFzIHdlbGwuDQoNClJlZ2FyZHMsDQpNaWNoYWVsDQoNCg0K

    --------------BOKxOjZV2kYE9dtFLhiHR0cP--

    -----BEGIN PGP SIGNATURE-----

    wsF5BAABCAAjFiEECbOsLssWnJBDRcxUauHfDWCPItwFAmGXtrgFAwAAAAAACgkQauHfDWCPItzK dQ/8DMWddf4rywTh9dsZKYvOgTBt+uabgl5zIw1KFuDQJ1ndEr5OGWI8pkvMk6T3Z/03ue/TWYkX BeMNlgYOGyOyfEx30b0X4z2et1Ep3oWRYDvQCkfN6rQbZXYbqEFe1BiwzlpsxGJwrAosy6G9guJY a2MMfswwmnQzs4t6bJjUC3zwkedTqPhF/Vy7bZnM/Lf1xGWPKRL18Icm7MtHoTSv2wdmqF13F/KP +baKQjiTb38hHLAnB7q2BANL2bPPR0XNWBLDP8vb9dxtOOfuZZqtqkcAdE+EMjEMINy3PkziWliT TtG6tw0PnsAS3QXx+nOIGsggVt3AgOSJx6FlWI6WCyCGpJR1dqwfaDIF2+pzAtT0E9KSDckvn1ca zq1hrdIWtn7DgRwXQ3vaZoWLGU3PHShvtsllUWYJEqcFHH7H0TzDmh55C+AwEuN21BamrJw7Yy0F QMftB0zPU7Iy4ZNbMViT+3fJyuv8TxLkctukDYpMe5lUAK8JWde2pkIsR6dbu27BILO2+qhJNbvm mzfkvymYOmfgGLjavWdSvyp9g6YiRrLBdLvP0kjp42OElxDk+ntCb7dTcb+Mb0eCry9r4gxNqa7J xkBZoAp1VBFdWVYZ5ClQhvqSs3k6Xwrm2JegH17FBwgd2Jc1YljX0uYxCBNyI9iwtchA91KfDWrt pdU=
    =L3JP
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marc Haber@21:1/5 to All on Fri Nov 19 20:10:05 2021
    On Fri, 19 Nov 2021 11:58:39 +0100, Philip Hands <phil@hands.com>
    wrote:
    Given that these bugs are going to be utter bastards to reproduce, and
    you can be sure that we'll have enough diversity in installed systems
    that some people are going to manage to be sufficiently unlucky, it
    would be nice to know the sort of damage we might expect.

    It strikes me that we ought to be able to screen our own repos for
    packages that could be able to tickle this bug. That would give us the
    chance to look at what sorts of files we might realistically expect to
    be clobbered, it should give some indication of how many packages we
    should expect to be able to trigger this, and knowing this might suggest >plausible work-arounds.

    This is one of the cases where I wish that Debian would be a more
    centrally organized project. Red Hat or SuSE would just fix their
    package management and go on with their business.

    It's a pity that we have actually THINK about alternatives to that
    trivial and obvious approach because we leave our core package
    maintainers too much freedom to stall.

    Grüße
    Marc
    --
    -------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " |
    Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sven Joachim@21:1/5 to Michael Biebl on Fri Nov 19 20:20:02 2021
    On 2021-11-19 15:37 +0100, Michael Biebl wrote:

    On 19.11.21 11:58, Philip Hands wrote:
    Ansgar <ansgar@43-1.org> writes:

    * doing this will, in a non-negligible number of cases, trigger the
    bug to manifest on systems where that package is upgraded from a
    version where the move had not taken place to one where it has.

    Why do you claim that?

    Given packages already did such moves in the last years and you claim
    this happens in a non-negligible number of cases, could you please
    point to some examples where this already happens in practice?
    My understanding is that in order to trigger this bug you need at
    least
    to both move a file from one place to the other, and also to rename the
    package that contains that file or move ownership to another package.
    I suspect that you might also need to be unlucky with the order that
    apt/dpkg decides to do the installation and, depending upon how far
    apart the move and the rename happens, also unlucky with your choice of
    from and to versions of the packages in question.

    Right. I think it would be immensely useful to have an actual
    reproducer so one could study the issue more closely or at least a bug report, which describes the issue in more detail, like the exact circumstances when this can happen.
    So far this is merely theoretical, right?
    Or do we have a documented instance of this happening?

    #953562 seems to be such an instance.

    Cheers,
    Sven

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Fri Nov 19 20:30:01 2021
    "Marco" == Marco d'Itri <md@linux.it> writes:

    Marco> On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:
    >> Are you seriously claiming that that phenomenon is not a
    >> severity:critical bug?
    Marco> I am seriously claming that whatever you are referring to, if
    Marco> true, is such a contrived example that does not actually
    Marco> happen in real life (or at least, it does not happen
    Marco> frequently enough).

    It was convincing enough to convince me, the TC and several other
    participants in the discussion.

    --Sam

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Gevers@21:1/5 to Marco d'Itri on Fri Nov 19 20:50:01 2021
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qLJT5lJRzf1bkurSlas2lHL7wUNXZFNMT
    Content-Type: text/plain; charset=utf-8; format=flowed
    Content-Language: en-US
    Content-Transfer-Encoding: quoted-printable

    Hi

    On 18-11-2021 22:44, Marco d'Itri wrote:
    On Nov 18, Zack Weinberg <zack@owlfolio.org> wrote:

    Are you seriously claiming that that phenomenon is not a severity:critical bug?
    I am seriously claming that whatever you are referring to, if true, is
    such a contrived example that does not actually happen in real life
    (or at least, it does not happen frequently enough).

    I'm thinking of bug #972936 and bug #953562. Did I remember those
    wrongly as being an example of this?

    Paul


    --qLJT5lJRzf1bkurSlas2lHL7wUNXZFNMT--

    -----BEGIN PGP SIGNATURE-----

    wsB5BAABCAAjFiEEWLZtSHNr6TsFLeZynFyZ6wW9dQoFAmGX/agFAwAAAAAACgkQnFyZ6wW9dQoz XQf+LmiXlCtY9V7xAAGXLX4xC3osgeTeW63yi0+rTJk9UxgaecUWKraFem4sWS/EsCR/t9fgEFyz /XdD5hRgAZoDTLdvrN0hCNf+j6VJ/BxmSTllvdyKIHMOt/h41daCEpZj/pbJ5guPMXec29u64XlE R/xp7xtBJOkPvmqPsysJwye2fCKHkoHAqvt7DMNXKYPnXG67jkzwbHm+q+k5RyyN6V5k8d++rTXh 6eoQmihEjoRw+Xz05rWC0LljnRi+rTBeOEWFtlRveuwOxAmYeq9xnPkWXc7Ksmdbt1tMiA+phgMh 41SDdufwAx352yTc4cHnvVOcACRKRhGW+bPQFP5cSw==
    =O2Yl
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Richard Laager on Fri Nov 19 20:50:01 2021
    Richard Laager <rlaager@debian.org> writes:

    Is this particularly hard to fix, though?

    Can't we just do something like the following pseudo-code:

    [...]

    (I've used /bin -> /usr/bin as an example, but the canonicalization must
    deal with the other merged paths too.)

    The second bit ensures that all new operations write canonicalized paths (e.g. /usr/bin/foo) into the database regardless of whether the .deb
    shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in sync with the filesystem moving forward.

    The first bit performs a one-time canonicalization of paths in the
    existing database. An on-disk flag ensures that this is run only once
    (for performance reasons only, because it's already idempotent). This corrects the existing database-filesystem mismatch.

    The one-time canonicalization can be removed once non-usrmerged systems
    are no longer supported. The second bit needs to live (probably a lot) longer, until it's no longer reasonable to install a .deb containing non-usrmerged paths (which might be old or from a third party).

    Am I missing something here?

    I also don't understand why this isn't the correct solution. It seems
    obvious to me that we should simply teach dpkg about path aliases and have
    it update both its internal state and its processing of new packages to canonicalize paths so that it has an accurate representation of the world,
    and then ensure dpkg is upgraded first.

    Are we missing something? If not, what is blocking this solution? Is it simply a matter of someone digging into dpkg's code sufficiently to put
    these changes in the correct location? Or is there some other issue?

    It seems like a huge waste of resources to me to do archive-wide package inspection to try to find patterns that might cause problems, and to ask
    all maintainers to remember this rather obscure rule, when we could just
    fix dpkg to make the problem go away.

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Luca Boccassi on Fri Nov 19 21:50:02 2021
    On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
    On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
    Luca Bocassi wrote:
    ...
    nobody has actually seen [the file disappearance bug]
    happen, to the best of my knowledge.

    I already explained why that doesn't prove the bug is a non-issue.
    To the contrary; it means there is an enormous installed base of
    systems where the bug is latent, waiting to cause problems under
    conditions which we can reasonably expect to occur shortly after
    the release of bookworm.

    Why would the release of bookworm make any difference?

    Up until the release of bookworm, all Debian packages must be
    constructed on the assumption that they _might_ be unpacked on a
    system that has not yet been converted to merged /usr. Particularly
    for priority-required packages, this means that no one will be moving
    files from /bin, /lib, etc to /usr in the bookworm cycle.

    Post-bookworm, if nothing changes, that assumption will no longer be
    in force, and people who maintain packages that install files into /
    will want to simplify their packaging by installing everything in /usr
    instead. If they also want to change the binary packages that ships
    some of those files at any point in the same release cycle -- kaboom.

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zack Weinberg@21:1/5 to Zack Weinberg on Fri Nov 19 22:00:02 2021
    On Fri, Nov 19, 2021, at 3:23 PM, Zack Weinberg wrote:
    On Thu, Nov 18, 2021, at 8:06 PM, Luca Boccassi wrote:
    On Thu, 2021-11-18 at 16:23 -0500, Zack Weinberg wrote:
    Luca Bocassi wrote:
    ...
    nobody has actually seen [the file disappearance bug]
    happen, to the best of my knowledge.

    I already explained why that doesn't prove the bug is a non-issue.
    To the contrary; it means there is an enormous installed base of
    systems where the bug is latent, waiting to cause problems under
    conditions which we can reasonably expect to occur shortly after
    the release of bookworm.

    Why would the release of bookworm make any difference?

    Up until the release of bookworm, all Debian packages must be
    constructed on the assumption that they _might_ be unpacked on a
    system that has not yet been converted to merged /usr. Particularly
    for priority-required packages, this means that no one will be moving
    files from /bin, /lib, etc to /usr in the bookworm cycle.

    Post-bookworm, if nothing changes, that assumption will no longer be
    in force, and people who maintain packages that install files into /
    will want to simplify their packaging by installing everything in /usr instead. If they also want to change the binary packages that ships
    some of those files at any point in the same release cycle -- kaboom.

    After having caught up on the thread, I see that the conditions required
    to trigger the bug are somewhat more complicated, but the point remains: particularly for the packages where a lost file could render the system unbootable, changes that would trigger the bug have been deferred until post-bookworm, *and* we can reasonably expect the maintainers of those
    packages to *want* to make changes with a high risk of triggering the bug.
    I imagine the coreutils maintainers are looking forward to getting rid
    of their list of programs that go in /bin, and the extra debian/rules
    logic to go with it, for instance (https://sources.debian.org/src/coreutils/8.32-4.1/debian/rules/#L16).

    zw

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Timo =?utf-8?Q?R=C3=B6hling?=@21:1/5 to All on Sat Nov 20 01:00:02 2021
    * Russ Allbery <rra@debian.org> [2021-11-19 11:46]:
    I also don't understand why this isn't the correct solution. It seems >obvious to me that we should simply teach dpkg about path aliases and have
    it update both its internal state and its processing of new packages to >canonicalize paths so that it has an accurate representation of the world, >and then ensure dpkg is upgraded first.
    +1

    Are we missing something? If not, what is blocking this solution? Is it >simply a matter of someone digging into dpkg's code sufficiently to put
    these changes in the correct location? Or is there some other issue?
    AFAIR, Guillem has not commented on the solution when it was brought
    up on d-devel [1], so I'm not sure if he would accept patches, let
    alone support anyone working on this. I'd be willing to assist him
    and contribute code, but I'm not going to spend a good chunk of my
    spare time just to see me work summarily rejected.

    Cheers
    Timo

    [1] https://lists.debian.org/debian-devel/2021/08/msg00438.html


    --
    ⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
    ⣾⠁⢠⠒⠀⣿⡁ │ Timo Röhling │
    ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
    ⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯

    -----BEGIN PGP SIGNATURE-----

    iQGzBAABCgAdFiEEJvtDgpxjkjCIVtam+C8H+466LVkFAmGYOBoACgkQ+C8H+466 LVms2gwAzR4MFB9GDm+liSYc2XyWkhWgz3enANFErqQll8xy3PKx/1Y9/mx0suGk fDzdPjxXyzACPsFF4peM4dz6z+GWMqz0kLwJ2rwVsuRhCKgz2qtws+yyF6GBVGM9 ns2ywFJxDIYoqCQ8ykgp1eJMZqKXcq3t8aQxJTqnu0LKeQ+kk9diDjSNHzYMoGKN 039tQtTzGHRF1WPPpAEuRF2XnocARs7wg57cAqiX4lFSHdtRz1PK3dQbsDCAd8Y5 R5gSOrO0Jo+bzEtP9TuCE6qFw2U/lWrYvfM8gt+l5SSl+ZvqfdVozYay5OOYVP8T JTQ0crvnuWktNDEx6A1AwBc8IdCp6F1dyXZb7FaZUK5
  • From Russ Allbery@21:1/5 to Simon Richter on Sat Nov 20 03:00:02 2021
    Simon Richter <sjr@debian.org> writes:

    I think the main blocker at the moment is that the dpkg change *also*
    has the potential to break a lot of things if it isn't carefully
    designed.

    I think you can simplify that problem space considerably because we know exactly what symlinks we care about and we don't intend to support going backwards (in other words, once you have merged /usr, it's not supported
    to turn /bin, etc., back into regular directories again).

    Therefore, dpkg can be configured with the list of the symlinks that
    *must* exist for it to be willing to operate on the system. Once it has
    been put into merged-/usr mode (which includes updating its state database
    to reflect the new canonical paths of all installed files), if those
    symlinks do not exist or do not have the correct targets, the system state
    is not safe and dpkg can (and probably should) abort and refuse to operate
    on that system.

    That modifies the problem of systems in various weird configurations from "weird things may happen" to "dpkg will (safely) refuse to run" and we can provide a tool to fix such a system by creating the correct symlinks.
    (This tool may well just be the existing usrmerge tool.) Getting into
    that state will be hopefully rare if we structure the upgrade process so
    that first one converts the system to merged-/usr if this has not already
    been done, then upgrades dpkg, then tells dpkg to switch into merged-/usr
    mode, and then safely upgrades the remaining packages.

    From dpkg's perspective, we now (kind of) change the policy for directory-vs-symlink conflicts, which dpkg currently resolves in favour
    of the directory.

    I don't think we should be doing anything of the sort. I think we should introduce a new concept specifically for this case and have the remappings
    of files for merged-/usr, once dpkg is in merged-/usr mode, take
    precedence over any rules like that. We do not want dpkg to be applying
    some policy or heuristic about directory vs. symlink conflicts for
    symlinks related to merged-/usr. We know exactly what we want the rule
    and behavior to be. That sort of remapping should happen way before any
    other dpkg analysis of the package or system state.

    In other words, I argue that once dpkg is in merged-/usr mode, it should
    treat a package with files in /bin exactly as if the files were in
    /usr/bin, at an earlier stage than when it starts looking at the file
    system.

    We're not trying to create a generalized feature here. Merged-/usr has a
    very specific definition, with very specific and clear behavior that we
    need from the package manager. Trying to generalize this is going to make
    a mess and make it much harder to reason about the behavior or test it.

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Richter@21:1/5 to Sam Hartman on Sat Nov 20 02:40:02 2021
    Hi,

    On Fri, Nov 19, 2021 at 12:28:56PM -0700, Sam Hartman wrote:

    But we could you know fix dpkg:-)
    I'm sure that will be painful at the political level, but personally I
    think it needs doing.

    I think the main blocker at the moment is that the dpkg change *also*
    has the potential to break a lot of things if it isn't carefully
    designed.

    The problem space right now is huge:

    - for upgraded systems, the system could be pre-usrmerge, or the
    conversion could have been performed by any usrmerge version that
    ever existed (and the usrmerge package could have been deinstalled
    and reinstalled since, so the patches to the conffiles it performs
    could be inconsistently applied).
    - for upgraded systems, any version of usrmerge since the last stable
    release could be installed
    - for upgraded systems, unless the usrmerge package is discontinued,
    it could be upgraded at any point during the update.
    - for freshly installed systems, dpkg is first invoked on a
    non-usrmerged tree, and should convert the installation as soon as it
    becomes aware that conversion is desired (which might not be
    something we want to hardcode in dpkg, but possibly configure from a
    separate, Debian-specific package like base-files).
    - for freshly installed systems, initial unpack might be controlled by
    debootstrap from stable, which is usrmerge aware, so some of its
    workarounds may be active, and we need to properly capture this
    state, too.
    - freshly installed systems might be created with cdebootstrap or
    multistrap, and could be generated in --foreign mode
    - the dynamic loader is specified to be on the root filesystem, and that
    is part of the ABI standard and not under our control, so we cannot get
    a conflict-free dpkg database in bookworm.

    From dpkg's perspective, we now (kind of) change the policy for directory-vs-symlink conflicts, which dpkg currently resolves in favour
    of the directory. That was done in ancient times because it was somewhat
    easy to do this in a package by accident. Not sure if that still
    applies, but if it does, then we need to formulate the new policy so
    that we don't catch a regression here (which is why my original idea to
    just ship the symlinks in base-files is a bad idea).

    From the way dpkg and the Debian Policy are initially designed, it is
    obvious to me that they started out as specification documents, and only
    when these were hammered out, they were poured into code and rules, and
    we've been operating from that stable base since, with two exceptions
    where technical facts were created without updating Policy accordingly,
    and which has both times been controversial.

    My interpretation of the "political" situation is that we need this to be a group effort, because no single person has the necessary time to do a
    thorough enough job that they would feel comfortable signing off on it. I'd
    be reluctant to do so even if I had three months of uninterrupted time -- that's the level of complexity I see here, and I suspect Guillem feels the same.

    So I believe the way forward will be writing a specification and giving it enough eyeballs to identify problem cases. Finding the adversarial example
    for the status quo was easy, since I had to find just one -- the goal now
    is to get to a state where we don't find such an example easily anymore.

    The specification should explain how the transition can be finished with
    the bookworm release for all the different ways packages can be installed,
    and it should describe the necessary code changes and new test cases to get full coverage, and the document should be signed off by multiple people who jointly take responsibility, to avoid placing all of that on a single
    person -- because that's what's impeding progress right now, that no one
    wants to be that person or even feels able to.

    The description of the problem space I put above is likely incomplete and overzealous at the same time (for example, I haven't checked how different
    the previous usrmerge packages are, and which of them ever made it into a stable release), so this is also just a starting point. I'll happily spend
    time refining this or any other proposal, but I'm too time-constrained to
    be the main driving force here -- this is not my day job, after all.

    Simon

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Simon Richter on Sat Nov 20 03:40:01 2021
    Simon Richter <sjr@debian.org> writes:

    Bootstrapping a new Debian system works by unpacking Essential packages
    with ar+tar, i.e. not using dpkg. These packages will always be unpacked
    to the file names listed inside the packages.

    Well, bootstrapping a new Debian system involves running a tool that
    bootstraps a new Debian system. I think you're constraining the problem
    too much.

    It's a nice property that everything on the system comes straight from a
    Debian package, but it's not a strict requirement, nor is it currently generally the case (maintainer scripts do things, for instance). Those symlinks are very special from a dpkg perspective; dpkg needs to refuse to
    mess with them even when upgrading a package that provides them, which
    would mean some irritating special-casing I suspect. It's not clear to me
    that shipping them as tar members of a package is the right way to go, as opposed to creating them separately as part of early system configuration.

    Dpkg has an elaborate per-file tracking to recover from errors, and this change introduces several new states and transitions,

    Why? I am not convinced this is true, apart from the obviously-needed
    one-time conversion of the state database.

    It involves a new filter, applied whenever a package is installed, that
    changes all the paths in the package to their merged-/usr paths, something
    that seems like it should be done before any other package processing.
    Once that's done, why would any new states or transitions be required?

    You could convince me that writing the filter is complicated because there
    may not be one place in dpkg where you can currently put such path
    rewriting since dpkg probably wasn't designed with that operation in mind.
    But it's going to be harder to convince me that there are new states or transitions; that smells like an over-complicated design.

    That is the goal, yes, but this is a massive undertaking.

    I'm still not convinced of this.

    We already have a quick hack for usrmerge support in dpkg, which is the
    check that allows moving a file from /bin to /usr/bin within the same
    package without the file being deleted in the end (because deletions are processed last for obvious reasons), and the obscure rule is precisely
    the limitation of this hack.

    This already sounds conceptually more complicated than just solving the
    problem properly, with the caveat as mentioned above that writing the
    necessary filter may be difficult.

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Richter@21:1/5 to All on Sat Nov 20 03:20:01 2021
    Hi,

    (I've used /bin -> /usr/bin as an example, but the canonicalization must deal with the other merged paths too.)

    /lib and /lib64 are a bit special, because they are part of the ELF ABI,
    and any manipulation of these paths needs to be atomic.

    Bootstrapping a new Debian system works by unpacking Essential packages
    with ar+tar, i.e. not using dpkg. These packages will always be unpacked
    to the file names listed inside the packages.

    So, new systems will either need a package that ships the /lib ->
    usr/lib symlink in the Essential set, or we need to ship the dynamic
    linker in /lib, which would be a conflict during unpacking.

    The second bit ensures that all new operations write canonicalized paths (e.g. /usr/bin/foo) into the database regardless of whether the .deb shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in sync with the filesystem moving forward.

    This loses the information about the original path, so this would be a
    trapdoor operation that can not be undone without affected packages
    going into reinstallation-required state (and we probably couldn't even identify those).

    Ideally, I'd rather not hardcode the list of top-level symlinks into
    dpkg, because they might be architecture-specific, or change over time,
    so there should be a mechanism to add and remove transformations.

    The one-time canonicalization can be removed once non-usrmerged systems
    are no longer supported. The second bit needs to live (probably a lot) longer, until it's no longer reasonable to install a .deb containing non-usrmerged paths (which might be old or from a third party).

    My expectation would be that this code will be here to stay, because it
    will be required for bootstrap.

    Are we missing something? If not, what is blocking this solution? Is it simply a matter of someone digging into dpkg's code sufficiently to put
    these changes in the correct location? Or is there some other issue?

    If it were that simple, someone would have done that already. Dpkg has
    an elaborate per-file tracking to recover from errors, and this change introduces several new states and transitions, last but not least
    because we also have filters and diversions as transformations on the
    database -- but the usrmerge transformation is supposed to be orthogonal
    to that.

    It seems like a huge waste of resources to me to do archive-wide package inspection to try to find patterns that might cause problems, and to ask
    all maintainers to remember this rather obscure rule, when we could just
    fix dpkg to make the problem go away.

    That is the goal, yes, but this is a massive undertaking. We already
    have a quick hack for usrmerge support in dpkg, which is the check that
    allows moving a file from /bin to /usr/bin within the same package
    without the file being deleted in the end (because deletions are
    processed last for obvious reasons), and the obscure rule is precisely
    the limitation of this hack.

    I've stumbled across this hack while investigating if it was possible to
    "just" fix dpkg. :P

    Simon

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Russ Allbery on Sat Nov 20 18:30:01 2021
    Russ Allbery <rra@debian.org> writes:

    Well, bootstrapping a new Debian system involves running a tool that bootstraps a new Debian system. I think you're constraining the problem
    too much.

    It's a nice property that everything on the system comes straight from a Debian package, but it's not a strict requirement, nor is it currently generally the case (maintainer scripts do things, for instance). Those symlinks are very special from a dpkg perspective; dpkg needs to refuse
    to mess with them even when upgrading a package that provides them,
    which would mean some irritating special-casing I suspect. It's not
    clear to me that shipping them as tar members of a package is the right
    way to go, as opposed to creating them separately as part of early
    system configuration.

    Having slept on this, let me make this problem concrete.

    I'm going to make the following assumptions:

    * We have some mechanism to put dpkg into what I've been calling
    merged-/usr mode. In this mode, it pre-filters all input paths from
    whatever source (including arguments to dpkg-divert,
    update-alternatives, etc.) and canonicalizes them in a very specific
    way: the directories that become symlinks in merged /usr are replaced in
    each path with their canonical paths. So /bin/ls becomes /usr/bin/ls,
    /lib64/ld-linux-x86-64.so.2 becomes /usr/lib64/ld-linux-x86-64.so.2, and
    so forth.

    * When bootstrapping a new Debian system, we want to put dpkg into
    merged-/usr mode as early as possible.

    * If dpkg is in merged-/usr mode, the first thing it does is checks the
    file system on which its operating and ensures that the expected
    symlinks already exist. If they do not, it aborts because operating on
    that file system is unsafe. (Leaving aside for the moment whether there
    should be some -force option, etc.)

    This produces a bootstrapping problem: all dynamic binaries on, say, an
    amd64 system refer to /lib64/ld-linux-x86-64.so.2 [1]. Therefore, before
    the first binary is run from within the context of the newly-installed
    system, either that path must exist as-is (which we don't want because we
    want to create a merged-/usr system where it belongs in /usr/lib64) or the symlink from /lib64 to /usr/lib64 must already exist.

    I think it's fairly obvious that we don't want a long-term design in which
    the libc6 package has to continue to ship /lib64/ld-linux-x86-64.so.2, we originally unpack that file in that path during bootstrap, and then
    something has to come along later and move it to /usr/lib64 and create a symlink. This has numerous drawbacks: weird special cases people have to remember, not being able to reconcile the contents of packages with their canonical paths in the long run, a window where we have to do file system surgery atomically, etc. Instead, we want to live in a world in which
    libc6 can ship /usr/lib64/ld-linux-x86-64.so.2, dpkg sees it as shipping
    that path, but everything works fine during bootstrap because the /lib64
    to /usr/lib64 symlink is already established before we have to execute
    binaries in the new environment. (Obviously it may be some time before we actually change the contents of the libc6 package; that's fine, the point
    of adding a path filter to dpkg is that we can take our time on that.)

    So, assuming we have a libc6 package that contains either /usr/lib64/ld-linux-x86-64.so.2 or /lib64/ld-linux-x86-64.so.2, how do we bootstrap this system?

    I can see a few approaches:

    * Put the required symlinks into base-files. This preserves the nice
    property that every file on the system comes from and is registered as
    belonging to a Debian package (which we don't fully meet but which we
    always aspire to). I haven't studied all the various tools for
    bootstrapping a Debian system, but I presume that unpacking base-files
    is the first thing that any of them do. This will therefore lay down
    the symlinks from the start and it won't matter what path under which
    the libc6 package ships the files if it's extracted over the resulting
    file system with ar+tar.

    The drawback here is that dpkg is going to rewrite all paths like /lib64
    to /usr/lib64, which would naively *also* apply to the base-files
    package when it looks at that package, but that can't be allowed because
    now we're back to the situation where dpkg's state database is
    inconsistent with the file system and dpkg thinks that base-files
    contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

    I think in this approach there would need to be some special-case code
    directly in dpkg that recognizes the usrmerge symlinks (and only
    specifically those symlinks) as special and preserves them as-is in the
    installed package database. (I think it's probably better to
    special-case the specific symlink than to special-case base-files the
    package.) We will then need rules that base-files must always contain
    those symlinks and they can't move between packages, etc., (or the
    system is likely to break horribly), but "base-files cannot do weird
    things" is probably already a constraint that we have.

    Another big drawback of this approach is that now we have to handle
    upgrades of the base-files package from older Debian installs very
    carefully. base-files should *not* be doing the usrmerge transition, so
    somehow we have to arrange for that transition to happen before
    base-files is upgraded. This may be doable, but feels rather messy;
    it's setting off my complexity radar.

    * Create a new essential package that contains these symlinks and that
    needs to be unpacked before any binaries are executed in the target file
    system. This has many of the advantages and drawbacks of the approach
    of putting the symlinks in base-files, but may make it easier to handle
    the upgrade problem. Ideally an upgrade would then involve installing
    usrmerge, letting it run, and then installing this new essential package
    so that it takes over ownership of those symlinks.

    This still feels kind of complex and messy to me, but maybe less so.

    * Create the symlinks directly in the bootstrapping script. In other
    words, after unpacking base-files, the bootstrapping script would
    directly create the required symlinks in the target file system, before
    unpacking any other package.

    This has the obvious drawback of moving things outside the packaging
    system and creating a new special case that every bootstrapping script
    needs to be aware of (and I know we have at least four or five that
    would need modifications). It has the advantage that the usrmerge
    symlinks are now not in the dpkg database and thus not subject to
    rewriting, and therefore won't need to be special-cased. However, that
    comes with the obvious disadvantage that they're not in the dpkg
    database, so for instance dpkg -S /lib wouldn't find that symlink unless
    it was added as some sort of dpkg-query special case (which doesn't seem
    like a great idea).

    The advantage of this approach is that it closely mimicks what's already
    happening now with the usrmerge package, and for which we therefore have
    a lot of existing experience.

    Presumably Fedora had the same problem with their bootstrapping and the
    RPM database. Does anyone know how they handled it?

    I realized when writing this that I've not previously dove into the
    details of the bootstrapping process and therefore don't know how the
    hand-off between packages manually installed with ar+tar to using dpkg in
    the target file system is handled. Are all the essential packages
    re-installed with dpkg again so that dpkg will have touched every package including base-files? Or does something write out the dpkg database files
    in /var/lib/dpkg/info for the manually-installed packages so that dpkg is already aware of them the first time it runs?

    If it's the latter, we're going to either need the same path rewriting in
    the process that writes /var/lib/dpkg/info, or we're going to need an
    explicit dpkg bootstrapping step the first time it runs, since some of
    those packages are going to ship files in /lib, etc., in their ar+tar representation but we want the database to show all of those files as
    installed in /usr/lib, etc. (with the exception of the /lib symlink as discussed above). That presumably means running the dpkg convert-to-merged-/usr code during bootstrap to convert the database.

    If it's the former, presumably we can just configure dpkg in merged-/usr
    before the first time it runs and then everything will be correct from the start, but we do have to figure out what piece of the bootstrapping
    process installs the necessary dpkg configuration saying to operate in merged-/usr mode. For the first two options above, putting it in
    base-files or in the new package that holds the usrmerge symlinks seems
    like an obvious choice (which would argue for making the merged-/usr dpkg configuration file a separate file, so that those packages can provide it without stomping on dpkg configuration that should be owned by the dpkg package), but again we have to carefully handle upgrades. For the third approach, it probably makes sense for the bootstrapping script to drop the configuration file in directly (and then, in the long run, use the
    oft-proposed but not-yet-implemented mechanism for packages to register
    their configuration files with dpkg that aren't shipped with the package
    so that the configuration file eventually ends up in the dpkg database as
    owned by dpkg).

    [1] amd64 used as an example; every architecture has some version of this
    same problem. I'm also using the dynamic loader as an example because
    I think it makes the problem most obvious and concrete, but I suspect
    there are other critical paths of this type that pose similar issues.
    Note that the exact list of paths that have to be symlinked is
    architecture-dependent (see directories_to_merge in convert-usrmerge).

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Gabor Gombas on Sun Nov 21 00:00:02 2021
    Gabor Gombas <gombasg@digikabel.hu> writes:

    If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to /usr/lib64/*", then this can easily be avoided.

    True, although I think you still want to ensure that dpkg never messes
    with those links. I don't think you want base-files to be able to drop
    one of those links from the package and cause it to be removed by dpkg,
    for instance.

    Maybe I'm borrowing trouble for a scenario that won't happen (and I think
    dpkg may have some special-casing of symlinks to directories anyway).

    Talking about "special casing" in dpkg is bothering me for a while. And
    there is a relatively simple way to avoid any kind of special casing:
    move the information out to a configuration file (which would _not_ be a conffile) - and now the code has no special casing, just
    configuration-driven logic.

    All that I mean by special-case is that I think it would be a mistake to
    try to shoehorn support for merged-/usr into some pre-existing dpkg
    concept. The merged-/usr links are special relative to other files that
    dpkg is managing and need to be treated with special care. We should add
    a new concept to dpkg for these sorts of links.

    I agree with making the exact set of links configurable. This also allows handling the architecture-specific links in a clean way.

    This new configuration file could be shipped by base-files itself, to
    ensure it does not get out of sync with the filesystem structure shipped
    in base-files. Then base-files' postinst would more or less need to
    include the current usrmerge package, plus whatever is needed to convert dpkg's database.

    I don't think we want base-files to take over the logic of usrmerge.

    usrmerge is both complex (in part because it relies on Perl) and not 100% hardened against weird issues, being interrupted in the middle of running,
    etc. That's all fine for what it's doing as a one-time migration tool
    that's run with care during an upgrade when the administrator already
    knows Big Things are happening to the system. Putting the logic into base-files makes it more of a routine thing and I think it should be a lot
    more hardened if we do that (and I don't think that's a good use of
    anyone's time). We ideally want usrmerge to be a tool each system runs
    once, under controlled circumstances, and then never again (and isn't
    needed at all for bootstrapping).

    We have a migration strategy that's working. Let's stick with that
    migration strategy without messing with it and instead just add a dpkg
    database conversion step to the end of that migration.

    I do agree that something needs to handle installing the configuration
    file and laid out some options for that in another message.

    This new configuration file would not be consumed by dpkg directly when installing packages, but only when the database conversion is called,
    and dpkg would keep an internal list of the rewriting rules which are
    active. Doing so would allow enforcing rewriting rules can only be added
    but never removed, which would avoid potential issues if base-files gets downgraded.

    This is a very good idea. I was starting to think along those lines, but hadn't taken it quite far enough. That would make it akin to dpkg --add-architecture, which feels like a good model (although as you mention
    I don't think we want --remove-architecture).

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gabor Gombas@21:1/5 to Russ Allbery on Sat Nov 20 23:20:02 2021
    Hi,

    On Sat, Nov 20, 2021 at 09:22:27AM -0800, Russ Allbery wrote:

    The drawback here is that dpkg is going to rewrite all paths like /lib64
    to /usr/lib64, which would naively *also* apply to the base-files
    package when it looks at that package, but that can't be allowed because
    now we're back to the situation where dpkg's state database is
    inconsistent with the file system and dpkg thinks that base-files
    contains some nonsensical /usr/lib64 to /usr/lib64 symlink.

    If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to /usr/lib64/*", then this can easily be avoided.

    I think in this approach there would need to be some special-case code
    directly in dpkg that recognizes the usrmerge symlinks [...]

    Talking about "special casing" in dpkg is bothering me for a while. And
    there is a relatively simple way to avoid any kind of special casing:
    move the information out to a configuration file (which would _not_ be a conffile) - and now the code has no special casing, just
    configuration-driven logic.

    This new configuration file could be shipped by base-files itself, to
    ensure it does not get out of sync with the filesystem structure shipped
    in base-files. Then base-files' postinst would more or less need to
    include the current usrmerge package, plus whatever is needed to convert
    dpkg's database.

    This new configuration file would not be consumed by dpkg directly when installing packages, but only when the database conversion is called,
    and dpkg would keep an internal list of the rewriting rules which are
    active. Doing so would allow enforcing rewriting rules can only be added
    but never removed, which would avoid potential issues if base-files gets downgraded.

    Gabor

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Russ Allbery on Mon Nov 22 13:30:01 2021
    On Sat, 20 Nov 2021 at 14:54:30 -0800, Russ Allbery wrote:
    That would make it akin to dpkg
    --add-architecture, which feels like a good model (although as you mention
    I don't think we want --remove-architecture).

    Instead of dpkg --add-architecture, which is a top-level "verb" like
    dpkg --remove or dpkg --install [1], probably a better thing to resemble
    would be the options that already exist to change dpkg's behaviour,
    and in particular --path-exclude and --path-include?

    Options to make path A an alias for path B would be a lot like --path-exclude in that they really only make sense in a configuration file in /etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
    have to exist as command-line options just like --path-exclude does.

    Here's a possible design (not fully thought out, and maybe wrong or unimplementable):

    --path-alias ALIAS=TARGET
    When unpacking or querying packages, paths below ALIAS will be
    transformed into the corresponding path below CANONICALIZED, where
    CANONICALIZED is the result of resolving TARGET relative to the
    directory containing ALIAS. This transformation is done before
    checking for file conflicts.

    ALIAS must be an absolute path.

    If the file list of any unpacked package contains a directory at
    ALIAS and/or CANONICALIZED, then ALIAS will be created as a symbolic
    link to TARGET, with its target directory created at CANONICALIZED.

    It is an error to have multiple --path-alias directives for
    the same ALIAS with different TARGET paths.

    This option should usually be set via /etc/dpkg/dpkg.cfg.d rather
    than as a command-line option.

    For example, a series of directives like path-alias=/bin=usr/bin
    can be used to implement the "/usr merge", causing /bin/bash to be
    unpacked as /usr/bin/bash while creating a symbolic link
    /bin -> usr/bin.

    With that, I believe a full implementation of merged-usr for all known architectures would be to install something like this, perhaps via
    base-files:

    # in /etc/dpkg/dpkg.cfg.d/merged-usr
    path-alias=/bin=usr/bin
    path-alias=/lib32=usr/lib32
    path-alias=/lib64=usr/lib64
    path-alias=/lib=usr/lib
    path-alias=/libo32=usr/libo32
    path-alias=/libx32=usr/libx32
    path-alias=/sbin=usr/sbin

    Rationale for TARGET being allowed to be relative: I believe the compat symlinks in merged-/usr are preferred in the form /bin -> usr/bin, to
    make sysroots (containers, chroots, btrfs root filesystem snapshots,
    etc.) easier to manipulate from outside.

    Rationale for ALIAS only being created if ALIAS and/or CANONICALIZED
    really exist in a package: we would probably prefer the multilib
    directories to be created "lazily", rather than having a /libo32 on
    systems that will never install a mips* package and a /libx32 on systems
    that will never install an x32 package. This is slightly nicer than the
    current implementation in convert-usrmerge and debootstrap, which creates
    those directories "eagerly" whenever the appropriate architectures are configured as either primary or foreign architectures.

    smcv

    [1] if dpkg had been designed with more recent CLI conventions in mind,
    I expect these "verbs" might have been subcommands like "dpkg remove",
    etc. as seen in apt/git/flatpak, which makes it a bit more obvious
    that only one is allowed

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Johannes Schauer Marin Rodrigues@21:1/5 to All on Mon Nov 22 17:00:01 2021
    Quoting Russ Allbery (2021-11-20 18:22:27)
    * Create a new essential package that contains these symlinks and that
    needs to be unpacked before any binaries are executed in the target file
    system. This has many of the advantages and drawbacks of the approach
    of putting the symlinks in base-files, but may make it easier to handle
    the upgrade problem. Ideally an upgrade would then involve installing
    usrmerge, letting it run, and then installing this new essential package
    so that it takes over ownership of those symlinks.

    This still feels kind of complex and messy to me, but maybe less so.

    * Create the symlinks directly in the bootstrapping script. In other
    words, after unpacking base-files, the bootstrapping script would
    directly create the required symlinks in the target file system, before
    unpacking any other package.

    This has the obvious drawback of moving things outside the packaging
    system and creating a new special case that every bootstrapping script
    needs to be aware of (and I know we have at least four or five that
    would need modifications). It has the advantage that the usrmerge
    symlinks are now not in the dpkg database and thus not subject to
    rewriting, and therefore won't need to be special-cased. However, that
    comes with the obvious disadvantage that they're not in the dpkg
    database, so for instance dpkg -S /lib wouldn't find that symlink unless
    it was added as some sort of dpkg-query special case (which doesn't seem
    like a great idea).

    The advantage of this approach is that it closely mimicks what's already
    happening now with the usrmerge package, and for which we therefore have
    a lot of existing experience.

    Please don't. Since 2014 it is possible to create a Debian chroot without taking care of the unpack order and without running any special setup beforehand that cannot be put into apt or dpkg itself (like creating /var/lib/dpkg/info which is not needed anymore since dpkg 1.20.0). One of my biggest motivations in writing mmdebstrap was to get rid of all the quirks that currently live in debootstrap, move them into apt and dpkg so that we end up with a declarative bootstrap process that is driven by the package contents, their metadata and maintainer scripts but does not need magic from the outside other than standardized interfaces. Requiring the merged-/usr setup to be done by the bootstrap script would be a step backwards. Think about the following points:

    * you create a Debian unstable chroot using snapshot.d.o -- now you have to
    somewhere put code that identifies that this is a chroot from the past and
    that has to decide whether that chroot should be merged-/usr or not. If
    merged-/usr setup was living somewhere in the Essential:yes package set
    the chroots would be created the right way automatically
    * same for derivatives -- now every bootstrap script has to learn which
    derivative defaults to merged-/usr, which ones do not and when they switched
    * we somehow need to convert installations of users that only run testing
    or unstable but never do full stable installation to merged-/usr -- one way
    to do that is to let one package in the Essential:yes set drive the
    conversion at which point the bootstrap script doesn't need to know about
    merged-/usr because the Essential:yes set knows how to set it up
    * think about it from a software engineering point of view: Debian as a
    component based software distribution should move specific functionality
    into its packages and global functionality into declarative metadata.

    Just as we are currently trying to reduce maintainer scripts and replace them by declarative approaches, we should also try to reduce the magic in our bootstrap scripts and move it either to our package managers or replace them by declarative solutions. We have been moving into the right direction during the last few years as apt and dpkg have picked up functionality I had earlier put into mmdebstrap (thanks a lot to guillem, DonKult and juliank for picking up my patches or writing the functionality themselves). It would mean a step backwards if all bootstrap scripts would have to carry the setup_merged_usr() function from debootstrap and then decide by some $magic when this function should be executed or not.

    Thanks!

    cheers, josch
    --==============)61198262470966361=MIME-Version: 1.0
    Content-Transfer-Encoding: 7bit
    Content-Description: signature
    Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmGbviEACgkQ8sulx4+9 g+EXXA//QNRkp9OC9704zkKOYt6TIHbpTpXpNKINqguSCrAQpdxgQUBVDY23eMFw FKK5qB9FQeV7MTe2paEmYzdqWTOG7PQ6jPX4KbypLoa+m09EVakjaESrhCsbz228 IuUlRUrF4kQQeLe+VKLvvws08rB3mjPXuV+hlYu3FI0xZqBGPR5/fSEJqo2QPQpk rjDlaEMBlwdPyKQHx2pr4upu3XWwBg8M/6EwlzDd8/7A4VAz3SyDhbE/PZG8Syqj XoBIVIX4N6XhmakOMRrHLavZY3zNRBS47BbL+C3izT6mEOCvwiTp+ie0HwzbswAu 9yJzeRjOLyV0qYV40SdqrNVgppIym8LdRhh/nIT1TDREs4wfZwyshnqv9kKctH48 1Y4ABsnwGElrzBvCb2bK9Yy119LmsbEQQfbpvzp+U4dnVax/8fIbIw3KbY11zPiK m0MoI1wSLUOEY4t1DLQCze86imONrpegnAuOzwfUUl5KCdYyOVzP930SU3gk8vrH ZnYOQMerDYaOyadn2xGSZxkKr4Sx99kkxtPH1z3B3YOKD29ZmzB8nbiokHAj84KS jfpivBL4Xmbuqvx5NI44sqzKp4iyq2l6IEw5iD0MaCfol9wcJm1fcz3fPCJkRPl6 xnPywRK8Td3Q9Y5pGwtcA7euj8lMh0Q2Ktad2Yfq430yD7cI4j0=
    =ce0I
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Johannes Schauer Marin Rodrigues on Mon Nov 22 18:00:02 2021
    Johannes Schauer Marin Rodrigues <josch@debian.org> writes:

    Please don't. Since 2014 it is possible to create a Debian chroot
    without taking care of the unpack order and without running any special
    setup beforehand that cannot be put into apt or dpkg itself (like
    creating /var/lib/dpkg/info which is not needed anymore since dpkg
    1.20.0). One of my biggest motivations in writing mmdebstrap was to get
    rid of all the quirks that currently live in debootstrap, move them into
    apt and dpkg so that we end up with a declarative bootstrap process that
    is driven by the package contents, their metadata and maintainer scripts
    but does not need magic from the outside other than standardized
    interfaces. Requiring the merged-/usr setup to be done by the bootstrap script would be a step backwards.

    I understand why you want this, but unfortunately it doesn't answer the question of how to safely perform the upgrade and reconfigure dpkg. Do
    you have any thoughts on the drawbacks of the approach of putting the
    symlinks in base-files that led me to lean away from doing that?

    I'd love to get all of the machinery into packages as well, but I think
    safely upgrading existing systems is even more important than that, and
    I'm worried about our ability to do so if we put the machinery in
    base-files.

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ansgar@21:1/5 to Simon McVittie on Mon Nov 22 20:20:03 2021
    On Mon, 2021-11-22 at 12:24 +0000, Simon McVittie wrote:
    Options to make path A an alias for path B would be a lot like --path- exclude in that they really only make sense in a configuration file in /etc/dpkg/dpkg.cfg.d, although for symmetry they would presumably also
    have to exist as command-line options just like --path-exclude does.

    I would expect using --path-exclude when installing individual packages
    to work: you just don't get the files from that package. (I haven't
    tried.)

    Merged-/usr (or --path-alias as you propose) does not work like this:
    Unlike --path-exclude, you would almost always need to specify a
    superset of --path-alias options for all future operations.

    So I do not think there should be a command-line option for this;
    unless it would behave like `--add-architecture` and register the
    setting somewhere.

        # in /etc/dpkg/dpkg.cfg.d/merged-usr
        path-alias=/bin=usr/bin
        path-alias=/lib32=usr/lib32
        path-alias=/lib64=usr/lib64
        path-alias=/lib=usr/lib
        path-alias=/libo32=usr/libo32
        path-alias=/libx32=usr/libx32
        path-alias=/sbin=usr/sbin

    I do not think this should be a configuration file (unless you suggest supporting other values or removing individual lines; after all a
    configuration file exists to be changed).

    (Though we have various other settings that don't make sense as
    configuration files in /etc, such as Debian's bugtracker and other
    settings in /etc/dpkg/origins/debian, the version in
    /etc/debian_version and so on.)

    Ansgar

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Ansgar on Mon Nov 22 21:00:01 2021
    Ansgar <ansgar@43-1.org> writes:

    So I do not think there should be a command-line option for this; unless
    it would behave like `--add-architecture` and register the setting
    somewhere.

    Agreed.

    I also think it needs to be a two-phase thing, because dpkg has to convert
    its internal database to respect the new aliases. So there should be a
    desired configuration and then an applied configuration, and the latter is
    only updated once the database conversion is complete, with dpkg refusing
    to perform package operations if the desired configuration doesn't match
    the applied configuration.

    Even if the command that changes the configuration also does the database conversion, I think we need this, since I don't believe it's possible to
    do the database conversion atomically (or at least it would involve extra complexity that I don't think we want), so we have to handle the database conversion being interrupted.

    I do not think this should be a configuration file (unless you suggest supporting other values or removing individual lines; after all a configuration file exists to be changed).

    Are you thinking of perhaps shipping this configuratino somewhere in
    /usr/lib instead so that it's not marked as a configuration file?

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sam Hartman@21:1/5 to All on Mon Nov 22 23:20:02 2021
    "Zack" == Zack Weinberg <zack@owlfolio.org> writes:
    Zack> Maybe it was not obvious to anyone in 2016 that the package
    Zack> database being inconsistent with the filesystem could cause
    Zack> actual data loss. However, I think it was the responsibility
    Zack> of the developers of usrmerge to find out how bad that problem
    Zack> actually was, rather than dismissing it. And, as it has
    Zack> proven to be a genuinely critical problem, it is also their
    Zack> responsibility to fix it, per the 'no one can be forced to do
    Zack> anything' rule.

    I agree with the above, but very definitely not with the further
    conclusions you draw.

    I think that the usrmerge proponents need to fix the problem before
    encouraging moving files from / to /usr in packages.
    They can do that either by:

    * proposing fixes to dpkg

    or

    * As Michael has pointed out, proposing mechanisms to guarantee that no
    Debian release both includes file migrations and package migrations.

    We disagree about the rollback.
    If we're ever communicating in a more realtime context I'd love to chat
    about the balances around making sure we do not reward bad behavior.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)