• RFC: New dpkg-build-api levels

    From Guillem Jover@21:1/5 to All on Fri Apr 28 03:30:01 2023
    Hi!

    Some time ago I started work on a new dpkg build API level system
    resembling the one used by debhelper. My current WIP is at <https://git.hadrons.org/git/debian/dpkg/dpkg.git/log/?h=pu/perl-Dpkg-BuildAPI>,
    which I'd like to finalize and merge for 1.22.x.

    This works by making packages build-depend on «dpkg-build-api (= N)»
    (which dpkg-dev then provides), or by setting the environment variable DPKG_BUILD_API=N (which overrides the value in the file, but might emit
    a warning if it's different). This will control what the dpkg build API
    is for that package. There is also a make fragment file (buildapi.mk),
    and a companion tool to extract the build API level called dpkg-buildapi(1)
    to be used from scripts or similar.

    The main reason behind the environment variable is to avoid having to
    re-parse the control file all over the place, or possibly for testing
    purposes (local or global). Unfortunately I don't see a way (that does
    not involve layer violations) to delegate driving the level to a helper
    like debhelper (which could then automatically set specific dpkg-build-api levels from its own levels) because this needs to be set before something
    like debhelper is even in the picture, but it could still tie them by
    making them a requirement and otherwise failing, so it could get some
    baseline guarantees.

    This should make it possible to change current defaults that we could not otherwise change. The idea would be that level 0 would keep the current semantics, where behavior might be changed after specific global
    transitions, or the usual deprecation cycles or similar, but for N >= 1,
    the behavior will be set in stone once defined and can only be changed
    by adding a new level.

    The current list of behavior change candidates for N=1 includes:

    * dpkg-shlibdeps no longer uses the LD_LIBRARY_PATH environment variable.
    * dpkg-buildpackage defaults to Rules-Requires-Root to no.
    * dpkg-buildpackage requires all required debian/rules targets.
    * vendor.mk defaults to using dpkg_vendor_derives_from_v1 for the
    dpkg_vendor_derives_from macro.
    * default.mk defaults to including buildtools.mk.
    * dpkg-buildflags changes default build flags (TBD).

    And that last one I'm still working my way through, but I'll send a
    separate mail for that one.

    So I'm interested in feedback on whether people see any issue with the
    above, and in particular Niels input (although I've already floated
    this to him in the past) on whether this plays nicely with debhelper,
    or whether there is something specific that could be improved or
    changed or similar for its usage, or further proposals for behavior
    changes.

    Thanks,
    Guillem

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Niels Thykier@21:1/5 to All on Sun May 14 10:30:01 2023
    Guillem Jover:
    Hi!


    Hi!

    Some time ago I started work on a new dpkg build API level system
    resembling the one used by debhelper. My current WIP is at <https://git.hadrons.org/git/debian/dpkg/dpkg.git/log/?h=pu/perl-Dpkg-BuildAPI>,
    which I'd like to finalize and merge for 1.22.x.

    This works by [...description omitted...].

    The main reason behind the environment variable is to avoid having to re-parse the control file all over the place, or possibly for testing purposes (local or global). Unfortunately I don't see a way (that does
    not involve layer violations) to delegate driving the level to a helper
    like debhelper (which could then automatically set specific dpkg-build-api levels from its own levels) because this needs to be set before something like debhelper is even in the picture, but it could still tie them by
    making them a requirement and otherwise failing, so it could get some baseline guarantees.


    I think this is fine as a starting point. If you had found a solution
    for the delegation, I think we should have seen it with
    Rules-Requires-Root.

    If we want to solve the delegation issue, we have to change the layers
    in our build system and that is a topic for another day.

    For the ENV, seems fine. Might also be useful for debhelper to know what
    it can expect from dpkg further down the line.

    This should make it possible to change current defaults that we could not otherwise change. The idea would be that level 0 would keep the current semantics, where behavior might be changed after specific global
    transitions, or the usual deprecation cycles or similar, but for N >= 1,
    the behavior will be set in stone once defined and can only be changed
    by adding a new level.

    The current list of behavior change candidates for N=1 includes:

    * dpkg-shlibdeps no longer uses the LD_LIBRARY_PATH environment variable.
    * dpkg-buildpackage defaults to Rules-Requires-Root to no.
    * dpkg-buildpackage requires all required debian/rules targets.
    * vendor.mk defaults to using dpkg_vendor_derives_from_v1 for the
    dpkg_vendor_derives_from macro.
    * default.mk defaults to including buildtools.mk.
    * dpkg-buildflags changes default build flags (TBD).


    Seems reasonable.

    I have a few minor suggestions for the upgrade documentation.

    I would concrete recommend that you swap the order of the upgrade
    levels, so the latest is in the top.

    By having the latest compat level in the top, the readers will only
    scroll over compat levels that are relevant to them. Right now, it makes
    no difference, but when you have 10+ compat levels (which debhelper accumulated) this will change.
    At such a time, most people will only be 1-2 versions behind but
    would have to scroll over 8 irrelevant versions for every package they
    upgrade. In my view, it is a bigger disservice to the user than having
    the list be "reverse ordered".

    Note whether "latest" can be "latest stable" with a dedicated section
    for experimental or unstable compat versions separately. Personally, I
    keep "latest known" in the top, so then people also get to read about
    upcoming changes.
    I would like to think it gives me feedback on upcoming changes before
    I stabilize a new compat level. However, it is not clear to me to which
    extent it works in that regard.


    I would also recommend that you also describe how to opt-out of an
    upgrade where the opt-out is trivial and possible. E.g., with `Rules-Requires-Root` you could say "To retain the previous behaviour, explicitly set `Rules-Requires-Root` to `binary-targets`".
    For me, it is generally more valuable to have 90% to upgrade with
    opt-outs than have 60% upgrade completely and have 40% that cannot
    upgrade due to one feature they have not figured out how to deal with.
    That is where the "opt-out" helps.
    It is often also easier to spot the opt-out than it is to tell why
    people are not upgrading (the documented method is likely going to be copy-pasted verbatim in most cases). In that case, it will also enable
    to get better feedback on which changes did not work so well for people
    via codesearch (etc.).

    With change like the `dpkg-shlibdeps` one, I would personally point
    people to the `-l` option already in the upgrade checklist. Like "If you
    have used LD_LIBRARY_PATH previously, then in most cases you want to use
    `-l` in dpkg-shlibdeps (or via dh_shlibdeps's `-l` or `-L` options if
    you use debhelper)".
    That kind of guiding hopefully results in more people successfully
    migrating - via the path you intended - with less user support landing
    on your desk.

    And that last one I'm still working my way through, but I'll send a
    separate mail for that one.

    So I'm interested in feedback on whether people see any issue with the
    above, and in particular Niels input (although I've already floated
    this to him in the past) on whether this plays nicely with debhelper,
    or whether there is something specific that could be improved or
    changed or similar for its usage, or further proposals for behavior
    changes.

    Thanks,
    Guillem


    To me, dpkg-build-api looks like it follows a tried and successful path
    and I suspect it will be successful as well.
    As long as we have the layering issue, it can only complement
    debhelper. However, I think complementing is completely fine. Even if we
    never solve the layering problem, I still think having a pattern for
    achieving improved defaults in dpkg over time is better than status quo.

    All in all, I hope you will go ahead with this proposal.

    Best regards,
    Niels

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Guillem Jover@21:1/5 to Niels Thykier on Wed May 17 01:40:01 2023
    Hi!

    On Sun, 2023-05-14 at 10:25:19 +0200, Niels Thykier wrote:
    Guillem Jover:
    The current list of behavior change candidates for N=1 includes:
    ·
    * dpkg-shlibdeps no longer uses the LD_LIBRARY_PATH environment variable.
    * dpkg-buildpackage defaults to Rules-Requires-Root to no.
    * dpkg-buildpackage requires all required debian/rules targets.
    * vendor.mk defaults to using dpkg_vendor_derives_from_v1 for the
    dpkg_vendor_derives_from macro.
    * default.mk defaults to including buildtools.mk.
    * dpkg-buildflags changes default build flags (TBD).

    Seems reasonable.

    Ah perhaps also the default paths for debian/files and
    debian/substvars, AFAIR that didn't cause much concern, so I might
    bring that up to confirm.

    I have a few minor suggestions for the upgrade documentation.
    ·
    I would concrete recommend that you swap the order of the upgrade levels, so the latest is in the top.

    Ack, done.

    By having the latest compat level in the top, the readers will only scroll over compat levels that are relevant to them. Right now, it makes no difference, but when you have 10+ compat levels (which debhelper
    accumulated) this will change.
    At such a time, most people will only be 1-2 versions behind but would
    have to scroll over 8 irrelevant versions for every package they upgrade. In my view, it is a bigger disservice to the user than having the list be "reverse ordered".
    ·
    Note whether "latest" can be "latest stable" with a dedicated section for experimental or unstable compat versions separately. Personally, I keep "latest known" in the top, so then people also get to read about upcoming changes.
    I would like to think it gives me feedback on upcoming changes before I stabilize a new compat level. However, it is not clear to me to which extent it works in that regard.

    Right, fully agree.

    I would also recommend that you also describe how to opt-out of an upgrade where the opt-out is trivial and possible. E.g., with `Rules-Requires-Root` you could say "To retain the previous behaviour, explicitly set `Rules-Requires-Root` to `binary-targets`".
    For me, it is generally more valuable to have 90% to upgrade with opt-outs than have 60% upgrade completely and have 40% that cannot upgrade due to one feature they have not figured out how to deal with. That is where the "opt-out" helps.
    It is often also easier to spot the opt-out than it is to tell why people are not upgrading (the documented method is likely going to be copy-pasted verbatim in most cases). In that case, it will also enable to get better feedback on which changes did not work so well for people via codesearch (etc.).
    ·
    With change like the `dpkg-shlibdeps` one, I would personally point people
    to the `-l` option already in the upgrade checklist. Like "If you have used LD_LIBRARY_PATH previously, then in most cases you want to use `-l` in dpkg-shlibdeps (or via dh_shlibdeps's `-l` or `-L` options if you use debhelper)".
    That kind of guiding hopefully results in more people successfully migrating - via the path you intended - with less user support landing on your desk.

    Ack, I've tried to do this for the various options, and expanded what
    some implied (the required rules targets). For the debhelper
    reference, I'm always conflicted, because I recognize that this is
    useful information, but it is also a layer violation (even though only documentation-wise) and it feels wrong for dpkg to document how to use debhelper. :/ So I've left that part out for now, it can always be added.

    I've force pushed to the branch with these and other updates and
    cleanups and fixes.

    To me, dpkg-build-api looks like it follows a tried and successful path and
    I suspect it will be successful as well.
    As long as we have the layering issue, it can only complement debhelper. However, I think complementing is completely fine. Even if we never solve
    the layering problem, I still think having a pattern for achieving improved defaults in dpkg over time is better than status quo.

    Yes, agreed.

    All in all, I hope you will go ahead with this proposal.

    I'll leave some more time for comments, but otherwise I think I'll start merging the infrastructure in a bit. Thanks for the feedback!

    Thanks,
    Guillem

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Guillem Jover@21:1/5 to Guillem Jover on Sat Jul 29 18:50:02 2023
    Hi!

    On Wed, 2023-05-17 at 01:31:57 +0200, Guillem Jover wrote:
    On Sun, 2023-05-14 at 10:25:19 +0200, Niels Thykier wrote:
    All in all, I hope you will go ahead with this proposal.
    ·
    I'll leave some more time for comments, but otherwise I think I'll start merging the infrastructure in a bit. Thanks for the feedback!

    I'm going to be merging the initial batch of patches for the next
    push. Anything else can be added later or on another API level.

    Thanks,
    Guillem

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)