• Debian and ISO C++ package management conversations

    From Daniel Ruoso@21:1/5 to All on Thu Nov 4 21:10:12 2021
    Hello,

    I have been involved in the conversations in the ISO WG21 (C++) SG15
    (Tooling Study Group). Particularly in the context of trying to get a
    story straight on how C++ modules will interact with package managers.

    Over the past two months, I have released a couple[1] papers[2],
    trying to outline the requirements on how C++ modules could be
    consumed in non-monorepo environments. Particularly, I think that
    affects how GNU/Linux distributions will have to deal with C++.

    During early discussion in the SG15 mailing list[3], it became clear
    that some folks are interested in getting a bit further in the package management side. During cppcon last week I managed to sit down with
    some of those folks, and got to an agreement to explore the
    possibility of moving the C++ language a bit ahead in the package
    management space, instead of just "accepting defeat" and making
    modules behave like headers in the way they are distributed.

    So I recently started another conversation[4] where I am trying to
    summarize what are the different requirements on how package managers
    and build systems need to cooperate in the context of C++.

    The main thing, however, is that we're not getting a lot of input from
    folks involved in the package management side. So I'm reaching out to
    folks that have experience maintaining C++ libraries in Debian --
    although, realistically, this applies to C, C++ and Fortran.

    This is particularly important because the main risk for any
    conversation in that area is that we can come up with a spec that
    "doesn't stick". So in a sense, we need to think about this in terms
    of "here's an additional thing that folks packaging C++ code will need
    to do in order to make them consumable by other build systems".

    Do folks think there's space for us to create more interoperability
    between package managers and build systems? What are the requirements
    that should drive an effort on that space?

    Daniel

    [1]: https://isocpp.org/files/papers/P2409R0.pdf - Although it is
    written from Bloomberg's perspective, our understanding is that it
    matches the requirements for most distributions (in fact, bloomberg
    uses dpkg for its internal libraries deployed as a "partial
    distribution" into /opt/bb)
    [2]: https://isocpp.org/files/papers/P2473R0.pdf - This is a "let's
    make modules look like headers" proposal.
    [3]: https://lists.isocpp.org/sg15/2021/10/1083.php
    [4]: https://lists.isocpp.org/sg15/2021/10/1145.php is where I try to
    summarize things into a coherent document.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Thaddeus H. Black@21:1/5 to Daniel Ruoso on Sat Nov 6 21:00:01 2021
    Daniel:

    Not an expert on the subject, I should be reluctant to reply on
    list, except that 24 hours have passed and I see no other
    replies on list yet; so here goes.

    On Thu, Nov 04, 2021 at 04:03:31PM -0400, Daniel Ruoso wrote:
    During early discussion in the SG15 mailing list[3], it became clear
    that some folks are interested in getting a bit further in the package management side. During cppcon last week I managed to sit down with
    some of those folks, and got to an agreement to explore the
    possibility of moving the C++ language a bit ahead in the package
    management space, instead of just "accepting defeat" and making
    modules behave like headers in the way they are distributed.

    I did not know that to make modules behave like headers were to
    accept defeat! Is it? I had thought that *the original idea*
    was to make modules indeed behave like headers, except without
    bringing the preprocessor into it.

    Someone has probably asked such questions on your committee a
    long time ago, but I was not there, so may I ask: what would
    the drawback be in letting modules behave like headers at least
    for now, and then revisiting the question for C++26 or C++29 in
    view of accumulated experience?

    The answer to the last question is unclear to me from your
    papers. It has something to do with Bloomberg and
    monorepo (the last being a word I had to go to look up).
    I didn't get it, to tell the truth.

    Another way to ask the question is: wouldn't you rather
    implement something like your paper [2] for now, and then
    gradually expand upon that later to move cautiously in the
    direction of paper [1]?

    Or am I missing the point?

    [1]: https://isocpp.org/files/papers/P2409R0.pdf - Although it is
    written from Bloomberg's perspective, our understanding is that it
    matches the requirements for most distributions (in fact, bloomberg
    uses dpkg for its internal libraries deployed as a "partial
    distribution" into /opt/bb)
    [2]: https://isocpp.org/files/papers/P2473R0.pdf - This is a "let's
    make modules look like headers" proposal.

    Except for the part on discovery tooling, which admittedly
    looks to me as though it might be slightly overengineered
    for C++23, paper [2] looks pretty good to me at first glance.
    (Admittedly, this may be merely because I understand paper [2]
    but do not understand paper [1].)

    So here is an alternate, tangential question which gets in a
    roundabout way at the same point, if you wish: if your more
    advanced proposals such as in paper [1] were implemented, then
    what would happen to the preprocessor? Admittedly, I have not
    directly used the preprocessor to do anything interesting since
    the C++98/03 era, but the thought of eliminating the
    preprocessor entirely seems unsettlingly un-C++-like to me.
    Doesn't it, to you, too?

    Or am I again missing the point?

    Meanwhile, it occurs to me that, in Debian, C++20 has only
    reached sid a few months ago; and even then, to the extent to
    which GCC is used, the compiler falls back to C++17 unless
    explicitly asked not to; so it could be that there exists
    limited practical experience in the Debian Project *at this
    time* to treat with you on questions like the questions you are
    asking regarding C++ modules.

    I am glad that you are asking the questions, nevertheless.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEEM1APDU+pwMhnuF4GcGMmQy2FIrwFAmGG04UACgkQcGMmQy2F Iryn3Q/+PEJksbl9k0L/vP+rqmQL3hiQg/fm7VcVqs3wTizwnWPs0R/+uqEvxWAq ltrQnDiR+Mzn9/kWb9Ce8yCANJeSQtpC8VpguYnQ8JC8MwPKUJA0neWW/trj+FyC LLROT+ItrHkuremS9P1KuPHHkiYcuKEhd9WCj0DCr4LeiiJDo2QK5MJyVgEbLdNb dCC8CXohjPJQ/QNtwztSoMlHXqIUl3sSfieNIE4sowNyBrs5Ubv3NUAlHdtE42Ov gEBSX1dFERWVw96/wfWrMOvpUupv4C9aS1K5t41VWwwwo2pBnvKPTJv/5MWpCZEV KxdfXkLv+we5j1kh6Dd+kfgVVWqdJN+ou1+R2Rka+c4VEkgM5Ck2dfwjF+bwSSiu Vfp4j89DY/93nBOzM6sdQ7ReugqOCFDYRAXwfKw6ii9SDYhRvS86dLHLaMOuR6nH x3DvmXxdWgDDAhqjMAsuMDWb/sHB6wLO00txxmw/75hKo352njoL17qBqRwFy9yh /YLvPlRpVbVNQQzpxVzdXQ8RcMelVjh7Nc1pQIgrmYYeC1h0PYmDKL8d8whSNvBf RpA3nfBAtC1u1ISbDlDA3cAopUXiCwiFhtFh0Me8A989RTu9ok3DEO1Bj4EPSReD 5M1eDX5kzbGKu/DtGI940QO0EmCiPP7lS35vveG4U7EvMezsEF0=
    =AB3z
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Daniel Ruoso@21:1/5 to thb@debian.org on Sat Nov 6 22:00:01 2021
    On Sat, Nov 6, 2021 at 3:52 PM Thaddeus H. Black <thb@debian.org> wrote:
    Not an expert on the subject, I should be reluctant to reply on
    list, except that 24 hours have passed and I see no other
    replies on list yet; so here goes.

    Thanks, your comments are appreciated.

    I did not know that to make modules behave like headers were to
    accept defeat! Is it?
    [...]
    what would
    the drawback be in letting modules behave like headers at least
    for now, and then revisiting the question for C++26 or C++29 in
    view of accumulated experience?

    The main concerns being raised are:

    1. Forcing file system semantics on module names, when module names
    are not meant to be hierarchical. The "." is just a word separator, it
    doesn't imply hierarchy.

    2. Impact of having to traverse the file system when trying to get to
    module files (although I'm personally skeptical on that front).

    3. Potential issues caused by case-insensitive file systems as well
    as on the mapping from unicode identifiers to file names.

    But perhaps more importantly, if we had support at a higher level, the
    package manager would be able to communicate to the build system the
    complete mapping from module names to interface files, and no lookup
    of any kind would be necessary. That's the context where making
    modules behave like headers is "admitting defeat".

    Another way to ask the question is: wouldn't you rather
    implement something like your paper [2] for now, and then
    gradually expand upon that later to move cautiously in the
    direction of paper [1]?
    Or am I missing the point?

    The main thing that may be causing confusion is that I seem to be
    presenting contradictory views here. Let me explain.

    I presented both papers 1 and 2.

    The point of the first paper was to raise the issue that the current implementation of modules in Clang and g++ would *not* work for
    environments like the Debian GNU/Linux distribution, for instance.
    It's written from Bloomberg's perspective since that's where we're
    coming from, but we actually have pretty much the same requirements as
    Debian has.

    The second paper presents a very simple implementation that addresses
    most of the issues raised by the first paper. In other words: I,
    personally, would be fine if we commit to the "behave like headers"
    approach.

    However, there has been some opposition to that approach, and I took
    it to myself to do an honest evaluation of what it would take to do a higher-level solution, where the package manager would be able to
    communicate to the build system what is the mapping between module
    names to interface files without any expectation of how the files are
    named. That's what the document linked from the other thread tries to
    address.

    Except for the part on discovery tooling, which admittedly
    looks to me as though it might be slightly overengineered
    for C++23, paper [2] looks pretty good to me at first glance.

    The reason why it needs to be a bit more over-engineered is that
    parsing modules must be done in topological-sort order of
    dependencies. Meaning you need to produce a full dependency graph of
    all modules, and make sure you process the module interfaces in
    dependency order. Only after the module interfaces are processed can
    you proceed to the normal "embarrassingly parallel" step of producing
    the object files. And the dependency discovery needs to be done by
    reading the module interface files with preprocessing capabilities.

    what would happen to the preprocessor? Admittedly, I have not
    directly used the preprocessor to do anything interesting since
    the C++98/03 era, but the thought of eliminating the
    preprocessor entirely seems unsettlingly un-C++-like to me.
    Doesn't it, to you, too?

    The preprocessor doesn't really go away with this. Modules actually
    raise another problem that you need not one set of preprocessor flags
    for your translation unit, but you also need to know the preprocessor
    flags required for every module interface you consume.

    But the point of modules, to a large extent, is to offer a
    non-preprocessor way of reusing code. And at that, the spec is very
    successful.

    Meanwhile, it occurs to me that, in Debian, C++20 has only
    reached sid a few months ago;

    Ok, that is a point that bears clarification. Modules are a feature in
    C++20. It's already part of the language, and there are already usages
    of modules in some specific organizations. However, we don't yet have
    any established convention on how anyone would be able to ship a C++
    library that exports modules.

    In fact, nothing that we're discussing in SG15 will actually change
    the spec of the language. What we're trying to do is create guidelines
    for interoperability between compilers, build systems, package
    managers, organizations shipping pre-built libraries, organizations
    consuming pre-built libraries.

    In that sense, we're actually very much behind, because Modules are
    not actually expected to change in the language in any meaningful way,
    but the tooling is nowhere near where it needs to be for modules to
    actually be useful.

    The worst case scenario is that C++ libraries that export modules will
    not be interoperable across build systems, and you'll end up in a
    situation where we can't meaningfully ship C++ libraries pre-built.

    Daniel

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)