• Build and run-time triplets

    From Julian Gilbey@21:1/5 to All on Wed Jun 8 23:50:01 2022
    I'd like to ask for some help. I'm working on packaging pydevd, which
    builds a private .so library. Ordinary extensions built using cython
    or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so",
    but this library, which is not dependent on the Python version, should presumably be called "bar.x86_64-linux-gnu.so".

    Question 1: How do I determine (within Python) the triplet to use when
    building the library?

    Question 2: How do I determine (within Python) the triplet to use when
    loading the library at runtime?

    Thanks!

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Thu Jun 9 08:30:01 2022
    On Wed, Jun 08, 2022 at 10:43:57PM +0100, Julian Gilbey wrote:
    I'd like to ask for some help. I'm working on packaging pydevd, which
    builds a private .so library. Ordinary extensions built using cython
    or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so",
    but this library, which is not dependent on the Python version, should presumably be called "bar.x86_64-linux-gnu.so".
    If it's just a private library and not a Python module it should be called bar.so.

    Question 1: How do I determine (within Python) the triplet to use when building the library?
    You don't.

    Question 2: How do I determine (within Python) the triplet to use when loading the library at runtime?
    You don't, but also how are you actually loading it?

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmKhkd4tFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh 1nkP/RWTQfNfrADCd7Pmgzi2LOV4t+y6pzmXxYrxPJmHbtI9RdMEgx0XIi9eyNJZ 5qSe/GKP5sxCn6DZXyveOlKdc754s9AUgE2lDxmdl6T/sLBP7jApOQ3iERHCGz4k 6DBjjDgB15H0hKMsNIBRYe+mJJZd/AcKfwFSmYFbO+/vdCdv5zBY4sqcrRPqsdlt Uq2dR1ziT5sUhsfcuxeQqNA+C2+fQtjHfSt/7aLL6RZirHC6WBt2fWxB4twnhA8h FR5/wOTnxTPgO0j+PIlDMuUlK+WH+qdtSM3dP8TGmbqCBpPygAj/MOC+1S35AXoP l+d8ctlV30rGAcZx06U86p6zPYXQoCI3QiiaO+c7Lbn+Jgm8c1b7qy4m7kKpBt83 4wZ0Xb08mgfcAKVpf3mIuXceZUzbnBUc5/iQJbqY/RFTZWhQ/mnhmwuP4K+fQ3vH sWbZV1T8U4M3n5vsdWoY0nxVjmAsfOEuL2+M63CMLNhYQsl39bFXwEBHV4xx2asU K53pHUhdisQ/2f0kcTZ7xZlvQUi72EgGC9g5JBOxMRnR8Hnd4IRPydunC+//nUE9 Kamr7CSAv+gE7T9k577b0L7ZD77atgrg2K8r1jrlqKY2vvQDY/QHet9RSgodhEFk QQFixWzg/SU/Geu3nVP5B/QsaThkDzCtrN0xGygEPqZTk1ii
    =AAIK
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Thu Jun 9 10:10:01 2022
    On Thu, Jun 09, 2022 at 08:42:28AM +0100, Julian Gilbey wrote:
    I'd like to ask for some help. I'm working on packaging pydevd, which builds a private .so library. Ordinary extensions built using cython
    or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so",
    but this library, which is not dependent on the Python version, should presumably be called "bar.x86_64-linux-gnu.so".
    If it's just a private library and not a Python module it should be called bar.so.

    Question 1: How do I determine (within Python) the triplet to use when building the library?
    You don't.

    Question 2: How do I determine (within Python) the triplet to use when loading the library at runtime?
    You don't, but also how are you actually loading it?

    Well, the upstream wanted to compile two versions of the library, one
    for 64 bit architectures and one for 32 bit architectures. I don't
    really want to build two different arch libraries in a single build,
    because that seems very contrary to the way the Debian architectures
    work, and would also limit it to the amd64/i386 architectures for no obviously good reason. I had imagined that if there is some sort of multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names.
    The normal way for this is putting it into
    /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll
    need the full path at the run time so you indeed need the triplet at both
    build and run time. You can get the triplet in d/rules, not sure how
    should you pass it to the build system as that depends on the build system used. For the run time, https://wiki.debian.org/Python/MultiArch suggests sys.implementation._multiarch (note that you cannot use it during the
    build as that would break cross-compilation etc.), not sure if there are
    better ways.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmKhqU0tFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh DS0QAJuwQH4YZmlz9QhE+o8u47qO1x6Hocm9HGRfImyapWxzBGa1PDbMsXeEPtsp lCO4JNcRM+vL3pvcGGSOatW/B8kMAnAKO7KuemljJ4ClQP2/1tL5f/w1ra2o7gj7 a+n9cuiZtFYza0MQ8WgSaxcLV0t85N/BB7VgUBtbnsz/hO+XzZnRS3e4oKojWB2F Xw4I3KVZSikdGbAvII5rtbUp+3loJWyG9++wVWolEh/gMSQVScssjUIOsS2ktsUp UCnrfGTJKr5nsYy/4HG/+JSwhpqiQdagOlCmsxTeutvgNRVLygtyY/uLrZVuAoyd XH4k1u3TFI95xeIbVKa7LrCoO3sswB+/bF1e7BJHDJxvPFXLHTfsmEyA7zvllL4O HgxNsUqPoE6t40gKtO3olMgtDPYg6tbq2Wmnk9KFdcBhrLHy1mAyZzZI1O1zEe2b GCG4Obnnaqc1ZvZtE7+phIN3DmfhIVBs9SFgq/hCltRy/JwMVFSAmL0J6eKOBRtI frQodDXoG+/78z++KaIpWDLUQ7fWqiIe9qWuXtLFAwv7TIWGymLBGytc/fq4FuYO GPi33amYyUl8gbkT0iILCIBeTLXMJAqtLKFtE2vQowg716mPkxoLCMPQzvUyawv+ CP9rt1tnk4VcvAD6jELVtWY3c8TNzdR4L8P045R+s4IN1lRC
    =Epyp
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Thu Jun 9 09:50:01 2022
    On Thu, Jun 09, 2022 at 11:23:26AM +0500, Andrey Rahmatullin wrote:
    On Wed, Jun 08, 2022 at 10:43:57PM +0100, Julian Gilbey wrote:
    I'd like to ask for some help. I'm working on packaging pydevd, which builds a private .so library. Ordinary extensions built using cython
    or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so",
    but this library, which is not dependent on the Python version, should presumably be called "bar.x86_64-linux-gnu.so".
    If it's just a private library and not a Python module it should be called bar.so.

    Question 1: How do I determine (within Python) the triplet to use when building the library?
    You don't.

    Question 2: How do I determine (within Python) the triplet to use when loading the library at runtime?
    You don't, but also how are you actually loading it?

    Well, the upstream wanted to compile two versions of the library, one
    for 64 bit architectures and one for 32 bit architectures. I don't
    really want to build two different arch libraries in a single build,
    because that seems very contrary to the way the Debian architectures
    work, and would also limit it to the amd64/i386 architectures for no
    obviously good reason. I had imagined that if there is some sort of
    multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names. But I've never
    done that myself, so I've no idea if it's even meaningful to do this.

    The library is loaded into gdb using this code:

    cmd.extend([
    "--eval-command='call (void*)dlopen(\"%s\", 2)'" % target_dll,
    "--eval-command='sharedlibrary %s'" % target_dll_name,
    "--eval-command='call (int)DoAttach(%s, \"%s\", %s)'" % (
    is_debug, python_code, show_debug_info)
    ])

    where cmd is a list containing the gdb call, and target_dll and
    target_dll_name point to the shared library.

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Thu Jun 9 11:00:01 2022
    On Thu, Jun 09, 2022 at 01:03:25PM +0500, Andrey Rahmatullin wrote:
    [...]
    Well, the upstream wanted to compile two versions of the library, one
    for 64 bit architectures and one for 32 bit architectures. I don't
    really want to build two different arch libraries in a single build, because that seems very contrary to the way the Debian architectures
    work, and would also limit it to the amd64/i386 architectures for no obviously good reason. I had imagined that if there is some sort of multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names.
    The normal way for this is putting it into
    /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time. You can get the triplet in d/rules, not sure how
    should you pass it to the build system as that depends on the build system used. For the run time, https://wiki.debian.org/Python/MultiArch suggests sys.implementation._multiarch (note that you cannot use it during the
    build as that would break cross-compilation etc.), not sure if there are better ways.

    Thanks for your help!

    OK (and yes, it does require the full path at runtime). What triplet
    do I use in d/rules? dpkg-architecture offers 6 different ones: DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
    to confirm that.

    About the location, though: why do compiled Python libraries live in /usr/lib/python3/dist-packages/<pkgname> and not
    /usr/lib/<triplet>/<pkgname>? And is there a good reason not to do
    the same with this Python-package-specific library? It's not for
    general use, so I can't see why I shouldn't put it in the python3
    directory with the other compiled Python module libraries.

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Simon McVittie on Thu Jun 9 11:50:01 2022
    On Thu, Jun 09, 2022 at 10:26:13AM +0100, Simon McVittie wrote:
    On Thu, 09 Jun 2022 at 13:03:25 +0500, Andrey Rahmatullin wrote:
    The normal way for this is putting it into /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time.

    You can do something like

    handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);
    [...]

    Then you'd install the private library into what Autotools would refer to
    as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems)
    and it will usually end up in the correct place. This assumes that
    ${libdir} is configured to something like
    ${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate for the distribution, but that's normally true anyway, and in particular should be true in debhelper.

    Thanks Simon!

    The build system here is the standard Python setup.py, except for this
    library. That is built by the following script:

    ---
    g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
    mv attach_linux_amd64.so ../attach_linux_amd64.so
    echo Compiled amd64
    ---

    There's not even an attempt at working out ${libdir} or so on. It
    seems like overkill to set up a whole Autotools environment for this
    one file :-(

    I'm still unsure why I shouldn't just put it in the Python package
    area near the cython files such as /usr/lib/python3/dist-packages/_pydevd_bundle/pydevd_cython.cpython-310-x86_64-linux-gnu.so

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Andrey Rahmatullin on Thu Jun 9 11:30:01 2022
    On Thu, 09 Jun 2022 at 13:03:25 +0500, Andrey Rahmatullin wrote:
    The normal way for this is putting it into
    /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time.

    You can do something like

    handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);

    (that's a literal string passed to dlopen!) and ld.so/libdl will expand
    ${LIB} at runtime to the token that was configured into our glibc, which
    in Debian's case is "lib/x86_64-linux-gnu" or similar. On non-Debian distributions, ${LIB} typically expands to "lib64" or "lib32" or "lib"
    instead, whichever one is most appropriate for the architecture and the distro's layout.

    Then you'd install the private library into what Autotools would refer to
    as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems)
    and it will usually end up in the correct place. This assumes that
    ${libdir} is configured to something like
    ${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate
    for the distribution, but that's normally true anyway, and in particular
    should be true in debhelper.

    Replace /usr with the ${exec_prefix} determined at compile-time if you
    want to send code upstream that supports custom installation prefixes.

    (Historical note: this is broken on ancient/EOL versions of Debian and
    maybe some of the Extended Security Maintenance versions of Ubuntu,
    where ${LIB} mistakenly expanded to the multiarch tuple without the
    "lib/" prefix. Non-FHS distributions like Exherbo and NixOS might also
    need to adjust it a bit, but they have to adjust everything else anyway,
    so they should be used to that...)

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Thu Jun 9 12:00:01 2022
    On Thu, Jun 09, 2022 at 09:56:42AM +0100, Julian Gilbey wrote:
    [...]
    Well, the upstream wanted to compile two versions of the library, one
    for 64 bit architectures and one for 32 bit architectures. I don't really want to build two different arch libraries in a single build, because that seems very contrary to the way the Debian architectures work, and would also limit it to the amd64/i386 architectures for no obviously good reason. I had imagined that if there is some sort of multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names.
    The normal way for this is putting it into /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time. You can get the triplet in d/rules, not sure how
    should you pass it to the build system as that depends on the build system used. For the run time, https://wiki.debian.org/Python/MultiArch suggests sys.implementation._multiarch (note that you cannot use it during the
    build as that would break cross-compilation etc.), not sure if there are better ways.

    Thanks for your help!

    OK (and yes, it does require the full path at runtime). What triplet
    do I use in d/rules? dpkg-architecture offers 6 different ones: DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
    to confirm that.
    It's always DEB_HOST_MULTIARCH. *_TARGET_* are for Canadian cross. See
    TERMS in dpkg-architecture(1).

    About the location, though: why do compiled Python libraries live in /usr/lib/python3/dist-packages/<pkgname> and not /usr/lib/<triplet>/<pkgname>?
    That's where the import machinery expects them. They also have special
    names.

    And is there a good reason not to do the same with this Python-package-specific library?
    Yes, it's not Python-related.

    It's not for general use, so I can't see why I shouldn't put it in the python3 directory with the other compiled Python module libraries.
    It's not a module.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmKhxEctFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh 89UP/2vzK0fIE/UTdKrXL/5b9lJPW0ij+S0+pl9mk6a3pKTvKu1Cc8I0VBWi2PtN QK7OJyVAKmOMlob9Oos473LLsx9I1zowOsMX9lkXvs64W5p2QKvOV6WzC0XiYMlc lIbtX+S9yy67nJ7eQQjQtbjBnpY/t01e6SDJJUNg2+n8sdhpc7o+gtDM5LIsSrg9 d6vnZ1COlbdsDhgSuJF+I7XjIRvlnbx+XpYthiEfhMNIBRzvwQxo2IhMxINm0BxJ O06nCJdmQEoMenPKDGgngkCbvI78WpHNao4wF4BLoM/wIBguGgXAb3gTHEGd0WSE LI0+HVRuVErg/pmaXhAwpxeE0gfv7nBo6gssPRIL9/Fl8ewqJHAop9iJXVhYpn1p CDfSVjxixKxjOm6qjJeWrX3eVvAQBTkbvGVda4X6a0Oam3Bja4nG1Wdal6ROwUda HAdb9mekdPP6UC4PTxAShmFnEvqdqp8MJM9KVEd2OrewrE7vAVuidGJdgc0oIWM4 pSD0TqxDYq3Lrk92q5tEOguWYjvFNxPbzBb4vqf1FIopiz9DlCG0GTr9gXc0t55p 5uKu4b7ynHlVB3Vi9J5KPaTnSt96nXGtnd2945sCqpLoQafw3y2tcF4aEF3ZJkbn +LGw0JSoLrn197zyNUAsZ7aSrZqo03b3giS7+EtUB2jy8rLC
    =42Dp
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Julian Gilbey on Thu Jun 9 12:10:01 2022
    On Thu, 09 Jun 2022 at 09:56:42 +0100, Julian Gilbey wrote:
    OK (and yes, it does require the full path at runtime). What triplet
    do I use in d/rules? dpkg-architecture offers 6 different ones: DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
    to confirm that.

    You'd want DEB_HOST_MULTIARCH here (or use ${LIB} as I mentioned in a
    previous message to this thread).

    DEB_BUILD_* is the architecture you are building *on*. For example,
    if you cross-build ARM packages on an x86 machine, DEB_BUILD_* is x86.
    Only use this rarely, for instance if you are compiling a program that
    you will run during the build but not install into the package. If you
    don't know that this is the correct choice then it probably isn't.

    DEB_HOST_* is the architecture you will run the built code on. For example,
    if you cross-build ARM packages on an x86 machine, DEB_HOST_* is ARM.
    This is usually the interesting one in practice.

    DEB_TARGET_* is almost never relevant: it only matters if you are
    compiling a compiler.

    *_MULTIARCH is the normalized multiarch tuple, for example i386-linux-gnu
    on the i386 architecture.

    *_GNU_TYPE is the GNU tuple to be used by Autotools build systems and
    to choose a cross-compiler, for example i686-linux-gnu on the i386 architecture.

    About the location, though: why do compiled Python libraries live in /usr/lib/python3/dist-packages/<pkgname> and not /usr/lib/<triplet>/<pkgname>?

    The Python jargon for a native C/C++ library that can be loaded to
    provide a Python module is an *extension*.

    If a program will load a shared object as a plugin, then the shared object needs to be installed into whatever directory the program has chosen as
    the location where it looks for plugins, otherwise the program will not
    find it: it is the program's choice, not the plugin's choice.

    Python extensions are like plugins for Python, and Python has chosen /usr/lib/python3/dist-packages as one of several locations where it looks
    for extensions (specifically, extensions that come from a package other
    than Python itself), so that is where you have to put them. If the extension that implements the "foo" module is somewhere else, then Python code that
    does "import foo" won't find it.

    (Analogous: if a package wants to provide a plugin for GStreamer, it has
    to put it in /usr/lib/<triplet>/gstreamer-1.0. If it puts the plugin
    anywhere else, GStreamer will not find it and the plugin will be useless.)

    The shared object also needs to follow the naming convention and API that
    the program that will load it has chosen. The naming convention that Python
    has chosen is that if you will load the module with "import foo", then
    the filename is something like "foo.cpython-310-x86_64-linux-gnu.so" or "foo.abi3.so", and it must export a symbol named "PyInit_foo" with a
    particular signature.

    Something that might be confusing you is that for a lot of Python modules
    that are implemented in C, the thing that API users are meant to import directly is Python code, and the Python code internally imports an
    extension with a different name. For instance, to use python3-dbus,
    you "import dbus", but the lower-level C code that implements that is
    actually in a private extension named "_dbus_bindings" (so the filename is _dbus_bindings.cpython-310-x86_64-linux-gnu.so). As an API user, you are
    not meant to "import _dbus_bindings", but the Python code in python3-dbus
    needs to be able to do that in order to do its job.

    And is there a good reason not to do
    the same with this Python-package-specific library?

    If it isn't a Python extension (cannot be loaded into Python with
    "import foo" to provide a module API to Python code) then it would seem inappropriate to put it in the directory that is reserved for Python extensions.

    It's not for
    general use, so I can't see why I shouldn't put it in the python3
    directory with the other compiled Python module libraries.

    Because it isn't a Python extension (what you are calling a "compiled
    Python module library"), but instead some other sort of library with a different purpose (from your other messages, it seems like it's being
    injected into a program that is being debugged with gdb).

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Thu Jun 9 12:10:01 2022
    On Thu, Jun 09, 2022 at 10:43:01AM +0100, Julian Gilbey wrote:
    The normal way for this is putting it into /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time.

    You can do something like

    handle = dlopen("/usr/${LIB}/pkgname/foo.so", flags);
    [...]

    Then you'd install the private library into what Autotools would refer to as ${libdir}/pkgname/foo.so (adjust as necessary for other build systems) and it will usually end up in the correct place. This assumes that ${libdir} is configured to something like ${exec_prefix}/lib/x86_64-linux-gnu or ${exec_prefix}/lib64 as appropriate for the distribution, but that's normally true anyway, and in particular should be true in debhelper.

    Thanks Simon!

    The build system here is the standard Python setup.py, except for this library. That is built by the following script:

    ---
    g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
    mv attach_linux_amd64.so ../attach_linux_amd64.so
    echo Compiled amd64
    ---

    There's not even an attempt at working out ${libdir} or so on.
    Sure, it's not necessary to know the installation path at the compile and
    link time. It's only needed at the install time.


    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmKhxLMtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh f/sP/1rXrMIUePZQ5FNOarPwRfqzz3ehtmjy34/XI2VZCmmwSG5qYW8lPe9aikaP I/s1djEYZ6pUHuN0/nlYAn0hRjg00DYgAKagWu9Jsw65hTDd0LyEkphBbEZwtiGb t5bWvNVWUmbjhuMcHNqp1+DkbqH4fi4Jp7crI7VhDy3oyo8SXmB5SGDGmaDFP4Wt u3NIk62+Fd7wrfhtIq63W7l77bzQdpnRT0Mzv3gCInRVMGZ1xvK6PQXiA6W3erdz Y4vlyRNyuAo1d80/9HUa/xwSZDyLQRAVNrOnf3cg1qKx4uHnUUhqsVxEV1Wcd5nz uI8Mk55PYCW1qex+rgBfV2GMLp1Lw3jGU4Xv4t2ISVVlTeyC2gRvIxBGQiLchFmb c7o2eCrB5FUE3pU6unPBJBIa6rE3qZfBLsM6hjTlxsS87mOVoQvSukr3C6Bc24cK LaRJ44yDS+vnRzWcbN/X6o5z8edBmX6njy+mSgvjonwu1KC58sRr5a/Wyf/zjDL+ TOVFjbG2HHoVN3xMriLWEQf9VgAaOrnXNG6ppWxe7y6PfNCeCPLcnB63jVnqruje 7MRvmbM6bmcPGIeQUS5j3YWDKsDl1Tu3C/yrwXrvWp56qLBSq9vd7FIovb42fs5O IXMGDaCexIyv9Kpdi6UQsSYH/bjxUykNKrpNUzgUF04TX2WZ
    =oStb
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Thu Jun 9 12:50:01 2022
    On Thu, Jun 09, 2022 at 03:00:24PM +0500, Andrey Rahmatullin wrote:
    The build system here is the standard Python setup.py, except for this library. That is built by the following script:

    ---
    g++ -m64 -shared -o attach_linux_amd64.so -fPIC -nostartfiles attach.cpp
    mv attach_linux_amd64.so ../attach_linux_amd64.so
    echo Compiled amd64
    ---

    There's not even an attempt at working out ${libdir} or so on.
    Sure, it's not necessary to know the installation path at the compile and link time. It's only needed at the install time.

    Ah, to clarify: this script compiles the library and sticks it into
    the parent directory. It is then installed with the rest of the
    Python module via setup.py into /usr/lib/python3/dist-packages/...
    So if I want to install it somewhere else, I will have to handle that
    manually.

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Simon McVittie on Thu Jun 9 12:50:01 2022
    On Thu, Jun 09, 2022 at 11:00:28AM +0100, Simon McVittie wrote:
    On Thu, 09 Jun 2022 at 09:56:42 +0100, Julian Gilbey wrote:
    OK (and yes, it does require the full path at runtime). What triplet
    do I use in d/rules? dpkg-architecture offers 6 different ones: DEB_{BUILD,HOST,TARGET}_{GNU_TYPE,MULTIARCH}? I'm guessing DEB_TARGET_MULTIARCH, but I'm really not certain, so it would be great
    to confirm that.

    You'd want DEB_HOST_MULTIARCH here (or use ${LIB} as I mentioned in a previous message to this thread).
    [...]

    Thanks for this great explanation - I'm learning so much in this
    thread!

    About the location, though: why do compiled Python libraries live in /usr/lib/python3/dist-packages/<pkgname> and not /usr/lib/<triplet>/<pkgname>?

    The Python jargon for a native C/C++ library that can be loaded to
    provide a Python module is an *extension*.
    [...]

    And is there a good reason not to do
    the same with this Python-package-specific library?

    If it isn't a Python extension (cannot be loaded into Python with
    "import foo" to provide a module API to Python code) then it would seem inappropriate to put it in the directory that is reserved for Python extensions.

    I hear this argument, though we do find data files and other sorts of
    things other than Python files and compiled extensions that are needed
    by Python modules in that same directory ("data_files" in setup.py);
    for example numpy has dozens of other files in /usr/lib/python3/dist-packages/numpy.

    I think the counter-argument is exactly what you said: as long as the
    shared library is in a place where the code that uses it knows where
    to locate it, all will work fine. The Python code as-written expects
    the library to be in the same directory as the Python code.

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Sun Jul 24 19:40:01 2022
    Well, here's an update on this old thread...

    On Thu, Jun 09, 2022 at 01:03:25PM +0500, Andrey Rahmatullin wrote:
    On Thu, Jun 09, 2022 at 08:42:28AM +0100, Julian Gilbey wrote:
    I'd like to ask for some help. I'm working on packaging pydevd, which builds a private .so library. Ordinary extensions built using cython or similar end up being called "foo.cpython-310-x86_64-linux-gnu.so", but this library, which is not dependent on the Python version, should presumably be called "bar.x86_64-linux-gnu.so".
    If it's just a private library and not a Python module it should be called
    bar.so.

    Question 1: How do I determine (within Python) the triplet to use when building the library?
    You don't.

    Question 2: How do I determine (within Python) the triplet to use when loading the library at runtime?
    You don't, but also how are you actually loading it?

    Well, the upstream wanted to compile two versions of the library, one
    for 64 bit architectures and one for 32 bit architectures. I don't
    really want to build two different arch libraries in a single build, because that seems very contrary to the way the Debian architectures
    work, and would also limit it to the amd64/i386 architectures for no obviously good reason. I had imagined that if there is some sort of multiarch setup, one might have the amd64 and i386 packages installed simultaneously, hence the need for different names.
    The normal way for this is putting it into
    /usr/lib/<triplet>/pkgname/foo.so, and according to the code below you'll need the full path at the run time so you indeed need the triplet at both build and run time. You can get the triplet in d/rules, not sure how
    should you pass it to the build system as that depends on the build system used. For the run time, https://wiki.debian.org/Python/MultiArch suggests sys.implementation._multiarch (note that you cannot use it during the
    build as that would break cross-compilation etc.), not sure if there are better ways.

    I got all of the triplets working, but was then stymied when I tried
    to specify Multi-Arch: same in the control file. I got a lintian
    warning: multi-arch-same-package-calls-pycompile
    It seems that since the pybuild system (via dh_python3) adds a
    py3compile command to the postinst of the package, then I can't safely
    use Multi-Arch: same.

    I don't know if this is the case for all python3 Arch: any packages
    with compiled extensions; are they all effectively Multi-Arch: no? Is
    this worth thinking about in the longer term?

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Sun Jul 24 20:50:01 2022
    On Sun, Jul 24, 2022 at 06:36:57PM +0100, Julian Gilbey wrote:
    I got all of the triplets working, but was then stymied when I tried
    to specify Multi-Arch: same in the control file. I got a lintian
    warning: multi-arch-same-package-calls-pycompile
    It seems that since the pybuild system (via dh_python3) adds a
    py3compile command to the postinst of the package, then I can't safely
    use Multi-Arch: same.
    Actually, #812228, mentioned in the tag description, was fixed in 2021 so
    it's possible that this is no longer a problem.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmLdk3YtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh IhEQAJ2ZT+pmEzkmSEvd8tGcysT9+OGWol0zUqfW+mh7O0iF2A19+39Pj9pcHjHq rLaRJVay0H36ney9ZURJQ0wtMbAMprMXXBv2gL87NT5kP+mFkh92Zk5cVTesLbjD hG0Ns6gCiIUxlvG6/FkX1R8POPfi+BfO7pCG+O9LQ4W3GOJwHMLBwmqw2p0K2WOc 3CsTZwqp/YoROgWB1U3uC7HsgELgt7JCLWMwDmL93HJlO6SIIDiBAPbAfip6X7yo ANQTz/A0N/8rlq8AUaadq8TrA7mNPtgeWtc5JCG1WY/UmJJecBu3afJkncc5kwSD wt3Wz0HsNwUjpaHYJFbJIikczmLhzcQCrk1GLQWqWqNj1uwVNaqJxTPudlgj7nzY l1OJLmmIpqFHVhyAEKwrQaj4y4WeeME/C8xUAMEoQIE5RWQIB4RUZtM1qN+cDW8U j0GlZoix9wxDkW9zKiaGYvJw83NtFdL6Px82Lt3O63HY8AkDrOJf1NG51A7fP0ne iYbsPjuRN9cvMOmHCyvte253yFi+6z91G1OaLYHB0wKJ64hW9Jx8HZWjWw91OCaS GsKej1exsGYcV2JOJxls+OGCKwgtwqYKW/Zj6Oup7oYFgJreMtgGB7Uqk170Rkxs owJqtiLljGbFIb7MKqKweYdBqDUk/H/uUqlUc/d39cNg6WUs
    =L2i7
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Sun Jul 24 21:40:02 2022
    On Sun, Jul 24, 2022 at 11:41:56PM +0500, Andrey Rahmatullin wrote:
    [...]

    I got all of the triplets working, but was then stymied when I tried
    to specify Multi-Arch: same in the control file. I got a lintian
    warning: multi-arch-same-package-calls-pycompile
    It seems that since the pybuild system (via dh_python3) adds a
    py3compile command to the postinst of the package, then I can't safely
    use Multi-Arch: same.

    I don't know if this is the case for all python3 Arch: any packages
    with compiled extensions;
    I think the tag desciption has a good step-by-step explanation why does
    the tag exists.
    But your package is not a "package with compiled extensions", is it?

    Yes, it's got compiled extensions (Cython) in addition to this
    non-Python .so library file.

    are they all effectively Multi-Arch: no? Is this worth thinking about
    in the longer term?
    What do you propose?

    I think the fix to bug #812228 might have done the job nicely ;-)

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Sun Jul 24 21:30:01 2022
    On Sun, Jul 24, 2022 at 11:46:14PM +0500, Andrey Rahmatullin wrote:
    On Sun, Jul 24, 2022 at 06:36:57PM +0100, Julian Gilbey wrote:
    I got all of the triplets working, but was then stymied when I tried
    to specify Multi-Arch: same in the control file. I got a lintian
    warning: multi-arch-same-package-calls-pycompile
    It seems that since the pybuild system (via dh_python3) adds a
    py3compile command to the postinst of the package, then I can't safely
    use Multi-Arch: same.
    Actually, #812228, mentioned in the tag description, was fixed in 2021 so it's possible that this is no longer a problem.

    Ah, that's exciting thanks! So maybe this lintian tag should be
    dropped?

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Julian Gilbey on Sun Jul 24 21:50:02 2022
    On Sun, Jul 24, 2022 at 08:30:42PM +0100, Julian Gilbey wrote:
    On Sun, Jul 24, 2022 at 11:41:56PM +0500, Andrey Rahmatullin wrote:
    [...]

    I got all of the triplets working, but was then stymied when I tried
    to specify Multi-Arch: same in the control file. I got a lintian warning: multi-arch-same-package-calls-pycompile
    It seems that since the pybuild system (via dh_python3) adds a
    py3compile command to the postinst of the package, then I can't safely use Multi-Arch: same.

    I don't know if this is the case for all python3 Arch: any packages
    with compiled extensions;
    I think the tag desciption has a good step-by-step explanation why does
    the tag exists.
    But your package is not a "package with compiled extensions", is it?

    Yes, it's got compiled extensions (Cython) in addition to this
    non-Python .so library file.

    are they all effectively Multi-Arch: no? Is this worth thinking about
    in the longer term?
    What do you propose?

    I think the fix to bug #812228 might have done the job nicely ;-)
    If it actually ships extensions, the "it should usually get a dependency
    on the Python interpreter for the same architecture" part should still
    apply as far as I understand it.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmLdoFstFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh ri4P/R8fgjkFOmv4+R390r6vKktoDR4Zla78It2nL0eLFgX2YyaIS4oQxw76vXzJ xPfW7YGzP4gM/Cpd2NWyMg7qzVMUeJ4O9yqxJoObGzNb/qHACdvEoTfALMD6dDw3 EqTgNq7vjQnWyHLyQYVmGP4/NlZFMsi+3sMt1jgHCp5ucbG9rbfGMGGfCFJ4eV+p tQ5VLTevjlG8jb6lYRyGRFDm/7VKuVGZ7vLPGHJGozneTotlccLIXt1uNJcBgo0q 0aSdZ5h/hD8E6xxHAiT93gEZiP1iuLD5acsSfyTwAQCpW7KDv5zeRRyNUJYACERE FYxLXGBe5OLBCiHl1GGACaeuPE2oQyVAds0zWdJORioAn+i2uCAt4CWmuTS656iB ZYYfA+tDP51ey6Q1aAbxBESz2o/yrkY55m7OoIz6skuFa079Ecgq0b118hYS6p6N xIgP7iMDDm+s9tY2xiNbv8VN907Qv+fzD9BOod0JIvtzjxDIY3Lo+8kyEukrdUAc UB2zScyA2KJnRzw88neBVskU8L8PpiTJ65hLr/dEGaZ5zJsTzZSwA61EsJpC2KXk 7ygS5aXcC7/8uQFIqhoud1Lzi6WIGW3NoCBRppDeNdMcLWb3PJO2M+vSsD1tXKai pvBSbbSISL7HRTdNaH2neSgGFvSw5A3zGrm12bgFxy9JOt//
    =1BAA
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Julian Gilbey@21:1/5 to Andrey Rahmatullin on Sun Jul 24 23:10:02 2022
    On Mon, Jul 25, 2022 at 12:41:16AM +0500, Andrey Rahmatullin wrote:
    On Sun, Jul 24, 2022 at 08:30:42PM +0100, Julian Gilbey wrote:
    [...]
    are they all effectively Multi-Arch: no? Is this worth thinking about in the longer term?
    What do you propose?

    I think the fix to bug #812228 might have done the job nicely ;-)
    If it actually ships extensions, the "it should usually get a dependency
    on the Python interpreter for the same architecture" part should still
    apply as far as I understand it.

    Thanks Andrey!

    OK. So let's dissect this tag info and see where we're currently at.

    Explanation: This <code>Multi-Arch: same</code> package uses <code>pycompile</code> or
    <code>py3compile</code> in the specified maintainer script.
    .
    <code>py{,3}compile</code> are tools used to byte-compile Python source
    files. It is typically run on installation of Debian packages that ship
    Python modules. However, they do not support installing several
    architectures of the same package and this is not Multi-Arch: safe.

    This is now out-of-date: firstly, we can presumably get rid of the
    pycompile mention, as there are only a tiny handful of Python 2
    packages still around, and we're trying to get rid of them.

    Secondly, py3compile now supports installing several architectures of
    the same package; see the closing changelog message on bug 812228:

    Architecture-qualify py*compile and py*clean calls in maintainer scripts,
    for architecture-specific Python packages. This allows co-installation
    (and even concurrent unpacking) of different architectures of a package.

    So the rest of the paragraph is also out of date.

    If the contents of the package is not architecture dependent, it should
    usually be made binary-all.

    That is still certainly true.

    If the contents of the package is architecture dependent, it should
    usually get a dependency on the Python interpreter for the same
    architecture. This is a dependency in the form of <code>python3</code>, not
    an architecture-qualified dependency such as <code>python3:any</code> (which
    can be fulfilled by the Python interpreter for any architecture).

    This is interesting; dh-python gives the dependency:
    python3 (<< 3.11), python3 (>= 3~), python3:any
    which has both same-architecture and qualified architecture
    dependencies; obviously the same-architecture one "wins". But this
    paragraph is probably unnecessary for most dh-python-using packages
    (though it doesn't seem to harm).

    If a dependency on the Python interpreter for the same architecture
    exists (usually generated by <code>dh-python</code>), the
    <code>Multi-Arch: same</code> has no effect and should be dropped.

    Ah. I see the point. Because python3 and python3-minimal are
    Multi-Arch: allowed, the different arches of python3 are not
    co-installable, and so there is no point in labelling the
    arch-dependent module packages as Multi-Arch: same; they still could
    not be co-installed.

    See-Also: pycompile(1), py3compile(1), Bug#812228

    This list can probably be trimmed down to py3compile.


    I hope this reasoning is useful; shall I pass it on to the lintian
    folk?

    Best wishes,

    Julian

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)