• Re: sysconfig default scheme change in Python 3.10

    From Simon McVittie@21:1/5 to Stefano Rivera on Mon Mar 28 23:50:01 2022
    On Mon, 28 Mar 2022 at 21:17:40 +0000, Stefano Rivera wrote:
    We've fixed this issue in pybind11 and automake-1.16, so far.

    Does this mean that packages that build using Automake, but use the pregenerated configure/Makefile.in provided by the upstream maintainer
    (often on an older or non-Debian distro) instead of regenerating the
    build system with dh_autoreconf or equivalent, are going to be broken now?

    If true, this will hopefully not affect too many packages, because using dh_autoreconf is the debhelper default and has been considered a Debian best-practice for a while; but it seems worth being aware of, because I'm reasonably sure there are still some packages that deliberately do not regenerate the build system, and some of those probably build CPython
    modules.

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Mon Mar 28 23:20:01 2022
    I realise we made a consequential changes in Python 3.10, that was never announced to the team. At the time, I didn't realise how consequential
    it would be, but it has forced a few packages to change in response.

    The change was in python3.10 3.10.2-3 (25 Feb 2022).

    The default sysconfig change has changed from posix_prefix to
    posix_local. This matches the long-existing behaviour in Debian's distutils.sysconfig. distutils is being removed from Python's stdlib in
    3.12 [PEP-632], so setuptools etc. are being migrated to be configured
    by sysconfig instead.

    The practical reason for this is to make "python3 setup.py install"
    install to /usr/local/lib/... rather than into the generally
    package-managed /usr/lib/python3/.

    $ python3.10 -m sysconfig
    ...
    Current installation scheme: "posix_local"

    Paths:
    data = "/usr/local"
    include = "/usr/include/python3.10"
    platinclude = "/usr/include/python3.10"
    platlib = "/usr/local/lib/python3.10/dist-packages"
    platstdlib = "/usr/lib/python3.10"
    purelib = "/usr/local/lib/python3.10/dist-packages"
    scripts = "/usr/local/bin"
    stdlib = "/usr/lib/python3.10"
    ...

    This has caused some changes for some Debian packages. Generally it can
    be fixed at a build-tool level, and shouldn't break other Python
    packages, but for non-standard tools that haven't been fixed yet, you
    may notice that modules install into /usr/local instead of /usr.
    dh_python3 will move modules from /usr/local back to /usr, but it's
    called after dh_install, so dh_install configuration can get broken by
    this change.

    The solution in build tools is to explicitly select the posix_prefix
    scheme, and specify the appropriate prefix (/usr or /usr/local). For
    build tools that default to /usr/local, this should be all you need to
    do.

    We've fixed this issue in pybind11 and automake-1.16, so far.
    If you find any more tools that need work, feel free to ask me to help.

    There have been some tweaks to this, since 3.10.2-3, to reduce breakage.
    One of the significant wants was to not prefix the include paths with
    /local/, as they are defined as referencing the location of cPython's
    headers, not the install location for headers [bpo-44445]. But some
    build tools may have their own interpretations of this...

    Related to this change is something else worth noting:

    Because distutils is being removed, modern setuptools now carries its
    own distutils fork, internally. This doesn't have Debian's patches
    implementing these scheme differences. Instead, it's configured by the
    presence of a _distutils_system_mod module, which monkey-patches
    distutils.

    [PEP-632]: https://peps.python.org/pep-0632/
    [bpo-44445]: https://bugs.python.org/issue44445

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Tue Mar 29 01:30:01 2022
    Hi Simon (2022.03.28_21:44:38_+0000)

    On Mon, 28 Mar 2022 at 21:17:40 +0000, Stefano Rivera wrote:
    We've fixed this issue in pybind11 and automake-1.16, so far.

    Does this mean that packages that build using Automake, but use the pregenerated configure/Makefile.in provided by the upstream maintainer
    (often on an older or non-Debian distro) instead of regenerating the
    build system with dh_autoreconf or equivalent, are going to be broken now?

    The change is to python.m4, so it gets included in aclocal.m4 and
    configure. If you rebuild those, you'll get the correct behaviour.

    I don't think the generated makefiles are affected, just the configured
    paths.

    We have upstreamed the patch, https://debbugs.gnu.org/54412 so
    eventually upstreams will ship something that works correctly, again.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)