Perhaps someone else wants to comment on that conversation

https://discuss.python.org/t/publish-linux-installer-on-python-org/53760/2
--
Salvo Tomaselli

"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di senso, ragione ed intelletto intendesse che noi ne facessimo a meno."
-- Galileo Galilei

https://ltworf.codeberg.page/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sunday, 26 May 2024 03:33:09 CEST Ian Norton wrote:

I am puzzled about some of the responses there, how can anyone expect to randomly update packages on the system using pip and not have it go wrong
on any distribution? This is why things like pipenv exist.

Or whatever today's tool is called. That's just one of the problems that the core Python packaging scene has. To be fair, though, I found myself using pip in a job a couple of years back just to "go with the flow", and it mostly worked. My expectations were not particularly high, however.

Where I was just installing things casually, I made pip install things using the --user option, which employs that unpleasant but pervasive approach of stashing huge volumes of software inside a hidden directory (something things like Plasma also do), where one wonders where all the disk space has gone. There's also the issue of uninstalling unwanted software, which probably still isn't solved.

Where I was installing things in a Web application environment, I made pip install in a designated directory, but I seem to remember having to do a fair amount of environmental configuration to make sure that Python and Apache were able to find all the modules. I guess I could have used virtualenv, venv, pipenv (see what I mean?), but one is left wondering how all of these things interact with the rest of the system. How does one make Apache "enter" the virtual environment of whichever flavour, for instance?

(Yes, "containers" exist, but I am reminded of the saying about solving problems with regular expressions: now you have two problems. Particularly
with the inelegant way Linux provides support for containers.)

With the fundamental Python packaging technology, the one thing that seems to have improved somewhat is the availability of wheels - whose implementation is probably heavily underappreciated - which avoids the issue I previously had where a source package is downloaded by pip and then fails to compile in a bad way. Twenty or so years ago, people wanted to replicate the CPAN experience
for Python, but I doubt that this was what they meant.

The Python core developers have always complained about distribution packaging and distributions "ruining the experience for users", all reported rather selectively, but this is a situation much of their own making. They have
always wanted to do everything themselves, traditionally with the
justification that Windows and the Mac have to be catered for. People used to band around the term "stop energy", but pandering to Microsoft and Apple
seemed to be a significant source of that energy.

Meanwhile, there are competing but incompatible tools like Conda that do everything themselves, too. When deploying things conservatively using pip, I had it suggested to me that I should use Conda. Conda's shared libraries are incompatible with the way Apache is built on Red Hat systems, so the proposed "solution" was to run the Web application using the built-in Python Web server and to proxy it using mod_proxy in Apache. What a joke!

So, the core developers make their own package repository, downloader, dependency resolver, installer, environment manager. At many an opportunity, they had the opportunity to leave this particular highway of madness, but they raised the stakes and now are in the business of effectively running their own "app" store, all without Big Tech's legions of low-cost workers trying to prevent malware being part of everyone's next invocation of "pip install".

I could be rather cynical about some of the motivations of people in the core development community. I see that a few of the discussion participants have worked for Red Hat and at least one of them was involved in the enthusiastic deprecation of Python 2 on the basis that people could always pay for
continued support. No coincidence that RHEL is where people will go for that support, I suppose.

And these discussions have always had the tone of people pontificating about what "the users" want or do, even speaking for them, and yet never really engaging with the actual users. Combined with a lack of appreciation that most users run other software that isn't written in (or for) Python, probably because Windows doesn't provide such software out of the box. Some actual
users are still running Python 2, believe it or not, regardless of whether
that can be seen from the ivory tower.

In case anyone thinks this rant was a bit unfair, I followed the Python
mailing lists for almost a couple of decades and saw plenty of discussions on such matters on python-dev. Then again, I think the stewardship record of the Python language over the last twenty years rather speaks for itself. That is a statement left for one to interpret in the way one feels most comfortable
with.

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

------MABR01ZRXBC9WUO57H59J2M37ETU64
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi,

I have always found most of issues mentioned in the thread a welcome change instead, and if anything the far superior ability of apt to automatically remove all the installed dependencies in case of the removal of a package compared to the complete
inability of pip has always left me in awe. I doubt they are really weighing the tradeoffs enough before choosing to hand out the blame.

Best,
Ananthu
------MABR01ZRXBC9WUO57H59J2M37ETU64
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><body><div dir="auto">Hi,<br><br>I have always found most of issues mentioned in the thread a welcome change instead, and if anything the far superior ability of apt to automatically remove all the installed dependencies in case of
the removal of a package compared to the complete inability of pip has always left me in awe. I doubt they are really weighing the tradeoffs enough before choosing to hand out the blame.<br><br>Best,<br>Ananthu</div></body></html>
------MABR01ZRXBC9WUO57H59J2M37ETU64--

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Ian (2024.05.26_01:33:09_+0000)

I am puzzled about some of the responses there, how can anyone expect to randomly update packages on the system using pip and not have it go wrong
on any distribution? This is why things like pipenv exist.

People don't understand that stuff until they dig to the details. And
even then, sometimes they forget and/or assume we have the resources to massively revamp our stack.

There are long-standing grievances here (see tiran's gist for example).
I have very little experience with Fedora/RH, but from their grumbling,
I assume they solve some problems for developers that we don't:

1. It appears that multiple versions of cPython are trivially available.
Each version has its own site-packages.
2. Their python packaging is more developer-centric than user-centric.
More modules included in the install.

We're pretty constrained on 1 by the debian security team policy. But
maybe there is discussion room there for non-supported Pythons?

We have taken steps to improve 2 by adding the python3-full package.

I could see a long term strategy for having a system-python package that provides a python3 binary used by Debian packages that need Python. And
a separate python package for developers that installs all the bells and whistles immediately.

Achieving this would require reorganizing the way we package almost all
Python, and I don't think we have the interest and resources to do that.

If pyenv was packaged in Debian (ITP #978149) it would probably be great
for new Python developers on Debian.

Not sure what other small tactical steps we could take?

Perhaps someone else wants to comment on that conversation

Replied. I've engaged with these guys on this stuff before. Let's see
where it goes...

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I'll just say that the equal amount of FUD and ranting can be easily
generated abotu Debian or even just about Debian Python packaging.

--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmZTN2MtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh SFsP+wRgJHLoA4ZnPvvemI/+9tIPk1tA7VRr0r4gZF2sb1O15RZNZlWjrheNDhm1 TovWEoquDbzJiuiHfeWvZ/Zj62ZN6fomv8604gmb2rvMcpW3RPInnfTDTx1Jr7/o FKkO7zV6rPL+E8mdOymMBm7I8CW29hCLCIbAYLdQyg7hzWrrzbgxGjmuqJzgx3fj 4zLrNXTOTp09IwncUB+cllYozLfiKAPrmrG2E+YaLmtrTP/Ack8MUQT4mp0e5/98 0fiwDLf2OLZyZbj6HbE4qrmwqhc6SUuexlDcNMoCuv+IBi3VwTq+KHWHb83kt0bJ nUyGhmCO7LLeZu/LsrFJajQGDC69zp3ph5bRqhZQeOSzlagecypGxJeogkIX3lvd NAqB5K26GofbHesmlCFPrpb9mlFck7dPkNZjW5wyRtFlbHi+OHVByVSnQKDQpEtY xG2wuRfwQ7AtfYVicNJDUShWwQCO2LPzlh1bs5Czl6OT4z+pzC/yjnAKUmKbJqCJ 2stNod6ce4bo+t5tVRQfS1CdbRXmgjHLCds8SjfhNE4oxouC/JEC1ZDXAVxFKRyh Ll/hCLcvlK6snwGItavzBzt1xiZ3JD2iGaUVDSbSHQIQavox2mQh2I+chkYj2A9A IK6PFhIr6V6LHkLg/9AIhXmjokcQnRLreqY9G+Kmcs1QW3Oa
=8P2p
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

That's just too crazy to let change a core system dependency so easily
from outside.



<div id="geary-body" dir="auto">That's just too crazy to let change a core system dependency so easily from outside.</div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I happen to be subscribed here, so figured I'd comment :)

FWIW I think the way the discussions are going... really in both locations.. is needlessly taking shots at each other.

I've commented on discuss.python.org, but figured I'd repeat myself here.

I think the way these discussions devolve into each "side" taking shots at each other and arguing over nonsense that doesn't matter does everyone a disservice. 

The facts of the matter, as far as I can tell:

- Distributors ship Python and a number of people find great value in that and it works great or at least fine for them.
- A number of users have needs or wants that are not well served by the Python that distributors ship, for one reason or another.

People in either of those groups sniping at each other for wanting the "wrong" things is completely unproductive. 

It's probably "fine" if Python.org wants to ship a linux binary, I suspect it'll have very little impact on distributors of Python (and might even make things better-- since some of the problems flow from an impedance mismatch, and it provides a
different escape hatch to point people towards if the distro Python doesn't suit their needs).

On 5/26/2024 9:39:25 AM, Stefano Rivera <stefanor@debian.org> wrote:
Hi Ian (2024.05.26_01:33:09_+0000)

I am puzzled about some of the responses there, how can anyone expect to randomly update packages on the system using pip and not have it go wrong
on any distribution? This is why things like pipenv exist.

People don't understand that stuff until they dig to the details. And
even then, sometimes they forget and/or assume we have the resources to massively revamp our stack.

There are long-standing grievances here (see tiran's gist for example).
I have very little experience with Fedora/RH, but from their grumbling,
I assume they solve some problems for developers that we don't:

1. It appears that multiple versions of cPython are trivially available.
Each version has its own site-packages.
2. Their python packaging is more developer-centric than user-centric.
More modules included in the install.

We're pretty constrained on 1 by the debian security team policy. But
maybe there is discussion room there for non-supported Pythons?

We have taken steps to improve 2 by adding the python3-full package.

I could see a long term strategy for having a system-python package that provides a python3 binary used by Debian packages that need Python. And
a separate python package for developers that installs all the bells and whistles immediately.

Achieving this would require reorganizing the way we package almost all
Python, and I don't think we have the interest and resources to do that.

If pyenv was packaged in Debian (ITP #978149) it would probably be great
for new Python developers on Debian.

Not sure what other small tactical steps we could take?

Perhaps someone else wants to comment on that conversation

Replied. I've engaged with these guys on this stuff before. Let's see
where it goes...

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272


[163d8144-a584-4d1d-9d71-f86da864fde2]
<div id="__MailbirdStyleContent" style="font-size: 12pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
I happen to be subscribed here, so figured I'd comment :)<div><br></div><div>FWIW I think the way the discussions are going... really in both
locations.. is needlessly taking shots at each other.</div><div><br></div><div>I've commented on discuss.python.org, but figured I'd repeat myself here.</div><div><br></div><div>I think the way these discussions devolve into each "side" taking shots at
each other and arguing over nonsense that doesn't matter does everyone a disservice.&nbsp;</div><div><br></div><div>The facts of the matter, as far as I can tell:</div><div><br></div><div>- Distributors ship Python and a number of people find great value
in that and it works great or at least fine for them.</div><div>- A number of users have needs or wants that are not well served by the Python that distributors ship, for one reason or another.</div><div><br></div><div>People in either of those groups
sniping at each other for wanting the "wrong" things is completely unproductive.&nbsp;</div><div><br></div><div>It's probably "fine" if Python.org wants to ship a linux binary, I suspect it'll have very little impact on distributors of Python (and might
even make things better-- since some of the problems flow from an impedance mismatch, and it provides a different escape hatch to point people towards if the distro Python doesn't suit their needs).</div><div><br></div><blockquote class="history_
container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px"> <p style="color: #AAAAAA; margin-top: 10px;">On 5/26/2024 9:39:25 AM, Stefano Rivera &
lt;stefanor@debian.org&gt; wrote:</p><div style="font-family:Arial,Helvetica,sans-serif">Hi Ian (2024.05.26_01:33:09_+0000)<br>&gt; I am puzzled about some of the responses there, how can anyone expect to<br>&gt; randomly update packages on the system
using pip and not have it go wrong<br>&gt; on any distribution? This is why things like pipenv exist.<br><br>People don't understand that stuff until they dig to the details. And<br>even then, sometimes they forget and/or assume we have the resources
to<br>massively revamp our stack.<br><br>There are long-standing grievances here (see tiran's gist for example).<br>I have very little experience with Fedora/RH, but from their grumbling,<br>I assume they solve some problems for developers that we
don't:<br><br>1. It appears that multiple versions of cPython are trivially available.<br> Each version has its own site-packages.<br>2. Their python packaging is more developer-centric than user-centric.<br> More modules included in the install.
<br><br>We're pretty constrained on 1 by the debian security team policy. But<br>maybe there is discussion room there for non-supported Pythons?<br><br>We have taken steps to improve 2 by adding the python3-full package.<br><br>I could see a long
term strategy for having a system-python package that<br>provides a python3 binary used by Debian packages that need Python. And<br>a separate python package for developers that installs all the bells and<br>whistles immediately.<br><br>Achieving
this would require reorganizing the way we package almost all<br>Python, and I don't think we have the interest and resources to do that.<br><br>If pyenv was packaged in Debian (ITP #978149) it would probably be great<br>for new Python developers on
Debian.<br><br>Not sure what other small tactical steps we could take?<br><br>&gt; &gt; Perhaps someone else wants to comment on that conversation<br><br>Replied. I've engaged with these guys on this stuff before. Let's see<br>where it goes...<br>
<br>Stefano<br><br>-- <br>Stefano Rivera<br> http://tumbleweed.org.za/<br> +1 415 683 3272<br><br></div></blockquote></div><img class="mailbird" width="1" height="1" style="border:0; width:1; height:1; display: none;" src="https://tracking.
getmailbird.com/OpenTrackingPixel/?messageId=Mailbird-49833c7e-fe6a-4af4-89c7-4b962879bffd@stufft.io&senderHash=57787FD98DE4EB8FA6FBC42D73E38B5233F15C50A1E6C93FEFCDDA0F25910CC6&recipientHash=
A680344B9C0D740131327222AC7C19830233B381EF4AD31267DCE32D1054E966&internalId=fdb3ba26-d564-43e8-bf82-f89b75d082f0" alt="163d8144-a584-4d1d-9d71-f86da864fde2">

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Overall, I think that's quite reasonable, but I think I need to pick at the both sides way this is framed.

I don't recall anyone here suggesting that it's a problem is Python upstream wants to ship Python binaries. While I think that there are issues with some of the upstream design choices, I understand why upstream packaging exists. Debian leverages it to
deliver our system.

I do find the common view upstream of distributions shipping anything Python as a problem to be super annoying. Not you, to be clear. I think you are one of the few people that does see both sides of the problem.

Personally, I've given up on any upstream interactions. I don't need pointless hostility. While there are technical concerns on both sides, socially I think the Python community isn't that interested in outside perspectives.

Thanks for sharing your thoughts,

Scott K

On May 26, 2024 11:23:03 PM UTC, Donald Stufft <donald@stufft.io> wrote:

I happen to be subscribed here, so figured I'd comment :)

FWIW I think the way the discussions are going... really in both locations.. is needlessly taking shots at each other.

I've commented on discuss.python.org, but figured I'd repeat myself here.

I think the way these discussions devolve into each "side" taking shots at each other and arguing over nonsense that doesn't matter does everyone a disservice. 

The facts of the matter, as far as I can tell:

- Distributors ship Python and a number of people find great value in that and it works great or at least fine for them.
- A number of users have needs or wants that are not well served by the Python that distributors ship, for one reason or another.

People in either of those groups sniping at each other for wanting the "wrong" things is completely unproductive. 

It's probably "fine" if Python.org wants to ship a linux binary, I suspect it'll have very little impact on distributors of Python (and might even make things better-- since some of the problems flow from an impedance mismatch, and it provides a

different escape hatch to point people towards if the distro Python doesn't suit their needs).

On 5/26/2024 9:39:25 AM, Stefano Rivera <stefanor@debian.org> wrote:
Hi Ian (2024.05.26_01:33:09_+0000)

I am puzzled about some of the responses there, how can anyone expect to
randomly update packages on the system using pip and not have it go wrong
on any distribution? This is why things like pipenv exist.

People don't understand that stuff until they dig to the details. And
even then, sometimes they forget and/or assume we have the resources to >massively revamp our stack.

There are long-standing grievances here (see tiran's gist for example).
I have very little experience with Fedora/RH, but from their grumbling,
I assume they solve some problems for developers that we don't:

1. It appears that multiple versions of cPython are trivially available.
Each version has its own site-packages.
2. Their python packaging is more developer-centric than user-centric.
More modules included in the install.

We're pretty constrained on 1 by the debian security team policy. But
maybe there is discussion room there for non-supported Pythons?

We have taken steps to improve 2 by adding the python3-full package.

I could see a long term strategy for having a system-python package that >provides a python3 binary used by Debian packages that need Python. And
a separate python package for developers that installs all the bells and >whistles immediately.

Achieving this would require reorganizing the way we package almost all >Python, and I don't think we have the interest and resources to do that.

If pyenv was packaged in Debian (ITP #978149) it would probably be great
for new Python developers on Debian.

Not sure what other small tactical steps we could take?

Perhaps someone else wants to comment on that conversation

Replied. I've engaged with these guys on this stuff before. Let's see
where it goes...

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

[163d8144-a584-4d1d-9d71-f86da864fde2]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:

While there are technical concerns on both sides, socially I think the
Python community isn't that interested in outside perspectives.

I managed to dig up these notes from the packaging summit at PyCon:

https://hackmd.io/@pradyunsg/pycon2024-pack-summit

Here's the summit page itself:

https://us.pycon.org/2024/events/packaging-summit/

There is some fixation on the "system Python" in distributions, and the following remarks:

"At least one distro team is working on moving their own Python out of the way, so users can install their own Python packages [...] Fedora tried platform-python and it broke everything, so it didn't really work"

Given the proliferation of "virtual environments" around Python, where you just pick your own Python version and accompanying packages, I find it odd that the Python packaging community gets so hung up on the system Python. Do they want it to just go away or not be on the path or something? Wouldn't having a singular upstream Python just cause the same problems when someone finds that it isn't the version they need?

For my own amusement and to confirm my own memories, I went back in time to check the Python Web site in the 1990s, and back then there was no problem in providing a binary for Linux and the Unix products of choice from that era. AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4, plus a Linux binary supplied either as an RPM or in a plain archive:

https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/ binaries-1.4/

What is stopping anyone from doing that all over again? The user downloads the binary, puts it in their current directory, and runs their software. Could it be that the burden of support is perhaps a little greater than one might expect?

Because from that starting point, you have to build multiple versions, and then you have to build accompanying libraries, and then you have to support third-party packages which need third-party libraries. It wasn't a surprise that things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to the proprietary platforms, whereas distributions emerged around Linux to manage this complexity and provide all this software.

It is easy for the various language communities to focus only on their own ecosystems, but everybody's software has to work together. And then there are companies targeting various markets that demand software built on a selection of different technologies, so you get perspectives like these:

"Why did the PyPI and Conda ecosystem get created? It was originally created as an educational teaching language. If all of your tools are in Python, all of the things in your ecosystem are supposed to work well together. However, the tools that scientists and data scientists use are very commonly written in C, C++, etc. and so there’s something called a “native binary problem”. Making
this stuff compatible across the board is an incredibly challenging issue! Conda was created to resolve those binary compatibility issues."

I honestly don't know what these people think operating system distributions are doing if not solving the "native binary problem".

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Consider that they are the same people that recently removed
"distutils" from the standard library, because it was not maintained.
When they have well enough funding to assign someone to maintain it,
instead of relying on external projects to install packages.

I think they are in the bubble of "people who are here on discuss" and
forgot the 99.9% who is not.

Il giorno lun 3 giu 2024 alle ore 00:08 Paul Boddie
<paul@boddie.org.uk> ha scritto:

On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:

While there are technical concerns on both sides, socially I think the Python community isn't that interested in outside perspectives.

I managed to dig up these notes from the packaging summit at PyCon:

https://hackmd.io/@pradyunsg/pycon2024-pack-summit

Here's the summit page itself:

https://us.pycon.org/2024/events/packaging-summit/

There is some fixation on the "system Python" in distributions, and the following remarks:

"At least one distro team is working on moving their own Python out of the way, so users can install their own Python packages [...] Fedora tried platform-python and it broke everything, so it didn't really work"

Given the proliferation of "virtual environments" around Python, where you just pick your own Python version and accompanying packages, I find it odd that the Python packaging community gets so hung up on the system Python. Do they want it to just go away or not be on the path or something? Wouldn't having a singular upstream Python just cause the same problems when someone finds that it isn't the version they need?

For my own amusement and to confirm my own memories, I went back in time to check the Python Web site in the 1990s, and back then there was no problem in providing a binary for Linux and the Unix products of choice from that era. AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4, plus a Linux binary supplied either as an RPM or in a plain archive:

https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/ binaries-1.4/

What is stopping anyone from doing that all over again? The user downloads the
binary, puts it in their current directory, and runs their software. Could it be that the burden of support is perhaps a little greater than one might expect?

Because from that starting point, you have to build multiple versions, and then you have to build accompanying libraries, and then you have to support third-party packages which need third-party libraries. It wasn't a surprise that things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to
the proprietary platforms, whereas distributions emerged around Linux to manage this complexity and provide all this software.

It is easy for the various language communities to focus only on their own ecosystems, but everybody's software has to work together. And then there are companies targeting various markets that demand software built on a selection of different technologies, so you get perspectives like these:

"Why did the PyPI and Conda ecosystem get created? It was originally created as an educational teaching language. If all of your tools are in Python, all of the things in your ecosystem are supposed to work well together. However, the tools that scientists and data scientists use are very commonly written in
C, C++, etc. and so there’s something called a “native binary problem”. Making
this stuff compatible across the board is an incredibly challenging issue! Conda was created to resolve those binary compatibility issues."

I honestly don't know what these people think operating system distributions are doing if not solving the "native binary problem".

Paul

--
Salvo Tomaselli

"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di senso, ragione ed intelletto intendesse che noi ne facessimo a meno."
-- Galileo Galilei

http://ltworf.github.io

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

It's not an accurate characterization that distutils was removed simply because it wasn't maintained.

It was as fragile library, and it was difficult to make any changes to it because a number of things had implemented themselves by reaching into distutils and randomly monkeypatching various aspects. This made it hard to maintain it _at all_, because
when people had tried to improve it, it caused ecosystem wide breakages until things like setuptools, numpy.distutils, etc got patched again.

So the unofficial policy became "do not touch distutils", for fear of causing these breakages.

In the interim the packaging toolchain evolved to the point that having distutils in the stdlib was no longer of general benefit, and in fact made things worse because people had grown accustomed to things like `from distutils import setup` being
transparently monkeypatched to be setuptools under the covers.

Ultimately, distutils had diverged so far from modern packaging tooling (due to the unofficial policy of not touching it), that it's existence was more or less a footgun, and it's only real purpose anymore was to be an implementation detail of other
libraries, which you had to install from PyPI, so it was decided it was better to remove it rather than leave it around.
On 6/3/2024 1:37:32 AM, Salvo Tomaselli <tiposchi@tiscali.it> wrote:
Consider that they are the same people that recently removed
"distutils" from the standard library, because it was not maintained.
When they have well enough funding to assign someone to maintain it,
instead of relying on external projects to install packages.

I think they are in the bubble of "people who are here on discuss" and
forgot the 99.9% who is not.

Il giorno lun 3 giu 2024 alle ore 00:08 Paul Boddie
ha scritto:

On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:

While there are technical concerns on both sides, socially I think the Python community isn't that interested in outside perspectives.

I managed to dig up these notes from the packaging summit at PyCon:

https://hackmd.io/@pradyunsg/pycon2024-pack-summit

Here's the summit page itself:

https://us.pycon.org/2024/events/packaging-summit/

There is some fixation on the "system Python" in distributions, and the following remarks:

"At least one distro team is working on moving their own Python out of the way, so users can install their own Python packages [...] Fedora tried platform-python and it broke everything, so it didn't really work"

Given the proliferation of "virtual environments" around Python, where you just pick your own Python version and accompanying packages, I find it odd that the Python packaging community gets so hung up on the system Python. Do they want it to just go away or not be on the path or something? Wouldn't having a singular upstream Python just cause the same problems when someone finds that it isn't the version they need?

For my own amusement and to confirm my own memories, I went back in time to check the Python Web site in the 1990s, and back then there was no problem in providing a binary for Linux and the Unix products of choice from that era. AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4, plus a Linux binary supplied either as an RPM or in a plain archive:

https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/ binaries-1.4/

What is stopping anyone from doing that all over again? The user downloads the
binary, puts it in their current directory, and runs their software. Could it be that the burden of support is perhaps a little greater than one might expect?

Because from that starting point, you have to build multiple versions, and then you have to build accompanying libraries, and then you have to support third-party packages which need third-party libraries. It wasn't a surprise that things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to
the proprietary platforms, whereas distributions emerged around Linux to manage this complexity and provide all this software.

It is easy for the various language communities to focus only on their own ecosystems, but everybody's software has to work together. And then there are companies targeting various markets that demand software built on a selection of different technologies, so you get perspectives like these:

"Why did the PyPI and Conda ecosystem get created? It was originally created as an educational teaching language. If all of your tools are in Python, all of the things in your ecosystem are supposed to work well together. However, the tools that scientists and data scientists use are very commonly written in
C, C++, etc. and so there’s something called a “native binary problem”. Making
this stuff compatible across the board is an incredibly challenging issue! Conda was created to resolve those binary compatibility issues."

I honestly don't know what these people think operating system distributions are doing if not solving the "native binary problem".

Paul

--
Salvo Tomaselli

"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di senso, ragione ed intelletto intendesse che noi ne facessimo a meno."
-- Galileo Galilei

http://ltworf.github.io


[937b435c-411d-4ec4-8588-691dce8610f2]
<div id="__MailbirdStyleContent" style="font-size: 12pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
It's not an accurate characterization that distutils was removed simply because it wasn't maintained.<div><br></div><div>It was as fragile
library, and it was difficult to make any changes to it because a number of things had implemented themselves by reaching into distutils and randomly monkeypatching various aspects. This made it hard to maintain it _at all_, because when people had tried
to improve it, it caused ecosystem wide breakages until things like setuptools, numpy.distutils, etc got patched again.</div><div><br></div><div>So the unofficial policy became "do not touch distutils", for fear of causing these breakages.</div><div><br><
/div><div>In the interim the packaging toolchain evolved to the point that having distutils in the stdlib was no longer of general benefit, and in fact made things worse because people had grown accustomed to things like `from distutils import setup`
being transparently monkeypatched to be setuptools under the covers.</div><div><br></div><div>Ultimately, distutils had diverged so far from modern packaging tooling (due to the unofficial policy of not touching it), that it's existence was more or less
a footgun, and it's only real purpose anymore was to be an implementation detail of other libraries, which you had to install from PyPI, so it was decided it was better to remove it rather than leave it around.</div><blockquote class="history_container"
type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left: 10px;min-width: 500px"> <p style="color: #AAAAAA; margin-top: 10px;">On 6/3/2024 1:37:32 AM, Salvo Tomaselli &lt;
tiposchi@tiscali.it&gt; wrote:</p><div style="font-family:Arial,Helvetica,sans-serif">Consider that they are the same people that recently removed<br>"distutils" from the standard library, because it was not maintained.<br>When they have well enough
funding to assign someone to maintain it,<br>instead of relying on external projects to install packages.<br><br>I think they are in the bubble of "people who are here on discuss" and<br>forgot the 99.9% who is not.<br><br>Il giorno lun 3 giu 2024
alle ore 00:08 Paul Boddie<br><paul@boddie.org.uk> ha scritto:<br>&gt;<br>&gt; On Monday, 27 May 2024 04:07:34 CEST Scott Kitterman wrote:<br>&gt; &gt;<br>&gt; &gt; While there are technical concerns on both sides, socially I think the<br>&gt; &gt;
Python community isn't that interested in outside perspectives.<br>&gt;<br>&gt; I managed to dig up these notes from the packaging summit at PyCon:<br>&gt;<br>&gt; https://hackmd.io/@pradyunsg/pycon2024-pack-summit<br>&gt;<br>&gt; Here's the summit
page itself:<br>&gt;<br>&gt; https://us.pycon.org/2024/events/packaging-summit/<br>&gt;<br>&gt; There is some fixation on the "system Python" in distributions, and the<br>&gt; following remarks:<br>&gt;<br>&gt; "At least one distro team is working
on moving their own Python out of the<br>&gt; way, so users can install their own Python packages [...] Fedora tried<br>&gt; platform-python and it broke everything, so it didn't really work"<br>&gt;<br>&gt; Given the proliferation of "virtual
environments" around Python, where you<br>&gt; just pick your own Python version and accompanying packages, I find it odd<br>&gt; that the Python packaging community gets so hung up on the system Python. Do<br>&gt; they want it to just go away or not
be on the path or something? Wouldn't<br>&gt; having a singular upstream Python just cause the same problems when someone<br>&gt; finds that it isn't the version they need?<br>&gt;<br>&gt; For my own amusement and to confirm my own memories, I went
back in time to<br>&gt; check the Python Web site in the 1990s, and back then there was no problem in<br>&gt; providing a binary for Linux and the Unix products of choice from that era.<br>&gt; AIX, FreeBSD, HP-UX, Irix, OSF/1, Solaris, and SunOS 4,
plus a Linux binary<br>&gt; supplied either as an RPM or in a plain archive:<br>&gt;<br>&gt; https://web.archive.org/web/20021030010019/http://www.python.org/ftp/python/<br>&gt; binaries-1.4/<br>&gt;<br>&gt; What is stopping anyone from doing that
all over again? The user downloads the<br>&gt; binary, puts it in their current directory, and runs their software. Could it<br>&gt; be that the burden of support is perhaps a little greater than one might<br>&gt; expect?<br>&gt;<br>&gt; Because
from that starting point, you have to build multiple versions, and<br>&gt; then you have to build accompanying libraries, and then you have to support<br>&gt; third-party packages which need third-party libraries. It wasn't a surprise<br>&gt; that
things like Sun Freeware (http://www.sunfreeware.com) emerged to cater to<br>&gt; the proprietary platforms, whereas distributions emerged around Linux to<br>&gt; manage this complexity and provide all this software.<br>&gt;<br>&gt; It is easy for
the various language communities to focus only on their own<br>&gt; ecosystems, but everybody's software has to work together. And then there are<br>&gt; companies targeting various markets that demand software built on a selection<br>&gt; of
different technologies, so you get perspectives like these:<br>&gt;<br>&gt; "Why did the PyPI and Conda ecosystem get created? It was originally created<br>&gt; as an educational teaching language. If all of your tools are in Python, all<br>&gt; of
the things in your ecosystem are supposed to work well together. However,<br>&gt; the tools that scientists and data scientists use are very commonly written in<br>&gt; C, C++, etc. and so there’s something called a “native binary problem”.
Making<br>&gt; this stuff compatible across the board is an incredibly challenging issue!<br>&gt; Conda was created to resolve those binary compatibility issues."<br>&gt;<br>&gt; I honestly don't know what these people think operating system
distributions<br>&gt; are doing if not solving the "native binary problem".<br>&gt;<br>&gt; Paul<br>&gt;<br>&gt;<br><br><br>-- <br>Salvo Tomaselli<br><br>"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di<br>senso,
ragione ed intelletto intendesse che noi ne facessimo a meno."<br> -- Galileo Galilei<br><br>http://ltworf.github.io<br><br></paul@boddie.org.uk></div></blockquote></div><img class="mailbird" width="1" height="1" style="border:0;
width:1; height:1; display: none;" src="https://tracking.getmailbird.com/OpenTrackingPixel/?messageId=Mailbird-e59afa0e-82f5-4a81-97a8-457e817c9be3@stufft.io&senderHash=57787FD98DE4EB8FA6FBC42D73E38B5233F15C50A1E6C93FEFCDDA0F25910CC6&recipientHash=
A680344B9C0D740131327222AC7C19830233B381EF4AD31267DCE32D1054E966&internalId=e233c051-c56e-4126-8d52-29296d4fc7a0" alt="937b435c-411d-4ec4-8588-691dce8610f2">

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Monday, 3 June 2024 16:27:29 CEST Donald Stufft wrote:

In the interim the packaging toolchain evolved to the point that having distutils in the stdlib was no longer of general benefit, and in fact made things worse because people had grown accustomed to things like `from distutils import setup` being transparently monkeypatched to be setuptools under the covers.

The way that distutils could not be relied upon to behave in a sensible way is entirely the fault of the developer(s) of setuptools having free rein to corrupt the Python packaging stack. In environments where I wanted to install to a particular location only the software I had already acquired, which is largely what the deployment element involved in distribution packaging can be reduced to, I did not want to be dealing with setuptools when distutils would do, nor with random hacks introduced to make distutils behave like setuptools.

For a while, I routinely stripped out unnecessary setuptools references from setup.py files, put distutils support back in, and mostly got the desired effect. But in the Python world, once someone teases some fancy new features and they catch on, everybody else has to hold on for the wild ride and budget for the consequences.

(For some software I have been trying to package, I see now that there is no setup.py or anything else, with some more "magic" introduced to be processed
by yet another tool. I apparently have to get with it, or something to that effect, which severely diminishes my interest in packaging that software at all. The outcome actually affects the Debian project directly, not that very many people seem to care, however.)

I suppose it also didn't help that distutils entered the standard library in the era where it was apparently acceptable to get one's code included and to then declare the job done. Back when Python 3 was initially introduced, I suggested that the standard library be reviewed and fixed up, especially since there was going to be a compatibility break anyway, but there was no appetite for it.

Still, I appreciate you engaging with this forum, even if it probably means having to defend decisions made by others.

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I remember doing this pull request over a decade ago:

https://github.com/broadinstitute/xtermcolor/pull/12/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7

to a project that used setuptools + something to download setuptools
to be able to use it.

I fail to see how that was in any way better than just using what was
in the stdlib.

Setuptools has always been a weird thing to possibly get rid of… and
why have 485 lines of completely unmaintained script to download
setuptools when distutils would do? Well I think the author just
followed some tutorial or guide at that time. But it was problematic
from the beginning that setuptools emerged. And now things are even
worse. Even if my post with the link to xkcd got removed from the
python discuss, the situation remains the same, with new things to do
the same thing being invented all the time.

Il giorno lun 3 giu 2024 alle ore 17:57 Paul Boddie
<paul@boddie.org.uk> ha scritto:

On Monday, 3 June 2024 16:27:29 CEST Donald Stufft wrote:

In the interim the packaging toolchain evolved to the point that having distutils in the stdlib was no longer of general benefit, and in fact made things worse because people had grown accustomed to things like `from distutils import setup` being transparently monkeypatched to be setuptools under the covers.

The way that distutils could not be relied upon to behave in a sensible way is
entirely the fault of the developer(s) of setuptools having free rein to corrupt the Python packaging stack. In environments where I wanted to install to a particular location only the software I had already acquired, which is largely what the deployment element involved in distribution packaging can be reduced to, I did not want to be dealing with setuptools when distutils would do, nor with random hacks introduced to make distutils behave like setuptools.

For a while, I routinely stripped out unnecessary setuptools references from setup.py files, put distutils support back in, and mostly got the desired effect. But in the Python world, once someone teases some fancy new features and they catch on, everybody else has to hold on for the wild ride and budget for the consequences.

(For some software I have been trying to package, I see now that there is no setup.py or anything else, with some more "magic" introduced to be processed by yet another tool. I apparently have to get with it, or something to that effect, which severely diminishes my interest in packaging that software at all. The outcome actually affects the Debian project directly, not that very many people seem to care, however.)

I suppose it also didn't help that distutils entered the standard library in the era where it was apparently acceptable to get one's code included and to then declare the job done. Back when Python 3 was initially introduced, I suggested that the standard library be reviewed and fixed up, especially since
there was going to be a compatibility break anyway, but there was no appetite for it.

Still, I appreciate you engaging with this forum, even if it probably means having to defend decisions made by others.

Paul

--
Salvo Tomaselli

"Io non mi sento obbligato a credere che lo stesso Dio che ci ha dotato di senso, ragione ed intelletto intendesse che noi ne facessimo a meno."
-- Galileo Galilei

http://ltworf.github.io

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)