Hey.

I hope this is not too off topic.

As far as I understand, dh-python, when building packages somehow
automatically uses the Debian package names and even prevents e.g.
setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.


I wondered whether it's possible to make tools like pip and setuptools
directly use the Debian python packages when resolving dependencies.

The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).

But then of course it also fails to e.g. do an editable install of a
locally developed package, when it tries to resolve the dependencies.

So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively
consider them as being resolved) *if* the package is locally installed
from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)


Thanks,
Philippe.

PS: Please keep me CCed.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607

On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

Hey.

I hope this is not too off topic.

As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g.
setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.

I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).

But then of course it also fails to e.g. do an editable install of a
locally developed package, when it tries to resolve the dependencies.

So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively
consider them as being resolved) *if* the package is locally installed
from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)

Thanks,
Philippe.

PS: Please keep me CCed.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
essentially a sha256-based manifest of files and some others.

On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:

You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?

On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:

I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607

On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

Hey.

I hope this is not too off topic.

As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.

I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).

But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)

Thanks,
Philippe.

PS: Please keep me CCed.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

An example using alpine (it seems some packages sporadically include
the dist-info folders): There is some more recent info here https://peps.python.org/pep-0627/

/ # apk add py3-pip
(1/6) Installing py3-six (1.16.0-r3)
(2/6) Installing py3-retrying (1.3.3-r3)
(3/6) Installing py3-parsing (3.0.9-r0)
(4/6) Installing py3-packaging (21.3-r2)
(5/6) Installing py3-setuptools (65.6.0-r0)
(6/6) Installing py3-pip (22.3.1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 82 MiB in 34 packages
/ # pip install pyparsing==3.0.9
Requirement already satisfied: pyparsing==3.0.9 in /usr/lib/python3.10/site-packages (3.0.9)

In the above example, pip refuses to double-install pyparsing only
because of the dist-info file, however lots of directly pip installed
packages produce egg-info folders instead as a I guess those are
installed from source by pip. Debian could produce dist-info folders
for those instead.

Just thinking out-loud

Ian

On Sun, 12 Feb 2023 at 08:18, Ian Norton <inorton@gmail.com> wrote:

https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is essentially a sha256-based manifest of files and some others.

On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:

You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?

On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:

I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested similar, such as https://github.com/pypa/pip/issues/11607

On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

Hey.

I hope this is not too off topic.

As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully not running) proxy.

I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).

But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)

Thanks,
Philippe.

PS: Please keep me CCed.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.

The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:

- pip doesn't know how to invoke apt (and yum, and Conda, and everything else). - how do we translate "foo" on PyPI to the equivalent package in Debian (and Fedora, and Conda, etc)
- how do we handle virtual environments?

Nobody has ever come up with a particularly good answer to those problems, so pip upstream hasn't been able to even consider doing something like that.

Of course Debian could patch it's copy of pip to do that, that would make some of those questions easier, but not all of them, and would drastically diverge it's behavior from the "expected" behavior, which makes it a rough idea in general for Debian to
diverge that drastically.
On 2/12/2023 3:18:55 AM, Ian Norton <inorton@gmail.com> wrote: https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
essentially a sha256-based manifest of files and some others.

On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote:

You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?

On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote:

I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607

On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote:

Hey.

I hope this is not too off topic.

As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.

I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).

But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)

Thanks,
Philippe.

PS: Please keep me CCed.

[88f0dcef-4969-48af-98f1-0c69549b4875]
<div id="__MailbirdStyleContent" style="font-size: 12pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are
installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.<div class="mb_sig"></div> <div><br></div><div>The question of
having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:</div><div><br></div><div>- pip doesn't know how to invoke apt (and yum, and Conda, and everything else).</div><div>- how do we translate "foo"
on PyPI to the equivalent package in Debian (and Fedora, and Conda, etc)</div><div>- how do we handle virtual environments?</div><div><br></div><div>Nobody has ever come up with a particularly good answer to those problems, so pip upstream hasn't been
able to even consider doing something like that.</div><div><br></div><div>Of course Debian could patch it's copy of pip to do that, that would make some of those questions easier, but not all of them, and would drastically diverge it's behavior from the "
expected" behavior, which makes it a rough idea in general for Debian to diverge that drastically.</div><blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left:
10px;min-width: 500px"> <p style="color: #AAAAAA; margin-top: 10px;">On 2/12/2023 3:18:55 AM, Ian Norton &lt;inorton@gmail.com&gt; wrote:</p><div style="font-family:Arial,Helvetica,sans-serif">https://packaging.python.org/en/
latest/specifications/recording-installed-packages/<br>defines the python spec where a package such as pyparsing would create<br>a tree of files under:<br>site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is<br>essentially a sha256-
based manifest of files and some others.<br><br>On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:<br>&gt;<br>&gt; You've made me wonder if it would be feasible to have a debian-centric<br>&gt; tool that populates .dist-info from
debs?<br>&gt;<br>&gt; On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:<br>&gt; &gt;<br>&gt; &gt; I requested this kind of thing from the pip folks as<br>&gt; &gt; https://github.com/pypa/pip/issues/11644 and others have requested<

&gt; &gt; similar, such as https://github.com/pypa/pip/issues/11607<br>&gt; &gt;<br>&gt; &gt; On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; Hey.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;

I hope this is not too off topic.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; As far as I understand, dh-python, when building packages somehow<br>&gt; &gt; &gt; automatically uses the Debian package names and even prevents e.g.<br>&gt; &gt; &gt; setuptools
from downloading any dependencies by setting a (hopefully<br>&gt; &gt; &gt; not running) proxy.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; I wondered whether it's possible to make tools like pip and setuptools<br>&gt; &gt; &gt; directly
use the Debian python packages when resolving dependencies.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; The main motivation are security constraints, so I had to configure<br>&gt; &gt; &gt; pip so that it cannot just download packages from PyPI (which is<br>&
gt; &gt; &gt; rather easy, simply setting no-index in pip.conf).<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; But then of course it also fails to e.g. do an editable install of a<br>&gt; &gt; &gt; locally developed package, when it tries to resolve the
dependencies.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; So I wondered whether it's possible to prevent pip from downloading<br>&gt; &gt; &gt; any remote stuff, while still resolving dependencies (respectively<br>&gt; &gt; &gt; consider them as being
resolved) *if* the package is locally installed<br>&gt; &gt; &gt; from the Debian archive?<br>&gt; &gt; &gt; (If a dependency isn't installed from a package it may of course fail.)<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; Thanks,<br>&
gt; &gt; &gt; Philippe.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; PS: Please keep me CCed.<br>&gt; &gt; &gt;<br><br></philcerf@gmail.com></inorton@gmail.com></inorton@gmail.com></div></blockquote></div><img class="mailbird" width="1" height="1" style="
border:0; width:1; height:1; display: none;" src="https://tracking.getmailbird.com/OpenTrackingPixel/?messageId=Mailbird-3d2f35ae-00ad-4864-a8f9-a39876df65ae@stufft.io&senderHash=57787FD98DE4EB8FA6FBC42D73E38B5233F15C50A1E6C93FEFCDDA0F25910CC6&
recipientHash=A680344B9C0D740131327222AC7C19830233B381EF4AD31267DCE32D1054E966&internalId=9e1e3ba6-db12-46a9-85ae-4abc99011e73" alt="88f0dcef-4969-48af-98f1-0c69549b4875">

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Feb 12, 2023 at 9:05 AM Ian Norton <inorton@gmail.com> wrote:

I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607

While they wrote it would be discussed, it doesn't really seem as if
they would care too much or as if anything would have happened since
then? :-(

I cannot really comment too much about the rest you wrote, as I'm not
really a Python or Debian packaging expert.

What I'd like to have is "simply" ;-) to prevent installation of any
remote code (as pip and similar tools, also for other languages, do)
and still use them as far as I have the packages from Debian
installed.
So for example, even if the pyproject.toml would specify some version
of a dependency that is not in Debian, it should rather fail than
downloading anything from somewhere.

Regards,
Philippe

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey.


On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.

Are you sure? I have a test project with a pyproject.toml that contains: [build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

i.e. no version requirement on setuptools.

Also:
$ dpkg -l python3-setuptools
...
ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements

Yet when I do e.g.:
$ pip install --editable .
Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example
Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
ERROR: Could not find a version that satisfies the requirement
setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a
problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a
problem with pip.

Now the particular error probably comes from my pip.conf, because I
set --no-index, at least when I try to install some package that is
not installed as Debian package, e.g.:
$ pip install acme
Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement acme
(from versions: none)
ERROR: No matching distribution found for acme

But I need something like that to actually disable any remote downloads.

Perhaps you're right however, cause when I run with an empty pip.conf,
it does e.g. download files to:
~/.local/lib/python3.11/site-packages/natsort*
(which I set as a project dependency), but it doesn't create something
like that for setuptools.

On the other hand:
I've just installed Debian's python3-natsort, rm -rf ~/.local/ and did
pip install --editable . again (without pip.conf), and it still seems
to download and install natsort to ~/.local .
So that doesn't look, as if it would recognize the one from Debian.

But even if that would actually work, why does it need the index at https://pypi.org/simple to use the local packages from Debian?

The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:

That might be nice for some users, but I personally wouldn't even need
or want that. It should perhaps rather just tell which Debian packages
need to be installed.


Thanks,
Philippe.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Philippe (2023.02.13_01:11:28_+0000)

On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

I'm pretty sure that most if not all debian packages already ship
the required information for pip to see them as installed, and if
they are installed and they satisfy the dependency constraints that
pip has for those projects, then they'll be used.

Yeah, most packages should ship .egg-info/.dist-info.

Also:
$ dpkg -l python3-setuptools
...
ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements

Yet when I do e.g.:
$ pip install --editable .
Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example
Installing build dependencies ... error
error: subprocess-exited-with-error

You sure it isn't doing an isolated build? Try --no-build-isolation.

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

My suggestion to the pip folks was a plugin system and extension point for "install x" package that distros could provide implementations for

On Wed, 15 Feb 2023, 16:35 Stefano Rivera, <stefanor@debian.org> wrote:

Hi Philippe (2023.02.13_01:11:28_+0000)

On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

I'm pretty sure that most if not all debian packages already ship
the required information for pip to see them as installed, and if
they are installed and they satisfy the dependency constraints that
pip has for those projects, then they'll be used.

Yeah, most packages should ship .egg-info/.dist-info.

Also:
$ dpkg -l python3-setuptools
...
ii python3-setuptools 66.1.1-1 all Python3 Distutils

Enhancements

Yet when I do e.g.:
$ pip install --editable .
Defaulting to user installation because normal site-packages is not

writeable

Obtaining file:///home/test/example
Installing build dependencies ... error
error: subprocess-exited-with-error

You sure it isn't doing an isolated build? Try --no-build-isolation.

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

<div dir="auto">My suggestion to the pip folks was a plugin system and extension point for &quot;install x&quot; package that distros could provide implementations for</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 15 Feb
2023, 16:35 Stefano Rivera, &lt;<a href="mailto:stefanor@debian.org">stefanor@debian.org</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Philippe (2023.02.13_01:11:28_+0000)<

&gt; On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft &lt;<a href="mailto:donald@stufft.io" target="_blank" rel="noreferrer">donald@stufft.io</a>&gt; wrote:<br>
&gt; &gt;<br>
&gt; &gt; I&#39;m pretty sure that most if not all debian packages already ship<br>
&gt; &gt; the required information for pip to see them as installed, and if<br> &gt; &gt; they are installed and they satisfy the dependency constraints that<br>
&gt; &gt; pip has for those projects, then they&#39;ll be used.<br>

Yeah, most packages should ship .egg-info/.dist-info.<br>

&gt; Also:<br>
&gt; $ dpkg -l python3-setuptools<br>
&gt; ...<br>
&gt; ii  python3-setuptools 66.1.1-1     all          Python3 Distutils Enhancements<br>
&gt; <br>
&gt; Yet when I do e.g.:<br>
&gt; $ pip install --editable .<br>
&gt; Defaulting to user installation because normal site-packages is not writeable<br>
&gt; Obtaining file:///home/test/example<br>
&gt;   Installing build dependencies ... error<br>
&gt;   error: subprocess-exited-with-error<br>

You sure it isn&#39;t doing an isolated build? Try --no-build-isolation.<br>

SR<br>

-- <br>
Stefano Rivera<br>
  <a href="http://tumbleweed.org.za/" rel="noreferrer noreferrer" target="_blank">http://tumbleweed.org.za/</a><br>
  +1 415 683 3272<br>
</blockquote></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Ian (2023.02.15_18:07:39_+0000)

My suggestion to the pip folks was a plugin system and extension point for "install x" package that distros could provide implementations for

Yeah, something like that could work. I don't know how useful it would
be, though.

Obviously, only root could use it (or root inside a container).

And our selection of Python modules is far from complete. It's not
Debian's intent to provide a mirror of PyPI within Debian. Generally
speaking, we package the modules that we find useful for supporting
building and shipping other python modules and applications.

We'll only have a single version of each package. And they're usually
not the versions developers want, because in any stable release they're probably out of date.

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

As someone who does Python software development on Debian constantly
for their $dayjob, my best advice is to just install things from
PyPI into and run them from venvs/virtualenvs. The default "--user"
install mode pip offers is fragile and leaves you with potential
conflicts anyway if you need different versions of dependencies for
different things.

To your original question, if you really want to use some
Debian-packaged libraries mixed with things installed from source or
from PyPI, make your venv with the --system-site-packages option.
--
Jeremy Stanley

-----BEGIN PGP SIGNATURE-----

iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmPtVf9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCkCpg//Rk39BAqwEaQfhacPJrYTzmdp9uSRPDnwVR3zQVQo+oUscK/ktZXm8xzx R/eF3voG60L0ZHkLTO3AnkUhGHShwy/wXLi7B9vvFpmZ45jrvIkNOAASmHh+Nb54 7FQ+w/rUrKcMP0p9v6PvDW7U66nsYM8vMmVmQmBpCCPBC42rTgDlRMtvCn8iWu/J fG4gdf3KFMMXAynAJHeuMJN2hrp3qn7L0NjCRk37VYPGUM9ws7HsQWXuCbkbNWm/ R+Z07sPReBynNhzQaoh5vbxjkXKOeE1pdnDwh4hDRBeXOMRHMJ4RI6WX6Oz3ZyCs 2x0yABhwWS4s/fFwqFiUkLAtvy6eCot8ynEtAEBrcZO2x4WeJGyM6XO1lOKKLvy/ nThEp5O5I9ktAgVROkjuZqyUau0tlqq9l29y4vMfbSzKiCEbsiPy4biyp87S8J8E 3uS+kTQ2TbNGhBqQQpL5gccNSlxaXyH6PbtpTg4R3Dr9NqA1wkoZ7PPzt7WFmIbS MGggsA2i+rt2Lj2fBJDqmZV6q6Qz5QTW1f42ADrXRx0ajBZRn5jH+r02xDVBPfB1 XgCk0TPwt3vKWunSKuQcmTXli9xTEkKsG+VCRLOwrh7EH7RcxItnL4LCsocTHlop g0CG0cXWFMYlKiesVWSZbCsA+xYOWm2wXDOSdaHdOC9sqKqryLk=
=kwwo
-----END PGP SIGNATURE-----

--- SoupGate-Win32

I agree that is "easiest" but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad
as old CPAN when it comes to packages disappearing or being broken or
depending on totally random versions

On Wed, 15 Feb 2023, 22:01 Jeremy Stanley, <fungi@yuggoth.org> wrote:

As someone who does Python software development on Debian constantly
for their $dayjob, my best advice is to just install things from
PyPI into and run them from venvs/virtualenvs. The default "--user"
install mode pip offers is fragile and leaves you with potential
conflicts anyway if you need different versions of dependencies for
different things.

To your original question, if you really want to use some
Debian-packaged libraries mixed with things installed from source or
from PyPI, make your venv with the --system-site-packages option.
--
Jeremy Stanley

<div dir="auto">I agree that is &quot;easiest&quot; but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad as old CPAN when it comes to packages disappearing or being broken or
depending on totally random versions</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 15 Feb 2023, 22:01 Jeremy Stanley, &lt;<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>&gt; wrote:<br></div><blockquote class="gmail_
quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">As someone who does Python software development on Debian constantly<br>
for their $dayjob, my best advice is to just install things from<br>
PyPI into and run them from venvs/virtualenvs. The default &quot;--user&quot;<br>
install mode pip offers is fragile and leaves you with potential<br>
conflicts anyway if you need different versions of dependencies for<br> different things.<br>

To your original question, if you really want to use some<br>
Debian-packaged libraries mixed with things installed from source or<br>
from PyPI, make your venv with the --system-site-packages option.<br>
-- <br>
Jeremy Stanley<br>
</blockquote></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Feb 16, 2023 at 01:12:49AM +0000, Ian Norton wrote:

I agree that is "easiest" but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad
as old CPAN when it comes to packages disappearing or being broken or depending on totally random versions

These, or comparable, problems also happen in Debian. For example, you
cannot expect any given module to be packaged (or not disappear in the
next release), sometimes the version in the repo is several years old and
of course packages in Debian can be broken, even if it's rare.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-02-16 01:12:49 +0000 (+0000), Ian Norton wrote:

I agree that is "easiest" but what I was after was the ability to
restrict myself to the curated and signed packages from debian,
pypi is just as bad as old CPAN when it comes to packages
disappearing or being broken or depending on totally random
versions

[...]

I think you missed my point, which was to explicitly create a venv
and install your project there instead of relying on pip's --user
default (which seemed to be resulting in errors for you). If you
create the venv with --system-site-packages enabled then it will
still use any Debian-packaged Python libraries you've installed.
--
Jeremy Stanley

-----BEGIN PGP SIGNATURE-----

iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmPuM7RfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCn23w/7B7WSHE/gouXczYjyPskU97seSJMbmCWRBfQOaUOSbQKttxm80VP8p/cr tUDfhGUsYfZTcFLap8lCfHXSbRqpW3zzdt2H4T7/xFTwpZYlk7iG8+QpC0dT1lzz GPt/uHESv8MpPeIN+W1/9CDU/wzAe4kAm+mdpwna8JfTjadYWpv/4wltcN2hRRmO xNMZtzLFuMdGiutB3z94+9CVxF4W1flcEKU9pfpnmfRPVXzgfv6q4Tw1btauPx3y MTUJIniSdCCQJj36ezsDGsec824jwKm+/kPQNoP0OldvlN39xvmuuyDn1S3GJc4O A5k8aYwhokffFSQrUz+QxwAqL7SlSpcBS/7oMYRbiZjB7TMX4aJdN3vfstnF4Kdc mL1e9EszybDC1tyHtAzZgMPhd5Z7gByM5sxYUewJzNVjt8uOMh1p1tkIjOGTTPwZ XAC/FIOfoKR+zEpRde8EpGRwbJe+6SLKAiQWU7WatNWEGQEZ81zdQl9N8fdQROt1 Dc8LLnQbRU3Vjudt3CdqglXZgyqMPOKrLFf2Hyos4S5Xr3QENwTD3gPB70CNbFgh d88u4dTZR8aVvj5AFlABKyFIu1CVzUE6S+8R7dyfmDLENHLFuKO5ehUMeTCGtLij dUDIsZkqNBC6fV3Vvn8Gr1lAUiHJ0MLekjCkGb+7wSX/nPg6his=
=39Lx
-----END PGP SIGNATURE-----

--- SoupGate-Win32

On Wed, Feb 15, 2023 at 5:35 PM Stefano Rivera <stefanor@debian.org> wrote:

You sure it isn't doing an isolated build? Try --no-build-isolation.

I absolutely am not :D (I'm really not a Python expert... or about
it's various build systems, pip, etc.).

And that seems to have done the trick.
And it further seems that there's a counter part with which I can do
python3 -m build
namely: --no-isolation

Am I right that these options simply mean that it doesn't use a venv
for the build, thereby "seeing" Debian's already installed packages
(or better said the eggs/wheels/whatsoever from them)?


Just for my confirmation:
If my goal is to simply never ever have pip/build/etc. download
anything from some location, but only use local sources (e.g. stuff
from Debian package) is the following safe?

I set /etc/pip.conf to:
[global]
no-index = true
index-url = https://localhost/
cert = /usr/local/share/ca-certificates/dummy.crt

(guess in principle the no-index should already be enough)

It seems to work (in the sense of blocking downloads) for both, pip
and python -m build ... but I don't quite understand why it also works
for build (thought that was independent of pip).


Awesome! Thanks guys :-)


Philippe

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Philippe (2023.02.17_02:17:49_+0000)

Well in my case the main motivation was security (i.e. only using
code) that has security support by Debian.

There is probably some value there. You're safer from a variety of
attacks that *could* theoretically happen on PyPI.

But, let me deflate Debian's reputation a bit here.
Debian security support doesn't mean you're completely protected. There
is probably a human behind a Debian upload that has vetted the upload
and thinks it is safe. They thought this thing was useful to package for
Debian (so probably not malware), and did some review to see that it
installed itself correctly. They may have reviewed the upstream code,
they may not have. They may review new upstream version diffs, they may
not. (Generally, small things are easy to review, big complex things are impossible to.)

For the security support, it's largely reliant on security issues being reported as CVEs, which security researchers usually do, but upstreams
often fail to do. And then it needs a volunteer to find/figure out the
fix and apply it to the version in Debian.

So, again, there is definitely value here. If you're just using software
from Debian stable releases, you know that some people have reviewed
some of it. And you can be reasonably confident that you're using the
same stack as some other people.

But, on balance, for many problems the gains here aren't worth the pain
of restricting yourself to Python modules published in Debian stable
releases.

But shouldn't that use case also be interesting for Debian
Maintainers? Whenever their pip would need to download something from
PyPI, it would mean that some dependency is likely not fulfilled in
Debian (unless of course that Debian package is simply not installed).

Generally speaking when I'm working on code, I install libraries in virtualenvs. This is what the upstream tooling expects and so it makes everything more convenient. All the work may be done in a container, but
I'm not restricting myself to Debian packages.

If I am using Debian packages for something, I'll install them with apt.
I don't need pip involved. This is where I don't find the pip plugin
idea that useful.

Some people try to write software specifically to run on Debian stable,
without any third party packages. For simple projects, this can work
well. But, there are downsides. You often find you have to couple code
changes to Debian's release cycle, which can get problematic. And nobody
will understand what you're trying to do :)

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Feb 17, 2023 at 03:17:49AM +0100, Philippe Cerfon wrote:

But shouldn't that use case also be interesting for Debian
Maintainers? Whenever their pip would need to download something from
PyPI, it would mean that some dependency is likely not fulfilled in
Debian (unless of course that Debian package is simply not installed).

Do you mean the scenario with packaging some new software? In that case
a packager would most likely review the deps, not run pip.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey.

On Fri, Feb 17, 2023 at 3:35 PM Stefano Rivera <stefanor@debian.org> wrote:

There is probably some value there. You're safer from a variety of
attacks that *could* theoretically happen on PyPI.

Well such language specific package repos (like pypi, npmjs, etc.)
have already been (numerous) times been victims from such attacks. So
it's not just a theoretical issue, I think.

But, let me deflate Debian's reputation a bit here.
Debian security support doesn't mean you're completely protected. There
is probably a human behind a Debian upload that has vetted the upload
and thinks it is safe. They thought this thing was useful to package for Debian (so probably not malware), and did some review to see that it installed itself correctly. They may have reviewed the upstream code,
they may not have. They may review new upstream version diffs, they may
not. (Generally, small things are easy to review, big complex things are impossible to.)

Sure, there are no code audits by Debian maintainers, there's no
guarantee that the maintainer retrieved the code in a secure way
(whatever that would be).

Still, even the download method of PyPI (and friends) - https - has
all kinds of issues. Typically one has something of around 150 root
CAs trusted, plus several thousands(?) of intermediate CAs from them.
Many of those in the hands quite questionable countries or organizations. Again, the DM might just as well download the code via https (and not
verify some upstream gpg, if present at all). But at least this would
cause *all* Debian users (of that package) to be compromised, which in
turn makes it much more likely that any compromise would get noticed.
Attacks specific to a single person are no longer easily possible.

But, on balance, for many problems the gains here aren't worth the pain
of restricting yourself to Python modules published in Debian stable releases.

Well I guess it's clear there's no 100% protection. In the end it's
simply just nice if one can easily choose whichever one wants. If
someone wants to use PyPI code, fine, if someone wants to restrict
himself to Debian-only, fine too.

Anyway, thanks :-)

Regards,
Philippe.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)