Hey.
I hope this is not too off topic.
As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g.
setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.
I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.
The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).
But then of course it also fails to e.g. do an editable install of a
locally developed package, when it tries to resolve the dependencies.
So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively
consider them as being resolved) *if* the package is locally installed
from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)
Thanks,
Philippe.
PS: Please keep me CCed.
You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?
On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:
I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607
On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:
Hey.
I hope this is not too off topic.
As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.
I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.
The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).
But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.
So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)
Thanks,
Philippe.
PS: Please keep me CCed.
https://packaging.python.org/en/latest/specifications/recording-installed-packages/
defines the python spec where a package such as pyparsing would create
a tree of files under:
site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is essentially a sha256-based manifest of files and some others.
On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:
You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?
On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:
I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested similar, such as https://github.com/pypa/pip/issues/11607
On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:
Hey.
I hope this is not too off topic.
As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully not running) proxy.
I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.
The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).
But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.
So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)
Thanks,
Philippe.
PS: Please keep me CCed.
You've made me wonder if it would be feasible to have a debian-centric
tool that populates .dist-info from debs?
On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote:
I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607
On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote:
Hey.
I hope this is not too off topic.
As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
not running) proxy.
I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.
The main motivation are security constraints, so I had to configure
pip so that it cannot just download packages from PyPI (which is
rather easy, simply setting no-index in pip.conf).
But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.
So I wondered whether it's possible to prevent pip from downloading
any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
(If a dependency isn't installed from a package it may of course fail.)
Thanks,
Philippe.
PS: Please keep me CCed.
> > similar, such as https://github.com/pypa/pip/issues/11607<br>> ><br>> > On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:<br>> > ><br>> > > Hey.<br>> > ><br>> > >I hope this is not too off topic.<br>> > ><br>> > > As far as I understand, dh-python, when building packages somehow<br>> > > automatically uses the Debian package names and even prevents e.g.<br>> > > setuptools
I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
similar, such as https://github.com/pypa/pip/issues/11607
I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.
The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:
On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:
I'm pretty sure that most if not all debian packages already ship
the required information for pip to see them as installed, and if
they are installed and they satisfy the dependency constraints that
pip has for those projects, then they'll be used.
Also:
$ dpkg -l python3-setuptools
...
ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements
Yet when I do e.g.:
$ pip install --editable .
Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example
Installing build dependencies ... error
error: subprocess-exited-with-error
Hi Philippe (2023.02.13_01:11:28_+0000)
On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:
I'm pretty sure that most if not all debian packages already ship
the required information for pip to see them as installed, and if
they are installed and they satisfy the dependency constraints that
pip has for those projects, then they'll be used.
Yeah, most packages should ship .egg-info/.dist-info.
Also:Enhancements
$ dpkg -l python3-setuptools
...
ii python3-setuptools 66.1.1-1 all Python3 Distutils
Yet when I do e.g.:writeable
$ pip install --editable .
Defaulting to user installation because normal site-packages is not
Obtaining file:///home/test/example
Installing build dependencies ... error
error: subprocess-exited-with-error
You sure it isn't doing an isolated build? Try --no-build-isolation.
SR
--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272
My suggestion to the pip folks was a plugin system and extension point for "install x" package that distros could provide implementations for
As someone who does Python software development on Debian constantly
for their $dayjob, my best advice is to just install things from
PyPI into and run them from venvs/virtualenvs. The default "--user"
install mode pip offers is fragile and leaves you with potential
conflicts anyway if you need different versions of dependencies for
different things.
To your original question, if you really want to use some
Debian-packaged libraries mixed with things installed from source or
from PyPI, make your venv with the --system-site-packages option.
--
Jeremy Stanley
I agree that is "easiest" but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as badThese, or comparable, problems also happen in Debian. For example, you
as old CPAN when it comes to packages disappearing or being broken or depending on totally random versions
I agree that is "easiest" but what I was after was the ability to[...]
restrict myself to the curated and signed packages from debian,
pypi is just as bad as old CPAN when it comes to packages
disappearing or being broken or depending on totally random
versions
You sure it isn't doing an isolated build? Try --no-build-isolation.
Well in my case the main motivation was security (i.e. only using
code) that has security support by Debian.
But shouldn't that use case also be interesting for Debian
Maintainers? Whenever their pip would need to download something from
PyPI, it would mean that some dependency is likely not fulfilled in
Debian (unless of course that Debian package is simply not installed).
But shouldn't that use case also be interesting for DebianDo you mean the scenario with packaging some new software? In that case
Maintainers? Whenever their pip would need to download something from
PyPI, it would mean that some dependency is likely not fulfilled in
Debian (unless of course that Debian package is simply not installed).
There is probably some value there. You're safer from a variety of
attacks that *could* theoretically happen on PyPI.
But, let me deflate Debian's reputation a bit here.
Debian security support doesn't mean you're completely protected. There
is probably a human behind a Debian upload that has vetted the upload
and thinks it is safe. They thought this thing was useful to package for Debian (so probably not malware), and did some review to see that it installed itself correctly. They may have reviewed the upstream code,
they may not have. They may review new upstream version diffs, they may
not. (Generally, small things are easy to review, big complex things are impossible to.)
But, on balance, for many problems the gains here aren't worth the pain
of restricting yourself to Python modules published in Debian stable releases.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 350 |
Nodes: | 16 (2 / 14) |
Uptime: | 09:20:25 |
Calls: | 7,625 |
Files: | 12,793 |
Messages: | 5,686,482 |