• can pip be made using local Debian packages for any dependencies

    From Philippe Cerfon@21:1/5 to All on Sun Feb 12 06:00:02 2023
    Hey.

    I hope this is not too off topic.

    As far as I understand, dh-python, when building packages somehow
    automatically uses the Debian package names and even prevents e.g.
    setuptools from downloading any dependencies by setting a (hopefully
    not running) proxy.


    I wondered whether it's possible to make tools like pip and setuptools
    directly use the Debian python packages when resolving dependencies.

    The main motivation are security constraints, so I had to configure
    pip so that it cannot just download packages from PyPI (which is
    rather easy, simply setting no-index in pip.conf).

    But then of course it also fails to e.g. do an editable install of a
    locally developed package, when it tries to resolve the dependencies.

    So I wondered whether it's possible to prevent pip from downloading
    any remote stuff, while still resolving dependencies (respectively
    consider them as being resolved) *if* the package is locally installed
    from the Debian archive?
    (If a dependency isn't installed from a package it may of course fail.)


    Thanks,
    Philippe.

    PS: Please keep me CCed.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ian Norton@21:1/5 to Philippe Cerfon on Sun Feb 12 09:10:01 2023
    I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
    similar, such as https://github.com/pypa/pip/issues/11607

    On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

    Hey.

    I hope this is not too off topic.

    As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g.
    setuptools from downloading any dependencies by setting a (hopefully
    not running) proxy.


    I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

    The main motivation are security constraints, so I had to configure
    pip so that it cannot just download packages from PyPI (which is
    rather easy, simply setting no-index in pip.conf).

    But then of course it also fails to e.g. do an editable install of a
    locally developed package, when it tries to resolve the dependencies.

    So I wondered whether it's possible to prevent pip from downloading
    any remote stuff, while still resolving dependencies (respectively
    consider them as being resolved) *if* the package is locally installed
    from the Debian archive?
    (If a dependency isn't installed from a package it may of course fail.)


    Thanks,
    Philippe.

    PS: Please keep me CCed.


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ian Norton@21:1/5 to Ian Norton on Sun Feb 12 09:20:01 2023
    https://packaging.python.org/en/latest/specifications/recording-installed-packages/
    defines the python spec where a package such as pyparsing would create
    a tree of files under:
    site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
    essentially a sha256-based manifest of files and some others.

    On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:

    You've made me wonder if it would be feasible to have a debian-centric
    tool that populates .dist-info from debs?

    On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:

    I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
    similar, such as https://github.com/pypa/pip/issues/11607

    On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

    Hey.

    I hope this is not too off topic.

    As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
    not running) proxy.


    I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

    The main motivation are security constraints, so I had to configure
    pip so that it cannot just download packages from PyPI (which is
    rather easy, simply setting no-index in pip.conf).

    But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

    So I wondered whether it's possible to prevent pip from downloading
    any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
    (If a dependency isn't installed from a package it may of course fail.)


    Thanks,
    Philippe.

    PS: Please keep me CCed.


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ian Norton@21:1/5 to Ian Norton on Sun Feb 12 09:30:01 2023
    An example using alpine (it seems some packages sporadically include
    the dist-info folders): There is some more recent info here https://peps.python.org/pep-0627/

    / # apk add py3-pip
    (1/6) Installing py3-six (1.16.0-r3)
    (2/6) Installing py3-retrying (1.3.3-r3)
    (3/6) Installing py3-parsing (3.0.9-r0)
    (4/6) Installing py3-packaging (21.3-r2)
    (5/6) Installing py3-setuptools (65.6.0-r0)
    (6/6) Installing py3-pip (22.3.1-r1)
    Executing busybox-1.35.0-r29.trigger
    OK: 82 MiB in 34 packages
    / # pip install pyparsing==3.0.9
    Requirement already satisfied: pyparsing==3.0.9 in /usr/lib/python3.10/site-packages (3.0.9)

    In the above example, pip refuses to double-install pyparsing only
    because of the dist-info file, however lots of directly pip installed
    packages produce egg-info folders instead as a I guess those are
    installed from source by pip. Debian could produce dist-info folders
    for those instead.

    Just thinking out-loud

    Ian

    On Sun, 12 Feb 2023 at 08:18, Ian Norton <inorton@gmail.com> wrote:

    https://packaging.python.org/en/latest/specifications/recording-installed-packages/
    defines the python spec where a package such as pyparsing would create
    a tree of files under:
    site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is essentially a sha256-based manifest of files and some others.

    On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:

    You've made me wonder if it would be feasible to have a debian-centric
    tool that populates .dist-info from debs?

    On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:

    I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested similar, such as https://github.com/pypa/pip/issues/11607

    On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:

    Hey.

    I hope this is not too off topic.

    As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully not running) proxy.


    I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

    The main motivation are security constraints, so I had to configure
    pip so that it cannot just download packages from PyPI (which is
    rather easy, simply setting no-index in pip.conf).

    But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

    So I wondered whether it's possible to prevent pip from downloading
    any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
    (If a dependency isn't installed from a package it may of course fail.)


    Thanks,
    Philippe.

    PS: Please keep me CCed.


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Donald Stufft@21:1/5 to Ian Norton on Sun Feb 12 19:40:02 2023
    I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.

    The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:

    - pip doesn't know how to invoke apt (and yum, and Conda, and everything else). - how do we translate "foo" on PyPI to the equivalent package in Debian (and Fedora, and Conda, etc)
    - how do we handle virtual environments?

    Nobody has ever come up with a particularly good answer to those problems, so pip upstream hasn't been able to even consider doing something like that.

    Of course Debian could patch it's copy of pip to do that, that would make some of those questions easier, but not all of them, and would drastically diverge it's behavior from the "expected" behavior, which makes it a rough idea in general for Debian to
    diverge that drastically.
    On 2/12/2023 3:18:55 AM, Ian Norton <inorton@gmail.com> wrote: https://packaging.python.org/en/latest/specifications/recording-installed-packages/
    defines the python spec where a package such as pyparsing would create
    a tree of files under:
    site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is
    essentially a sha256-based manifest of files and some others.

    On Sun, 12 Feb 2023 at 08:12, Ian Norton wrote:

    You've made me wonder if it would be feasible to have a debian-centric
    tool that populates .dist-info from debs?

    On Sun, 12 Feb 2023 at 08:05, Ian Norton wrote:

    I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
    similar, such as https://github.com/pypa/pip/issues/11607

    On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon wrote:

    Hey.

    I hope this is not too off topic.

    As far as I understand, dh-python, when building packages somehow automatically uses the Debian package names and even prevents e.g. setuptools from downloading any dependencies by setting a (hopefully
    not running) proxy.


    I wondered whether it's possible to make tools like pip and setuptools directly use the Debian python packages when resolving dependencies.

    The main motivation are security constraints, so I had to configure
    pip so that it cannot just download packages from PyPI (which is
    rather easy, simply setting no-index in pip.conf).

    But then of course it also fails to e.g. do an editable install of a locally developed package, when it tries to resolve the dependencies.

    So I wondered whether it's possible to prevent pip from downloading
    any remote stuff, while still resolving dependencies (respectively consider them as being resolved) *if* the package is locally installed from the Debian archive?
    (If a dependency isn't installed from a package it may of course fail.)


    Thanks,
    Philippe.

    PS: Please keep me CCed.



    [88f0dcef-4969-48af-98f1-0c69549b4875]
    <div id="__MailbirdStyleContent" style="font-size: 12pt;font-family: Arial;color: #000000;text-align: left" dir="ltr">
    I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are
    installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.<div class="mb_sig"></div> <div><br></div><div>The question of
    having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:</div><div><br></div><div>- pip doesn't know how to invoke apt (and yum, and Conda, and everything else).</div><div>- how do we translate "foo"
    on PyPI to the equivalent package in Debian (and Fedora, and Conda, etc)</div><div>- how do we handle virtual environments?</div><div><br></div><div>Nobody has ever come up with a particularly good answer to those problems, so pip upstream hasn't been
    able to even consider doing something like that.</div><div><br></div><div>Of course Debian could patch it's copy of pip to do that, that would make some of those questions easier, but not all of them, and would drastically diverge it's behavior from the "
    expected" behavior, which makes it a rough idea in general for Debian to diverge that drastically.</div><blockquote class="history_container" type="cite" style="border-left-style: solid;border-width: 1px;margin-top: 20px;margin-left: 0px;padding-left:
    10px;min-width: 500px"> <p style="color: #AAAAAA; margin-top: 10px;">On 2/12/2023 3:18:55 AM, Ian Norton &lt;inorton@gmail.com&gt; wrote:</p><div style="font-family:Arial,Helvetica,sans-serif">https://packaging.python.org/en/
    latest/specifications/recording-installed-packages/<br>defines the python spec where a package such as pyparsing would create<br>a tree of files under:<br>site-packages/pyparsing-3.0.9-dist-info/ including RECORD which is<br>essentially a sha256-
    based manifest of files and some others.<br><br>On Sun, 12 Feb 2023 at 08:12, Ian Norton <inorton@gmail.com> wrote:<br>&gt;<br>&gt; You've made me wonder if it would be feasible to have a debian-centric<br>&gt; tool that populates .dist-info from
    debs?<br>&gt;<br>&gt; On Sun, 12 Feb 2023 at 08:05, Ian Norton <inorton@gmail.com> wrote:<br>&gt; &gt;<br>&gt; &gt; I requested this kind of thing from the pip folks as<br>&gt; &gt; https://github.com/pypa/pip/issues/11644 and others have requested<
    &gt; &gt; similar, such as https://github.com/pypa/pip/issues/11607<br>&gt; &gt;<br>&gt; &gt; On Sun, 12 Feb 2023 at 04:56, Philippe Cerfon <philcerf@gmail.com> wrote:<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; Hey.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;
    I hope this is not too off topic.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; As far as I understand, dh-python, when building packages somehow<br>&gt; &gt; &gt; automatically uses the Debian package names and even prevents e.g.<br>&gt; &gt; &gt; setuptools
    from downloading any dependencies by setting a (hopefully<br>&gt; &gt; &gt; not running) proxy.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; I wondered whether it's possible to make tools like pip and setuptools<br>&gt; &gt; &gt; directly
    use the Debian python packages when resolving dependencies.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; The main motivation are security constraints, so I had to configure<br>&gt; &gt; &gt; pip so that it cannot just download packages from PyPI (which is<br>&
    gt; &gt; &gt; rather easy, simply setting no-index in pip.conf).<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; But then of course it also fails to e.g. do an editable install of a<br>&gt; &gt; &gt; locally developed package, when it tries to resolve the
    dependencies.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; So I wondered whether it's possible to prevent pip from downloading<br>&gt; &gt; &gt; any remote stuff, while still resolving dependencies (respectively<br>&gt; &gt; &gt; consider them as being
    resolved) *if* the package is locally installed<br>&gt; &gt; &gt; from the Debian archive?<br>&gt; &gt; &gt; (If a dependency isn't installed from a package it may of course fail.)<br>&gt; &gt; &gt;<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; Thanks,<br>&
    gt; &gt; &gt; Philippe.<br>&gt; &gt; &gt;<br>&gt; &gt; &gt; PS: Please keep me CCed.<br>&gt; &gt; &gt;<br><br></philcerf@gmail.com></inorton@gmail.com></inorton@gmail.com></div></blockquote></div><img class="mailbird" width="1" height="1" style="
    border:0; width:1; height:1; display: none;" src="https://tracking.getmailbird.com/OpenTrackingPixel/?messageId=Mailbird-3d2f35ae-00ad-4864-a8f9-a39876df65ae@stufft.io&senderHash=57787FD98DE4EB8FA6FBC42D73E38B5233F15C50A1E6C93FEFCDDA0F25910CC6&
    recipientHash=A680344B9C0D740131327222AC7C19830233B381EF4AD31267DCE32D1054E966&internalId=9e1e3ba6-db12-46a9-85ae-4abc99011e73" alt="88f0dcef-4969-48af-98f1-0c69549b4875">

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Philippe Cerfon@21:1/5 to inorton@gmail.com on Mon Feb 13 02:20:01 2023
    On Sun, Feb 12, 2023 at 9:05 AM Ian Norton <inorton@gmail.com> wrote:
    I requested this kind of thing from the pip folks as https://github.com/pypa/pip/issues/11644 and others have requested
    similar, such as https://github.com/pypa/pip/issues/11607

    While they wrote it would be discussed, it doesn't really seem as if
    they would care too much or as if anything would have happened since
    then? :-(

    I cannot really comment too much about the rest you wrote, as I'm not
    really a Python or Debian packaging expert.

    What I'd like to have is "simply" ;-) to prevent installation of any
    remote code (as pip and similar tools, also for other languages, do)
    and still use them as far as I have the packages from Debian
    installed.
    So for example, even if the pyproject.toml would specify some version
    of a dependency that is not in Debian, it should rather fail than
    downloading anything from somewhere.

    Regards,
    Philippe

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Philippe Cerfon@21:1/5 to donald@stufft.io on Mon Feb 13 02:30:02 2023
    Hey.


    On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

    I'm pretty sure that most if not all debian packages already ship the required information for pip to see them as installed, and if they are installed and they satisfy the dependency constraints that pip has for those projects, then they'll be used.


    Are you sure? I have a test project with a pyproject.toml that contains: [build-system]
    requires = ["setuptools"]
    build-backend = "setuptools.build_meta"

    i.e. no version requirement on setuptools.

    Also:
    $ dpkg -l python3-setuptools
    ...
    ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements

    Yet when I do e.g.:
    $ pip install --editable .
    Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example
    Installing build dependencies ... error
    error: subprocess-exited-with-error

    × pip subprocess to install build dependencies did not run successfully.
    │ exit code: 1
    ╰─> [2 lines of output]
    ERROR: Could not find a version that satisfies the requirement
    setuptools (from versions: none)
    ERROR: No matching distribution found for setuptools
    [end of output]

    note: This error originates from a subprocess, and is likely not a
    problem with pip.
    error: subprocess-exited-with-error

    × pip subprocess to install build dependencies did not run successfully.
    │ exit code: 1
    ╰─> See above for output.

    note: This error originates from a subprocess, and is likely not a
    problem with pip.

    Now the particular error probably comes from my pip.conf, because I
    set --no-index, at least when I try to install some package that is
    not installed as Debian package, e.g.:
    $ pip install acme
    Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement acme
    (from versions: none)
    ERROR: No matching distribution found for acme

    But I need something like that to actually disable any remote downloads.

    Perhaps you're right however, cause when I run with an empty pip.conf,
    it does e.g. download files to:
    ~/.local/lib/python3.11/site-packages/natsort*
    (which I set as a project dependency), but it doesn't create something
    like that for setuptools.

    On the other hand:
    I've just installed Debian's python3-natsort, rm -rf ~/.local/ and did
    pip install --editable . again (without pip.conf), and it still seems
    to download and install natsort to ~/.local .
    So that doesn't look, as if it would recognize the one from Debian.

    But even if that would actually work, why does it need the index at https://pypi.org/simple to use the local packages from Debian?


    The question of having pip automatically install a debian package instead of using PyPI basically comes down to a few problems:


    That might be nice for some users, but I personally wouldn't even need
    or want that. It should perhaps rather just tell which Debian packages
    need to be installed.


    Thanks,
    Philippe.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Wed Feb 15 17:40:02 2023
    Hi Philippe (2023.02.13_01:11:28_+0000)
    On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

    I'm pretty sure that most if not all debian packages already ship
    the required information for pip to see them as installed, and if
    they are installed and they satisfy the dependency constraints that
    pip has for those projects, then they'll be used.

    Yeah, most packages should ship .egg-info/.dist-info.

    Also:
    $ dpkg -l python3-setuptools
    ...
    ii python3-setuptools 66.1.1-1 all Python3 Distutils Enhancements

    Yet when I do e.g.:
    $ pip install --editable .
    Defaulting to user installation because normal site-packages is not writeable Obtaining file:///home/test/example
    Installing build dependencies ... error
    error: subprocess-exited-with-error

    You sure it isn't doing an isolated build? Try --no-build-isolation.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ian Norton@21:1/5 to stefanor@debian.org on Wed Feb 15 19:10:01 2023
    My suggestion to the pip folks was a plugin system and extension point for "install x" package that distros could provide implementations for

    On Wed, 15 Feb 2023, 16:35 Stefano Rivera, <stefanor@debian.org> wrote:

    Hi Philippe (2023.02.13_01:11:28_+0000)
    On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft <donald@stufft.io> wrote:

    I'm pretty sure that most if not all debian packages already ship
    the required information for pip to see them as installed, and if
    they are installed and they satisfy the dependency constraints that
    pip has for those projects, then they'll be used.

    Yeah, most packages should ship .egg-info/.dist-info.

    Also:
    $ dpkg -l python3-setuptools
    ...
    ii python3-setuptools 66.1.1-1 all Python3 Distutils
    Enhancements

    Yet when I do e.g.:
    $ pip install --editable .
    Defaulting to user installation because normal site-packages is not
    writeable
    Obtaining file:///home/test/example
    Installing build dependencies ... error
    error: subprocess-exited-with-error

    You sure it isn't doing an isolated build? Try --no-build-isolation.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272


    <div dir="auto">My suggestion to the pip folks was a plugin system and extension point for &quot;install x&quot; package that distros could provide implementations for</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 15 Feb
    2023, 16:35 Stefano Rivera, &lt;<a href="mailto:stefanor@debian.org">stefanor@debian.org</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Philippe (2023.02.13_01:11:28_+0000)<

    &gt; On Sun, Feb 12, 2023 at 7:31 PM Donald Stufft &lt;<a href="mailto:donald@stufft.io" target="_blank" rel="noreferrer">donald@stufft.io</a>&gt; wrote:<br>
    &gt; &gt;<br>
    &gt; &gt; I&#39;m pretty sure that most if not all debian packages already ship<br>
    &gt; &gt; the required information for pip to see them as installed, and if<br> &gt; &gt; they are installed and they satisfy the dependency constraints that<br>
    &gt; &gt; pip has for those projects, then they&#39;ll be used.<br>

    Yeah, most packages should ship .egg-info/.dist-info.<br>

    &gt; Also:<br>
    &gt; $ dpkg -l python3-setuptools<br>
    &gt; ...<br>
    &gt; ii  python3-setuptools 66.1.1-1     all          Python3 Distutils Enhancements<br>
    &gt; <br>
    &gt; Yet when I do e.g.:<br>
    &gt; $ pip install --editable .<br>
    &gt; Defaulting to user installation because normal site-packages is not writeable<br>
    &gt; Obtaining file:///home/test/example<br>
    &gt;   Installing build dependencies ... error<br>
    &gt;   error: subprocess-exited-with-error<br>

    You sure it isn&#39;t doing an isolated build? Try --no-build-isolation.<br>

    SR<br>

    -- <br>
    Stefano Rivera<br>
      <a href="http://tumbleweed.org.za/" rel="noreferrer noreferrer" target="_blank">http://tumbleweed.org.za/</a><br>
      +1 415 683 3272<br>
    </blockquote></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Wed Feb 15 21:20:01 2023
    Hi Ian (2023.02.15_18:07:39_+0000)
    My suggestion to the pip folks was a plugin system and extension point for "install x" package that distros could provide implementations for

    Yeah, something like that could work. I don't know how useful it would
    be, though.

    Obviously, only root could use it (or root inside a container).

    And our selection of Python modules is far from complete. It's not
    Debian's intent to provide a mirror of PyPI within Debian. Generally
    speaking, we package the modules that we find useful for supporting
    building and shipping other python modules and applications.

    We'll only have a single version of each package. And they're usually
    not the versions developers want, because in any stable release they're probably out of date.

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeremy Stanley@21:1/5 to All on Wed Feb 15 23:10:01 2023
    As someone who does Python software development on Debian constantly
    for their $dayjob, my best advice is to just install things from
    PyPI into and run them from venvs/virtualenvs. The default "--user"
    install mode pip offers is fragile and leaves you with potential
    conflicts anyway if you need different versions of dependencies for
    different things.

    To your original question, if you really want to use some
    Debian-packaged libraries mixed with things installed from source or
    from PyPI, make your venv with the --system-site-packages option.
    --
    Jeremy Stanley

    -----BEGIN PGP SIGNATURE-----

    iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmPtVf9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCkCpg//Rk39BAqwEaQfhacPJrYTzmdp9uSRPDnwVR3zQVQo+oUscK/ktZXm8xzx R/eF3voG60L0ZHkLTO3AnkUhGHShwy/wXLi7B9vvFpmZ45jrvIkNOAASmHh+Nb54 7FQ+w/rUrKcMP0p9v6PvDW7U66nsYM8vMmVmQmBpCCPBC42rTgDlRMtvCn8iWu/J fG4gdf3KFMMXAynAJHeuMJN2hrp3qn7L0NjCRk37VYPGUM9ws7HsQWXuCbkbNWm/ R+Z07sPReBynNhzQaoh5vbxjkXKOeE1pdnDwh4hDRBeXOMRHMJ4RI6WX6Oz3ZyCs 2x0yABhwWS4s/fFwqFiUkLAtvy6eCot8ynEtAEBrcZO2x4WeJGyM6XO1lOKKLvy/ nThEp5O5I9ktAgVROkjuZqyUau0tlqq9l29y4vMfbSzKiCEbsiPy4biyp87S8J8E 3uS+kTQ2TbNGhBqQQpL5gccNSlxaXyH6PbtpTg4R3Dr9NqA1wkoZ7PPzt7WFmIbS MGggsA2i+rt2Lj2fBJDqmZV6q6Qz5QTW1f42ADrXRx0ajBZRn5jH+r02xDVBPfB1 XgCk0TPwt3vKWunSKuQcmTXli9xTEkKsG+VCRLOwrh7EH7RcxItnL4LCsocTHlop g0CG0cXWFMYlKiesVWSZbCsA+xYOWm2wXDOSdaHdOC9sqKqryLk=
    =kwwo
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32
  • From Ian Norton@21:1/5 to fungi@yuggoth.org on Thu Feb 16 02:20:01 2023
    I agree that is "easiest" but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad
    as old CPAN when it comes to packages disappearing or being broken or
    depending on totally random versions

    On Wed, 15 Feb 2023, 22:01 Jeremy Stanley, <fungi@yuggoth.org> wrote:

    As someone who does Python software development on Debian constantly
    for their $dayjob, my best advice is to just install things from
    PyPI into and run them from venvs/virtualenvs. The default "--user"
    install mode pip offers is fragile and leaves you with potential
    conflicts anyway if you need different versions of dependencies for
    different things.

    To your original question, if you really want to use some
    Debian-packaged libraries mixed with things installed from source or
    from PyPI, make your venv with the --system-site-packages option.
    --
    Jeremy Stanley


    <div dir="auto">I agree that is &quot;easiest&quot; but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad as old CPAN when it comes to packages disappearing or being broken or
    depending on totally random versions</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 15 Feb 2023, 22:01 Jeremy Stanley, &lt;<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>&gt; wrote:<br></div><blockquote class="gmail_
    quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">As someone who does Python software development on Debian constantly<br>
    for their $dayjob, my best advice is to just install things from<br>
    PyPI into and run them from venvs/virtualenvs. The default &quot;--user&quot;<br>
    install mode pip offers is fragile and leaves you with potential<br>
    conflicts anyway if you need different versions of dependencies for<br> different things.<br>

    To your original question, if you really want to use some<br>
    Debian-packaged libraries mixed with things installed from source or<br>
    from PyPI, make your venv with the --system-site-packages option.<br>
    -- <br>
    Jeremy Stanley<br>
    </blockquote></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rakhmatullin@21:1/5 to Ian Norton on Thu Feb 16 10:20:01 2023
    On Thu, Feb 16, 2023 at 01:12:49AM +0000, Ian Norton wrote:
    I agree that is "easiest" but what I was after was the ability to restrict myself to the curated and signed packages from debian, pypi is just as bad
    as old CPAN when it comes to packages disappearing or being broken or depending on totally random versions
    These, or comparable, problems also happen in Debian. For example, you
    cannot expect any given module to be packaged (or not disappear in the
    next release), sometimes the version in the repo is several years old and
    of course packages in Debian can be broken, even if it's rare.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeremy Stanley@21:1/5 to Ian Norton on Thu Feb 16 14:50:01 2023
    On 2023-02-16 01:12:49 +0000 (+0000), Ian Norton wrote:
    I agree that is "easiest" but what I was after was the ability to
    restrict myself to the curated and signed packages from debian,
    pypi is just as bad as old CPAN when it comes to packages
    disappearing or being broken or depending on totally random
    versions
    [...]

    I think you missed my point, which was to explicitly create a venv
    and install your project there instead of relying on pip's --user
    default (which seemed to be resulting in errors for you). If you
    create the venv with --system-site-packages enabled then it will
    still use any Debian-packaged Python libraries you've installed.
    --
    Jeremy Stanley

    -----BEGIN PGP SIGNATURE-----

    iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmPuM7RfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCn23w/7B7WSHE/gouXczYjyPskU97seSJMbmCWRBfQOaUOSbQKttxm80VP8p/cr tUDfhGUsYfZTcFLap8lCfHXSbRqpW3zzdt2H4T7/xFTwpZYlk7iG8+QpC0dT1lzz GPt/uHESv8MpPeIN+W1/9CDU/wzAe4kAm+mdpwna8JfTjadYWpv/4wltcN2hRRmO xNMZtzLFuMdGiutB3z94+9CVxF4W1flcEKU9pfpnmfRPVXzgfv6q4Tw1btauPx3y MTUJIniSdCCQJj36ezsDGsec824jwKm+/kPQNoP0OldvlN39xvmuuyDn1S3GJc4O A5k8aYwhokffFSQrUz+QxwAqL7SlSpcBS/7oMYRbiZjB7TMX4aJdN3vfstnF4Kdc mL1e9EszybDC1tyHtAzZgMPhd5Z7gByM5sxYUewJzNVjt8uOMh1p1tkIjOGTTPwZ XAC/FIOfoKR+zEpRde8EpGRwbJe+6SLKAiQWU7WatNWEGQEZ81zdQl9N8fdQROt1 Dc8LLnQbRU3Vjudt3CdqglXZgyqMPOKrLFf2Hyos4S5Xr3QENwTD3gPB70CNbFgh d88u4dTZR8aVvj5AFlABKyFIu1CVzUE6S+8R7dyfmDLENHLFuKO5ehUMeTCGtLij dUDIsZkqNBC6fV3Vvn8Gr1lAUiHJ0MLekjCkGb+7wSX/nPg6his=
    =39Lx
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32
  • From Philippe Cerfon@21:1/5 to stefanor@debian.org on Fri Feb 17 03:40:01 2023
    On Wed, Feb 15, 2023 at 5:35 PM Stefano Rivera <stefanor@debian.org> wrote:
    You sure it isn't doing an isolated build? Try --no-build-isolation.

    I absolutely am not :D (I'm really not a Python expert... or about
    it's various build systems, pip, etc.).

    And that seems to have done the trick.
    And it further seems that there's a counter part with which I can do
    python3 -m build
    namely: --no-isolation

    Am I right that these options simply mean that it doesn't use a venv
    for the build, thereby "seeing" Debian's already installed packages
    (or better said the eggs/wheels/whatsoever from them)?


    Just for my confirmation:
    If my goal is to simply never ever have pip/build/etc. download
    anything from some location, but only use local sources (e.g. stuff
    from Debian package) is the following safe?

    I set /etc/pip.conf to:
    [global]
    no-index = true
    index-url = https://localhost/
    cert = /usr/local/share/ca-certificates/dummy.crt

    (guess in principle the no-index should already be enough)

    It seems to work (in the sense of blocking downloads) for both, pip
    and python -m build ... but I don't quite understand why it also works
    for build (thought that was independent of pip).


    Awesome! Thanks guys :-)


    Philippe

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Stefano Rivera@21:1/5 to All on Fri Feb 17 15:40:01 2023
    Hi Philippe (2023.02.17_02:17:49_+0000)
    Well in my case the main motivation was security (i.e. only using
    code) that has security support by Debian.

    There is probably some value there. You're safer from a variety of
    attacks that *could* theoretically happen on PyPI.

    But, let me deflate Debian's reputation a bit here.
    Debian security support doesn't mean you're completely protected. There
    is probably a human behind a Debian upload that has vetted the upload
    and thinks it is safe. They thought this thing was useful to package for
    Debian (so probably not malware), and did some review to see that it
    installed itself correctly. They may have reviewed the upstream code,
    they may not have. They may review new upstream version diffs, they may
    not. (Generally, small things are easy to review, big complex things are impossible to.)

    For the security support, it's largely reliant on security issues being reported as CVEs, which security researchers usually do, but upstreams
    often fail to do. And then it needs a volunteer to find/figure out the
    fix and apply it to the version in Debian.

    So, again, there is definitely value here. If you're just using software
    from Debian stable releases, you know that some people have reviewed
    some of it. And you can be reasonably confident that you're using the
    same stack as some other people.

    But, on balance, for many problems the gains here aren't worth the pain
    of restricting yourself to Python modules published in Debian stable
    releases.

    But shouldn't that use case also be interesting for Debian
    Maintainers? Whenever their pip would need to download something from
    PyPI, it would mean that some dependency is likely not fulfilled in
    Debian (unless of course that Debian package is simply not installed).

    Generally speaking when I'm working on code, I install libraries in virtualenvs. This is what the upstream tooling expects and so it makes everything more convenient. All the work may be done in a container, but
    I'm not restricting myself to Debian packages.

    If I am using Debian packages for something, I'll install them with apt.
    I don't need pip involved. This is where I don't find the pip plugin
    idea that useful.

    Some people try to write software specifically to run on Debian stable,
    without any third party packages. For simple projects, this can work
    well. But, there are downsides. You often find you have to couple code
    changes to Debian's release cycle, which can get problematic. And nobody
    will understand what you're trying to do :)

    SR

    --
    Stefano Rivera
    http://tumbleweed.org.za/
    +1 415 683 3272

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rakhmatullin@21:1/5 to Philippe Cerfon on Fri Feb 17 19:30:01 2023
    On Fri, Feb 17, 2023 at 03:17:49AM +0100, Philippe Cerfon wrote:
    But shouldn't that use case also be interesting for Debian
    Maintainers? Whenever their pip would need to download something from
    PyPI, it would mean that some dependency is likely not fulfilled in
    Debian (unless of course that Debian package is simply not installed).
    Do you mean the scenario with packaging some new software? In that case
    a packager would most likely review the deps, not run pip.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Philippe Cerfon@21:1/5 to stefanor@debian.org on Sat Feb 18 06:20:01 2023
    Hey.

    On Fri, Feb 17, 2023 at 3:35 PM Stefano Rivera <stefanor@debian.org> wrote:
    There is probably some value there. You're safer from a variety of
    attacks that *could* theoretically happen on PyPI.

    Well such language specific package repos (like pypi, npmjs, etc.)
    have already been (numerous) times been victims from such attacks. So
    it's not just a theoretical issue, I think.


    But, let me deflate Debian's reputation a bit here.
    Debian security support doesn't mean you're completely protected. There
    is probably a human behind a Debian upload that has vetted the upload
    and thinks it is safe. They thought this thing was useful to package for Debian (so probably not malware), and did some review to see that it installed itself correctly. They may have reviewed the upstream code,
    they may not have. They may review new upstream version diffs, they may
    not. (Generally, small things are easy to review, big complex things are impossible to.)

    Sure, there are no code audits by Debian maintainers, there's no
    guarantee that the maintainer retrieved the code in a secure way
    (whatever that would be).

    Still, even the download method of PyPI (and friends) - https - has
    all kinds of issues. Typically one has something of around 150 root
    CAs trusted, plus several thousands(?) of intermediate CAs from them.
    Many of those in the hands quite questionable countries or organizations. Again, the DM might just as well download the code via https (and not
    verify some upstream gpg, if present at all). But at least this would
    cause *all* Debian users (of that package) to be compromised, which in
    turn makes it much more likely that any compromise would get noticed.
    Attacks specific to a single person are no longer easily possible.

    But, on balance, for many problems the gains here aren't worth the pain
    of restricting yourself to Python modules published in Debian stable releases.

    Well I guess it's clear there's no 100% protection. In the end it's
    simply just nice if one can easily choose whichever one wants. If
    someone wants to use PyPI code, fine, if someone wants to restrict
    himself to Debian-only, fine too.

    Anyway, thanks :-)

    Regards,
    Philippe.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)