• pip/pip3 confusion and keeping up to date

    From Chris Green@21:1/5 to All on Thu Nov 2 10:58:00 2023
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    There's still both /usr/bin/pip and /usr/bin/pip3 but they're
    identical so presuably I can now simply use pip and it will be a
    python3 pip.


    So, going on from this, how do I do the equivalent of "apt update; apt
    upgrade" for my globally installed pip packages?

    --
    Chris Green
    ·

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Chris Green on Thu Nov 2 14:55:55 2023
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    There's still both /usr/bin/pip and /usr/bin/pip3 but they're
    identical so presuably I can now simply use pip and it will be a
    python3 pip.


    So, going on from this, how do I do the equivalent of "apt update; apt upgrade" for my globally installed pip packages?

    I'm not sure what that question has to do with everything that preceded
    it, but you don't want to install python packages globally using pip.
    Either install them with 'apt', or install them in a virtual environment.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dieter Maurer@21:1/5 to Chris Green on Thu Nov 2 17:18:09 2023
    Chris Green wrote at 2023-11-2 10:58 +0000:
    ...
    So, going on from this, how do I do the equivalent of "apt update; apt >upgrade" for my globally installed pip packages?

    `pip list -o` will tell you for which packages there are upgrades
    available.
    `pip install -U ...` will upgrade packages.

    Be careful, though.
    With `apt`, you usually have (`apt`) sources representing a consistent
    package universe. Someone tests that package upgrades in this
    universe do not break other packages (in this universe).
    Because of this, upgrading poses low risk.

    `PyPI` does not guarantes consistency. A new package version
    may be incompatible to a previous one -- and with other
    package you have installed.

    I do not think that you would want to auto-upgrade all installed
    packages.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Dieter Maurer on Thu Nov 2 16:35:28 2023
    On 2023-11-02, Dieter Maurer <dieter@handshake.de> wrote:
    Chris Green wrote at 2023-11-2 10:58 +0000:
    ...
    So, going on from this, how do I do the equivalent of "apt update; apt >>upgrade" for my globally installed pip packages?

    `pip list -o` will tell you for which packages there are upgrades
    available.
    `pip install -U ...` will upgrade packages.

    Be careful, though.
    With `apt`, you usually have (`apt`) sources representing a consistent package universe. Someone tests that package upgrades in this
    universe do not break other packages (in this universe).
    Because of this, upgrading poses low risk.

    `PyPI` does not guarantes consistency. A new package version
    may be incompatible to a previous one -- and with other
    package you have installed.

    I do not think that you would want to auto-upgrade all installed
    packages.

    Indeed. What you're describing is a very unfortunate failing of pip.
    'Upgrade' doesn't even follow requirements when you tell it what to
    upgrade - e.g. if you do "pip install foo" and foo requires "bar<2"
    so you end up with:

    Package Version
    ---------------------- ---------
    foo 1.0.0
    bar 1.2.0

    and then a new version 1.3.0 of bar comes out and you do
    "pip install -U foo", pip will not upgrade bar even though it could
    and should, because foo is already at the latest version so pip won't
    even look at its dependencies.

    Indeed there is no way of knowing that you should upgrade bar without
    manually following all the dependency graphs. ("pip list -o" will tell
    you there's a newer version, but that isn't the same - e.g. if the new
    version of bar was 2.0.0 then "pip list -o" will list it, but you should
    not upgrade to it.)

    You can do "pip install -I foo", which will pointlessly reinstall foo
    and then presumably upgrade bar as well, thus probably getting to the
    right result via a rather roundabout route, but I'm not sure if that
    does indeed work properly and if it is a reliable and recommended way
    of doing things.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Green@21:1/5 to Jon Ribbens on Thu Nov 2 21:16:49 2023
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    There's still both /usr/bin/pip and /usr/bin/pip3 but they're
    identical so presuably I can now simply use pip and it will be a
    python3 pip.


    So, going on from this, how do I do the equivalent of "apt update; apt upgrade" for my globally installed pip packages?

    I'm not sure what that question has to do with everything that preceded
    it, but you don't want to install python packages globally using pip.
    Either install them with 'apt', or install them in a virtual environment.

    Why in a virtual environment? When I install a package whether from
    apt or from pip I want everyone/everything on my system to be able to
    use it.

    I do only install a few things using pip.

    --
    Chris Green
    ·

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Green@21:1/5 to Jon Ribbens on Thu Nov 2 21:19:43 2023
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Dieter Maurer <dieter@handshake.de> wrote:
    Chris Green wrote at 2023-11-2 10:58 +0000:
    ...
    So, going on from this, how do I do the equivalent of "apt update; apt >>upgrade" for my globally installed pip packages?

    `pip list -o` will tell you for which packages there are upgrades available.
    `pip install -U ...` will upgrade packages.

    Be careful, though.
    With `apt`, you usually have (`apt`) sources representing a consistent package universe. Someone tests that package upgrades in this
    universe do not break other packages (in this universe).
    Because of this, upgrading poses low risk.

    `PyPI` does not guarantes consistency. A new package version
    may be incompatible to a previous one -- and with other
    package you have installed.

    I do not think that you would want to auto-upgrade all installed
    packages.

    Indeed. What you're describing is a very unfortunate failing of pip. 'Upgrade' doesn't even follow requirements when you tell it what to
    upgrade - e.g. if you do "pip install foo" and foo requires "bar<2"
    so you end up with:

    Package Version
    ---------------------- ---------
    foo 1.0.0
    bar 1.2.0

    and then a new version 1.3.0 of bar comes out and you do
    "pip install -U foo", pip will not upgrade bar even though it could
    and should, because foo is already at the latest version so pip won't
    even look at its dependencies.

    Indeed there is no way of knowing that you should upgrade bar without manually following all the dependency graphs. ("pip list -o" will tell
    you there's a newer version, but that isn't the same - e.g. if the new version of bar was 2.0.0 then "pip list -o" will list it, but you should
    not upgrade to it.)

    You can do "pip install -I foo", which will pointlessly reinstall foo
    and then presumably upgrade bar as well, thus probably getting to the
    right result via a rather roundabout route, but I'm not sure if that
    does indeed work properly and if it is a reliable and recommended way
    of doing things.

    It is a bit of a minefield isn't it. I try to minimise my use of
    packages installed using pip for this very reason. Maybe the safest
    route would simply be to uninstall everything and then re-install it.
    ·

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Chris Green on Thu Nov 2 21:35:43 2023
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    There's still both /usr/bin/pip and /usr/bin/pip3 but they're
    identical so presuably I can now simply use pip and it will be a
    python3 pip.


    So, going on from this, how do I do the equivalent of "apt update; apt
    upgrade" for my globally installed pip packages?

    I'm not sure what that question has to do with everything that preceded
    it, but you don't want to install python packages globally using pip.
    Either install them with 'apt', or install them in a virtual environment.

    Why in a virtual environment? When I install a package whether from
    apt or from pip I want everyone/everything on my system to be able to
    use it.

    Because pip barely plays well by itself, let alone with other package
    managers at the same time.

    I do only install a few things using pip.

    Are they not available in your system's package manager?
    I guess you might get away with "sudo -H pip install -U foo"
    for a couple of things, if they don't have many dependencies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Chris Green on Thu Nov 2 21:38:33 2023
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Dieter Maurer <dieter@handshake.de> wrote:
    Chris Green wrote at 2023-11-2 10:58 +0000:
    ...
    So, going on from this, how do I do the equivalent of "apt update; apt
    upgrade" for my globally installed pip packages?

    `pip list -o` will tell you for which packages there are upgrades
    available.
    `pip install -U ...` will upgrade packages.

    Be careful, though.
    With `apt`, you usually have (`apt`) sources representing a consistent
    package universe. Someone tests that package upgrades in this
    universe do not break other packages (in this universe).
    Because of this, upgrading poses low risk.

    `PyPI` does not guarantes consistency. A new package version
    may be incompatible to a previous one -- and with other
    package you have installed.

    I do not think that you would want to auto-upgrade all installed
    packages.

    Indeed. What you're describing is a very unfortunate failing of pip.
    'Upgrade' doesn't even follow requirements when you tell it what to
    upgrade - e.g. if you do "pip install foo" and foo requires "bar<2"
    so you end up with:

    Package Version
    ---------------------- ---------
    foo 1.0.0
    bar 1.2.0

    and then a new version 1.3.0 of bar comes out and you do
    "pip install -U foo", pip will not upgrade bar even though it could
    and should, because foo is already at the latest version so pip won't
    even look at its dependencies.

    Indeed there is no way of knowing that you should upgrade bar without
    manually following all the dependency graphs. ("pip list -o" will tell
    you there's a newer version, but that isn't the same - e.g. if the new
    version of bar was 2.0.0 then "pip list -o" will list it, but you should
    not upgrade to it.)

    You can do "pip install -I foo", which will pointlessly reinstall foo
    and then presumably upgrade bar as well, thus probably getting to the
    right result via a rather roundabout route, but I'm not sure if that
    does indeed work properly and if it is a reliable and recommended way
    of doing things.

    It is a bit of a minefield isn't it. I try to minimise my use of
    packages installed using pip for this very reason. Maybe the safest
    route would simply be to uninstall everything and then re-install it.

    That is literally what I do quite often - completely erase the
    virtual env and then re-create it from scratch - because it seems
    to be the only / easiest way to upgrade the packages to the latest
    versions consistent with given dependencies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mats Wichmann@21:1/5 to Chris Green via Python-list on Thu Nov 2 16:07:33 2023
    On 11/2/23 04:58, Chris Green via Python-list wrote:
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    FWIW, for this you install the little stub package python-is-python3. Especially if you want to keep a python2 installation around - "python"
    will still be python3 in this case.

    So, going on from this, how do I do the equivalent of "apt update; apt upgrade" for my globally installed pip packages

    Odds are you don't want to. The internet is full of surprises about
    dependency problems when stuff is blindly updated; the set of Python
    packages in the apt repositories is carefully curated to avoid these
    problems - and this is part of the reason why sometimes certain such
    packages are irritatingly down-rev.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Green@21:1/5 to Jon Ribbens on Fri Nov 3 10:05:09 2023
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-02, Chris Green <cl@isbd.net> wrote:
    I have a couple of systems which used to have python2 as well as
    python3 but as Ubuntu and Debian verions have moved on they have
    finally eliminated all dependencies on python2.

    So they now have only python3 and there is no python executable in
    PATH.

    There's still both /usr/bin/pip and /usr/bin/pip3 but they're
    identical so presuably I can now simply use pip and it will be a
    python3 pip.


    So, going on from this, how do I do the equivalent of "apt update; apt >> > upgrade" for my globally installed pip packages?

    I'm not sure what that question has to do with everything that preceded
    it, but you don't want to install python packages globally using pip.
    Either install them with 'apt', or install them in a virtual environment.

    Why in a virtual environment? When I install a package whether from
    apt or from pip I want everyone/everything on my system to be able to
    use it.

    Because pip barely plays well by itself, let alone with other package managers at the same time.

    Well that's a criticism of pip rather than of how I use it! :-)

    OK, there are risks.


    I do only install a few things using pip.

    Are they not available in your system's package manager?
    I guess you might get away with "sudo -H pip install -U foo"
    for a couple of things, if they don't have many dependencies.

    I obviously check the package manager and also other sources which
    work through apt before resorting to using pip.

    The sort of thing I have to use pip for is software for odd I2C sensor
    devices and such. These rarely have any dependencies or they are all
    the same dependencies as the other I2C device software. So,
    hopefully, I won't hit any big problems. I have to say that so far I
    haven't been bitten.

    I would point out that this is mostly for a headless Beaglebone Black
    single board computer (comparable with a Raspberry Pi) with only a
    minimal 'console' installation of Debian so it's a pretty minimal
    system.

    --
    Chris Green
    ·

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Fri Nov 3 14:47:18 2023
    Am Thu, Nov 02, 2023 at 09:35:43PM -0000 schrieb Jon Ribbens via Python-list:

    Regardless of ...

    Because pip barely plays well by itself, let alone with other package managers at the same time.

    ... being true ...

    I do only install a few things using pip.

    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Karsten Hilbert on Fri Nov 3 13:53:32 2023
    On 2023-11-03, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
    Am Thu, Nov 02, 2023 at 09:35:43PM -0000 schrieb Jon Ribbens via Python-list:

    Regardless of ...

    Because pip barely plays well by itself, let alone with other package
    managers at the same time.

    ... being true ...

    I do only install a few things using pip.

    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Yes, as previously mentioned, use virtual environments.

    These days they don't even need to be "activated". For package 'foo'
    for example you could create /usr/local/lib/foo, under which you would
    create a virtual environment and install the 'foo' package inside it,
    and then you could do:

    ln -s /usr/local/lib/foo/env/bin/foo /usr/local/bin/foo

    and then you could just type 'foo' to run it.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dieter Maurer@21:1/5 to Karsten Hilbert on Fri Nov 3 17:26:06 2023
    Karsten Hilbert wrote at 2023-11-3 14:47 +0100:
    ...
    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Complex applications may maintain a set of "known workable versions"
    associated with the application's releases.
    They may describe those "known workable versions" in a `pip` constraint file. In this case, you can upgrade to a new application release
    by using this constraint file.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Fri Nov 3 14:44:27 2023
    Am Thu, Nov 02, 2023 at 04:07:33PM -0600 schrieb Mats Wichmann via Python-list:

    So they now have only python3 and there is no python executable in
    PATH.

    FWIW, for this you install the little stub package python-is-python3. Especially if you
    want to keep a python2 installation around - "python" will still be python3 in this
    case.

    Since you seem knowledgeable in this area: Do you know of a
    resource for learning the *canonical* way of packaging a
    Python application for installation via apt which

    - needs some packages available via apt
    - needs some packages only available via pip
    - needs some packages newer than what is available via apt

    ?

    Thanks,
    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Karsten Hilbert on Sun Nov 5 12:59:12 2023
    On 2023-11-03, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
    Am Thu, Nov 02, 2023 at 04:07:33PM -0600 schrieb Mats Wichmann via Python-list:
    So they now have only python3 and there is no python executable in
    PATH.

    FWIW, for this you install the little stub package python-is-python3.
    Especially if you want to keep a python2 installation around -
    "python" will still be python3 in this case.

    Since you seem knowledgeable in this area: Do you know of a
    resource for learning the *canonical* way of packaging a
    Python application for installation via apt which

    - needs some packages available via apt
    - needs some packages only available via pip
    - needs some packages newer than what is available via apt

    ?

    I suspect the answer to that is that you would have to:

    * create packages yourself for the unpackaged dependencies
    * create a dependency graph of *every* Python package in the package
    repository (whether or not the package is relevant to what you're doing)
    * work out what versions of every Python package are required in order
    to have a dependency graph that can be successfully resolved, taking
    into account the requirements of your new package also
    * contact every single maintainer of every single one of the packages
    that needs updating and persuade them to update their packages and
    reassure them that you are getting all the other package maintainers
    to update their packages accordingly and that you have a plan and
    that you know what you're doing

    ... screen fades to black, title card "3 years later", fade in to ...

    * publish your package

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Chris Green@21:1/5 to Jon Ribbens on Sun Nov 5 15:00:41 2023
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-03, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
    Am Thu, Nov 02, 2023 at 04:07:33PM -0600 schrieb Mats Wichmann via Python-list:
    So they now have only python3 and there is no python executable in
    PATH.

    FWIW, for this you install the little stub package python-is-python3.
    Especially if you want to keep a python2 installation around -
    "python" will still be python3 in this case.

    Since you seem knowledgeable in this area: Do you know of a
    resource for learning the *canonical* way of packaging a
    Python application for installation via apt which

    - needs some packages available via apt
    - needs some packages only available via pip
    - needs some packages newer than what is available via apt

    ?

    I suspect the answer to that is that you would have to:

    * create packages yourself for the unpackaged dependencies
    * create a dependency graph of *every* Python package in the package
    repository (whether or not the package is relevant to what you're doing)
    * work out what versions of every Python package are required in order
    to have a dependency graph that can be successfully resolved, taking
    into account the requirements of your new package also
    * contact every single maintainer of every single one of the packages
    that needs updating and persuade them to update their packages and
    reassure them that you are getting all the other package maintainers
    to update their packages accordingly and that you have a plan and
    that you know what you're doing

    ... screen fades to black, title card "3 years later", fade in to ...

    * publish your package

    Surely it's not that bad, the vast bulk of Debian, Ubuntu and other distributions are installed via systems that sort out dependencies once
    given a particular package's requirements. Python is surely not
    unique in its dependency requirements.

    --
    Chris Green
    ·

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Chris Green on Sun Nov 5 16:04:21 2023
    On 2023-11-05, Chris Green <cl@isbd.net> wrote:
    Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
    On 2023-11-03, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
    Am Thu, Nov 02, 2023 at 04:07:33PM -0600 schrieb Mats Wichmann via Python-list:
    So they now have only python3 and there is no python executable in
    PATH.

    FWIW, for this you install the little stub package python-is-python3.
    Especially if you want to keep a python2 installation around -
    "python" will still be python3 in this case.

    Since you seem knowledgeable in this area: Do you know of a
    resource for learning the *canonical* way of packaging a
    Python application for installation via apt which

    - needs some packages available via apt
    - needs some packages only available via pip
    - needs some packages newer than what is available via apt

    ?

    I suspect the answer to that is that you would have to:

    * create packages yourself for the unpackaged dependencies
    * create a dependency graph of *every* Python package in the package
    repository (whether or not the package is relevant to what you're doing) >> * work out what versions of every Python package are required in order
    to have a dependency graph that can be successfully resolved, taking
    into account the requirements of your new package also
    * contact every single maintainer of every single one of the packages
    that needs updating and persuade them to update their packages and
    reassure them that you are getting all the other package maintainers
    to update their packages accordingly and that you have a plan and
    that you know what you're doing

    ... screen fades to black, title card "3 years later", fade in to ...

    * publish your package

    Surely it's not that bad, the vast bulk of Debian, Ubuntu and other distributions are installed via systems that sort out dependencies once
    given a particular package's requirements. Python is surely not
    unique in its dependency requirements.

    I think there's a lot of work that goes on behind the scenes to keep the
    entire package set consistent so that you don't end up in the situation
    where, e.g. package A depends on D<2.0 and package B requires D>=2.0 and therefore you can't install A and B at the same time (and trying to
    avoid as much as possible the hacky situation where you have two
    packages for D, one for <2.0 and another called D2 for >=2.0).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Sun Nov 5 23:22:38 2023
    Am Fri, Nov 03, 2023 at 01:53:32PM -0000 schrieb Jon Ribbens via Python-list:

    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Yes, as previously mentioned, use virtual environments.

    These days they don't even need to be "activated". For package 'foo'
    for example you could create /usr/local/lib/foo, under which you would
    create a virtual environment and install the 'foo' package inside it,
    and then you could do:

    ln -s /usr/local/lib/foo/env/bin/foo /usr/local/bin/foo

    and then you could just type 'foo' to run it.

    This all being nice and well, but:

    How does one "fill" that venv with packages from pip during

    apt-get install python3-app-of-interest

    ?

    Is the suggested way really to pip-install into this venv
    during apt-get install ?

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Sun Nov 5 23:29:24 2023
    Am Sun, Nov 05, 2023 at 03:00:41PM +0000 schrieb Chris Green via Python-list:

    * contact every single maintainer of every single one of the packages
    that needs updating and persuade them to update their packages and
    reassure them that you are getting all the other package maintainers
    to update their packages accordingly and that you have a plan and
    that you know what you're doing

    ... screen fades to black, title card "3 years later", fade in to ...

    * publish your package

    Surely it's not that bad, the vast bulk of Debian, Ubuntu and other distributions are installed via systems that sort out dependencies once
    given a particular package's requirements. Python is surely not
    unique in its dependency requirements.

    Oh, Debian does just fine in resolving dependencies it knows
    about within its .deb package universe. The problem arises
    when there's unpackaged modules required. The only answer
    seems to be to package such modules oneself.

    If that's been conquered a venv isn't even needed anymore.

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Sun Nov 5 23:19:00 2023
    Am Fri, Nov 03, 2023 at 05:26:06PM +0100 schrieb Dieter Maurer:

    Karsten Hilbert wrote at 2023-11-3 14:47 +0100:
    ...
    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Complex applications may maintain a set of "known workable versions" associated with the application's releases.
    They may describe those "known workable versions" in a `pip` constraint file. In this case, you can upgrade to a new application release
    by using this constraint file.

    Hello Dieter,

    do you happen to know where to read up on how to fit a pip
    constraint file into a Debian package creation workflow ?

    Thanks,
    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jon Ribbens@21:1/5 to Karsten Hilbert on Mon Nov 6 01:17:11 2023
    On 2023-11-05, Karsten Hilbert <Karsten.Hilbert@gmx.net> wrote:
    Am Fri, Nov 03, 2023 at 01:53:32PM -0000 schrieb Jon Ribbens via Python-list:

    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Yes, as previously mentioned, use virtual environments.

    These days they don't even need to be "activated". For package 'foo'
    for example you could create /usr/local/lib/foo, under which you would
    create a virtual environment and install the 'foo' package inside it,
    and then you could do:

    ln -s /usr/local/lib/foo/env/bin/foo /usr/local/bin/foo

    and then you could just type 'foo' to run it.

    This all being nice and well, but:

    How does one "fill" that venv with packages from pip during

    apt-get install python3-app-of-interest

    ?

    Is the suggested way really to pip-install into this venv
    during apt-get install ?

    I don't know what you mean by that. But if you install the apt packages
    and then create your venv with --system-site-packages then I believe
    your venv should be able to see the apt packages and import them.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Dieter Maurer@21:1/5 to Karsten Hilbert on Mon Nov 6 08:58:00 2023
    Karsten Hilbert wrote at 2023-11-5 23:19 +0100:
    ...
    do you happen to know where to read up on how to fit a pip
    constraint file into a Debian package creation workflow ?

    I have only rudimentary `apt` knowledge.

    I know it is quite flexible, e.g. it used to handle `flash`
    in a special way. I expect it flexible enough to load from `PyPI`.

    `apt` is essentially controlled by a set of `sources`.
    Each "source" describes a list of package distributions.
    A description contains metadata about the distribution
    including name, version, description, dependencies, incompatibilities, etc.

    Any integration would require you to define one or more
    "source"s, listing the distributions which should come from `PyPI`.
    This would allow you to specify potential upgrades at a central
    place, the "source"s, you control.
    It would not solve the more fundamental problem: to determine
    which versions are compatible with the universe of (all the
    "sources" described) package versions.


    I know that debian packagers create debian packages
    from Python distributions not using the approach sketched above
    and likely they have their reasons.

    You might want to discuss this on an `apt` related mailing list.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Mon Nov 6 22:26:36 2023
    Am Mon, Nov 06, 2023 at 01:17:11AM -0000 schrieb Jon Ribbens via Python-list:

    Are they not available in your system's package manager?

    ... this clearly often answers to "no" for applications of
    any complexity.

    Is there a suggested proper path to deal with that (Debian is
    of interest to me here) ?

    Yes, as previously mentioned, use virtual environments.

    How does one "fill" that venv with packages from pip during

    apt-get install python3-app-of-interest

    ?

    Is the suggested way really to pip-install into this venv
    during apt-get install ?

    I don't know what you mean by that. But if you install the apt packages
    and then create your venv with --system-site-packages then I believe
    your venv should be able to see the apt packages and import them.

    The problem is when there's modules not available via apt as
    packages. In that case on needs to either package them or pip
    them into the venv suggested above.

    When they are apt-gettable no venv is needed in the first
    place. One would just install the application script like any
    other binary, and which loads apt-installed modules just so.

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Mon Nov 6 22:28:56 2023
    Am Mon, Nov 06, 2023 at 08:58:00AM +0100 schrieb Dieter Maurer:

    I know that debian packagers create debian packages
    from Python distributions not using the approach sketched above
    and likely they have their reasons.

    You might want to discuss this on an `apt` related mailing list.

    Yeah, I guess. I know all this stuff about python modules
    being packaged for Debian and how apt handles dependencies
    etc etc.

    I had just hoped someone here might have a handy pointer for
    how to deal with modules having to be installed from pip for
    use with an apt-installed python-based application.

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mats Wichmann@21:1/5 to Karsten Hilbert via Python-list on Mon Nov 6 14:43:47 2023
    On 11/6/23 14:28, Karsten Hilbert via Python-list wrote:

    I had just hoped someone here might have a handy pointer for
    how to deal with modules having to be installed from pip for
    use with an apt-installed python-based application.

    That just shouldn't happen - such packages are supposed to be dependency-complete within the packaging universe in question.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Karsten Hilbert@21:1/5 to All on Mon Nov 6 23:04:37 2023
    Am Mon, Nov 06, 2023 at 02:43:47PM -0700 schrieb Mats Wichmann via Python-list:

    I had just hoped someone here might have a handy pointer for
    how to deal with modules having to be installed from pip for
    use with an apt-installed python-based application.

    That just shouldn't happen - such packages are supposed to be dependency-complete within
    the packaging universe in question.

    Yep, that's the preferable ideal world.

    Which doesn't happen (but that's not the fault of anyone
    around here, no harm intended).

    .From all the posts I gather the answer to my question is
    "simply": unpackaged-but-needed modules need to be packaged.

    Karsten
    --
    GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Thomas Passin@21:1/5 to Karsten Hilbert via Python-list on Mon Nov 6 17:31:00 2023
    On 11/6/2023 5:04 PM, Karsten Hilbert via Python-list wrote:
    Am Mon, Nov 06, 2023 at 02:43:47PM -0700 schrieb Mats Wichmann via Python-list:

    I had just hoped someone here might have a handy pointer for
    how to deal with modules having to be installed from pip for
    use with an apt-installed python-based application.

    That just shouldn't happen - such packages are supposed to be dependency-complete within
    the packaging universe in question.

    Yep, that's the preferable ideal world.

    Which doesn't happen (but that's not the fault of anyone
    around here, no harm intended).

    .From all the posts I gather the answer to my question is
    "simply": unpackaged-but-needed modules need to be packaged.

    I think there is one aspect that isn't getting consideration here. And
    that is whether or not you want these packages installed in the default
    system Python install. You might not. Maybe you want to get the latest possible version of super-dooper-gui-helper, but one of its dependencies doesn't play well with the system Python libraries. Or ... but you get
    the point. There are probably many cases where you want *not* to
    install into the system Python world. So you would need to come up with
    an APT-based installer that doesn't do that.

    Obviously it's not unthinkable; it is just one more thing to figure out.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)