• [gentoo-user] python, my nemesis

    From Gerrit Kuehn@21:1/5 to All on Mon Sep 20 15:00:02 2021
    Hi,

    I'd like to update a system that is about 1 year old. I have python3.7
    and python 3.8 installed, 3.7 is the default. After updating the repo,
    it was suggested to update portage first (probably useful to support
    new EAPI versions).
    I read about updating to python 3.9, so I created

    ---
    ~ # cat /etc/portage/package.use/py
    */* PYTHON_TARGETS: -* python3_9 python3_8
    */* PYTHON_SINGLE_TARGET: -* python3_9
    ---

    as suggested. However, there are still collisions (and I don't really understand them):

    ---
    ~ # emerge -p --oneshot sys-apps/portage

    These are the packages that would be merged, in order:

    Calculating dependencies... done!
    [ebuild N ] acct-group/portage-0
    [ebuild N ] acct-user/portage-0
    [ebuild U ] sys-devel/automake-1.16.4 [1.16.1-r1] USE="-test%"
    [ebuild NS ] dev-lang/python-3.9.6_p2 [2.7.18-r2, 3.7.8-r2, 3.8.5] USE="sqlite* -verify-sig%"
    [ebuild U ] dev-python/certifi-10001-r1 [10001]
    PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
    [ebuild U ] dev-python/setuptools-57.4.0-r2 [46.4.0-r3] PYTHON_TARGETS="python3_8* python3_9* (-python3_10)"
    [ebuild N ] dev-python/toml-0.10.2 USE="-test"
    PYTHON_TARGETS="python3_8 python3_9 (-pypy3) (-python3_10)"
    [ebuild U ] dev-python/setuptools_scm-6.0.1-r1 [4.1.2-r1]
    PYTHON_TARGETS="
  • From Michael Orlitzky@21:1/5 to Gerrit Kuehn on Mon Sep 20 15:20:02 2021
    On Mon, 2021-09-20 at 14:50 +0200, Gerrit Kuehn wrote:
    Hi,

    I'd like to update a system that is about 1 year old. I have python3.7
    and python 3.8 installed, 3.7 is the default. After updating the repo,
    it was suggested to update portage first (probably useful to support
    new EAPI versions).
    I read about updating to python 3.9, so I created

    ---
    ~ # cat /etc/portage/package.use/py
    */* PYTHON_TARGETS: -* python3_9 python3_8
    */* PYTHON_SINGLE_TARGET: -* python3_9
    ---

    You should probably not mess with these variables until after your
    system is 100% updated and consistent. And even then, probably not.

    With our package manager written in python, you often need old python
    stuff to build the new python stuff, and disabling the old python stuff
    will throw a wrench into that. Even in situations where technically
    some upgrade path exists, the complexity of the python dependencies
    often means that the package manager will give up before it finds the
    solution unless the solution is obvious. By tweaking those variables,
    you make the solution less obvious to it.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Michael Orlitzky on Mon Sep 20 16:00:01 2021
    On Mon, 20 Sep 2021 09:18:23 -0400
    Michael Orlitzky <mjo@gentoo.org> wrote:

    ---
    ~ # cat /etc/portage/package.use/py
    */* PYTHON_TARGETS: -* python3_9 python3_8
    */* PYTHON_SINGLE_TARGET: -* python3_9
    ---

    You should probably not mess with these variables until after your
    system is 100% updated and consistent. And even then, probably not.

    With our package manager written in python, you often need old python
    stuff to build the new python stuff, and disabling the old python
    stuff will throw a wrench into that. Even in situations where
    technically some upgrade path exists, the complexity of the python dependencies often means that the package manager will give up before
    it finds the solution unless the solution is obvious. By tweaking
    those variables, you make the solution less obvious to it.

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    But also when trying "emerge -1vUD @world" (be it with or without the package.use settings), I get stuck in conflicts (mostly on perl and setuptools). perl issues would probably resolve once I have EAPI8
    support, i.e. get new portage.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Orlitzky@21:1/5 to Gerrit Kuehn on Mon Sep 20 16:40:03 2021
    On Mon, 2021-09-20 at 15:56 +0200, Gerrit Kuehn wrote:

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    It was only suggested for a period of about three weeks:

    We are planning to switch the default Python target of Gentoo systems
    on 2021-06-01, from Python 3.8 to Python 3.9. If you have not  
    changed the values of PYTHON_TARGETS or PYTHON_SINGLE_TARGET, the 
    change will have immediate effect on your system and the package 
    manager will try to switch automatically on the next upgrade 
    following the change.


    But also when trying "emerge -1vUD @world" (be it with or without the package.use settings), I get stuck in conflicts (mostly on perl and setuptools). perl issues would probably resolve once I have EAPI8
    support, i.e. get new portage.



    You can try --backtrack=200 (or a larger value) to fix that; it was a
    problem for a lot of people. Even though the end result of both
    approaches is "it doesn't work," I promise you you're better off not
    messing with the python stuff until the system is fully upgraded.
    Upgrading a year-old system is hard enough, and the introduction of
    EAPI=8 just made it worse.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael@21:1/5 to All on Mon Sep 20 15:29:30 2021
    On Monday, 20 September 2021 14:56:46 BST Gerrit Kuehn wrote:

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    But also when trying "emerge -1vUD @world" (be it with or without the package.use settings), I get stuck in conflicts (mostly on perl and setuptools). perl issues would probably resolve once I have EAPI8
    support, i.e. get new portage.


    cu
    Gerrit

    With a year old system you should question if reinstalling your system after a back up of configuration and data files would be a smarter approach. If you *must* upgrade your current installation for learning or as an experiment,
    then this is something which has been done before. You will need to walk through all the major upgrades of the last year, by using git to gradually catch up with the latest stable versions. This means a lot of packages will have to be updated a number of times.

    Personally, I'd back up /home /etc and world file, plus any databases or websites if stored under /var/, then untar the latest stage 3 tarball and update @system and @world. Overall it should be a *much* faster approach to allow you to bring your installation up to date.
    -----BEGIN PGP SIGNATURE-----

    iQIyBAABCAAdFiEEXqhvaVh2ERicA8Ceseqq9sKVZxkFAmFImsoACgkQseqq9sKV Zxlrrw/4pjmhc4HN+RIHRMg5LvkErunSuEMNujJ0yqYAWlZERqNae3O689u6N6ag fFSXOgP0WqPNUcabitEp5UmVPPHNuZ28mh4OB8A0SokDDAMXoKWzN8DDqMdMiZzc 9KpWwFwTKtHlcGHqsbPqdy8u5Ph5bKDu3K3p4bM4X0oa8sbcw1CGVM8iUhdIQ446 dOye2CALHOc4SqSB4QBoTXFFnnd+brAdqCkGiwAlExn4JTLyGUWEqfwq59/DlhDa PTRINBz5unGGEz0AWj6mLkectEqvrxLYEmSRGKypGwIm3KrC/dFT+eBLI2/oIpea wFaOdhh6AhmMVrAt5IWn4iUUOXNqy0iJT90JxvwgB4AbmtTMmmqOX+8jI2Kf0irI PXZ8MPF1dyujdcrq7oCuwKzLMKIcsPTBv0lhf7t8BbB2R6C/tLlKeTOxtoCW0fG3 EDLxBlFUX4PT1IUfZH2vEWzzBFXeTH75zZSIdqDnYOrEf1LSJk5dyoQcn1pydx+r dGtyIEUKFU1ns0eADbTDC2CawOlC7rFI64NXLhWh4WXIsjZN5dtNDm6EX8De4YRB uK+UgwMTYJX9GCQpeBEt3whDddhoXPdRZL2wwB5xlb5lZF2dJvoOkNw2w6f6+SC+ R6Eo9tvOplf/uALLZXDrtK5Fg1ZbgJiene9EVGd1EplNT3jnLQ==
    =LI/8
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Gerrit Kuehn on Mon Sep 20 16:50:02 2021
    On Mon, 20 Sep 2021 16:31:59 +0200
    Gerrit Kuehn <gerrit.kuehn@aei.mpg.de> wrote:

    This looks like portage is blocked by itself... so how to solve this
    one?

    Well, I simply installed the new virtual packages manually and used
    "--nodeps" on portage itself afterwards... and here we are. Looks like
    new portage will update the rest of my system without any issue now (at
    least it says so when checking with -p), so here we go. Still no idea
    why this was blocking so hard, though.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Arve Barsnes@21:1/5 to Gerrit Kuehn on Mon Sep 20 16:50:02 2021
    On Mon, 20 Sept 2021 at 16:32, Gerrit Kuehn <gerrit.kuehn@aei.mpg.de> wrote:
    (sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc)
    native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev
    (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
    -python3_6 -python3_8 -python3_9" pulled in by
    sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)]
    required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed)
    USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3)
    -python3_6 -python3_8"
    ---


    This looks like portage is blocked by itself... so how to solve this
    one?

    This is because your gentoolkit installed version has no support for
    python 3.9. You might be able to include gentoolkit in your emerge
    command, or you might need to build portage with python 3.8 support as
    well as 3.9. This type of problem might crop up for several other
    packages as you try to get python updated.

    Cheers,
    Arve

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Michael on Mon Sep 20 17:00:01 2021
    On Mon, 20 Sep 2021 15:29:30 +0100
    Michael <confabulate@kintzios.com> wrote:

    With a year old system you should question if reinstalling your
    system after a back up of configuration and data files would be a
    smarter approach. If you *must* upgrade your current installation
    for learning or as an experiment, then this is something which has
    been done before.

    I know. I'm a happy Gentoo user for more than 20 years now. However,
    this is the master installation for a diskless setup providing the
    baseline for more than a dozen servers. Of course, I can still reinstall
    but this would be more work than just one machine + a few config files.
    As one year is not too terribly old (imho) and this looked like
    something that might be easily solved, I decided to ask for advice here
    first.

    Personally, I'd back up /home /etc and world file, plus any databases
    or websites if stored under /var/, then untar the latest stage 3
    tarball and update @system and @world.

    Just extracting stage3 over everything that is already there?

    Overall it should be a *much*
    faster approach to allow you to bring your installation up to date.

    Well, yeah, your mileage may vary.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Arve Barsnes on Mon Sep 20 17:00:02 2021
    On Mon, 20 Sep 2021 16:41:17 +0200
    Arve Barsnes <arve.barsnes@gmail.com> wrote:

    This is because your gentoolkit installed version has no support for
    python 3.9.

    Yes, I must have been overlooking this all the time. That's why I came
    here first (more/better eyes on the issue :-).

    You might be able to include gentoolkit in your emerge
    command, or you might need to build portage with python 3.8 support as
    well as 3.9. This type of problem might crop up for several other
    packages as you try to get python updated.

    emerge -p1vUD @world

    lists about 200 ports to rebuild now, without showing any other issues
    ahead. I'll go for it (and keep my fingers crossed ;-). Thank you for
    your support.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Michael Orlitzky on Mon Sep 20 17:10:01 2021
    On Mon, 20 Sep 2021 10:34:16 -0400
    Michael Orlitzky <mjo@gentoo.org> wrote:

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    It was only suggested for a period of about three weeks:

    Uh, yes, another thing I overlooked. I should get more sleep, I guess.
    :) Thanks for pointing this out!


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael@21:1/5 to All on Mon Sep 20 16:20:58 2021
    On Monday, 20 September 2021 15:52:03 BST Gerrit Kuehn wrote:
    On Mon, 20 Sep 2021 15:29:30 +0100

    Michael <confabulate@kintzios.com> wrote:
    Personally, I'd back up /home /etc and world file, plus any databases
    or websites if stored under /var/, then untar the latest stage 3
    tarball and update @system and @world.

    Just extracting stage3 over everything that is already there?

    No, I move out of the way the config/data files I want to keep and move them back in after untarring the stage 3.


    Overall it should be a *much*
    faster approach to allow you to bring your installation up to date.

    Well, yeah, your mileage may vary.

    Quite, if you can get your existing installation to only run a minimal number of rebuilds to arrive at an upgraded toolchain, then the benefit of reinstalling wouldn't be there. This could have been the case on an average year, when python deprecations didn't accelerate and EAPI didn't change. If however you end up tying up yourself in knots with subsequent python upgrades and difficult to resolve conflicts, then the pain Vs gain calculus changes.


    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEEXqhvaVh2ERicA8Ceseqq9sKVZxkFAmFIptoACgkQseqq9sKV ZxnRRw//eAI3NvkWeFpx6Dc7eD0Lq2WCqt+CGQXFgaNSx8VhVQI9Qug1xqmr08v9 m6wQpgUmECDLftEAvQ0LXgZYX4ILaw7j3/bH6ZR9r9FZLpooewtDwq1oyVeMH3+A xrwN9rJdF2vhbDAkfTNFXmzm1hplOIjoisEgUQ1iiwuCvnhGSXT+hopaL2QMtQsW eQT1mOBjnbEk0dhpAOsAc8OdKzmT7jX5JD2pZ4hg4NwONoebEmEtCGu2EsWKYnGI rl0rNzYcp7OwwDMOi+1yzz0+I3sasBiKdQAL5A+aT+dtMZy9VT5kNEKLZPp+eb7h TjXYlZlDTm9kxAnq7oJZ3yh103ktYqG13Yf1Ich1iYAzbbUXbaXmHabghYlYjnOE W1S0xaPL7FHS0NozQFfo8NsFpGlUYZ2Mq9JpFPZ5yRqXfMMz/0GYUp4oTKfk2bAf mxZhDVhuk4Kiom5YwsErE6LM//+E/con8tiNzpTIDEzlAPcE0bXTFi12Mp9mnGwV Azvr7I4KfCnpFLMSUhWSenNu3dBLgMjX9Q+0GJv2oEgf6qWC+lH7dK4TccdxjrQ1 nVg+ziHPaSBKs9/ao0nFaNVuv2le9ezymutF4YpOcTDo+KowBGxEsWuJUvwl/ox3 LrAmDIhF0xcF7JfIISqwBMscTgSvbr5+Zr0zwZmF6AQ8A6UMHDo=
    =UphK
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Neil Bothwick@21:1/5 to Gerrit Kuehn on Mon Sep 20 17:20:01 2021
    On Mon, 20 Sep 2021 15:56:46 +0200, Gerrit Kuehn wrote:

    ~ # cat /etc/portage/package.use/py
    */* PYTHON_TARGETS: -* python3_9 python3_8
    */* PYTHON_SINGLE_TARGET: -* python3_9
    ---

    You should probably not mess with these variables until after your
    system is 100% updated and consistent. And even then, probably not.

    With our package manager written in python, you often need old python
    stuff to build the new python stuff, and disabling the old python
    stuff will throw a wrench into that. Even in situations where
    technically some upgrade path exists, the complexity of the python dependencies often means that the package manager will give up before
    it finds the solution unless the solution is obvious. By tweaking
    those variables, you make the solution less obvious to it.

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    That news item is about going from 3.8 to 3.9, you are on 3.7. I'd try
    removing the -* items are trying again.


    --
    Neil Bothwick

    "We demand rigidly defined areas of doubt and uncertainty!"

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCAAdFiEE8k9T/rX16EJxEKG692eFu0QSMJgFAmFIpf0ACgkQ92eFu0QS MJiXUQ/8DXm6sfJC8AUee7DIozETJxyOZfNmGIpEvSOTdtwtFWzFHYj5iahhAl1d Y5QchjPXLa+rDoEnxOwg/uRVz/YRWEhNQ6zdie6/BxJZhGc1S4RRxgr59xEBzVmn WQ6haT8z89YdH6QdI66P23KZESA6LUhqIAY0kijVPMKlyAmsm1vWdCzQRRbIMKuY y7sBVIEAhKa8QbX18d47fm5iNMlycD2Gi5GiZ4x2UdN8Gzw5vaxld9C4ayri8qAz Dz3UyhD0NLEeTzrvdqYbIGRXOiYNGjuVxT57FEpjilBoCqpj9Hd6JOqZp/un1MmG udMZ3sb7hW42h7J0aQwjR6lmllx8H0lv2JOsaiuXdyNhD5gV0vBpgJZvYtX/hfIL qHfPwm77berkbex0b9e5G0KEXcUBLWgBy271isIHf+552/wcWtZkbghDzaAzWJyH ISTp55E3z/OK7ramruVMODh3DneWAs8Hmnnb+wnoaPZaHoHtjK+lgTG6cxf2r2Tv 2NZpfHkWsZGRwN9v33PVB6kzR7KwK+d8b7ILTPJX5zv5tgvwnASGYgK2qJ+1gMGT lbHR/mLFKmoMdxXYogLw6SBdDroU40MYDw0bgvGYRK+0cp693M/+Kw00XK9P+g60 kkZNqwbKkp1ZB7FkPMKw8jPiHqI1I4vERYJjmLp7UQp+oRTz+aM=
    =ie9/
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Michael on Mon Sep 20 17:50:02 2021
    On Mon, 20 Sep 2021 16:20:58 +0100
    Michael <confabulate@kintzios.com> wrote:

    Well, yeah, your mileage may vary.

    Quite, if you can get your existing installation to only run a
    minimal number of rebuilds to arrive at an upgraded toolchain, then
    the benefit of reinstalling wouldn't be there. This could have been
    the case on an average year, when python deprecations didn't
    accelerate and EAPI didn't change. If however you end up tying up
    yourself in knots with subsequent python upgrades and difficult to
    resolve conflicts, then the pain Vs gain calculus changes.

    I was already wondering why so many things happened during the
    (comparatively) short time I didn't watch. Looks like people have been
    in lockdown with plenty of time to come around with new things.
    Fortunately, the base installation I have to update doesn't contain
    many applications (e.g., no X involved). Everything sits on an NFS
    server with ZFS below it, so it is quite easy to do snaphots and go
    back and forth between them, and we have plenty to CPU power to rebuild
    things.
    OTOH, the orchestration to have the setup re-done in a fully automated
    way is still under development, so I definitely wanted to try updating
    before really starting over.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gerrit Kuehn@21:1/5 to Neil Bothwick on Mon Sep 20 17:40:01 2021
    On Mon, 20 Sep 2021 16:17:17 +0100
    Neil Bothwick <neil@digimed.co.uk> wrote:

    Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html

    That news item is about going from 3.8 to 3.9, you are on 3.7.

    Yes and no. It started with

    ---
    Display-If-Installed: dev-lang/python:3.7
    Display-If-Installed: dev-lang/python:3.8
    ---

    So my impression was I might find myself in a similar situation with
    3.7.

    I'd try
    removing the -* items are trying again.

    Same issue (actually, I tried that first and only found the news item
    after this didn't work).
    But I think I fixed most issues now (see my previous posts). Thank you
    for taking time to look into this.


    cu
    Gerrit

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)