I've been having a somewhat interesting time with the python3.11-add transition. Python 3.11 has made some significant changes to its
bytecode representation, and also changed some of it's internal data
structures related to frames quite significantly.

In my corner of the Python world, several important packages use this information, including Spyder, IPython and Jupyter (which all
(recursively) depend on python3-parso and/or python3-bytecode), but
these low-level packages are not really ready for 3.11 yet (parso
claims "basic support for Python 3.11 and 3.12", so it's unlikely it
will actually yet work with 3.11, though I haven't tested this, and
bytecode makes no claims to yet be ready for 3.11).

Several packages have tests which now fail on 3.11, preventing
migration to testing (spyder-kernels is one). In pydevd (which does
not yet support 3.11, because it depends on python3-bytecode), I've
just disabled all the failing tests and put a note in README.Debian,
but this is not a great solution. I could add X-Python3-Version: <<
3.11 to debian/control, but that is surely not that desirable either.

I'm just flagging this up here, with a question about how we should
proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-11-21 02 h 08, Julian Gilbey wrote:

I'm just flagging this up here, with a question about how we should
proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty
good and it seemed reasonable to prioritize stability for our next
stable release :)

It's very frustrating to work on packaging python libraries and apps for
a whole release cycle, just to see all that work put in the bin at the
last minute because upstream doesn't support 3.11...

I've been told the current 3.11 transition was a test, and if it was
clear too many important things were broken and couldn't be fixed, we
would roll back and release using 3.10.

--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
⢿⡄⠘⠷⠚⠋ pollo@debian.org / veronneau.org
⠈⠳⣄

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Nov 21, 2022 at 12:03 PM Louis-Philippe Véronneau
<pollo@debian.org> wrote:

On 2022-11-21 02 h 08, Julian Gilbey wrote:

I'm just flagging this up here, with a question about how we should proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty
good and it seemed reasonable to prioritize stability for our next
stable release :)

It's very frustrating to work on packaging python libraries and apps for
a whole release cycle, just to see all that work put in the bin at the
last minute because upstream doesn't support 3.11...

this, 100 times

I've been told the current 3.11 transition was a test, and if it was
clear too many important things were broken and couldn't be fixed, we
would roll back and release using 3.10.

why are we running a "test" this close to the release? *who* are we
running this test for? who made this decision (i figure RT gave the go
ahead, but still)? is there any searchable source for this claim?

Thanks,
--
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On November 21, 2022 5:02:57 PM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:

On 2022-11-21 02 h 08, Julian Gilbey wrote:

I'm just flagging this up here, with a question about how we should
proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty good and it seemed reasonable to prioritize stability for our next stable release :)

It's very frustrating to work on packaging python libraries and apps for a whole release cycle, just to see all that work put in the bin at the last minute because upstream doesn't support 3.11...

I've been told the current 3.11 transition was a test, and if it was clear too many important things were broken and couldn't be fixed, we would roll back and release using 3.10.

Looks like you can add #1024521 to the list.

Scott K

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/21/22 18:30, Sandro Tosi wrote:

On Mon, Nov 21, 2022 at 12:03 PM Louis-Philippe Véronneau
<pollo@debian.org> wrote:

On 2022-11-21 02 h 08, Julian Gilbey wrote:

I'm just flagging this up here, with a question about how we should
proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty
good and it seemed reasonable to prioritize stability for our next
stable release :)

It's very frustrating to work on packaging python libraries and apps for
a whole release cycle, just to see all that work put in the bin at the
last minute because upstream doesn't support 3.11...

this, 100 times

I very much don't agree. I think it's going pretty well, and the number
of breakage isn't high. We just need a little bit of effort to make it
in good enough shape.

I've been told the current 3.11 transition was a test, and if it was
clear too many important things were broken and couldn't be fixed, we
would roll back and release using 3.10.

Though we're not there yet, as a point were we can say it's a lost battle.

why are we running a "test" this close to the release?

Let me fix the above sentence for you:
s/release/freeze/

Because Python 3.11 brings many nice feature, one of them being that
it's 15 to 50% faster, very often 20% faster. Also, releasing the last
Debian with an already obsolete interpreter version doesn't feel good.

*who* are we
running this test for? who made this decision (i figure RT gave the go
ahead, but still)? is there any searchable source for this claim?

This was discussed during Debconf in Prizren. You are always free to:
- join us during debconf
- join on IRC if you can't go to Debconf
- read the video and voice your concerns on the list

So the decision was made collectively. We also discussed this on IRC,
and in this very list, if I'm not mistaking (sorry, I will not search
for it).

Currently, Python 3.11 is only an "available interpreter version", not
the default. So it's kind of easy to revert (we would "only" need to
remove the 3.11 interpreter, and rebuild the packages that produce 3.11
so files). It would be a lot harder to revert having 3.11 as default,
Mathias said/wrote. So we're good...

I very much thank Stefano for the many fixes he already uploaded.

Now, out of *many* of my packages, only a very few broke. Complicated
packages like Eventlet for example, just worked. Others had upstream
patches I applied. And I am in the opinion that we should go ahead and
make 3.11 the default.

I'd be happy to have the opinion of the rest of the team, especially
Doko and Stefano.

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Nov 22, 2022 at 09:22:05AM +0100, Thomas Goirand wrote:

this, 100 times

I very much don't agree. I think it's going pretty well, and the number of breakage isn't high. We just need a little bit of effort to make it in good enough shape.
[...]
Now, out of *many* of my packages, only a very few broke. Complicated packages like Eventlet for example, just worked. Others had upstream patches I applied. And I am in the opinion that we should go ahead and make 3.11 the default.

If there are people with the expertise to help upstream update
bytecode and parso (and probably several other low-level packages) for
3.11 so that the software that depends on them works with 3.11, then
fine. (And it is a non-trivial update, AFAICT.) But until then, I'd
be very reluctant to make 3.11 the default.

I haven't decided what to do with packages which now FTBFS under 3.11
because of this. Should we just let them fall out of testing (that
certainly includes spyder, and quite possibly ipython as well)? Or
should we mark them as X-Python3-Version: << 3.11 so they can stay in
testing as long as Python 3.10 is the default? If we make 3.11 the
default, these packages will likely not be in bookworm, which might
upset our users even more.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Monday, November 21, 2022 12:25:05 PM EST Scott Kitterman wrote:

On November 21, 2022 5:02:57 PM UTC, "Louis-Philippe V�ronneau"

<pollo@debian.org> wrote:

On 2022-11-21 02 h 08, Julian Gilbey wrote:

I'm just flagging this up here, with a question about how we should
proceed. Certainly we are not ready to make Python 3.11 the default
Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty good >and it seemed reasonable to prioritize stability for our next stable >release :)

It's very frustrating to work on packaging python libraries and apps for a >whole release cycle, just to see all that work put in the bin at the last >minute because upstream doesn't support 3.11...

I've been told the current 3.11 transition was a test, and if it was clear >too many important things were broken and couldn't be fixed, we would roll >back and release using 3.10.

Looks like you can add #1024521 to the list.

This one is fixed.

Scott K
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEE53Kb/76FQA/u7iOxeNfe+5rVmvEFAmN82CUACgkQeNfe+5rV mvFEaA//Tf3ehCfmWR3Rt7ad326D9Vy+1Z85qhRtegfKeTgn1NfZ9TVKopCp/oJa TmSzl5UW7/QC+52H9Pw0k97M85ST8ofuEjgCUrRKhV4oy2po04NN72dMaAc6GVpM ygqmWnRfSFZpITGms7c4HHOpCYTn3zZ6Jx45oY2sLYwHL5jPk2VAyzJ8nH07Wu8a oKw88fgZcXtF9jzl/+cJO9UFZfpXFPxeX1sB2OYN+ZhI1AMawbdRuLnW7WJZGh/L 0jiTUTN5XGvJSo/KkrATgchvLpHogPavmeTXE2TxM0pCM9OX4Uakhc2hkMpfKmKV UCss+Ei9bZO08o5YpC3VIdPp4anH26rM0DY9TT0GdYVDbdbZocO5isgSxtjl+KzO fPTODrO1qM9IIEV8wRDSl/rrGuJUnil6ZGQ2sBuiNqkQQxh9Yujd7RVFfENFhxjD XeJA4k9GAZ6OmuoJHOdsJsSvgTd0zAszY/s0p5CDxh0mObJ4Jylzz2p7wfeNHLm5 jqf+09xy3I7biQ5GtE5AgMq5OmvgKm0/aMGmnTzgS6x16uK7I8HaAM8wv7KMP3x6 eAh3q7o4Xf+RAysws3my7oId/yfWFR3zXdXMimVqgiEzdH3si1mCOUaFlJ2yXxum zLbwCWxRaGDvCqStQ/By4TpDoRVUcBdISs7vX28oxiD0txiG63Y=
=zLWk
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/22/22 10:59, Julian Gilbey wrote:

On Tue, Nov 22, 2022 at 09:22:05AM +0100, Thomas Goirand wrote:

this, 100 times

I very much don't agree. I think it's going pretty well, and the number of >> breakage isn't high. We just need a little bit of effort to make it in good >> enough shape.
[...]
Now, out of *many* of my packages, only a very few broke. Complicated
packages like Eventlet for example, just worked. Others had upstream patches >> I applied. And I am in the opinion that we should go ahead and make 3.11 the >> default.

If there are people with the expertise to help upstream update
bytecode and parso (and probably several other low-level packages) for
3.11 so that the software that depends on them works with 3.11, then
fine. (And it is a non-trivial update, AFAICT.) But until then, I'd
be very reluctant to make 3.11 the default.

Have you tried this PR?
https://github.com/MatthieuDartiailh/bytecode/pull/107

I haven't decided what to do with packages which now FTBFS under 3.11
because of this. Should we just let them fall out of testing (that
certainly includes spyder, and quite possibly ipython as well)?
Or should we mark them as X-Python3-Version: << 3.11 so they can stay in testing as long as Python 3.10 is the default?

I don't think this is the way.

If we make 3.11 the
default, these packages will likely not be in bookworm, which might
upset our users even more.

We're not there yet. We have until January to fix, and we haven't
decided yet if 3.11 will be the default.

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Nov 22, 2022 at 05:01:03PM +0100, Thomas Goirand wrote:

If there are people with the expertise to help upstream update
bytecode and parso (and probably several other low-level packages) for
3.11 so that the software that depends on them works with 3.11, then
fine. (And it is a non-trivial update, AFAICT.) But until then, I'd
be very reluctant to make 3.11 the default.

Have you tried this PR? https://github.com/MatthieuDartiailh/bytecode/pull/107

As you can see by reading it, there is still at least one blocking
point (to use Matthieu's language), and Matthieu is the core bytecode developer. Once that is sorted, then pydevd needs to be updated to
use it. But this is far outside my expertise, and I'm not going to
apply a huge patch that I don't understand and that is known to still
be buggy. So I'm just going to have to wait.

I don't know the current state of parso; it would be interesting to
see whether parso 0.8.3 successfully works under 3.11, but that is
Piotr Ożarowski's package (not under the Python team); he is active,
but has not yet uploaded this newer version.

I haven't decided what to do with packages which now FTBFS under 3.11 because of this. Should we just let them fall out of testing (that certainly includes spyder, and quite possibly ipython as well)?
Or should we mark them as X-Python3-Version: << 3.11 so they can stay in testing as long as Python 3.10 is the default?

I don't think this is the way.

I'm sorry, I don't understand - which is not the way?

If we make 3.11 the
default, these packages will likely not be in bookworm, which might
upset our users even more.

We're not there yet. We have until January to fix, and we haven't decided
yet if 3.11 will be the default.

Fair enough.

But I still don't know what to do in the meantime with the spyder
ecosystem besides either wait for upstream to sort bytecode and pydevd
and Piotr (and possibly upstream) to sort parso, or to mark them as
Python 3.10 only.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/22/22 17:59, Julian Gilbey wrote:

Or should we mark them as X-Python3-Version: << 3.11 so they can stay in >>> testing as long as Python 3.10 is the default?

I don't think this is the way.

I'm sorry, I don't understand - which is not the way?

I don't think you should "mark them as X-Python3-Version: << 3.11",
because either we use 3.10 or 3.11 in Bookworm, I don't think that
there's a plan for having both interpreters available.

But I still don't know what to do in the meantime with the spyder
ecosystem besides either wait for upstream to sort bytecode and pydevd
and Piotr (and possibly upstream) to sort parso, or to mark them as
Python 3.10 only.

Well, hopefully for you, you'll get it fixed before next January, or we
go back to 3.10 only (or both?).

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Nov 23, 2022 at 12:21:38AM +0100, Thomas Goirand wrote:

On 11/22/22 17:59, Julian Gilbey wrote:

Or should we mark them as X-Python3-Version: << 3.11 so they can stay in
testing as long as Python 3.10 is the default?

I don't think this is the way.

I'm sorry, I don't understand - which is not the way?

I don't think you should "mark them as X-Python3-Version: << 3.11", because either we use 3.10 or 3.11 in Bookworm, I don't think that there's a plan
for having both interpreters available.

It's more about the present time: if I mark them as << 3.11, then they
will build fine and be able to stay in testing. If I don't, they'll
be pulled from testing in the next few weeks because of FTBFS bugs.
Obviously, once they do build OK with 3.11, then I can remove that
clause.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)