Control: unmerge 1058317 1063345
Control: reassign 1063345 python3.12
Control: affects 1063345 celery
Control: forwarded 1063345 https://github.com/python/cpython/issues/115874 Control: block 1058317 by 1063345
Control: tag 1063345 patch

On Tue, Feb 06, 2024 at 02:26:55PM +0100, Jérémy Lal wrote:

python3-celery test suite crashes with python 3.12 during gc before exit (see attached stack trace).

It can be reproduced quickly with

apt build-dep celery
dget -xu https://deb.debian.org/debian/pool/main/c/celery/celery_5.3.4-2.dsc cd celery-5.3.4
python3.12 -m pytest -vvvvv t/unit/tasks/test_canvas.py::test_group::test_apply_contains_chords_containing_empty_chain t/unit/tasks/test_canvas.py::test_group::test_apply

Results (0.56s):

1 passed
1 xfailed
Erreur de segmentation (core dumped)

There are two separate issues in the current celery FTBFS, so I'm
unmerging these two bugs again because they ought to be tracked
separately - and it turns out that this one really is a bug in Python
3.12.

I happened to notice that something very similar to #1063345 had
recently been reported to Python upstream as https://github.com/python/cpython/issues/115874, with a much more manageably-sized reproducer, and there's a patch for it in https://github.com/python/cpython/issues/115874#issuecomment-1965775536.
I've built the Debian python3.12 package with this patch, and I've
confirmed that Celery 5.3.6 passes its tests with this patch when it
previously failed.

Celery is unlucky to run into this, but it doesn't seem to be anything particularly intrinsic to Celery - it's an artifact of the particular
set of pytest plugins it happens to use, which manage to tickle this
particular bug. I think it would be worth applying this patch to
Debian's python3.12 package, as it could easily bite in other places and
this was a real time-sink for me before I managed to find the upstream
bug that somebody else had fortunately filed. (It would also be nice to
get Celery back into Debian testing and Ubuntu noble if at all possible,
the latter of which has a fairly close deadline.)

Separately, #1058317 still has the test failure in
test_AsyncResult.test_del. That was fixed in Celery 5.3.5, and 5.3.6 is currently on salsa, but there's only limited point in uploading it until
the Python bug is fixed.

Suggested Python patch follows, though you might want to check whether
it's progressed any further upstream in case there've been any
additional tweaks:

diff --git a/debian/changelog b/debian/changelog
index 0e16636..b57f7dc 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+python3.12 (3.12.2-1.1) UNRELEASED; urgency=medium
+
+ * Don't rely on module state in teedataobject_clear (from Brandt Bucher in
+ https://github.com/python/cpython/issues/115874; closes: #1063345).
+
+ -- Colin Watson <cjwatson@debian.org> Tue, 27 Feb 2024 10:15:37 +0000
+
python3.12 (3.12.2-1) unstable; urgency=medium

* Python 3.12.2 release.
diff --git a/debian/patches/itertools-clear-crash.diff b/debian/patches/itertools-clear-crash.diff
new file mode 100644
index 0000000..cdaeebb
--- /dev/null
+++ b/debian/patches/itertools-clear-crash.diff
@@ -0,0 +1,19 @@
+Description: Don't rely on module state in teedataobject_clear
+Origin: other, https://github.com/python/cpython/issues/115874#issuecomment-1965775536
+Bug: https://github.com/python/cpython/issues/115874
+Bug-Debian: https://bugs.debian.org/1063345
+
+Index: b/Modules/itertoolsmodule.c +=================================================================