Control: retitle -1 test_allenvs_print_extras tests are flaky

On Fri, Nov 03, 2023 at 08:57:57PM +0800, Bo YU wrote:

The bug was opened to track a potential bugs on armel(or others slow
archs, but it does not always happened).

Fix from upstream[0] may take a long time so if you use the package please keep in mind about this unless the bug was closed.

I recently uploaded tox/4.14.1-1. tox-current-env autopkgtests fail on
s390x[1] with a similar error, which in turn is currently preventing tox
from migrating into testing[2]

1: https://ci.debian.net/packages/t/tox-current-env/testing/s390x/44050718/
2: https://qa.debian.org/excuses.php?package=tox

This bug was originally for test_allenvs_print_extras_to_file(); for
this I would imagine think the fix is something like:
- assert prep_tox_output(result.stdout) == expected
+ assert sorted(prep_tox_output(result.stdout)) == sorted(expected)

The s390x autopkgtest above fails on test_allenvs_print_extras(), which
is very similar. However, that one seems to have a sorted() already:
45s def test_allenvs_print_extras(print_extras_stdout_arg):
45s result = tox(print_extras_stdout_arg)
45s expected = []
45s for env in envs_from_tox_ini():
45s expected.extend(("dev", "full", f"{env}: OK"))
45s expected.pop() # The last "py310: OK" is not there
45s expected.append(tox_footer(spaces=0))
45s expected = ("\n".join(expected)).splitlines()
45s > assert sorted(prep_tox_output(result.stdout).splitlines()) == sorted(expected)

I don't fully grok what this is supposed to be doing (but I've looked at
it for only a couple of minutes). That .pop() does look suspicious
though, in the sense that it operates before the output is sorted.

Faidon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Bo,

Thanks for the short reaction time and for spending your time on this
bug, much appreciated!

On Fri, Mar 22, 2024 at 01:26:35PM +0800, Bo YU wrote:

Thanks your detailed analysis for it. I have uploaded -3 to unstable to
hope to unblock tox migrating. But these are some unexpected maybe.

tox has migrated since. Honestly I'm not sure why, given the regression!
So there isn't as much urgency as there was when I filed this bug, I'd
say.

This will lead a autopkgtest on s390x[0] again from its tracker page[1] to get the link(UTC+8 13:06 2024/03/22).

But another retry on s390x it passed on s390x[2].

[0]: https://ci.debian.net/packages/t/tox-current-env/testing/s390x/44235641/ [1]: https://tracker.debian.org/pkg/tox-current-env
[2]: https://ci.debian.net/packages/t/tox-current-env/

Yup, looks like the same bug all over the place, effectively a race.

Do you think I will need to upload it to unstable again with `sorted`
for `test_allenvs_print_deps_to_file` as I analyzed above?

It's clearly a bug IMHO, so it'd be good to fix at some point. No
urgency with regards to the tox migration as I said, but this will
happen again on e.g. the next tox upload.

If I were you I'd fix it in git, submit all of these fixes to upstream,
wait a month or two so in case upstream releases a new version
(hopefully with these), and then upload regardless.

Thanks again,
Faidon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)