Package: autopkgtest
Version: 5.32
Severity: wishlist

tl;dr:

autopkgtest is slower than it needs to be for packages with many
tests. I can see at least one easy opportunity for a potentially
substantial improvement, and have another more doubtful idea.

Background and factual information:

src:dgit has over a hundred autopkgtests (115 in all right now, but
some don't run in CI due to Restrictions; only 108 run in CI). This
is convenient for isolating failures. The tests are grouped into
(currently) 26 stanzas in debian/tests/control.

These tests usually take around 40 minutes to run in a formal run
under autopkgtest. Sometimes the time taken exceeds the default
1-hour timeout in salsa CI.

Some of the tests are very fast - one or two seconds. Some take much
longer.

(During a manual run outside autopkgtest, a complete run of the tests
takes about 8 minutes, but that's with CPU parallelism so isn't
comparable. I haven't done a non-parallel benchmark run.)

Proposal 1:

Observing the logs, I see that even for tests in the same stanza,
autopkgtest prpeares and installs an autopkgtest-satdep package,
between every test.

For short tests, this dominates the execution time. In the logs eg
https://salsa.debian.org/dgit-team/dgit/-/jobs/5270905
it seems to take about 6 seconds.

So I think skipping the testbed re-preparation for tests within the
same stanza would save about 6 seconds per such test, or 530-ish
seconds for each run. Ie, it might cut 20% off the total runtime for
this package.

I'm hoping that this would be relatively straightforward to implement.

Proposal 2:

*Re*installing packages involved with many of the tests seems quite
expensive. For src:dgit, most of the tests involve installing
dgit.deb. dgit and its dependencies seem to take about 20s to
install.

If the test stanzas could be sequenced into "chains" where each test
is has a monotonically longer dependency list, the intervening satdep installation, to get from one test stanza in the chain, to the nest
stanza, wouldn't need to involve *reinstalling* anything.

I'm not sure how much time this would save. Also it's not clear to me
that this would always be 100% faithful. After dependency resolution
is not monotonic: an extra dependency might result in *deinstallation*
of packages.

So this proposal is more fanciful but I thought I would share it
anyway.

Regards,c
Ian.

--
Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own.

Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)