Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
It install by default the build dependencies. which is not what
peoples are doing when they install the packages.
It should be possible to define the autopkgtest dependencies. This
way we could catch missing dependencies in python-<modules>
dependencies.
Is it possible ?
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)
Cheers,
While I agree that running tests is good, it's not a universally
reasonable requirement.
2. There's at least one case where Debian has customizations that
cause the test suite to fail, but the failures don't seem to cause any
real problems. If anyone wants to make it so the weasyprint test suite
works on Debian, please knock yourself out.
3. I also maintain multiple packages which require network access
to run their test suite, so they can't run tests during build, only autopkgtests.
If we add this as a hard requirement for team packages, what do people
think will happen? Will people invest more volunteer time that they otherwise wouldn't have to add running tests? Will they leave it for
someone else on the team to address? Will they remove packages from
team maintenance?
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test >> a random package in this way.
Even if not minimal, at least correspond to the upstream declares dependencies.
by 'declare' I am not even sure of the meaning.
dependencies of the test can be different from the required installed ones.
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.
Is there a standard in the Python community for the test dependencies ?
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
[...]
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged,
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Hey,
Louis-Philippe Véronneau <pollo@debian.org> wrote on 29/07/2024 at 10:53:11+0200:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)
Cheers,
I spend a lot of time trying to make the upstream tests working in
package building, but I'm against enforcing this.
There are a lot of packages with good code but crappy tests, and we
don't want to force ourselves maintaining these, and dropping the
packages seem a bit too much.
On July 30, 2024 12:49:50 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
On 2024-07-29 21:07, Scott Kitterman wrote:
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends
out.
That sounds like a valid technical reason not to run the tests to me :)
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself
with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.
Again, as long as you document that, I don't think it would go against the proposed policy change.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Same.
Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem
On 2024-07-29 21:07, Scott Kitterman wrote:I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged,
That sounds like a valid technical reason not to run the tests to me :)out.
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself
Again, as long as you document that, I don't think it would go against the proposed policy change.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Same.
On 2024-07-30 11:09, Scott Kitterman wrote:packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
On July 30, 2024 12:49:50 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
On 2024-07-29 21:07, Scott Kitterman wrote:
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends
out.
That sounds like a valid technical reason not to run the tests to me :)
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself
with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.
Again, as long as you document that, I don't think it would go against the proposed policy change.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Same.
Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem
Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...
On July 30, 2024 2:47:08 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
On 2024-07-30 11:09, Scott Kitterman wrote:
On July 30, 2024 12:49:50 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
On 2024-07-29 21:07, Scott Kitterman wrote:
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends
out.
That sounds like a valid technical reason not to run the tests to me :) >>>>
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself
with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.
Again, as long as you document that, I don't think it would go against the proposed policy change.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Same.
Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem
specifications.
Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...
I guess it depends on how you look at the word. In the context of technical specifications, I tend to think of terms as defined in RFC 2119 [1]. I think that's pretty close to what you are suggesting:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
The way RFC 2119 defines MUST doesn't leave much room for exceptions. If you want to be clear, I suggest SHOULD with a reference to RFC 2119. While not universal, it is a widely used reference for definitions of these terms in technical
Scott K
[1] https://datatracker.ietf.org/doc/html/rfc2119#section-3
On 2024-07-30 12:58, Scott Kitterman wrote:packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.
On July 30, 2024 2:47:08 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
On 2024-07-30 11:09, Scott Kitterman wrote:
On July 30, 2024 12:49:50 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
On 2024-07-29 21:07, Scott Kitterman wrote:
On July 29, 2024 8:53:11 AM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)
I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:
1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends
yourself out.
That sounds like a valid technical reason not to run the tests to me :) >>>>>
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock
with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.
Again, as long as you document that, I don't think it would go against the proposed policy change.
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.
Same.
Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem
specifications.
Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...
I guess it depends on how you look at the word. In the context of technical specifications, I tend to think of terms as defined in RFC 2119 [1]. I think that's pretty close to what you are suggesting:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
The way RFC 2119 defines MUST doesn't leave much room for exceptions. If you want to be clear, I suggest SHOULD with a reference to RFC 2119. While not universal, it is a widely used reference for definitions of these terms in technical
Scott K
[1] https://datatracker.ietf.org/doc/html/rfc2119#section-3
Fair enough. I also made that change.
Those will correspond to build dependencies in a typical simple case, as
the upstream tests need all those optional deps, assuming you mean the
tests dependencies (and not the runtime dependencies which should already
be in Depends).
Maybe we should install only the python binaries and the dependencies marked
<!nocheck>.
In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.
So this is exactly what you wanted to avoid.
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test a random package in this way.
Even if not minimal, at least correspond to the upstream declares dependencies.
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.
In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.
On Mon, Jul 29, 2024 at 03:30:39PM +0200, PICCA Frederic-Emmanuel wrote:
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.
Maybe we indeed want a "minimal" autopkgtest environment
but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.
Maybe we indeed want a "minimal" autopkgtest environment
For a simple smoke-test that the Python packages import succesfully, isn't this exactly autopkgtest-pkg-python?
For thorough test coverage, I don't think this is possible: in general,
the test suite will legitimately require installation of packages that are only needed during testing (pytest or similar), or packages that are
optional for basic use of the code but mandatory for full test coverage (those might be Recommends or Suggests).
but many
upstream tests will fail in those and I don't see an automatic way to test a random package in this way
If we want to run upstream test-suites as autopkgtests, then I think
ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate autopkgtest test-cases with different dependencies.
But I don't think we can easily have that, because the Testsuite field
only takes one value.
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.
In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.
Erm, no, you need to B-D on all runtime dependencies so they are
correctly included in the computed python3:Depends substvar, unless
you list these runtime dependencies explictly by hand.
Yes, I want to be sure that all the runtime dependencies are rightfully declared in the Depends of the python module package.
sometimes the upstream forgot about dependencies, or mark them as optional, but they are not when running the tests...
Is it possible to achieve this automatically ? lintian ? dh-python ? python3-stdeb ?
On Wed, Jul 31, 2024 at 09:46:27AM +0100, Simon McVittie wrote:
If we want to run upstream test-suites as autopkgtests, then I think ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate autopkgtest test-cases with different dependencies.
I always thought the smoke test is redundant if there is the upstream
tests one.
What do you think?
But I don't think we can easily have that, because the Testsuite field
only takes one value.
It's been about 2 weeks since the policy change has been proposed. I believe I have taken in account the feedback and the discussion seems to have died down.
As the response from the team member was largely positive, I propose the change be merged.
On 2024-07-29 4 h 53 a.m., Louis-Philippe Véronneau wrote:
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
You can find the MR here:
https://salsa.debian.org/python-team/tools/python-modules/-/
merge_requests/24
People present at the BoF seemed to think this was a good idea. If
you don't please reply to this message and make yourself heard :)
Cheers,
Hi,
It's been about 2 weeks since the policy change has been proposed. I
believe I have taken in account the feedback and the discussion seems
to have died down.
As the response from the team member was largely positive, I propose
the change be merged.
Cheers,
Can you be more explicit on the reasons why you dislike the second sentence?
It only says you should document why tests aren't ran in a package and encourages you to open a bug report.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 349 |
Nodes: | 16 (2 / 14) |
Uptime: | 106:25:25 |
Calls: | 7,611 |
Calls today: | 2 |
Files: | 12,786 |
Messages: | 5,682,888 |