Is it better than
RESTRICT="test"
?
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
create mode 100644 glep-9999.ebuild
diff --git a/glep-9999.ebuild b/glep-9999.ebuild
new file mode 100644
index 0000000..9ee18ca
--- /dev/null
+++ b/glep-9999.ebuild
@@ -0,0 +1,132 @@
+---
+GLEP: 9999
+Title: TEST_SUITE_PRESENT variable
+Author: Michał Górny <mgorny@gentoo.org>
+Type: Standards Track
+Status: Draft
+Version: 1
+Created: 2023-02-19
+Last-Modified: 2023-02-19
+Post-History: 2023-02-19
+Content-Type: text/x-rst
+---
+
+
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite. It can be set either by the ebuild,
+the eclass or the default ``src_test`` implementation, and afterwards +included in the package manager logs. This can aid in analyzing
+the results of automated package testing.
+
+
+Motivation
+==========
+
+The deployment of new Python targets in Gentoo currently involves
+testing of a large number of Gentoo packages against the said target.
+This is currently done manually for the most part. It can be
+particularly time consuming if multiple individuals repeatedly test
+the same package only to determine that it remains incompatible with
+the new interpreter.
+
+The Python team wanted to explore the use of automation to aid this +testing. Unfortunately, this faces a major problem: for the vast
+of majority of packages, the incompatibilities with new Python versions
+do not exhibit during the installation and can only be detected through +running the test suite. The results of automated testing are therefore +only meaningful if the package features a test phase.
+
+For packages using ``distutils-r1`` eclass, the presence of test suite
+can usually be easily determined through grepping for +``distutils_enable_tests`` call or an explicit ``python_test()``
+function. Even then, it seems sensible to work towards a more generic +approach to tell whether a package had a test suite or not,
+and therefore whether a particular successful automated testing result +means that the package actually passed tests or only confirmed that
+the Python files were copied successfully.
+
+An explicit indication whether a test suite was present can be presented
+by the package manager as part of logs, along with the result of running +the test phase. Afterwards, these logs can be used to determine which +packages were actually tested.
+
+
+Specification
+=============
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
+by a ``src_test()`` implementation to indicate whether the package
+featured a test suite. It can take three values:
+
+- ``yes`` indicating that a test suite was run
+- ``indeterminate`` indicating that it was not possible to clearly
+ determine whether the test suite was present or not (this could be
+ a case e.g. when a generic test command is run and it does not
+ indicate whether any tests were found)
+- ``no`` indicating that no test suite was run
+
+This variable *should* be set by eclasses defining the ``src_test()`` +phase. If the package in question is using ``src_test()`` defined
+by an eclass that does not declare it explicitly, the PM must assume +``indeterminate``.
+
+The variable *may* be set by an ebuild defining the ``src_test()``
+phase. If the ebuild does not define it explicitly, the PM must assume +``yes``.
+
+The default ``src_test()`` implementation as defined by the PMS sets
+the value to ``indeterminate`` if it runs a ``check`` or ``test``
+target, and to ``no`` if neither of the targets is found.
+
+
+Rationale
+=========
+
+The use of ternary flag makes it possible to clearly represent all three +possible outcomes while navigating the defaults defined in the GLEP.
+The flag is set in ``src_test()``, so that runtime conditions (such
+as the results obtained from the actual test runner) can be used to +determine the actual value.
+
+The defaults were defined based on the following assumptions:
+
+1. The presence of ``check`` target is common in autotools projects but
+ it does not guarantee that the target actually does anything, let
+ alone run a proper test suite. However, the lack of any test target
+ clearly indicates that no tests were run.
+
+2. Eclass ``src_test`` implementations can be very generic and succeed
+ without actually performing any testing. It is therefore reasonable
+ to default to ``indeterminate`` result when they are used,
+ and recommend them to explicitly override the variable.
+
+3. Explicit ``src_test`` declared in ebuild can generally be assumed
+ to actually run tests, with the exception of declaring the function
+ to prevent ``default_src_test`` from running. It therefore makes
+ sense to default to ``yes`` but allow ebuilds to override the value
+ in the latter case.
+
+
+Backwards Compatibility
+=======================
+
+This GLEP is entirely optional. The package managers not implementing
+it will treat the variable as a regular bash variable set by the test
+phase and ignore it.
+
+
+Reference Implementation
+========================
+
+TODO
+
+
+Copyright
+=========
+
+This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 +International License. To view a copy of this license, visit +https://creativecommons.org/licenses/by-sa/4.0/.
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
create mode 100644 glep-9999.ebuild
diff --git a/glep-9999.ebuild b/glep-9999.ebuild
new file mode 100644
index 0000000..9ee18ca
--- /dev/null
+++ b/glep-9999.ebuild
@@ -0,0 +1,132 @@
+---
+GLEP: 9999
+Title: TEST_SUITE_PRESENT variable
+Author: Michał Górny <mgorny@gentoo.org>
+Type: Standards Track
+Status: Draft
+Version: 1
+Created: 2023-02-19
+Last-Modified: 2023-02-19
+Post-History: 2023-02-19
+Content-Type: text/x-rst
+---
+
+
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite. It can be set either by the ebuild,
+the eclass or the default ``src_test`` implementation, and afterwards +included in the package manager logs. This can aid in analyzing
+the results of automated package testing.
+
+
+Motivation
+==========
+
+The deployment of new Python targets in Gentoo currently involves
+testing of a large number of Gentoo packages against the said target.
+This is currently done manually for the most part. It can be
+particularly time consuming if multiple individuals repeatedly test
+the same package only to determine that it remains incompatible with
+the new interpreter.
+
+The Python team wanted to explore the use of automation to aid this +testing. Unfortunately, this faces a major problem: for the vast
+of majority of packages, the incompatibilities with new Python versions
+do not exhibit during the installation and can only be detected through +running the test suite. The results of automated testing are therefore +only meaningful if the package features a test phase.
+
+For packages using ``distutils-r1`` eclass, the presence of test suite
+can usually be easily determined through grepping for +``distutils_enable_tests`` call or an explicit ``python_test()``
+function. Even then, it seems sensible to work towards a more generic +approach to tell whether a package had a test suite or not,
+and therefore whether a particular successful automated testing result +means that the package actually passed tests or only confirmed that
+the Python files were copied successfully.
+
+An explicit indication whether a test suite was present can be presented
+by the package manager as part of logs, along with the result of running +the test phase. Afterwards, these logs can be used to determine which +packages were actually tested.
+
+
+Specification
+=============
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
+by a ``src_test()`` implementation to indicate whether the package
+featured a test suite. It can take three values:
+
+- ``yes`` indicating that a test suite was run
+- ``indeterminate`` indicating that it was not possible to clearly
+ determine whether the test suite was present or not (this could be
+ a case e.g. when a generic test command is run and it does not
+ indicate whether any tests were found)
+- ``no`` indicating that no test suite was run
+
+This variable *should* be set by eclasses defining the ``src_test()`` +phase. If the package in question is using ``src_test()`` defined
+by an eclass that does not declare it explicitly, the PM must assume +``indeterminate``.
+
+The variable *may* be set by an ebuild defining the ``src_test()``
+phase. If the ebuild does not define it explicitly, the PM must assume +``yes``.
+
+The default ``src_test()`` implementation as defined by the PMS sets
+the value to ``indeterminate`` if it runs a ``check`` or ``test``
+target, and to ``no`` if neither of the targets is found.
+
+
+Rationale
+=========
+
+The use of ternary flag makes it possible to clearly represent all three +possible outcomes while navigating the defaults defined in the GLEP.
+The flag is set in ``src_test()``, so that runtime conditions (such
+as the results obtained from the actual test runner) can be used to +determine the actual value.
+
+The defaults were defined based on the following assumptions:
+
+1. The presence of ``check`` target is common in autotools projects but
+ it does not guarantee that the target actually does anything, let
+ alone run a proper test suite. However, the lack of any test target
+ clearly indicates that no tests were run.
+
+2. Eclass ``src_test`` implementations can be very generic and succeed
+ without actually performing any testing. It is therefore reasonable
+ to default to ``indeterminate`` result when they are used,
+ and recommend them to explicitly override the variable.
+
+3. Explicit ``src_test`` declared in ebuild can generally be assumed
+ to actually run tests, with the exception of declaring the function
+ to prevent ``default_src_test`` from running. It therefore makes
+ sense to default to ``yes`` but allow ebuilds to override the value
+ in the latter case.
+
+
+Backwards Compatibility
+=======================
+
+This GLEP is entirely optional. The package managers not implementing
+it will treat the variable as a regular bash variable set by the test
+phase and ignore it.
+
+
+Reference Implementation
+========================
+
+TODO
+
+
+Copyright
+=========
+
+This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 +International License. To view a copy of this license, visit +https://creativecommons.org/licenses/by-sa/4.0/.
--
2.39.2
What if developer configured an ebuild in a way that it downloads the
test suite/files/data with USE=test?
IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should
be true (exists).
+AbstractI wonder if we could not simply use IUSE="test" for this purpose? That
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite.
On 20 Feb 2023, at 08:37, Florian Schmaus <flow@gentoo.org> wrote:
On 19/02/2023 18.32, Michał Górny wrote:
+AbstractI wonder if we could not simply use IUSE="test" for this purpose? That is, allow declaring the 'test' use flag, even though it is not used in a USE conditional, to indicate that the package has a test suite.
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite.
What if developer configured an ebuild in a way that it downloads the
test suite/files/data with USE=test?
IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should
be true (exists).
W dniu 19.02.2023 o 18:32, Michał Górny pisze:
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 132 insertions(+)
create mode 100644 glep-9999.ebuild
diff --git a/glep-9999.ebuild b/glep-9999.ebuild
new file mode 100644
index 0000000..9ee18ca
--- /dev/null
+++ b/glep-9999.ebuild
@@ -0,0 +1,132 @@
+---
+GLEP: 9999
+Title: TEST_SUITE_PRESENT variable
+Author: Michał Górny <mgorny@gentoo.org>
+Type: Standards Track
+Status: Draft
+Version: 1
+Created: 2023-02-19
+Last-Modified: 2023-02-19
+Post-History: 2023-02-19
+Content-Type: text/x-rst
+---
+
+
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether +the package features a test suite. It can be set either by the ebuild, +the eclass or the default ``src_test`` implementation, and afterwards +included in the package manager logs. This can aid in analyzing
+the results of automated package testing.
+
+
+Motivation
+==========
+
+The deployment of new Python targets in Gentoo currently involves
+testing of a large number of Gentoo packages against the said target. +This is currently done manually for the most part. It can be +particularly time consuming if multiple individuals repeatedly test
+the same package only to determine that it remains incompatible with
+the new interpreter.
+
+The Python team wanted to explore the use of automation to aid this +testing. Unfortunately, this faces a major problem: for the vast
+of majority of packages, the incompatibilities with new Python versions +do not exhibit during the installation and can only be detected through +running the test suite. The results of automated testing are therefore +only meaningful if the package features a test phase.
+
+For packages using ``distutils-r1`` eclass, the presence of test suite +can usually be easily determined through grepping for +``distutils_enable_tests`` call or an explicit ``python_test()`` +function. Even then, it seems sensible to work towards a more generic +approach to tell whether a package had a test suite or not,
+and therefore whether a particular successful automated testing result +means that the package actually passed tests or only confirmed that
+the Python files were copied successfully.
+
+An explicit indication whether a test suite was present can be presented +by the package manager as part of logs, along with the result of running +the test phase. Afterwards, these logs can be used to determine which +packages were actually tested.
+
+
+Specification
+=============
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
+by a ``src_test()`` implementation to indicate whether the package +featured a test suite. It can take three values:
+
+- ``yes`` indicating that a test suite was run
+- ``indeterminate`` indicating that it was not possible to clearly
+ determine whether the test suite was present or not (this could be
+ a case e.g. when a generic test command is run and it does not
+ indicate whether any tests were found)
+- ``no`` indicating that no test suite was run
+
+This variable *should* be set by eclasses defining the ``src_test()`` +phase. If the package in question is using ``src_test()`` defined
+by an eclass that does not declare it explicitly, the PM must assume +``indeterminate``.
+
+The variable *may* be set by an ebuild defining the ``src_test()``
+phase. If the ebuild does not define it explicitly, the PM must assume +``yes``.
+
+The default ``src_test()`` implementation as defined by the PMS sets
+the value to ``indeterminate`` if it runs a ``check`` or ``test``
+target, and to ``no`` if neither of the targets is found.
+
+
+Rationale
+=========
+
+The use of ternary flag makes it possible to clearly represent all three +possible outcomes while navigating the defaults defined in the GLEP.
+The flag is set in ``src_test()``, so that runtime conditions (such
+as the results obtained from the actual test runner) can be used to +determine the actual value.
+
+The defaults were defined based on the following assumptions:
+
+1. The presence of ``check`` target is common in autotools projects but
+ it does not guarantee that the target actually does anything, let
+ alone run a proper test suite. However, the lack of any test target
+ clearly indicates that no tests were run.
+
+2. Eclass ``src_test`` implementations can be very generic and succeed
+ without actually performing any testing. It is therefore reasonable
+ to default to ``indeterminate`` result when they are used,
+ and recommend them to explicitly override the variable.
+
+3. Explicit ``src_test`` declared in ebuild can generally be assumed
+ to actually run tests, with the exception of declaring the function
+ to prevent ``default_src_test`` from running. It therefore makes
+ sense to default to ``yes`` but allow ebuilds to override the value
+ in the latter case.
+
+
+Backwards Compatibility
+=======================
+
+This GLEP is entirely optional. The package managers not implementing
+it will treat the variable as a regular bash variable set by the test +phase and ignore it.
+
+
+Reference Implementation
+========================
+
+TODO
+
+
+Copyright
+=========
+
+This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
+International License. To view a copy of this license, visit +https://creativecommons.org/licenses/by-sa/4.0/.
--
Have a great day!
~ Maciej XGQT Barć
xgqt@gentoo.org
Gentoo Linux developer
(dotnet, emacs, math, ml, nim, scheme, sci) https://wiki.gentoo.org/wiki/User:Xgqt
9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C
How about using a new token in PROPERTIES instead?
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite. It can be set either by the ebuild,
+the eclass or the default ``src_test`` implementation, and afterwards +included in the package manager logs. This can aid in analyzing
+the results of automated package testing.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 407 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 15:42:00 |
| Calls: | 8,555 |
| Calls today: | 7 |
| Files: | 13,219 |
| Messages: | 5,925,786 |