Hello,

It has been quite some time since I contributed any packaging to Debian, but
in the last year or so I have been trying to put together some packaging for MoinMoin 2. As many of you will be aware, MoinMoin (colloquially referenced as "Moin" hereafter) is currently used by Debian for the Debian Wiki, but this service runs Moin 1.9 which requires Python 2.

After some discussion, I have put together a few packages for Moin 2 and its dependencies that are not currently packaged in Debian. These have been hosted on Salsa and are available here:

https://salsa.debian.org/moin-team/

A few of these packages, named using "old-" as a prefix, are discarded
attempts and only remain because I do not have privileges to delete them. The rest have been packaged using the gbp tooling and associated techniques.

Although I feel that this packaging effort is now fairly well advanced in that I can build packages locally, install them in a chroot, and the installed software appears to work correctly (perhaps with some additional fixes
required related to behaviour that may be inherent in the software itself), I find myself trying to satisfy various packaging requirements. One of these is the ability to run the software's test suite successfully.

Here, it would seem that the most prudent approach is to use the Salsa CI service instead of trying to get the test suite to run during the package
build process. One motivation for doing so involves not having to specify
build dependencies for packages only needed for test suite execution, which itself requires the invocation of gbp buildpackage with --extra-package arguments since some packages are completely new to Debian.

I have also found it difficult to persuade the tests to run successfully
during the build process. A few of these attempt to invoke the moin program, but this cannot be located since it is not installed in the build environment. (There may be other failure situations, but these will be dealt with eventually.) I find the different test features of the build tooling - dh_auto_test, autopkgtest, and so on - all rather confusing and poorly summarised. However, one conclusion is that testing a system, as some of the test cases appear to do, and as opposed to testing library functionality, is not necessarily appropriate when directed by something like dh_auto_test.

For another package I have been working on, the Salsa CI facility has proven
to be usable, configured using files in debian/test, particularly as it allows test-related dependencies to be specified independently. However, this other package has no dependencies that are currently unpackaged in Debian.
Meanwhile, the testing of this new Moin package depends on brand new packages somehow being made available.

So, I would appreciate guidance on how I might enable testing of this new Moin package, along with its new dependencies, in the Salsa environment or any
other appropriate environment that would satisfy Debian packaging needs and policies. It would also be quite helpful if built packages might be published somewhere for people to test, but this is a separate consideration even if
such packages would obviously need to be generated as part of the testing regime.

Thanks for reading this far and for any guidance that might be offered!

Regards,

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

I'm not a Debian developer but I have some experience on Salsa CI, so I
thought that I might be able to help... but then I was confused by a
specific part of the message:

On 17 Aug 2023 at 17:10:08, Paul Boddie wrote:

[...]

For another package I have been working on, the Salsa CI facility has proven to be usable, configured using files in debian/test, particularly as it allows
test-related dependencies to be specified independently. However, this other package has no dependencies that are currently unpackaged in Debian. Meanwhile, the testing of this new Moin package depends on brand new packages somehow being made available.

If this dependencies are available on the "build" step: could they be
made available on the autopkgtest? I didn't quite understand why this is
not possible. I've found the autopkgtest quite flexible (since the tests
are scripts that could prepare some environment)

So, I would appreciate guidance on how I might enable testing of this new Moin
package, along with its new dependencies, in the Salsa environment or any other appropriate environment that would satisfy Debian packaging needs and policies. It would also be quite helpful if built packages might be published somewhere for people to test, but this is a separate consideration even if such packages would obviously need to be generated as part of the testing regime.

For me, the packages on the build job are made available via a Salsa
artifact automatically (and easy to download as a .zip of the *.deb). I
think that this happens on all the "build" jobs on Salsa CI. They can be downloaded as a .zip file
(e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890,
right hand side "Job artifacts" and click "Download"). Would that be
enough?

I also created a repo and hosted it on a Salsa CI page for internal
testing but this is a bit of a workaround. This is in a new job but just download the artifacts (via a Salsa CI dependency) and run
dpkg-scanpackages and copy the files to the right place).

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote:

I'm not a Debian developer but I have some experience on Salsa CI, so I thought that I might be able to help... but then I was confused by a
specific part of the message:

On 17 Aug 2023 at 17:10:08, Paul Boddie wrote:

[...]

For another package I have been working on, the Salsa CI facility has proven to be usable, configured using files in debian/test, particularly
as it allows test-related dependencies to be specified independently. However, this other package has no dependencies that are currently unpackaged in Debian. Meanwhile, the testing of this new Moin package depends on brand new packages somehow being made available.

If this dependencies are available on the "build" step: could they be
made available on the autopkgtest? I didn't quite understand why this is
not possible. I've found the autopkgtest quite flexible (since the tests
are scripts that could prepare some environment)

The package has dependencies on installation but these dependencies are not strictly necessary when building. However, if I wanted to run the test suite when building, I would indeed need to pull in these dependencies as build dependencies so that the software being tested can run without import errors.

Since some of these dependencies are new packages, I had been specifying them using the --extra-package option of gbp buildpackage in order to satisfy such build dependencies, but these were really "test dependencies". I have since decided that I would rather distinguish between build and test dependencies,
as previously noted.

I have to add that the other package I refer to has a test suite that takes a long time to run, so that is another reason why I chose Salsa CI for that package instead of letting autopkgtest do its work:

https://salsa.debian.org/pboddie/shedskin

When I get the courage to do so, I will probably submit an intention-to- package for that package.

[...]

For me, the packages on the build job are made available via a Salsa
artifact automatically (and easy to download as a .zip of the *.deb). I
think that this happens on all the "build" jobs on Salsa CI. They can be downloaded as a .zip file
(e.g. https://salsa.debian.org/freexian-team/debusine/-/jobs/4564890,
right hand side "Job artifacts" and click "Download"). Would that be
enough?

I think the problem is that when running the test suite, the Moin package
would need to obtain the test dependencies specified in debian/test/control. These include various new packages that cannot be obtained from the usual Debian archives, and so some kind of equivalent to gbp's --extra-package would be needed in the Salsa CI environment.

One can imagine having a common storage area holding these newly introduced packages that the CI scripts could access in preference to the usual archives. In fact, this would be something that might also affect existing packages. Consider the situation where fixes to a dependency are required to fix functionality in a particular package. One would have to wait for the fixed dependency to become integrated into the unstable archive before the principal package's tests would start to work again.

I also created a repo and hosted it on a Salsa CI page for internal
testing but this is a bit of a workaround. This is in a new job but just download the artifacts (via a Salsa CI dependency) and run
dpkg-scanpackages and copy the files to the right place).

This sounds like something related to what might be required. In effect, you seem to be doing what I am doing when I actually install my built packages in
a chroot. I run apt-ftparchive (which runs dpkg-scanpackages) to generate Packages, Sources and Release files that tells apt about the new packages when added as another package source.

In the Salsa CI environment, I would need to have the built packages (found in the artefacts for each package's build job) copied somewhere that can then be found by the Moin package's pipeline jobs and the scripts creating a special repository of new packages.

Thanks for replying to my enquiry!

Regards,

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On 18 Aug 2023 at 13:16:15, Paul Boddie wrote:

On Friday, 18 August 2023 09:51:29 CEST Carles Pina i Estany wrote:

I'm not a Debian developer but I have some experience on Salsa CI, so I thought that I might be able to help... but then I was confused by a specific part of the message:

On 17 Aug 2023 at 17:10:08, Paul Boddie wrote:

[...]

For another package I have been working on, the Salsa CI facility has proven to be usable, configured using files in debian/test, particularly as it allows test-related dependencies to be specified independently. However, this other package has no dependencies that are currently unpackaged in Debian. Meanwhile, the testing of this new Moin package depends on brand new packages somehow being made available.

If this dependencies are available on the "build" step: could they be
made available on the autopkgtest? I didn't quite understand why this is not possible. I've found the autopkgtest quite flexible (since the tests are scripts that could prepare some environment)

The package has dependencies on installation but these dependencies
are not strictly necessary when building. However, if I wanted to run

I have the same situation on this side. Two solutions for two places.

Debusine uses https://salsa.debian.org/salsa-ci-team/pipeline approach (extending it)

the test suite when building, I would indeed need to pull in these dependencies as build dependencies so that the software being tested
can run without import errors.

In our repo I have two solutions (or hacks, or workarounds) for this
situation.

AUTOPKGTEST TIME (after build)
==============================
In the autopkgtest job (from salsa-ci/pipeline), autopkgtest runs a
shell script (below integration-tests-generic). Some of them set things
up:

----------
Tests: integration-tests-generic
Depends: debusine-client, debusine-server, debusine-worker, postgresql, redis-server, sudo, nginx
Restrictions: allow-stderr, needs-root
----------

It could pull things from the internet (Pypi, git...) if it also has, in Restrictions: "needs-internet"

Alternatively (and perhaps better, to keep things separated?), you could
use salsa CI variable: SALSA_CI_AUTOPKGTEST_ARGS with this format: SALSA_CI_AUTOPKGTEST_ARGS:
'--setup-commands=ci/pin-django-from-backports.sh

Instead of pinning you could install packages. I don't know if it needs "needs-internet" in Restrictions.

Note that the debusine-* packages are made available automatically
thanks to Salsa CI pipeline and that tests-autopkgtest has:
------
needs:
- job: build
artifacts: true
------

So I don't need to do any repository. The build job just creates the
*.deb packages (in a specific directory?), they are saved as artifacts
and made available to "autopkgtest" job.

UNIT-TESTS (before build)
=========================
We have a job before build doing (I'm simplifying):

-----
unit-tests:
stage: upstream-tests
dependencies: []
script:
- NON_INTERACTIVE=1 bin/quick-setup.sh install_packages
- make coverage # to run the tests with coverage
-----

In "bin/quick-setup.sh" it installs the required packages (some from
Debian, some from Pypi if need to be)

In here we execute tests before packaging them (because we are upstream
as well).

I have to add that the other package I refer to has a test suite that takes a long time to run, so that is another reason why I chose Salsa CI for that package instead of letting autopkgtest do its work:

Note that when I say "autopkgtest" above I always think of autopkgtest
ran by Salsa CI.

One can imagine having a common storage area holding these newly

Until now I see "the whole internet" as a "common storage area" (I know,
not ideal! could be closer to the process, more robust, faster, etc.)

introduced packages that the CI scripts could access in preference to
the usual archives. In fact, this would be something that might also
affect existing packages. Consider the situation where fixes to a
dependency are required to fix functionality in a particular package.
One would have to wait for the fixed dependency to become integrated
into the unstable archive before the principal package's tests would
start to work again.

Semi-offtopic: MMmmm... Debusine is the project that I mentioned before
where I implemented different things in .gitlab-ci.yml.

What you mentioned here is a thing that debusine, in the future (not impelmented yet) might help. But other people can talk better about this
than me... and it's not available right now.

I also created a repo and hosted it on a Salsa CI page for internal
testing but this is a bit of a workaround. This is in a new job but just download the artifacts (via a Salsa CI dependency) and run dpkg-scanpackages and copy the files to the right place).

This sounds like something related to what might be required. In effect, you seem to be doing what I am doing when I actually install my built packages in a chroot. I run apt-ftparchive (which runs dpkg-scanpackages) to generate Packages, Sources and Release files that tells apt about the new packages when
added as another package source.

Making the packages available on GitHub Pages is a workaround to make
them available to end users via a repository. I'm happy to discuss but
it should not be needed for the tests (or I didn't need it for the
tests).

For the jobs it is happening via https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-apt-repository

In the Salsa CI environment, I would need to have the built packages (found in
the artefacts for each package's build job) copied somewhere that can then be found by the Moin package's pipeline jobs and the scripts creating a special repository of new packages.

Archiving artifacts should happen automatically on the "build" step of
Salsa CI (salsa-ci/pipeline). If I understand correctly what you
wanted...

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote:

Here, it would seem that the most prudent approach is to use the Salsa CI service instead of trying to get the test suite to run during the package build process.

You should do both if possible, assuming that by "Salsa CI service" you
mean autopkgtests which you can, and IMO should, also run locally.

One motivation for doing so involves not having to specify
build dependencies for packages only needed for test suite execution, which itself requires the invocation of gbp buildpackage with --extra-package arguments since some packages are completely new to Debian.

This is fine. Not to mention that the same problem exists for
autopkgtests, as you say below.

I have also found it difficult to persuade the tests to run successfully during the build process. A few of these attempt to invoke the moin program, but this cannot be located since it is not installed in the build environment.

This should also be fine, unless it's completely impossible to run it
without installing into /.

However, one conclusion is that testing a system, as some of the
test cases appear to do, and as opposed to testing library functionality, is not necessarily appropriate when directed by something like dh_auto_test.

If there are tests that can't be run at build time you can skip those. You
can even ask the upstream to provide tool arguments to simplify that.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote:

For the jobs it is happening via https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-

apt-repository

Reviewing this documentation is actually more helpful than I thought it would be. I had noticed the "aptly" task in the YAML files, and I had started to wonder if that meant that some kind of repository publishing was occurring somewhere.

In the Salsa CI environment, I would need to have the built packages
(found in the artefacts for each package's build job) copied somewhere
that can then be found by the Moin package's pipeline jobs and the
scripts creating a special repository of new packages.

Archiving artifacts should happen automatically on the "build" step of
Salsa CI (salsa-ci/pipeline). If I understand correctly what you
wanted...

I think you have a good understanding of what I am trying to achieve. If I can get the new package dependencies (emeraldtree, feedgen, and so on) to yield installable packages when built that can then be referenced by Salsa CI as it runs the build jobs for Moin, I have a chance of running the test suite.

I'm currently persuading the CI system to run the "aptly" task and to publish package repositories. I will then augment a customised YAML file for the Moin package with references to these repositories and see if the test suite can be invoked somewhat more successfully as a consequence.

Thanks once again for the guidance!

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Friday, 18 August 2023 18:21:00 CEST Andrey Rakhmatullin wrote:

On Thu, Aug 17, 2023 at 05:10:08PM +0200, Paul Boddie wrote:

Here, it would seem that the most prudent approach is to use the Salsa CI service instead of trying to get the test suite to run during the package build process.

You should do both if possible, assuming that by "Salsa CI service" you
mean autopkgtests which you can, and IMO should, also run locally.

I'm not really clear on what autopkgtest really is, other than a tool that
uses some kind of test suite description that may reside in debian/test. I'm also not completely clear on what is supposed to invoke it, other than either the Salsa CI mechanism or dh_auto_test. In the Debian Wiki documentation...

https://wiki.debian.org/Python/LibraryStyleGuide

...it mentions a field in debian/control:

Testsuite: autopkgtest-pkg-python

My impression is that this calls autodep8 to generate some kind of test suite description which is then invoked by dh_auto_test. It doesn't help that there is an alternative to this that resembles it but behaves differently:

Testsuite: autopkgtest-pkg-pybuild

What I have previously managed to do with the Salsa CI mechanism is to run tests specified in debian/test, disabling these tests at build time using the following in debian/rules:

export PYBUILD_DISABLE=test

For another package which has limited dependencies, I do this because the
tests take a long time to run, and I even need to increase the timeout for the Salsa CI jobs. For this package, the tests don't take too long to run (those that do run), but some tests fail for reasons previously described, and this
is due to some configuration issue that is unpleasant to debug.

Success with testing using Salsa CI steers me away from using the Testsuite field in the normal build environment because the resulting test suite invocation failures suggest that some customisation is needed or that the environment is inappropriate in some way (even if I specify numerous build dependencies).

One motivation for doing so involves not having to specify
build dependencies for packages only needed for test suite execution,
which itself requires the invocation of gbp buildpackage with --extra-package arguments since some packages are completely new to
Debian.

This is fine. Not to mention that the same problem exists for
autopkgtests, as you say below.

I have also found it difficult to persuade the tests to run successfully during the build process. A few of these attempt to invoke the moin program, but this cannot be located since it is not installed in the
build environment.

This should also be fine, unless it's completely impossible to run it
without installing into /.

The moin program is made available in setup.py using an entry point. Maybe if there were some kind of script instead, it would work.

However, one conclusion is that testing a system, as some of the
test cases appear to do, and as opposed to testing library functionality, is not necessarily appropriate when directed by something like dh_auto_test.

If there are tests that can't be run at build time you can skip those. You can even ask the upstream to provide tool arguments to simplify that.

I may well discuss such matters with them. One challenge that is relevant in this situation is that upstream have been working in their own virtualenv (or venv, or whatever it is now) world for a few years, using plenty of dependencies that were not packaged in Debian. That hasn't been entirely conducive to adoption of Moin 2 which has in turn isolated its developers further.

Fortunately, the gulf between that world and Debian can be bridged, and some
of the more tedious packaging seems to have been done for many of the dependencies involved. It just needs someone to shepherd this effort to an acceptable conclusion, which is of concern to Debian itself since Moin runs
the Debian Wiki.

Thanks for the encouragement and advice.

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, Aug 18, 2023 at 07:19:18PM +0200, Paul Boddie wrote:

Here, it would seem that the most prudent approach is to use the Salsa CI service instead of trying to get the test suite to run during the package build process.

You should do both if possible, assuming that by "Salsa CI service" you mean autopkgtests which you can, and IMO should, also run locally.

I'm not really clear on what autopkgtest really is, other than a tool that uses some kind of test suite description that may reside in debian/test. I'm also not completely clear on what is supposed to invoke it, other than either the Salsa CI mechanism or dh_auto_test.

The maintainer is supposed to invoke it before uploading the package, and
the Debian infra invokes it on all packages that are uploaded.
Notably, dh_auto_test is unrelated.

In the Debian Wiki documentation...

https://wiki.debian.org/Python/LibraryStyleGuide

...it mentions a field in debian/control:

Testsuite: autopkgtest-pkg-python

My impression is that this calls autodep8 to generate some kind of test suite

Yes.
Though note that it generates a trivial test that only checks a top-level import (just like the wiki page says).

description which is then invoked by dh_auto_test.

It's not invoked by dh_auto_test. autopkgtests are not a part of the
package build process and they operate on built binary packages.

It doesn't help that there
is an alternative to this that resembles it but behaves differently:

Testsuite: autopkgtest-pkg-pybuild

It just generates a different (better) test.

I have also found it difficult to persuade the tests to run successfully during the build process. A few of these attempt to invoke the moin program, but this cannot be located since it is not installed in the build environment.

This should also be fine, unless it's completely impossible to run it without installing into /.

The moin program is made available in setup.py using an entry point. Maybe if there were some kind of script instead, it would work.

AFAIK there should be ways to work with this.
Ideally, of course, the upstream test suite should support this, but I understand that not all upstreams use common practices.

However, one conclusion is that testing a system, as some of the
test cases appear to do, and as opposed to testing library functionality, is not necessarily appropriate when directed by something like dh_auto_test.

If there are tests that can't be run at build time you can skip those. You can even ask the upstream to provide tool arguments to simplify that.

I may well discuss such matters with them. One challenge that is relevant in this situation is that upstream have been working in their own virtualenv (or venv, or whatever it is now) world for a few years, using plenty of dependencies that were not packaged in Debian.

Which is by itself not a problem from the technical side, as you could
just package those (I understand that this requires effort and may have
other problems, but by itself it's normal).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On 18 Aug 2023 at 19:03:48, Paul Boddie wrote:

On Friday, 18 August 2023 16:12:19 CEST Carles Pina i Estany wrote:

For the jobs it is happening via https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-

apt-repository

Reviewing this documentation is actually more helpful than I thought it would be. I had noticed the "aptly" task in the YAML files, and I had started to wonder if that meant that some kind of repository publishing was occurring somewhere.

Ha! I wasn't aware of the aptly option (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-apt-repository
and SALSA_CI_DISABLE_APTLY=0).
I think that I might have re-invented the wheel in a tiny part of
Debusine CI/CD.

I will point out at some things that might safe some time to you
(great) or not (ignore! :-) ):

debusine's .gitlab-ci.yml: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml

Latest CI build: https://salsa.debian.org/freexian-team/debusine/-/pipelines/566796

In the "build" job: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574451

You can see what's doing and it's using salsa-ci/pipeline's defaults in
the build. All comes from
https://salsa.debian.org/salsa-ci-team/pipeline/ , specifically from: https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/salsa-ci.yml#L201

(I sometimes grep salsa-ci-team/pipeline for commands that I see in
debusine's CI output to understand the flow...)

In the autopkgtest job it sets things up and invokes autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458

The *.deb from build are made available via "artifacts" (and that
autopkgtest needs the artifact of build).

The job autopkgtest, via debci command, runs autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65

And, to be honest, I didn't look into these details for long time. From
my point of view I just write debian/tests/control with the tests
(commands) to run: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/debian/tests/control

And that things run in a container/chroot (depends), etc.

You can invoke autopkgtest locally as well...

In the Salsa CI environment, I would need to have the built packages (found in the artefacts for each package's build job) copied somewhere that can then be found by the Moin package's pipeline jobs and the scripts creating a special repository of new packages.

Archiving artifacts should happen automatically on the "build" step of Salsa CI (salsa-ci/pipeline). If I understand correctly what you
wanted...

I think you have a good understanding of what I am trying to achieve. If I can
get the new package dependencies (emeraldtree, feedgen, and so on) to yield installable packages when built that can then be referenced by Salsa CI as it runs the build jobs for Moin, I have a chance of running the test suite.

yep!

I'm currently persuading the CI system to run the "aptly" task and to publish package repositories. I will then augment a customised YAML file for the Moin

I might look more about this on Monday (I will write it down now)
because I am using "dpkg-scanpackages" and I might be able to avoid it
via SALSA_CI_DISABLE_APTLY: 0

I see the salsa-CI like a fancy environment where to run scripts,
generate files, pass files between scripts, refactor scripts (in YAML in
my .gitlab-ci.yml or to upstream salsa-ci/pipeline), scheduled jobs,
etc. so almost anything that you can think of you can do it. Either with artifacts to keep and retrieve files between "jobs" or via GitHub Pages.

If you need help creating a new job (a new bubble in the pipeline) let
me know (even if you want off-list, if this is getting off-topic of debian-python). Or look at the https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml where were created the stage "upstream-tests" with things like
"code-linting". So You could even create dependencies of your task
before the tests are executed and pass the .deb files via artifacts (or
even repositories).

package with references to these repositories and see if the test
suite can be invoked somewhat more successfully as a consequence.

It's possible for sure. Other people in this list might come up with a different idea. I don't have almost any experience with Debian
packaging, but I have some experience on the salsa CI. So I might be
giving you solutions that might be sub-optimal! :-)

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote:

Ha! I wasn't aware of the aptly option (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> apt-repository and SALSA_CI_DISABLE_APTLY=0).
I think that I might have re-invented the wheel in a tiny part of
Debusine CI/CD.

It is certainly a way of propagating packages to those that might need them. However, the instructions indicating how a package might access these dependencies appear to be deficient.

It does not appear to be sufficient to merely specify the dependency package repositories and mark them as trusted. Doing that will just cause the repositories to be regarded as ignored and the GPG keys signalled as unrecognised.

So, the GPG keys need to be obtained. This is a hassle because something like wget is needed to do that, and then apt has to be persuaded not to fail in an opaque way. So the modified recipe is something like this:

before_script:
- apt-get update
- NON_INTERACTIVE=1 apt-get install -y wget
- echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list
...
- wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ public-key.asc
...
- apt-get update

This seems to make the various jobs happy, but the one that I was most concerned with remains unhappy! I don't actually need the dependencies for anything other than autopkgtest, but that job employs its own environment
where the above recipe has no effect.

So, what I now need to do is to find out how I can make the new packages available to autopkgtest specifically.

I will point out at some things that might safe some time to you
(great) or not (ignore! :-) ):

debusine's .gitlab-ci.yml: https://salsa.debian.org/freexian-team/debusine/-/blob/devel/.gitlab-ci.yml

This is definitely useful for examples of the CI definition syntax and how hidden jobs can be used to provide additional script fragments.

[...]

The job autopkgtest, via debci command, runs autopkgtest: https://salsa.debian.org/freexian-team/debusine/-/jobs/4574458#L65

The difficult thing appears to be the configuration of this testing environment. One approach might involve overriding the existing job
definitions to incorporate the injection of new packages.

[...]

It's possible for sure. Other people in this list might come up with a different idea. I don't have almost any experience with Debian
packaging, but I have some experience on the salsa CI. So I might be
giving you solutions that might be sub-optimal! :-)

I think that you are giving me some good ideas at least. I do feel that I am encountering problems that must have been encountered many times before, but I could imagine that people do not use these tools in this way or have different expectations of what they are able to accomplish.

As for Debian's tooling, I consider it rather chaotic. When I previously did packaging, things like pbuilder were still in use, being slowly pushed aside
by sbuild and probably other things that are now forgotten or obsolete. There always seems to be new things coming along, new mixtures of technologies, and the potential for more confusion and uncertainty.

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quick answer for now...

On 19 Aug 2023 at 19:04:09, Paul Boddie wrote:

On Friday, 18 August 2023 19:54:55 CEST Carles Pina i Estany wrote:

Ha! I wasn't aware of the aptly option (https://salsa.debian.org/salsa-ci-team/pipeline/#using-automatically-built-> apt-repository and SALSA_CI_DISABLE_APTLY=0).
I think that I might have re-invented the wheel in a tiny part of
Debusine CI/CD.

It is certainly a way of propagating packages to those that might need
them. However, the instructions indicating how a package might access
these dependencies appear to be deficient.

I haven't used aptly on salsa but on Monday I want to give it a try, to
see if I can replace my solution...

It does not appear to be sufficient to merely specify the dependency
package repositories and mark them as trusted. Doing that will just
cause the repositories to be regarded as ignored and the GPG keys
signalled as unrecognised.

So, the GPG keys need to be obtained. This is a hassle because something like wget is needed to do that, and then apt has to be persuaded not to fail in an opaque way. So the modified recipe is something like this:

before_script:
- apt-get update
- NON_INTERACTIVE=1 apt-get install -y wget
- echo "deb https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/ artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list
...
- wget -q -O /etc/apt/trusted.gpg.d/emeraldtree.asc https:// salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly/ public-key.asc

I'm ensure that this is correct. I have:
-----
deb [arch=amd64 signed-by=/usr/share/keyrings/oracle-virtualbox-2016.gpg] https://download.virtualbox.org/virtualbox/debian bookworm contrib
-----

So I thought that the "signed-by" was needed; and that the files that I
have there are binary files (not .asc) of this type:
carles@pinux:~$ file /etc/apt/trusted.gpg.d/skype.gpg /etc/apt/trusted.gpg.d/skype.gpg: OpenPGP Public Key Version 4, Created Wed Jun 22 09:36:35 2016, RSA (Encrypt or Sign, 2048 bits); User ID; Signature; OpenPGP Certificate
carles@pinux:~$

I don't know if a .asc files are allowed. Actually, from my
.bash_history:
---
wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg
---

(I was following instructions, I didn't try leaving the .asc file there)

...
- apt-get update

This seems to make the various jobs happy, but the one that I was most concerned with remains unhappy! I don't actually need the dependencies
for anything other than autopkgtest, but that job employs its own
environment where the above recipe has no effect.

You could try using:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh'
---
(and write and ship the ci/setup-the-repo.sh in the repo, do whatever
you need there)

Or, probably even better but less flexible:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE COMPONENT'
---
(I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS.
I haven't used add-apt-source, I don't know wha

On Saturday, 19 August 2023 20:32:59 CEST Carles Pina i Estany wrote:

Quick answer for now...

And a very quick reply from me... :-)

[...]

I don't know if a .asc files are allowed. Actually, from my
.bash_history:
---
wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg ---

(I was following instructions, I didn't try leaving the .asc file there)

They are allowed in recent versions of apt, as confirmed by the man page.

[...]

You could try using:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh'
---
(and write and ship the ci/setup-the-repo.sh in the repo, do whatever
you need there)

This could be very useful.

Or, probably even better but less flexible:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE COMPONENT' ---
(I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS.
I haven't used add-apt-source, I don't know what happens with the gpg
keys... but you could use [trusted=yes] :-| )

For the MIRROR you have salsa variables that might help if you need to specify the pipeline.

So could this.

So, what I now need to do is to find out how I can make the new packages available to autopkgtest specifically.

We will get there! (one way or another)

Well, I have engineered something very inelegant, revealed in full here:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml

It defines a package list file and uses wget to obtain each repository's
public key. Fortunately, wget works at the "script" level of the jobs
involved, whereas it didn't work at what must have been the global "before_script" level (suggested in the documentation) and needed to be installed first.

To make autopkgtest and piuparts find the packages, the repository details are then copied into the appropriate location in each case. However, since such customisation is not readily incorporated into the existing job descriptions,
I have had to duplicate the existing job description code and insert
additional statements in the script sections.

As a consequence of these changes, the extra packages are found and installed within the environments concerned. This is enough to make the piuparts job succeed, but autopkgtest still fails. Fortunately, that may be due only to problems with the actual tests: I think the upstream maintainers may have made some errors.

[...]

hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in salsa-ci/pipeline last year because I was trying to do a similar thing
to what you are doing (I had to pin a package from backports).

(I'm just a user of salsa-ci/pipeline, not a member of the team neither
I can speak for them!)

If this can simplify what I've done, which is really quite horrible, then I will adopt it instead. The way that the artefacts of the dependencies are
bound up in specifically numbered jobs is also particularly unfortunate. Of course, if the packages enter the Debian archives, all of this extra effort will be made largely redundant, anyway, even if coordinated testing of new package versions would still benefit from such effort.

[...]

I don't know if an FAQ, conventions, best practises or something might help...

I think that the documentation could be reworked, but that is largely true for a lot of the materials available for Debian. And since some of that documentation resides on the Debian Wiki, that is another reason why I am trying to move this particular packaging exercise forwards.

Thanks once again!

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On 20 Aug 2023 at 00:49:20, Paul Boddie wrote:

On Saturday, 19 August 2023 20:32:59 CEST Carles Pina i Estany wrote:

Quick answer for now...

And a very quick reply from me... :-)

[...]

I don't know if a .asc files are allowed. Actually, from my
.bash_history:
---
wget -O- https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg
--dearmor --yes --output /usr/share/keyrings/oracle-virtualbox-2016.gpg ---

(I was following instructions, I didn't try leaving the .asc file there)

They are allowed in recent versions of apt, as confirmed by the man page.

[...]

Cool! I'll try next time (I cannot see it on my man apt-get, apt or sources.list on bookworm but I might need a newer apt)

You could try using:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/setup-the-repo.sh'
---
(and write and ship the ci/setup-the-repo.sh in the repo, do whatever
you need there)

This could be very useful.

Also available (and also added last year for a similar case):
-----
variables:
SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'ci/pin-django-from-backports.sh'
-----

(and SALSA_CI_PIUPARTS_POST_INSTALL_SCRIPT)

Or, probably even better but less flexible:
---
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--add-apt-source="deb http://MIRROR SUITE COMPONENT' ---
(I found add-apt-source via "man autopkgtest" in TEST BED SETUP OPTIONS.
I haven't used add-apt-source, I don't know what happens with the gpg keys... but you could use [trusted=yes] :-| )

For the MIRROR you have salsa variables that might help if you need to specify the pipeline.

So could this.

So, what I now need to do is to find out how I can make the new packages available to autopkgtest specifically.

We will get there! (one way or another)

Well, I have engineered something very inelegant, revealed in full here:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml

I, more or less, see what you did!

Interesting approach, but if you could use the variables and ship the
shell script it might reduce some duplication between jobs (if possible,
I haven't look into too much detail in your case) and more importantly
you might be able to use the standard "autopkgtest" or "piuparts"
(instead of redefining them).

Just last week I created a new autopkgtest (I have it running on
bookworm and bullseye). It's done this way:
------
autopkgtest-bullseye:
extends: .test-autopkgtest
variables:
RELEASE: "bullseye-backports"
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh --skip-test=debusine-doc-linkcheck'
------

Or, for more context: https://salsa.debian.org/freexian-team/debusine/-/blob/add-bullseye-support/.gitlab-ci.yml#L84
It ends up having "autopkgtest" and "autopkgtest-bullseye". But using
the variable SALSA_CI_DISABLE_AUTOPKGTEST I could disable the one
without my extends.

[...]

hopefully SALSA_CI_AUTOPKGTEST_ARGS will help you. I added this in salsa-ci/pipeline last year because I was trying to do a similar thing
to what you are doing (I had to pin a package from backports).

(I'm just a user of salsa-ci/pipeline, not a member of the team neither
I can speak for them!)

If this can simplify what I've done, which is really quite horrible, then I will adopt it instead. The way that the artefacts of the dependencies are bound up in specifically numbered jobs is also particularly unfortunate. Of

If it's in the same pipeline: you don't need the numbers to get the
artifacts (the .deb files from build).

For the aptly generated artifact: I will investigate this tomorrow (to
fetch it and try to use) (I want to replace some code that I did with
the aptly repo, if possible).

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sunday, 20 August 2023 14:06:37 CEST Carles Pina i Estany wrote:

Interesting approach, but if you could use the variables and ship the
shell script it might reduce some duplication between jobs (if possible,
I haven't look into too much detail in your case) and more importantly
you might be able to use the standard "autopkgtest" or "piuparts"
(instead of redefining them).

I have continued along such a path.

Just last week I created a new autopkgtest (I have it running on
bookworm and bullseye). It's done this way:
------
autopkgtest-bullseye:
extends: .test-autopkgtest
variables:
RELEASE: "bullseye-backports"
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh --skip-test=debusine-doc-linkcheck' ------

Or, for more context: https://salsa.debian.org/freexian-team/debusine/-/blob/add-bullseye-support/ .gitlab-ci.yml#L84 It ends up having "autopkgtest" and "autopkgtest-bullseye". But using the variable SALSA_CI_DISABLE_AUTOPKGTEST
I could disable the one without my extends.

Previously, I had merely redefined the autopkgtest and piuparts jobs,
replacing them with ones that use the customised code:

autopkgtest:
extends: .test-autopkgtest-revised

piuparts:
extends: .test-piuparts-revised

I had assumed that such redefinition was possible. When people decide to introduce a new language, as GitLab appear to have done, they should realise that they need to match the considerable levels of documentation that existing languages may have. Fortunately, my assumptions were correct.

Following your lead, I have since found that the following works:

autopkgtest:
extends: .test-autopkgtest
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=debian/salsa/add- repositories.sh'

piuparts:
extends: .test-piuparts
variables:
SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'debian/salsa/add-repositories.sh'

You can see that by defining the variables to customise the tools, I am able
to work with the existing job definitions. This simplifies the CI description file considerably:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml

The script is pretty straightforward, too:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa/add-repositories.sh

It is so mundane that I could imagine it being replaced by standard functionality very easily.

[Job numbers]

If it's in the same pipeline: you don't need the numbers to get the
artifacts (the .deb files from build).

The .deb files are produced by the pipelines associated with other package repositories. Thus, Aptly is used to make the .deb files available to the Moin package's pipeline. I imagine that I could build the dependencies in the same pipeline, however. Apparently, GitLab's non-free editions support "multi- project" pipelines, but I suspect that Aptly has been introduced to get around the lack of such functionality in the free edition.

For the aptly generated artifact: I will investigate this tomorrow (to
fetch it and try to use) (I want to replace some code that I did with
the aptly repo, if possible).

Good luck with that task! :-)

As a result of this discussion and a lot of effort, I have managed to get autopkgtest and piuparts to see and to use the package dependencies needed by the Moin package. This allowed me to tackle the test suite that had caused me to look at all of this in the beginning. And now, it seems that I can get all the reasonable tests to pass:

https://salsa.debian.org/moin-team/moin/-/pipelines/567871

Once again, many thanks for all the guidance!

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On 20 Aug 2023 at 23:16:57, Paul Boddie wrote:

On Sunday, 20 August 2023 14:06:37 CEST Carles Pina i Estany wrote:

[...]

Thanks for sharing your path here, and I'm happy that you are reaching
your destination :-)

autopkgtest:
extends: .test-autopkgtest
variables:
SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=debian/salsa/add- repositories.sh'

piuparts:
extends: .test-piuparts
variables:
SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'debian/salsa/add-repositories.sh'

You can see that by defining the variables to customise the tools, I am able to work with the existing job definitions. This simplifies the CI description file considerably:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa-ci.yml

The script is pretty straightforward, too:

https://salsa.debian.org/moin-team/moin/-/blob/debian/master/debian/salsa/add-repositories.sh

If you want, you can simplify more (it's not exactly the same, so it
might or might not help). There is a way on GitLab to point to the
latest build of a job. For example, you have the following URL for one
of the git repos: https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly

You could use instead (to avoid the pipeline number): https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/master/raw/aptly?job=aptly

Which is a redirect to the latest pipeline. Currently:
----
$ curl -s -I "https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/master/raw/aptly?job=aptly" | grep -E -i "^(http|location)"
HTTP/2 302
location: https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/aptly
----

Follows this format:
BRANCH=debian/master
DIRECTORY=aptly
JOB_NAME=aptly https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/${BRANCH}/raw/${DIRECTORY}?job=${JOB_NAME}

Just a side note: be careful about expiring artifacts. In some projects (settings dependant) only the latest artifact is kept and older ones
might be expired (deleted) after some time. I don't think that this is
the case of the moin-team/emeraldtree after a quick check... but I'm
unsure where this is properly checked on GitLab.

[...]

Cheers,

--
Carles Pina i Estany
https://carles.pina.cat || Wiktionary translations: https://kamus.pina.cat

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Monday, 21 August 2023 15:01:26 CEST Carles Pina i Estany wrote:

If you want, you can simplify more (it's not exactly the same, so it
might or might not help). There is a way on GitLab to point to the
latest build of a job. For example, you have the following URL for one
of the git repos:

https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/

aptly

You could use instead (to avoid the pipeline number): https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/

master/raw/aptly?job=aptly

That is a good idea, since I always intend the latest job to be successful.
:-)

Which is a redirect to the latest pipeline. Currently:
----
$ curl -s -I "https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/debian/mas ter/raw/aptly?job=aptly" | grep -E -i "^(http|location)" HTTP/2 302
location: https://salsa.debian.org/moin-team/emeraldtree/-/jobs/4575438/artifacts/raw/

aptly

----

Follows this format:
BRANCH=debian/master
DIRECTORY=aptly
JOB_NAME=aptly https://salsa.debian.org/moin-team/emeraldtree/-/jobs/artifacts/${BRANCH}/ra w/${DIRECTORY}?job=${JOB_NAME}

Thanks for demonstrating this. I imagine that it is documented somewhere in
the GitLab manuals, too. One note of caution, though. It seems that using
these redirected URLs is not necessarily possible with apt, and so it might be advisible to obtain the redirected URL as shown in the output above.

In my script I now get curl (and not wget any more) to obtain the final URL to provide to the apt configuration, and I also use this job-specific URL to obtain each GPG key. I suppose I could change the apt configuration to allow redirects instead.

Just a side note: be careful about expiring artifacts. In some projects (settings dependant) only the latest artifact is kept and older ones
might be expired (deleted) after some time. I don't think that this is
the case of the moin-team/emeraldtree after a quick check... but I'm
unsure where this is properly checked on GitLab.

"CI/CD Settings", "Artifacts", "Keep artifacts from most recent successful jobs" is set, so I imagine that as long as the latest job is successful, everything will still work. Indeed, it is desirable to rely on the latest job under such conditions: one would not want a newer dependency build to break
the Moin package because an older set of artefacts has been deleted.

Thanks once again for the feedback and advice!

Paul

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)