Hey Rapha�l!

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

----- Forwarded message from cdimage creator role <debian-cd@casulana.debian.org> -----

From: cdimage creator role <debian-cd@casulana.debian.org>
To: steve@einval.com
Date: Mon, 08 Aug 2022 18:29:33 +0000
Subject: testingcds 3sidamd64 (amd64 dvd) has failed; log included
Message-Id: <E1oL7VN-002UJb-5v@casulana.debian.org>

Error when trying to create cds for 3sidamd64
Mon Aug 8 03:20:26 UTC 2022: Using CONF from /home/debian-cd/build.bookworm/CONF.sh.
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/CD[1234567890]*
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/tasks
rm -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/*.filelist* rm -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/packages-stamp /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/upgrade-stamp /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/checksum-check
echo "Cleaning the build directory"
Cleaning the build directory
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/apt
rm -rf /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64
make[1]: Leaving directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
make[1]: Entering directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm
mkdir -p /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/apt

...

Generating the complete list of packages to be included in /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list...
perl -ne 'chomp; next if /^\s*$/; \
print "$_\n" if not $seen{$_}; $seen{$_}++;' \
/srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/rawlist \
> /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list /home/debian-cd/build.bookworm/debian-cd/tools/check_backports_packages /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports
if [ -f /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports ]; then mv /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list.backports /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list; fi
for ARCH in amd64; do \
ARCH=$ARCH /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps /srv/cdbuilder.debian.org/src/deb-cd/tmp/3sidamd64/bookworm/list; \
done
Running sort_deps to sort packages for amd64:
Generating dependency tree with apt-cache depends...
Adding standard, required, important and base packages first
S/R/I/B packages take 124138086 bytes
Adding the rest of the requested packages
Deep recursion on subroutine "main::add_missing" at /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps line 1139, <LIST> line 52916.
Deep recursion on subroutine "main::add_missing" at /home/debian-cd/build.bookworm/debian-cd/tools/sort_deps line 1142, <LIST> line 52916.
Killed
make[1]: *** [Makefile:484: image-trees] Error 137
make[1]: Leaving directory '/srv/cdbuilder.debian.org/git/setup/bookworm/debian-cd'


----- End forwarded message -----
--
Steve McIntyre, Cambridge, UK. steve@einval.com < Aardvark> I dislike C++ to start with. C++11 just seems to be
handing rope-creating factories for users to hang multiple
instances of themselves.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Aug 08, 2022 at 07:56:49PM +0100, Steve McIntyre wrote:

Hey Rapha�l!

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Checking on the build machine just now, I could see that a number of
sort_deps processes were chewing up ~40G of memory each. I'm going to
revert your commit for now and re-trigger the weekly build.

--
Steve McIntyre, Cambridge, UK. steve@einval.com "I used to be the first kid on the block wanting a cranial implant,
now I want to be the first with a cranial firewall. " -- Charlie Stross

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Seems updated images are there now, thanks Sledge :)

On 8/8/22, Steve McIntyre <steve@einval.com> wrote:

On Mon, Aug 08, 2022 at 07:56:49PM +0100, Steve McIntyre wrote:

Hey Raphaėl!

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Checking on the build machine just now, I could see that a number of sort_deps processes were chewing up ~40G of memory each. I'm going to
revert your commit for now and re-trigger the weekly build.

--
Steve McIntyre, Cambridge, UK.
steve@einval.com
"I used to be the first kid on the block wanting a cranial implant,
now I want to be the first with a cranial firewall. " -- Charlie Stross

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Aug 08, 2022 at 07:56:49PM +0100, Steve McIntyre wrote:

Hey Rapha�l!

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Sorry, I meant 8513b237afbdb59a4a59d1bc707d37f0aa9138f7 of course.

--
Steve McIntyre, Cambridge, UK. steve@einval.com "I can't ever sleep on planes ... call it irrational if you like, but I'm
afraid I'll miss my stop" -- Vivek Das Mohapatra

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Steve,

On lun., 08 août 2022, Steve McIntyre wrote:

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.

Can you point me to the debian-cd configuration that I shall use to reproduce the issue?

Cheers,
--
Raphaël Hertzog ◈ Offensive Security ◈ Kali Linux Developer

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey Raphael

On Tue, Aug 09, 2022 at 09:00:59AM +0200, Raphael Hertzog wrote:

On lun., 08 ao�t 2022, Steve McIntyre wrote:

This weekly DVD run just tooke ~15h to run compared to the normal ~30m
or so. Checking the stuff in the log here, I'm thinking that this
"Deep recursion on subroutine" message is very likely due to your
changes in commit cc4e1fa450a074ef1428bf678a57cdf8b0d0f0e5. Could you
take a look please?

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.

Can you point me to the debian-cd configuration that I shall use to reproduce >the issue?

It's the weekly config that builds all the full media sets. The DVD
build on its own will show it:

#### B_amd64_dvd config start: ####
#### amd64.cfg start ####
export ARCH=amd64
export VARIANTS=xen
export MAXISOS=1
export MAXJIGDOS=ALL
export MAX_PKG_SIZE=999999999999
export DESKTOP=all
export INSTALLER_CD=3
export KERNEL_PARAMS="desktop=all"
export TASK=Debian-all
export DRYRUN_WAIT=0.5
export FORCE_CD_SIZE1=STICK4GB;;
#### dvd.cfg end ####
#### B_amd64_dvd config end: ####

--
Steve McIntyre, Cambridge, UK. steve@einval.com "... the premise [is] that privacy is about hiding a wrong. It's not.
Privacy is an inherent human right, and a requirement for maintaining
the human condition with dignity and respect."
-- Bruce Schneier

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello,

On mar., 09 août 2022, Raphael Hertzog wrote:

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.

Can you point me to the debian-cd configuration that I shall use to reproduce the issue?

So the issue was not trivial to reproduce. FWIW, it's only reproducible in bullseye, for some reason buster was coping fine with it.

But I got it reproduced and the problem was due to packages which have
strong "Recommends" like libapreq2-dev which has "Recommends:
libapreq2-doc (= 2.13-7+b3)" which is not satisfiable. It would never stop
its recursion because the version found was not good for it.

So in the end I have a new patch in the hertzog/bug601203 branch. I'm
doing some further tests but it seems to solve the issue: https://salsa.debian.org/images-team/debian-cd/-/commit/ca9ac8deac5c1436f4b311a22a34a56f236dfe05

But as I investigated I found more things to fix, like quite
some dead code: https://salsa.debian.org/images-team/debian-cd/-/commit/e77ade6033445571092e8663ad94a24c7e882b03

I'm going to push this soon but I would love if someone else could
do a test run and ensure it doesn't break anything else.

I have created a merge request to make it easier to review the code if
anyone wants to do it: https://salsa.debian.org/images-team/debian-cd/-/merge_requests/26

Cheers,
--
Raphaël Hertzog ◈ Offensive Security ◈ Kali Linux Developer

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey Raphael!

On Tue, Aug 09, 2022 at 07:55:21PM +0200, Raphael Hertzog wrote:

Hello,

On mar., 09 août 2022, Raphael Hertzog wrote:

Ah, sure, that seems quite likely. I did not have the issue in my
tests, but I tested with a partial mirror as created by simple-cdd so I
quite likely did not hit the problematic recursive dependency.

Can you point me to the debian-cd configuration that I shall use to reproduce
the issue?

So the issue was not trivial to reproduce. FWIW, it's only reproducible in >bullseye, for some reason buster was coping fine with it.

Hmmm, OK.

But I got it reproduced and the problem was due to packages which have
strong "Recommends" like libapreq2-dev which has "Recommends:
libapreq2-doc (= 2.13-7+b3)" which is not satisfiable. It would never stop >its recursion because the version found was not good for it.

Argh.

So in the end I have a new patch in the hertzog/bug601203 branch. I'm
doing some further tests but it seems to solve the issue: >https://salsa.debian.org/images-team/debian-cd/-/commit/ca9ac8deac5c1436f4b311a22a34a56f236dfe05

But as I investigated I found more things to fix, like quite
some dead code: >https://salsa.debian.org/images-team/debian-cd/-/commit/e77ade6033445571092e8663ad94a24c7e882b03

I'm going to push this soon but I would love if someone else could
do a test run and ensure it doesn't break anything else.

I have created a merge request to make it easier to review the code if
anyone wants to do it: >https://salsa.debian.org/images-team/debian-cd/-/merge_requests/26

I can have a look, but not *right* now I'm afraid - swamped with other
stuff. Hopefully by the end of the week...

--
Steve McIntyre, Cambridge, UK. steve@einval.com “Changing random stuff until your program works is bad coding
practice, but if you do it fast enough it’s Machine Learning.”
-- https://twitter.com/manisha72617183

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)