Hi,

I'm using qemu-user with a Debian unstable sh4 chroot on a 40-core Xeon machine and running dpkg-deb fails with the same issue reported in #846564 [1]:

dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
dh_builddeb: error: Aborting due to earlier error

Is there a way to tune lzma such that it reduces memory consumption in this case?

Thanks,
Adrian

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846564

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Thu, 2024-09-26 at 10:31:55 +0200, John Paul Adrian Glaubitz wrote:

I'm using qemu-user with a Debian unstable sh4 chroot on a 40-core Xeon machine
and running dpkg-deb fails with the same issue reported in #846564 [1]:

dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
dh_builddeb: error: Aborting due to earlier error

Hmm, the heuristics in the code should in theory prevent this, so it
would be nice to know what is going wrong with that, in case it could
be improved. See the filter_xz_get_memlimit() and
filter_xz_get_cputhreads() functions in lib/dpkg/.

Is there a way to tune lzma such that it reduces memory consumption in this case?

You should be able to globally reduce the amount of threads used with
the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
need to be passed from debian/rules or debhelper or similar).

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Guillem,

On Thu, 2024-09-26 at 13:20 +0200, Guillem Jover wrote:

dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
dh_builddeb: error: Aborting due to earlier error

Hmm, the heuristics in the code should in theory prevent this, so it
would be nice to know what is going wrong with that, in case it could
be improved. See the filter_xz_get_memlimit() and
filter_xz_get_cputhreads() functions in lib/dpkg/.

I thought so as well after reading through the bug report in #846564,
but unfortunately the issue is reliably reproducible.

Is there a way to tune lzma such that it reduces memory consumption in this case?

You should be able to globally reduce the amount of threads used with
the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
need to be passed from debian/rules or debhelper or similar).

OK, thanks. I will try whether this helps as a workaround.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 2024-09-28 at 08:22 +0200, John Paul Adrian Glaubitz wrote:

Is there a way to tune lzma such that it reduces memory consumption in this case?

You should be able to globally reduce the amount of threads used with
the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
need to be passed from debian/rules or debhelper or similar).

Thanks a lot for the suggestion. For anyone else running into this problem, the following workaround added to debian/rules did the trick to me:

# Workaround for #846564
override_dh_builddeb:
dh_builddeb -- --threads-max=16

Unfortunately, this workaround is not very reliable and the issue just showed again.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Guillem,

On Thu, 2024-09-26 at 13:20 +0200, Guillem Jover wrote:

On Thu, 2024-09-26 at 10:31:55 +0200, John Paul Adrian Glaubitz wrote:

I'm using qemu-user with a Debian unstable sh4 chroot on a 40-core Xeon machine
and running dpkg-deb fails with the same issue reported in #846564 [1]:

dpkg-deb: building package 'gcc-snapshot' in '../gcc-snapshot_20240922-1+sh4.1_sh4.deb'.
dpkg-deb: building package 'gcc-snapshot-dbgsym' in '../gcc-snapshot-dbgsym_20240922-1+sh4.1_sh4.deb'.
dpkg-deb (subprocess): compressing tar member: lzma error: Cannot allocate memory
dpkg-deb: error: <compress> from tar -cf subprocess returned error exit status 2
dh_builddeb: error: dpkg-deb --root-owner-group --build debian/.debhelper/gcc-snapshot/dbgsym-root .. returned exit code 2
dh_builddeb: error: Aborting due to earlier error

Hmm, the heuristics in the code should in theory prevent this, so it
would be nice to know what is going wrong with that, in case it could
be improved. See the filter_xz_get_memlimit() and
filter_xz_get_cputhreads() functions in lib/dpkg/.

You should be able to reproduce it with qemu-user on a machine with many cores. Any emulated 32-bit architecture should work.

Is there a way to tune lzma such that it reduces memory consumption in this case?

You should be able to globally reduce the amount of threads used with
the DPKG_DEB_THREADS_MAX envvar (as the --threads-max option would
need to be passed from debian/rules or debhelper or similar).

Thanks a lot for the suggestion. For anyone else running into this problem,
the following workaround added to debian/rules did the trick to me:

# Workaround for #846564
override_dh_builddeb:
dh_builddeb -- --threads-max=16

Thanks,
Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)