Hi,

XZ Utils 5.3.3alpha has been released and one of the highlights is a
threaded .xz decompressor. See
https://www.mail-archive.com/xz-devel@tukaani.org/msg00591.html

Assuming the final stable release by streams happens in December, would
there still be time to have dpkg support for it targeting Bookworm?
Would it help to have the current alpha version experimental?
Is there anything I could help with?

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Tue, 2022-09-27 at 21:44:36 +0200, Sebastian Andrzej Siewior wrote:

XZ Utils 5.3.3alpha has been released and one of the highlights is a
threaded .xz decompressor. See
https://www.mail-archive.com/xz-devel@tukaani.org/msg00591.html

Assuming the final stable release by streams happens in December, would
there still be time to have dpkg support for it targeting Bookworm?
Would it help to have the current alpha version experimental?
Is there anything I could help with?

Ah! I was looking into this a couple of weeks back, and was wondering
how to proceed. I merged all the needed patches into dpkg git main
except for the last patch enabling the threaded decompressor. The
problem is that I wanted to retest it, as the xz implementation
changed upstream, and I had adapted it to the new API, but I run out
of steam when I was confronted with having to update the xz-utils
packaging for the alpha branch.

So having that packaged in experimental would help. My main concern
was that given that this is in an alpha, I'm not sure what API
guarantees are there. But I was considering committing the patch (once
tested) anyway, which would end up being disabled until the xz-utils
version gets uploaded to sid, and would require just a binNMU (if
necessary).

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 2022-09-28 at 01:43 +0200, Guillem Jover wrote:

Hi!

On Tue, 2022-09-27 at 21:44:36 +0200, Sebastian Andrzej Siewior
wrote:

XZ Utils 5.3.3alpha has been released and one of the highlights is
a
threaded .xz decompressor. See
         https://www.mail-archive.com/xz-devel@tukaani.org/msg00591.html

Assuming the final stable release by streams happens in December,
would
there still be time to have dpkg support for it targeting Bookworm?
Would it help to have the current alpha version experimental?
Is there anything I could help with?

Ah! I was looking into this a couple of weeks back, and was wondering
how to proceed. I merged all the needed patches into dpkg git main
except for the last patch enabling the threaded decompressor. The
problem is that I wanted to retest it, as the xz implementation
changed upstream, and I had adapted it to the new API, but I run out
of steam when I was confronted with having to update the xz-utils
packaging for the alpha branch.

BTW not about xz , but Zstd, which is available on dpkg (1.21.1ubuntu1)

[..]
- Add Zstd compression and decompression support for binary
packages.

i.e. maybe also include Zstd to be compatible with Ubuntu

reference : https://bugzilla.redhat.com/show_bug.cgi?id=2112807

So having that packaged in experimental would help. My main concern
was that given that this is in an alpha, I'm not sure what API
guarantees are there. But I was considering committing the patch
(once
tested) anyway, which would end up being disabled until the xz-utils
version gets uploaded to sid, and would require just a binNMU (if
necessary).

Thanks,
Guillem

--
Sérgio M. B.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-28 01:43:34 [+0200], Guillem Jover wrote:

Hi!

Hi,

So having that packaged in experimental would help. My main concern
was that given that this is in an alpha, I'm not sure what API
guarantees are there. But I was considering committing the patch (once tested) anyway, which would end up being disabled until the xz-utils
version gets uploaded to sid, and would require just a binNMU (if
necessary).

Okay. Let me package the alpha for exp then.
Jonathan, are there any objections? While at it, I would also update the version in ustable to the current stable.

Thanks,
Guillem

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-29 13:05:15 [+0200], To debian-dpkg@lists.debian.org wrote:

On 2022-09-28 01:43:34 [+0200], Guillem Jover wrote:

Hi!

Hi,

Okay. Let me package the alpha for exp then.
Jonathan, are there any objections? While at it, I would also update the version in ustable to the current stable.

It has been uploaded to experimental.

Thanks,
Guillem

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Mon, 2022-10-03 at 10:58:10 +0200, Sebastian Andrzej Siewior wrote:

On 2022-09-29 13:05:15 [+0200], To debian-dpkg@lists.debian.org wrote:

Okay. Let me package the alpha for exp then.
Jonathan, are there any objections? While at it, I would also update the version in ustable to the current stable.

It has been uploaded to experimental.

Great, thanks much for this! I've rebased (with no changes needed) the
patch on top of git HEAD, built and executed the functional test suite,
and it seems to be working fine. :) And pushed it to <https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/xz-mt-decompressor>.


I'm a little wary about merging this though, given upstream mails
about it, and their non-guarantees for the API/ABI stability, and for
their fear of downstreams to potentially start using it by packaging
the alpha or something like that. Given that this will not be enabled
in sid/testing anyway as the required liblzma will not be there, perhaps
we can wait merging it until a non-alpha release hits sid? If you think
it would make your life way way easier for testing or similar, then
perhaps adding the patch shadowed behind AUTHOR_TESTING (just like the shared-library support) could do it. Or did you have in mind something
else?

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-10-07 05:07:08 [+0200], Guillem Jover wrote:

Hi!

Hi,

On Mon, 2022-10-03 at 10:58:10 +0200, Sebastian Andrzej Siewior wrote:

On 2022-09-29 13:05:15 [+0200], To debian-dpkg@lists.debian.org wrote:

Okay. Let me package the alpha for exp then.
Jonathan, are there any objections? While at it, I would also update the version in ustable to the current stable.

It has been uploaded to experimental.

Great, thanks much for this! I've rebased (with no changes needed) the
patch on top of git HEAD, built and executed the functional test suite,
and it seems to be working fine. :) And pushed it to <https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/xz-mt-decompressor>.

Thanks.

I'm a little wary about merging this though, given upstream mails
about it, and their non-guarantees for the API/ABI stability, and for
their fear of downstreams to potentially start using it by packaging
the alpha or something like that. Given that this will not be enabled
in sid/testing anyway as the required liblzma will not be there, perhaps
we can wait merging it until a non-alpha release hits sid? If you think
it would make your life way way easier for testing or similar, then
perhaps adding the patch shadowed behind AUTHOR_TESTING (just like the shared-library support) could do it. Or did you have in mind something
else?

Right. What about uploading it functional to experimental so it can be
tested by a larger fraction? I'm mostly concerned about memory limits on
small machines but I also haven't look at the code to be certain that it
is something to worry about ;)
I guess we could add breaks: to liblzma to ensure that dpkg and liblzma
will be updated in one go (in experimental) in case of an ABI change and
both need to be updated.
Will try to build the new dpkg over the weekend and give it a try.

Thanks,
Guillem

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-10-07 05:07:08 [+0200], Guillem Jover wrote:

Hi!

Hi,

Great, thanks much for this! I've rebased (with no changes needed) the
patch on top of git HEAD, built and executed the functional test suite,
and it seems to be working fine. :) And pushed it to <https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/xz-mt-decompressor>.

The 5.4 hit experimental as you noticed. Once you fine with your testing
we may move it to unstable.

Thanks,
Guillem

Sebaxtian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Wed, 2022-12-14 at 07:15:40 +0100, Sebastian Andrzej Siewior wrote:

On 2022-10-07 05:07:08 [+0200], Guillem Jover wrote:

Great, thanks much for this! I've rebased (with no changes needed) the patch on top of git HEAD, built and executed the functional test suite,
and it seems to be working fine. :) And pushed it to <https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/xz-mt-decompressor>.

The 5.4 hit experimental as you noticed. Once you fine with your testing
we may move it to unstable.

Yes, thank you! I'm running with it, and seems to be working fine. Did
you end up doing more tests? I think moving the library to unstable the
sooner the better, so that it can be exposed to more people. I guess I'd
do a dpkg upload in at most a couple of days or so.

I think one minor concern I have (but now realized it does not affect
Debian) is that the -T option for the xz tool (when used by libdpkg
instead of linking against the liblzma library), will fallback to the non-multithreaded encoder/decoder if passed 1. But passing +1 is not
supported by older versions. This would affect reproducibility. I
guess I could always pass +1 and document that the new version is
required when not linking against the library, but meh I guess.

I also realized that it would be nice to update the dpkg-source code
to use the threaded stuff. Will prepare a patch for that.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-12-15 05:50:34 [+0100], Guillem Jover wrote:

Hi!

Hi,

On Wed, 2022-12-14 at 07:15:40 +0100, Sebastian Andrzej Siewior wrote:

On 2022-10-07 05:07:08 [+0200], Guillem Jover wrote:

Great, thanks much for this! I've rebased (with no changes needed) the patch on top of git HEAD, built and executed the functional test suite, and it seems to be working fine. :) And pushed it to <https://git.hadrons.org/git/debian/dpkg/dpkg.git/commit/?h=next/xz-mt-decompressor>.

The 5.4 hit experimental as you noticed. Once you fine with your testing
we may move it to unstable.

Yes, thank you! I'm running with it, and seems to be working fine. Did
you end up doing more tests? I think moving the library to unstable the

I had the xz binary in use on a few systems and I had dpkg packaged to
use it. Nothing bad did happen.

sooner the better, so that it can be exposed to more people. I guess I'd
do a dpkg upload in at most a couple of days or so.

Okay. Let me do this today.
One side note: I had a bit of off-list discussion with Lasse regarding
the LZMA properties (lc/lp/pb) and the arm64 filter. It seems it
squeezes a few additional bytes during the compression. The arm64 filter requires 5.4.0 but changing the properties is transparent. Do you think,
that we could tweak a little? Also he mentioned that the used memory
(for the dictionary and so the block) could be increased today since we
do have more memory. What do you think? Is this something we should
touch or is the benefit probably small because most packages are tiny
with a few exceptions (gcc, kernel, texlive,...) so it is not worh the
trouble with people that have just a little bit of ram.

I think one minor concern I have (but now realized it does not affect
Debian) is that the -T option for the xz tool (when used by libdpkg
instead of linking against the liblzma library), will fallback to the non-multithreaded encoder/decoder if passed 1. But passing +1 is not supported by older versions. This would affect reproducibility. I
guess I could always pass +1 and document that the new version is
required when not linking against the library, but meh I guess.

Okay. Good point.

I also realized that it would be nice to update the dpkg-source code
to use the threaded stuff. Will prepare a patch for that.

Oh. I assumed it did it ;) Cool.

Thanks,
Guillem

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)