Control: tags 1065751 + patch
Control: tags 1065751 + pending

Dear maintainer,

I've prepared an NMU for pristine-tar (versioned as 1.50+nmu2) and
uploaded it to DELAYED/2. Please feel free to tell me if I
should delay it longer.

Could someone check this, please?

Regards.

Sebastian

diff -Nru pristine-tar-1.50+nmu1/debian/changelog pristine-tar-1.50+nmu2/debian/changelog
--- pristine-tar-1.50+nmu1/debian/changelog 2024-02-25 12:18:32.000000000 +0100
+++ pristine-tar-1.50+nmu2/debian/changelog 2024-03-10 21:38:16.000000000 +0100
@@ -1,3 +1,11 @@
+pristine-tar (1.50+nmu2) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Preoperly account -T parameter for xz. Thanks to Jia Tan for the hint.
+ (Closes: #1065751).
+
+ -- Sebastian Andrzej Siewior <sebastian@breakpoint.cc> Sun, 10 Mar 2024 21:38:16 +0100
+
pristine-tar (1.50+nmu1) unstable; urgency=medium

* Non-maintainer upload.
diff -Nru pristine-tar-1.50+nmu1/pristine-xz pristine-tar-1.50+nmu2/pristine-xz --- pristine-tar-1.50+nmu1/pristine-xz 2024-02-25 12:18:06.000000000 +0100
+++ pristine-tar-1.50+nmu2/pristine-xz 2024-03-10 21:38:12.000000000 +0100
@@ -416,11 +416,11 @@
next if $param eq '--check=crc64';
next if $param eq '--check=sha256';
next if $param =~ /^(--block-list=[0-9,]+)$/;
- next if $param =~ /^-T[0-9]+$/;
+
if ($param =~ /^-T[0-9]+$/)

Hi Sebastian, all,

Will this fix be enough for addressing all cases, though?

I'm thinking specifically of cases where tarball repacking
is involved, for example when using git-buildpackage's
"gbp import-orig --uscan" where uscan is used to download and
repack the upstream tarball, because the package at hand has
a Files-Excluded field in its debian/copyright header stanza.
As far as I can tell, Devscripts::Compression would need to be
updated to specify -T1 for xz compressions.

I believe there are also some cases where git-buildpackage
itself does repacking, so we'd probably want to update its
gbp.pkg.compressor's Opts to pass in -T1 for xz.

Thanks,
-a

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2024-03-11 00:05:54 [+0000], Amin Bandali wrote:

Hi Sebastian, all,

Hi,

Will this fix be enough for addressing all cases, though?

I think so. Do you have a test case for me to check?

I'm thinking specifically of cases where tarball repacking
is involved, for example when using git-buildpackage's
"gbp import-orig --uscan" where uscan is used to download and
repack the upstream tarball, because the package at hand has
a Files-Excluded field in its debian/copyright header stanza.
As far as I can tell, Devscripts::Compression would need to be
updated to specify -T1 for xz compressions.

I believe there are also some cases where git-buildpackage
itself does repacking, so we'd probably want to update its gbp.pkg.compressor's Opts to pass in -T1 for xz.

Who is handling the compression in the first place here?
The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's
delta information. If the -T argument then everything keeps working
as-is. If you use gbp to repack the tar archive then I would recommend
to no pass -T1 and to use multi-threaded compression. pristine-tar
will recongnise this and record this information.

Thanks,
-a

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On Mon, Mar 11, 2024 at 05:55:31PM +0100, Sebastian Andrzej Siewior wrote:

On 2024-03-11 00:05:54 [+0000], Amin Bandali wrote:

Hi Sebastian, all,

Hi,

Will this fix be enough for addressing all cases, though?

I think so. Do you have a test case for me to check?

Not about pristine-xz specifically; I meant more in the context of
other devel tools like gbp et al.

I'm thinking specifically of cases where tarball repacking
is involved, for example when using git-buildpackage's
"gbp import-orig --uscan" where uscan is used to download and
repack the upstream tarball, because the package at hand has
a Files-Excluded field in its debian/copyright header stanza.
As far as I can tell, Devscripts::Compression would need to be
updated to specify -T1 for xz compressions.

I believe there are also some cases where git-buildpackage
itself does repacking, so we'd probably want to update its gbp.pkg.compressor's Opts to pass in -T1 for xz.

Who is handling the compression in the first place here?

In the case of "gbp import-orig --uscan", gbp invokes uscan, part of
the devscripts package which has several perl modules including Devscripts::Compression which is a sort of a wrapper around dpkg's Dpkg::Compression, which will ultimately run the 'xz' executable.

In some other cases like "gbp import-orig --filter" mentioned by
Andrey, gbp does the compression itself. Which is why I suggested
that 'Opts' in gbp.pkg.compressor may need to be updated to add '-T1'
for calls to 'xz'.

The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's
delta information. If the -T argument then everything keeps working
as-is. If you use gbp to repack the tar archive then I would recommend
to no pass -T1 and to use multi-threaded compression. pristine-tar
will recongnise this and record this information.

Sorry I don't think I fully understood this bit. Could you please
explain again, perhaps a bit more verbosely?

To clarify, the use-cases described earlier involving gbp and
devscripts aren't necessarily related to pristine-xz, used for
regenerating pristine xz files; rather, about the generation or
repacking of xz files *before* they are handed to pristine-xz for
processing and storage in the repo. I was trying to imply that
similarly to how you sent patches for pristine-tar to adapt it for
changes in xz-utils, that similar patches are probably also needed
for gbp and devscripts. Does that make sense?

Sebastian

Thanks,
-amin

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2024-03-11 21:23:03 [+0000], Amin Bandali wrote:

Hi,

Hi,

On Mon, Mar 11, 2024 at 05:55:31PM +0100, Sebastian Andrzej Siewior wrote:

On 2024-03-11 00:05:54 [+0000], Amin Bandali wrote:

Hi Sebastian, all,

Hi,

Will this fix be enough for addressing all cases, though?

I think so. Do you have a test case for me to check?

Not about pristine-xz specifically; I meant more in the context of
other devel tools like gbp et al.

ah okay. pristine-tar was the only tool that had CI failures during the
upload of new xz-utils to exp. I wouldn't know other tools that require
to recreate the same binary file.

Who is handling the compression in the first place here?

In the case of "gbp import-orig --uscan", gbp invokes uscan, part of
the devscripts package which has several perl modules including Devscripts::Compression which is a sort of a wrapper around dpkg's Dpkg::Compression, which will ultimately run the 'xz' executable.

In some other cases like "gbp import-orig --filter" mentioned by
Andrey, gbp does the compression itself. Which is why I suggested
that 'Opts' in gbp.pkg.compressor may need to be updated to add '-T1'
for calls to 'xz'.

okay. I wouldn't recomment doing -T1. This forces xz doing a single
block and using a signle thread. The default (without passing the -T
argument) will allow xz to use multiple threads and compress into
multiple blocks which in turn can be decompressed using multiple
threads.
Forcing -T1 will force single threaded compression and decompression. pristine-tar can handle both cases.

The idea is to pass -T1 to xz if nothing was recorded in pristine-tar's delta information. If the -T argument then everything keeps working
as-is. If you use gbp to repack the tar archive then I would recommend
to no pass -T1 and to use multi-threaded compression. pristine-tar
will recongnise this and record this information.

Sorry I don't think I fully understood this bit. Could you please
explain again, perhaps a bit more verbosely?

If you do "pristine-tar gendelta" then pristine tar creates a .delta
file which is tar.gz file containing a few files including the actual
delta from `xdelta' and a file called `wrapper'. The `wrapper' file is
also a tar.gz file including files regarding the invocation of the
compressing tool which includes the arguments required to produce the
exact output of the resulting .xz (from the tar input). Prior 1.50+nmu1 pristine-tar didn't record here the -T argument unless multi-threaded compression was used and pristine-tar used -Tcpus and recorded this.
Since 1.50+nmu1 I made pristine-tar to always record the -T argument in
the wrapper file, either -Tcpus in the multi threaded case as it did or
by using -T1 in the single threaded one block case.
That means the reproduce case has always the fitting -T argument. If you
get an older archive which lacks the -T argument, pristine-tar will
assume -T1 which was the old default.

To clarify, the use-cases described earlier involving gbp and
devscripts aren't necessarily related to pristine-xz, used for
regenerating pristine xz files; rather, about the generation or
repacking of xz files *before* they are handed to pristine-xz for
processing and storage in the repo. I was trying to imply that
similarly to how you sent patches for pristine-tar to adapt it for
changes in xz-utils, that similar patches are probably also needed
for gbp and devscripts. Does that make sense?

So gbp and descripts should be able to deal with xz as-is since they
don't have any expectation in the resulting binary file. They are happy
once the input compressed/ decompressed. pristine-tar is the only tool,
to my best knowledge, that requires binary identical output. Therefore I
would keep gbp and devscripts as-is and prefer the multi-threaded
compression & decompression.
dpkg uses multi-threaded compression since a while and decompression
since Bookworm.

Thanks,
-amin

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2024-03-12 09:26:32 [-0400], Jeremy Bícha wrote:

Could someone check this, please?

Did you try running autopkgtests on this version? The autopkgtests fail for me.

autopkgtests were the first thing that pointed me here and they passed.
If you say they fail for you then I may have used the wrong xz version…

I assume that the largest use of pristine-tar in Debian is with git-buildpackage. The 1.50+nmu1 upload **caused** pristine-tar to
break in many cases for me. If I revert back to 1.50, I no longer get mismatched tarballs errors. Here are some test cases to demonstrate:

Test Case 1
==========
gbp clone --add-upstream-vcs https://salsa.debian.org/jbicha/pangomm2.48

cd pangomm2.48

gbp import-orig --uscan

gbp buildpackage

What happens
------
The exact hashes will probably vary but I get an error like this:

gbp:error: Pristine-tar couldn't verify
"pangomm2.48_2.50.2.orig.tar.xz": pristine-tar: /home/jeremy/build-area/pangomm2.48_2.50.2.orig.tar.xz does not match
stored hash (expected e99b6a9c89e9c284bf44f5ae8125c06515d6ab8f8577d75d2887726dacb5a372, got 826ad52f53ac8e15c9ceba4dc6e616efddae5e089f36bf4e60081c177d80d4b6)

| (sid)bigeasy@debbuildd:~/pristine$ gbp clone --add-upstream-vcs https://salsa.debian.org/jbicha/pangomm2.48
| gbp:info: Cloning from 'https://salsa.debian.org/jbicha/pangomm2.48'
| gbp:info: Adding upstream vcs at https://gitlab.gnome.org/GNOME/pangomm.git as additional remote
| (sid)bigeasy@debbuildd:~/pristine$ cd pangomm2.48
| (sid)bigeasy@debbuildd:~/pristine/pangomm2.48$ gbp import-orig --uscan
| gbp:info: Launching uscan...
| Newest version of pangomm2.48 on remote site is 2.50.2, local version is 2.50.1
| => Newer package available from:
| => https://download.gnome.org/sources/pangomm/2.50/pangomm-2.50.2.tar.xz
| Successfully repacked ../pangomm-2.50.2.tar.xz as ../pangomm2.48_2.50.2.orig.tar.xz, deleting 416 files from it.
| gbp:info: Using uscan downloaded tarball ../pangomm2.48_2.50.2.orig.tar.xz
| What is the upstream version? [2.50.2]
| gbp:info: Importing '../pangomm2.48_2.50.2.orig.tar.xz' to branch 'upstream/latest'...
| gbp:info: Source package is pangomm2.48
| gbp:info: Upstream version is 2.50.2
| gbp:info: Replacing upstream source on 'debian/latest'
| gbp:info: Running Postimport hook
| dch warning: neither DEBEMAIL nor EMAIL environment variable is set
| dch warning: building email address from username and FQDN
| dch: Did you see those 2 warnings? Press RETURN to continue...
|
| git commit -m 'New upstream release'
| [debian/latest f5c4f6d14a78] New upstream release
| 1 file changed, 5 insertions(+), 2 deletions(-)
| gbp:info: Successfully imported version 2.50.2 of ../pangomm2.48_2.50.2.orig.tar.xz

passed.

Other info
-----
pangomm2.48 uses Files-Excluded in debian/copyright so uscan will
rebuild a tarball and its hash will vary depending on the time it was created. (Perhaps the hash should be reproducible but that's not
relevant to this bug.)

Test Case 2
=========
gbp clone https://salsa.debian.org/gnome-team/gtk4
cd gtk4
gbp buildpackage

What happens
------------
gbp:error: Pristine-tar couldn't verify "gtk4_4.12.5+ds.orig.tar.xz": pristine-tar: /home/jeremy/devel/pkg-gnome/temp/build-area/gtk4_4.12.5+ds.orig.tar.xz
does not match stored hash (expected 3338a691d774ae031af65299e9a1c6207f543f13b256539717a1970f752358cb, got 70ac33e0f37dc1b657d6560f1b8a40b3f4b67e956936633ced495d8b880d3fb0)

| (sid)bigeasy@debbuildd:~/pristine$ gbp clone https://salsa.debian.org/gnome-team/gtk4
| gbp:info: Cloning from 'https://salsa.debian.org/gnome-team/gtk4'
| (sid)bigeasy@debbuildd:~/pristine$ cd gtk4
| (sid)bigeasy@debbuildd:~/pristine/gtk4$ gbp buildpackage
| gbp:info: Creating /home/bigeasy/pristine/gtk4_4.12.5+ds.orig.tar.xz
| gbp:info: Performing the build
| dpkg-buildpackage -us -uc -ui -i -I
| dpkg-buildpackage: info: source package gtk4
| dpkg-buildpackage: info: source version 4.12.5+ds-4

passed.

Other info
----
This pristine-tar tarball was committed January 19 so it did not use
either the new xz-utils or pristine-tar.

Test Case 3
=========
gbp clone https://salsa.debian.org/gnome-team/pango
cd pango
gbp buildpackage

What happens
-------------------
gbp:error: Pristine-tar couldn't verify
"pango1.0_1.52.1+ds.orig.tar.xz": pristine-tar: /home/jeremy/devel/pkg-gnome/temp/build-area/pango1.0_1.52.1+ds.orig.tar.xz does not match stored hash (expected 12d67d8182cbb2ae427406df9bab5ce2ff5619102bf2a0fc6331d80a9914b139, got a641d29d2d7df7843e44762a0733987dc8220d238b697b382dd96fafe5fc890a)

| (sid)bigeasy@debbuildd:~/pristine$ gbp clone https://salsa.debian.org/gnome-team/pango
| gbp:info: Cloning from 'https://salsa.debian.org/gnome-team/pango'
| (sid)bigeasy@debbuildd:~/pristine$ cd pango
| (sid)bigeasy@debbuildd:~/pristine/pango$ gbp buildpackage
| gbp:info: Creating /home/bigeasy/pristine/pango1.0_1.52.1+ds.orig.tar.xz
| gbp:info: Performing the build
| dpkg-buildpackage -us -uc -ui -i -I

Other info
-------------
This tarball was committed a few days ago with the new xz-utils and pristine-tar 1.50+nmu1.
pango also uses Files-Excluded

Conclusion
========
Test cases 1, 2, and 3 pass with pristine-tar 1.50 but fail with
pristine-tar 1.50+nmu1

tried with 1.50+nmu2 which is in deferred for the next 11h. So tomorrow
it should be all good.

Thank you,
Jeremy Bícha

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Mar 12, 2024 at 09:13:09PM +0100, Sebastian Andrzej Siewior wrote:

tried with 1.50+nmu2 which is in deferred for the next 11h. So tomorrow
it should be all good.

Please make sure to commit the patch to the pristine-tar git repository
as well.

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEst7mYDbECCn80PEM/A2xu81GC94FAmXwyEQACgkQ/A2xu81G C95JiA/7BalmS+tC+hpeyELN0zKUQYecAwR7EJ3we/vNO1KO9NBJwS2S8TC5iFWL tV8NK7FqtL0Xhue/ohNXGR1taonzrsm+uPU79WsGI3PRHTebwpDavmpoPjhMU/u4 CnVgvEZFIsln1G1WM78TFMtlzl4Au1Io421IAMH2XwJVZ+lZhcEcRosENoL/Higy 18xPd1oFYKUgtan/nC8Z5loNjNsfkCFNXVl1qiolMfuoasmpbXfWHhoQ8qvdpAM0 D74wUavi2N1BycXKYnbHFX6R0Ntoz51IC8V7NL8g1ZAAADiwrhZXMh3hFLgGzozo Hn/9evN935HEhwWbtW7G+jEsXgA0D/CxDyfOk3p5XG5ZcI/4qBXOfZwxl1VrB3qX Mj9xU08Hf4/tXoSvtYIrn1U6edQelefUPuEFP8C85yRu9UqmbAO/B59LMumjoJV/ 7cFYFZoCHSZzh5vGinEtMpezioTGNQadBPCPMvjTt0cKIFUjxmxMH8zdNuJObtMn vWdkqxa+5lIzv3I21mo7aAVdcHi/am4rzwDgf28APUYrW96jdj7OVo0wjsMKdcpL sF9OF24g6Hp0B3jiTMcKqEfgQ3BkoBTUf0SO6k1Q6NoPrmG/d0Y8Jy+fAkQu9jfO LnP97WyhnBzCDiCwRzxahzPmoLvkzY/UaHh5wS51H+UgVRubDds=
=RBEn
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Mar 12, 2024 at 5:16 PM Jeremy Bícha <jeremy.bicha@canonical.com> wrote:

On Tue, Mar 12, 2024 at 4:13 PM Sebastian Andrzej Siewior <sebastian@breakpoint.cc> wrote:

On 2024-03-12 09:26:32 [-0400], Jeremy Bícha wrote:

Could someone check this, please?

Did you try running autopkgtests on this version? The autopkgtests fail for me.

autopkgtests were the first thing that pointed me here and they passed.
If you say they fail for you then I may have used the wrong xz version…

I tried running the autopkgtests locally using a version of 1.50+nmu2
that I built using your patch and the autopkgtests failed.

None of the 3 test cases passed for me with that version or with 1.50+nmu1.

The 3 test cases pass for me now with the uploaded 1.50+nmu2. Maybe my
earlier test build used the old version of xz-utils. Thank you for
your upload!

Jeremy Bícha

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2024-03-31 19:42:24 [+0000], tony mancill wrote:

Given what has unfolded over the past few days regarding xz-utils and CVE-2024-3094 [0], should we revisit the patches applied here and for #1063252? Are they still needed?

Not with the fallback to pre 5.4.x series but *I* don't think this will
remain forever.
The requirement for the change was different default value for the -T parameter. Recording the -T parameter by default and relying on
defaults is good.

Thank you,
tony

Sebastian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)