"Simon" == Simon Richter <sjr@debian.org> writes:

Simon> Hi,
Simon> On 8/16/21 3:18 AM, Paul Wise wrote:

>> I'd like to suggest that we standardise on the upstream VCS for
>> our orig.tar.gz files and phase out use of upstream packaging
>> ecosystems.

Simon> This is also an additional burden on package maintainers:
Simon> explaining how they arrived at that particular "upstream"
Simon> package in a reproducible way, and why what we ship as "orig"
Simon> is different from upstream, and what the copyright and
Simon> licensing situation for that derived work is.

Simon> Upstream projects have gotten a lot sloppier in how they cut
Simon> releases, that is true, and that is making packaging more
Simon> difficult as we need to disable mechanisms and embedded code
Simon> copies that were included for our "convenience."

Simon> Rather than accept defeat,

I don't think it's accepting defeat to use an upstream vcs.
I think it's just better.
IT's closer to the development workflow I'd use to work on the upstream.
AS a maintainer, it makes it easier for me to forward or back port
patches.
It makes it easier for me to contribute upstream.

I acknowledge your concern about needing to justify why I picked a
particular git tag.

By this point I kind of think source tarballs are an anti-pattern.

I do agree with you that Debian should work with upstreams to understand
our needs and to produce high quality releases.
At least for me though, that's unrelated to this issue.
In my world high quality source releases are signed git tags not
tarballs.
I'd like Debian to embrace my world:-)

-----BEGIN PGP SIGNATURE-----

iHUEARYIAB0WIQSj2jRwbAdKzGY/4uAsbEw8qDeGdAUCYSa68AAKCRAsbEw8qDeG dH4JAP9wtMCoj99iFcsIISKpxPuuXvkDi76+4CsfbJJ+p3AgZgEA95Wp439E6IjK DwBzwxf7OPkunlxCda8gkFAdp/VJgAM=
=YlPn
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 0825, Simon Richter wrote:

Hi,

On 8/25/21 1:21 AM, Sean Whitton wrote:

From my point of view, signing git tags is no less well established a
best practice than signing tarballs -- in fact, to me, it seems *more*
well established.

That is ecosystem dependent.

FWIW, I'd love to see git bundles as a source archive format -- this would >allow shipping a (signed) tag, its commit, and the tree and blob objects for >that commit as a single file that can be built in a reproducible way and allows
changes on top to be easily tracked, including the branch point.

In the absence of an "official" upstream release tarball, using this format >also makes it clear that this is a git snapshot, so no explanation is needed >how that archive was created.

Ecosystem-dependent or not, I can see being able to verify who uploaded
the Git tag (or anything for that matter) as being increasingly valuably
in a world where there is a lot of uncaught or ignored plagiarism.
Uploaders and creators should have integrity so that their users can
rely on them and be confident to deliver quality work.

--
Best regards,

Brian T

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEE9fpVo96/flopdKOfgw2Ncu3Nhn0FAmEm5tMACgkQgw2Ncu3N hn0iwhAAnj7PGHhZ+s4ASJ5vJJn35ef/5AjhJOY3kSSMHV4Q6PkemZgWWpoAimoy /8x1NsZiwb0NmFE/1nuoI6a+Ce4eoEfuryZZJgo7WTT04X97Ru3MTzWfaGoNo7rK rTsxkeLSTqwrPd5sFWu84UXJ60SVrDFOLrt79D9Ot2Y8L63efzsuj3mLicaPF31O u5YRznB2csuy/L6pfhv+/GZZxPcmD4G4QNRRJHVvaAkRuvRcQWpVCDxrHi1q4aZ7 b7kMDFXUz4g2A97QhdBWFt5NbcJ2RilU/wCd5j5rq0ttgFy+pRoSWSONdWlcIrg9 J9fr3KXkQEXzPZwa+U+KA+xUPUNXABuNUA/qVfxOOX0Ber09BMcGfREnNWgAf7LJ uSRYJe9WgpKa9D6elo0wxTGyJ1WFID8T8fIG2kgvG5NdQiDKb+LrKJZ5O3hIlFsm ZbV1lXDyL4iKfDceiy/LLXj8v7KglMKWoh20JyoknD0CczAT/dYz+xYtkEMYaxOY d7CYfbx/qn6O3N4JG0P4VQPV1frCxZHurAKEG3ocrBSKR0Y78T3EvGTwcf868uZD qB2mdyBx1a/U/Kqzi+bxTIZ6ccjl5pG907zpybW1X1wqLDKo5T8//ubrZFH/YYAY nSMOdj1HPodB71MqnsZ6YCbAOS3Lv0WL/ItIuxbRBClrr53Ha3U=
=OCy0
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

To: debian-devel@lists.debian.org

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --iklOciN0r926TnC8rtpkkRyEjtW462Mag
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

Hi Thomas,

On 8/26/21 4:16 PM, Thomas Goirand wrote:

"git archive" is reproducible, for simplicity I wouldn't use a prefix
though. xz has some issues with reproducibility, AFAIK "-T2" makes it
disable some internal heuristics that are based on the machine it is
running on, and generates consistent output.

Excuse me, but -T in xz looks like to be the number of threads. Did you mistake it with something else?

No, enabling multithreading with a number of threads > 1 disables a
heuristic that optimizes memory usage based on memory available on the
machine it is running on, and uses fixed-size buffers.

Simon


--iklOciN0r926TnC8rtpkkRyEjtW462Mag--

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEEtjuqOJSXmNjSiX3Tfr04e7CZCBEFAmEnqCMACgkQfr04e7CZ CBHNpAgAi277yhlWKHMfut8+FX3cphsd5k15TVGMAE9bTItj5SprgacaRRMZ9ZMR VNUh+zshskY3OBdXSvadpePOWzlrz1a3XqlyC5Gw77Qce/IP0+M7qARKRTUlESqh GMl91b34IWGg9dPVaAbr28JhqBLm3mY2ploQgIC2srQ78aOtKokirgU/GB+pzRL3 9ou8YlO440euL7A49hvvI3jMBNFSIaYV7AC3xgXkJxc6TyArAzql7Dg7y16XVVWa 8m3KswFTjzDDyxcjXUxh1wqbGWDRaMEjojvuotdkkLScxNFB8cItMr+4EqsIYhpi WbWeh0jpmIaGIMx7IrR0qQ7fTBDAYQ==
=sYL5
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 8/25/21 4:35 PM, Simon Richter wrote:

Hi,

I wrote this many times, but I don't see why we should use any "upstream
tarball" when the Git repository itself contains the tarball with:

git archive --prefix=$(DEBPKGNAME)-$(VERSION)/ $(GIT_TAG) \
    | xz >../$(DEBPKGNAME)_$(VERSION).orig.tar.xz

(which leads to a .xz, which is nicer)

"git archive" is reproducible, for simplicity I wouldn't use a prefix
though. xz has some issues with reproducibility, AFAIK "-T2" makes it
disable some internal heuristics that are based on the machine it is
running on, and generates consistent output.

   Simon

Excuse me, but -T in xz looks like to be the number of threads. Did you
mistake it with something else?

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Aug 25, 2021 at 2:36 PM Simon Richter wrote:

"git archive" is reproducible

I'm told by the Bootstrappable Builds folks that `git archive` isn't deterministic in some cases to do with filtering, I lost the details
though.

--
bye,
pabs

https://wiki.debian.org/PaulWise

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello zigo,

On Wed 25 Aug 2021 at 04:11PM +02, Thomas Goirand wrote:

I wrote this many times, but I don't see why we should use any "upstream tarball" when the Git repository itself contains the tarball with:

git archive --prefix=$(DEBPKGNAME)-$(VERSION)/ $(GIT_TAG) \
| xz >../$(DEBPKGNAME)_$(VERSION).orig.tar.xz

(which leads to a .xz, which is nicer)

Not only then, only only has to merge the upstream tag in the Debian
branch to get the new release, but on top, no need to "gbp import" or "pristine-tar commit", and a single packaging branch becomes enough.

I very much wish this packaging workflow gained more traction, and the pristine-tar abomination dies...

I agree.

I'd like to suggest using 'git deborig' which is much shorter to type :)

--
Sean Whitton

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmEpUJUZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQFtsD/9QQx4hxkqYaHfYRbPP8U0h vsP3eDlxRFCWJ0pTylHLlzHIxpfoJ+m3akoZfdiaJgPkKT/cXw9x951jlSByTQWT vayfjxiCWv/0qxVpY7FXgh6EYuzSijEEVyT+ZVfMJERi/3i1OihU7DCuJflK68/b gLWALTJPvoYPhmXVmzYh7SfMLfr/UCxrn2eT89RbLWj2v8JyF/HrPXTbaBtSzAk3 boCyMOhjU1l1iAYyHYpsbrZqyo3Xcn89ekf4gnQMZcAuB1rgpjA5/HNnNvqHnyVH 0owbEyr+zHQuJsGxVKEIlEBpGbyPmA/YlofaxP/jBQDtZOv1HYXexdNP9XaVwaIy J3YM0C4wwPZqm/zsPWDVccOFXlth8JcEzaQ+8oV95T9tl3GFrPjc/GCbAPnSzi6u MapfPAdxAOr9XLXIPN+xmsoHmkaD+XkkvCaXzUlOXzfBn0omzd/0F0rtJU860bIA PlZxcFVucW7aKncIxK7/b/9YcL0MEVVs+twwAchbFQ5Z9SwzgFEZb6zoGHfboywz HrkvfJJW8hOW4OuPeizCsV0s6odjm7/ZOGLCys1Av0bCM3LmF+4o21KMTrHAJ66z sxiX5IJRuQgsZK3e1W8lxvSagsVEKqnly1vFex1cyH/w10ikdO7ZD7sKAuBv/poE cbj0mZtUgnQhCIRzwDX+5g=›A+
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usen

Hello,

On Fri 27 Aug 2021 at 07:58AM GMT, Paul Wise wrote:

On Wed, Aug 25, 2021 at 2:36 PM Simon Richter wrote:

"git archive" is reproducible

I'm told by the Bootstrappable Builds folks that `git archive` isn't deterministic in some cases to do with filtering, I lost the details
though.

git-deborig in devscripts has code to handle the filtering problem, I
believe.

--
Sean Whitton

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmEpUGsZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQLXDD/0XGbTSa8txM0j7BAmckQuI 12VJsUQOfw7fu9dlRK5t/otKcK1cXkXLd5NQQYVoNSyNctyOSX7KjjQpLTNdpbQv k/1lmFwyQjPGE9oIjLu2BCNfsg9Q7MoY5DV9mBrFIHkoX1kxK9tzxLo/uCi4DscD UjQnqrVoBD/jldxA1SYXe8Mewxv3BdjpD0T2Zqu4bLXe3gbN8g6M98Cq2NEwC3k2 yahS9hWT29mfc95TRlewulf5QZacSfUp0jIU9M3//Ee4skpOh4+4U1i3MwYRXOnn J4aKMBZJ0f4bVYabxDhDr9lGhzF+MSrZ/YrIMVnXtnxjy823+2nfl3AkcJLrHBSS izA0c7SEmH6XYNbISH4UrnuDtQb55cqLcAd8DeXwmFO4lFDL7ur70Ge0noJHBAnb u6ZFvhZP27QQzZ+hjNRlpx0hmUcSw2d+5ypiYUXNopGQN//eBMkIxBbtfI4X0h8A 6j/8y0dkZOaiWffkgAJaMaOkmtfmBh9Q+0l/eU2JBxXOli6hnFlaT+jdb13ulYiE Wtw+zTgbhohAnWmH2SWELCFyHWPR+w5NKOkjZ2isuHlXnPJhTUPHdONztLYyaKEh IGPbxOYQCi0ZuJV815U5X0SNVTZ1MN4yddXTFPmVxFHYDrlWyiHhyyTjDv/3qb0Z WzUZhjrNLoWuW+/rnzR9RA==v3Ah
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Us

Hello,

On Wed 25 Aug 2021 at 12:00PM +02, Simon Richter wrote:

Hi,

On 8/25/21 1:21 AM, Sean Whitton wrote:

From my point of view, signing git tags is no less well established a
best practice than signing tarballs -- in fact, to me, it seems *more*
well established.

That is ecosystem dependent.

Yes, that was my point. We're going to have upstreams who release
tarballs and upstreams who release tags for some time.

--
Sean Whitton

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmEpUWUZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQEVzD/4vPt0TBtp5Y5Pm7fPuEtlu rim0pqjI+gsCD66BVZtjvYNhCmOyyNeK97vuNR35VB1LEedvQPvBO+hQ0A6o3pfr WtCUxwIYVMxfhzL22Ou6ossMT9oPAmsvXMu6j6OkZRO/TKxX2bxBblTFDxz8mGcm m6cGtYli0JajjWjjHVdkuz2oLap4A2fSQ24SV7zrQc8w5+yoG8u8r0Jti0ekGz9q VVhnh2UoRHk7O9vCpWSAe16VlmAnZS/yzoOXCGTxVjlxMDxYkjp1/PAsa1XjhY/I pqkT4zSZJViSFiWmTmPaN4B2KeoWgbu8R2VbYZ9fdIQX+ehh+DydUOve1w0AmCEw uawzkPJOA7D+9Qbv2gJzbIQhW6kaSofDd53kJjPUwuVLuv8I4cYXbxUoLY3wZISw 3zk0w7IRlYO5gAAEnyyq/bKJCvRMnAo3AIH749c0HT9r6Wsn65PPa5ZByOLP1iEf Og6OZyVLbLcFTM36p3I9wSbnRON2bDVkCRiYaBZGoEDbbDko5VZV+spUHKNimxjM x3PaQ+SD+NjhP/DSY2J94phojzEBhdXXNmUFOp59RjJXmpjk6NRXYd7eScfg3kQB JbjDsyuH20BJhCeINrvU93K5zJ+lhCqfLccCwE0C2pzMT2fA9Uwr8K0ttXijSrIU be7xjAQRtFHYAqnbLfttHA==+c3S
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Us

Hi,

On 27.08.21 22:56, Sean Whitton wrote:

That is ecosystem dependent.

Yes, that was my point. We're going to have upstreams who release
tarballs and upstreams who release tags for some time.

My expectation for that state would be "indefinitely", and I don't see
that as a bad thing, we should be able to handle either in our tooling,
and we don't need to inflict a certain workflow on upstreams, and I
don't see the autoconf+automake folks deviate from "make distcheck" as a release process anytime soon.

Where I do see us pushing upstreams though is towards "making releases
at all, and committing to supporting them." The number of packages with
version numbers like "0.0.20190812+git01234567" is too damn high.

Simon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

git archive --prefix=$(DEBPKGNAME)-$(VERSION)/ $(GIT_TAG) \
| xz >../$(DEBPKGNAME)_$(VERSION).orig.tar.xz

I think you should add +ds version suffix or similar to indicate
repacking for Debian. Does it still make sense provided that upstream
does not care much of tarballs?


-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQRm7llN8yxifaG60cF2qh9JI3wlQUCYSulbwAKCRAF2qh9JI3w lSmHAPsHJTSmbUah74er9qCFFejxEIfpu1/+AWxO1H5qy0DYFAD/VMkoPPmeETZe QZ5+L3ifDuQDSnewCdk22m8cfJp35AQ=
=ohss
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: linux.debian.maint.perl

On Mon, 16 Aug 2021 09:18:34 +0800, Paul Wise wrote:

I noticed that sometimes Debian's choice of upstream source for
packaging can be suboptimal. This is especially apparent for the
different per-language upstream packaging ecosystems[1], where the
upstream packaging differs from the upstream VCS in some significant
ways, including missing files, prebuilt files, embedded copies etc.

While the upstream VCS also sometimes has these issues, it is often
much less problematic than the upstream packaging ecosystems.

I'd like to suggest that we standardise on the upstream VCS for our orig.tar.gz files and phase out use of upstream packaging ecosystems.

[…]

1. the ecosystems I'm talking about include cargo, npm, browser
extensions, rubygems, pypi, CPAN etc.

Sorry for being a bit late to the party; as you mention CPAN here, I
thought I'd share some thoughts about it.
(We briefly discussed the topic at the pkg-perl BoF during DebConf
[0] but this is not an offical team statement.)

I see the advantages of the proposal but I think for the perl
ecosystem it doesn't make a whole lot of sense, for two reasons:

- First, CPAN and Debian are quite similar (for better or worse :));
not only about the same age but for both projects the canonical
way of distribution is via tarballs from mirrors - the Debian
archive and the CPAN mirror network. And for both project there is
no requirement to use any VCS or even less a specific one or a
specific hosting for a VCS.
- Second, using only the VCS of a CPAN distribution is not ideal
because it misses information which is created at release time and
which we rely on. So taking the code from an upstream repo
basically means doing part of a release ourselves.

In general, the above mentioned problems of discrepancies between
upstream VCS (if they exist) and upstream tarballs are minor or close
to non-existant in the CPAN world. Hence switching to a VCS-based
approach wouldn't really solve any actual problem in almost all cases
and would create challenges for our tools and workflows.

There are exceptions where we do use the upstream VCS as the tarball
indeed contains undesirable artifacts; and we agreed in the BoF that
improving our tooling to work from a VCS instead of tarballs would be
nice.


Cheers,
gregor


[0] https://lists.debian.org/debian-perl/2021/08/msg00013.html

--
.''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
: :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
`. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
`-

-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEE0eExbpOnYKgQTYX6uzpoAYZJqgYFAmFQNpZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEQx RTEzMTZFOTNBNzYwQTgxMDREODVGQUJCM0E2ODAxODY0OUFBMDYACgkQuzpoAYZJ qgbX8BAAl6QPajp/xIZA/+6L+YoNHM2PN40EHmwRCWRjFkLvGeEbW3iJPM2w8iy5 gbmwp041ODAEi6RClLbLbhPUyXWIpL2/4KgIiL+szSNGM2USjabyvSx2yHQOMZMw ps4u6nM+z9VMC78i4kFMMDqU7lCjx+AZzvdXqxx8lHm6MK/Pw3AMF+thsM+ucJas zrkenwznYHjhKVwPsnJzPnUq6RGtT88AVmiDnKWqqC5+JKhJ+s7EigkaBv1+5HnB AIxy8DkIkYsk0ShcMUEIiASSMfdPUwI0CcXURunFxK2KtgzNV13c36NV6UuyXOj7 5LpZ+sLl6vX5vxZsF81NRmYouQrXbAsjvBav5C3NkdGSSya2JbsKs56DNbuys+4V s211T5X0UM2LIbfNjnwy/OLahwS4N8iy3Jx0ypSVrvGgKNboPpz53sFtzJNz9xLf i7i8Wn3HgudhYUwoJJH/oB0foIlxmoQsyL6xutfMe2OdntwL0G9NRFqzGJiJODAv
yKc5nYns