About two years ago, Guillem Jover added support for -ffile-prefix-map
to dpkg [1] (thanks!).
Related discussion [2] concluded that the option
should be disabled by default and enabled after Reproducible Builds had trialed it for a while. Having enabled it in some of my recent
packaging, Iām curious whether it may be time to switch this flag to be enabled by default. What do you think?
I believe maintainers can override this flag in their packages if
needed? If so, I'd be inclined to explore setting it by default!
On 2020-06-03, Benjamin Barenblat wrote:
Related discussion [2] concluded that the option
should be disabled by default and enabled after Reproducible Builds had trialed it for a while. Having enabled it in some of my recent
packaging, Iām curious whether it may be time to switch this flag to be enabled by default. What do you think?
It helps maybe hundreds of packages, and we've identified a small (16 at
the moment) number packages for which it causes issues:
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
But most of those packages still FTBFS on bullseye and buster where we
don't set the flag (or vary the build path)...
There may also be others that fail but haven't been identified.
I recall this thread where it broke test suites of some packages:
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html
Vagrant Cascadian wrote:
I believe maintainers can override this flag in their packages if
needed? If so, I'd be inclined to explore setting it by default!
Regarding maintainer control over this I have two remarks to make. To
answer Vagrant's question of whether they can override this (ie.
disable it in the case of it becoming a default) then unless I am
missing something that would be possible via the usual dpkg-buildflags mechanism.
Secondly (and related to the previous remark as well as being another
reason to enable this soon) many maintainers are explicitly enabling
the fixfilepath feature area of dpkg-buildflags right now as we
incentivise making their package appear reproducible on the
Reproducible Build's testing framework.
My point here is that if were to enable this flag anyway, doing it
before yet more packages add this boilerplate would avoid even more cluttering of debian/rules files. (Many packages are still parsing debian/changelog and manually exporting SOURCE_DATE_EPOCH.)
I recall this thread where it broke test suites of some packages:
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html
But I assume this would have been detected while building?
Most of which have already been done now in the repro infra, I think? So
it'd just mostly be a matter of someone sending a note to d-d proposing
the change, providing the numbers, waiting a bit in case someone has concerns, then I'm happy to flip the default. :)
On 2020-06-03, Benjamin Barenblat wrote:
Related discussion [2] concluded that the option
should be disabled by default and enabled after Reproducible Builds had
trialed it for a while. Having enabled it in some of my recent
packaging, Iām curious whether it may be time to switch this flag to be
enabled by default. What do you think?
It helps maybe hundreds of packages
and we've identified a small (16 at the moment) number packages for
which it causes issues:
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
But most of those packages still FTBFS on bullseye and buster where we
don't set the flag (or vary the build path)...
There may also be others that fail but haven't been identified.
I recall this thread where it broke test suites of some packages:
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html
I believe maintainers can override this flag in their packages if
needed? If so, I'd be inclined to explore setting it by default!
Hi Lucas!
I've appreciated all the work you've been doing with archive-wide
rebuilds of Debian over the years!
I'm looking into proposing to enable dpkg's DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default, further
background in this thread:
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20200601/012375.html
We would like to figure out how many packages it causes to FTBFS
(usually test suites expecting full paths); we've identified a small
handful through tests.reproducible-builds.org:
https://tests.reproducible-builds.org/debian/issues/bullseye/ftbfs_due_to_f-file-prefix-map_issue.html
https://tests.reproducible-builds.org/debian/issues/bullseye/ffile_prefix_map_passed_to_clang_issue.html
Those issues were identified through manually checking the build logs,
and our reproducible builds infrastructure is not great for isolating
out the effects of specific changes as each build has several variations tested.
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
automatically retry with -fixfilepath
, so that
we can flag which packages build fine without too much worry about false positives and in an automated fashion.
I've never experimented with anything on an archive-wide scale before,
so would appreciate any ideas, tooling or other help you might be able
to offer. Thanks!
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I've appreciated all the work you've been doing with archive-wide
rebuilds of Debian over the years!
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental)
with that enabled by default? And ideally a script similar to https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
to install that package?
automatically retry with -fixfilepath
That would be the current default, right? So a rebuild with vanilla
unstable would work as well?
Thanks for the quick response! More in-line below.
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I've appreciated all the work you've been doing with archive-wide
rebuilds of Debian over the years!
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental) with that enabled by default? And ideally a script similar to https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
to install that package?
I could build and upload an updated dpkg somewhere...
I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:
sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' /usr/share/perl5/Dpkg/Vendor/Debian.pm
Is this an acceptible way forward?
On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I've appreciated all the work you've been doing with archive-wide
rebuilds of Debian over the years!
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental)
with that enabled by default? And ideally a script similar to
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10 >> > to install that package?
I could build and upload an updated dpkg somewhere...
I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:
sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' /usr/share/perl5/Dpkg/Vendor/Debian.pm
Is this an acceptible way forward?
Ah, yes, totally!
Unrelated question: those rebuilds would only be for arch:any packages, right? I can't think of a reason to rebuild arch:all for this test, but
maybe I'm missing something.
Could you provide a dpkg package in a private repo (or in experimental) with that enabled by default? And ideally a script similar to https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10I could build and upload an updated dpkg somewhere...
to install that package?
On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental)
with that enabled by default? And ideally a script similar to
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10 >> > to install that package?
I could build and upload an updated dpkg somewhere...
I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:
sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' /usr/share/perl5/Dpkg/Vendor/Debian.pm
Is this an acceptible way forward?
Ah, yes, totally!
Proposed merge request adding a script that does just that:
https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
Vagrant Cascadian wrote:
Proposed merge request adding a script that does just that:
https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
2+ weeks on, any update on this? I keep coming across packages that
would be fixed by fixfilepath.
Hi,
On 07/09/20 at 20:42 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental)
with that enabled by default? And ideally a script similar to
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
to install that package?
I could build and upload an updated dpkg somewhere...
I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:
sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' /usr/share/perl5/Dpkg/Vendor/Debian.pm
Is this an acceptible way forward?
Ah, yes, totally!
Proposed merge request adding a script that does just that:
https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
In http://qa-logs.debian.net/2020/09/26.fixfilepath/ you can find:
00cmp.fixfilepath.only-fails-with-fixfilepath.txt: that's the list of packages that fail with fixfilepath, but don't fail without it.
00res.fixfilepath.txt: the full list of built packages
00cmp.fixfilepath.txt: the differences with a normal build
... and the logs for the packages listed in the first file.
Let me know if you need something else.
On 2020-08-31, Lucas Nussbaum wrote:
On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Could you provide a dpkg package in a private repo (or in experimental) >> > with that enabled by default? And ideally a script similar to
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10 >> > to install that package?
I could build and upload an updated dpkg somewhere...
I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:
sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' /usr/share/perl5/Dpkg/Vendor/Debian.pm
Is this an acceptible way forward?
Ah, yes, totally!
Proposed merge request adding a script that does just that:
https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
On 07/09/20 at 20:42 -0700, Vagrant Cascadian wrote:...
On 2020-08-31, Lucas Nussbaum wrote:
On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
On 2020-08-31, Lucas Nussbaum wrote:
On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:...
I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure,
Proposed merge request adding a script that does just that:
https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
In http://qa-logs.debian.net/2020/09/26.fixfilepath/ you can find:
00cmp.fixfilepath.only-fails-with-fixfilepath.txt: that's the list of packages that fail with fixfilepath, but don't fail without it.
At a quick glance, looks to be 33 packages, many of which were already identified in by our tags:
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
I've just added the missing ones, and will follow-up with bugs and
hopefully patches in the coming weeks. It is a pretty reasonable number
of packages to tackle.
At a quick glance, looks to be 33 packages, many of which were already
identified in by our tags:
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
I've just added the missing ones, and will follow-up with bugs and
hopefully patches in the coming weeks. It is a pretty reasonable number
of packages to tackle.
Vagrant, can you briefly elaborate on your plan of action here? In particular, at what point do you plan to request enabling
-ffile-prefix-map distribution-wide? I can think of a number of
potential answers, so would be interested to learn your current
intentions.
On 2020-10-15, Chris Lamb wrote:
At a quick glance, looks to be 33 packages, many of which were already
identified in by our tags:
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
I've just added the missing ones, and will follow-up with bugs and
hopefully patches in the coming weeks. It is a pretty reasonable number
of packages to tackle.
Vagrant, can you briefly elaborate on your plan of action here? In
particular, at what point do you plan to request enabling
-ffile-prefix-map distribution-wide? I can think of a number of
potential answers, so would be interested to learn your current
intentions.
I need to follow-up with bug reports for the packages that aren't
triggered by clang. Was hoping to have that done by now; it's on my
priority list... I would guess this is in the ballpark of 10-20 more
bugs. Of course, anyone can file those bug reports...
Once llvm-defaults updates the default clang, I think nearly half of the FTBFS issues will likely disappear, so don't think this is worth filing
for those bugs, presuming llvm-defaults gets updated soon.
The stupid fix for the remaining packages is to disable this feature for
a handful of packages, e.g. in debian/rules:
DEB_BUILD_MAIN_OPTIONS=reproducible=-fixfilepath,+fixdebugpath
(or maybe just -fixfilepath, depending on how the dpkg patch lands)
So, looking at:...
https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F
We're nearly ready to take it to debian-devel for comments...
I guess, at risk of a small breach of procedure, it would be worth
starting the conversation on debian-devel *now* even though the
remaining bugs aren't yet filed just to get the process started (while continuing to file the remaining bugs)... to hopefully make it in time
for bullseye.
All of the patches I submitted were merely disable fixfilepath, though
they could all use further exploration to find the root cause and a more ideal fix, but given time constraints and the small number of affected packages, I figured this was a reasonable short-term workaround.
Guillem (and all the other dpkg folks), do you see any impediment to
raising the question of enabling fixfilepath by default with the
Debian community at large?
I don't see either an issue with whether the bugs have been filed or
not, but I think that's just part of the conversation with
debian-devel at large (which I doubt would be a matter of contention
there either).
Guillem (and all the other dpkg folks), do you see any impediment to
raising the question of enabling fixfilepath by default with the
Debian community at large?
All of the patches I submitted were merely disable fixfilepath, though
they could all use further exploration to find the root cause and a more
ideal fix, but given time constraints and the small number of affected
packages, I figured this was a reasonable short-term workaround.
Thanks, Vagrant. I would agree that is a good workaround for now and
probably shouldn't prevent us from making this (IMHO) long-overdue
change.
Do you have a timeline on when you plan to raise this on debian-devel?
On 2020-10-22, Chris Lamb wrote:
Guillem (and all the other dpkg folks), do you see any impediment to
raising the question of enabling fixfilepath by default with the
Debian community at large?
All of the patches I submitted were merely disable fixfilepath, though
they could all use further exploration to find the root cause and a more >>> ideal fix, but given time constraints and the small number of affected
packages, I figured this was a reasonable short-term workaround.
Thanks, Vagrant. I would agree that is a good workaround for now and
probably shouldn't prevent us from making this (IMHO) long-overdue
change.
Do you have a timeline on when you plan to raise this on debian-devel?
Hope to craft and send it tomorrow after the Reproducible Builds IRC
meeting.
https://pad.sfconservancy.org/p/-NUkgkGAezz0wc7n54el
Feel free to comment or even edit directly.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 393 |
Nodes: | 16 (2 / 14) |
Uptime: | 35:11:13 |
Calls: | 8,256 |
Files: | 13,132 |
Messages: | 5,877,346 |