Hi!

As mentioned on the mail about the dpkg-build-api, buildflags and the
current interface they are exposed through have become problematic over
time. There are at least three areas which I think need to be reworked
or improved, which I'm listing below, and for which I'll send separate
mails to split the topics (two now and the last one later today or so):

* «*_FOR_BUILD» flags handling
* Multiple toolchains support
* Versioned buildflags interface

If there are other problems or sub-items in these, I'm also interested,
but in any case I'd appreciate feedback on any of the above, as for
some I have a rough idea how to go about, which might be flawed, and
in other cases I'm not so sure.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

[ See <https://lists.debian.org/debian-dpkg/2023/11/msg00031.html>. ]

This problem is related to the *_FOR_BUILD support (to specify flags for
the build instead of host system during cross-building), which got
recently partially improved (in dpkg 1.22.1), but there are still things
not covered.

These are at least two of the sub-problems that need to be tackled:

* Need to make features per-arch in output (such as --query), but that
might break users not expecting repeated entries. I guess the options
here might be to either:

a) try to look for users of the interfaces and make them cope (but
that would miss internal/private usage),
[ If starting from scratch I'd rather do this, but it has now
the potential for breakage. ]
b) version the interface via a CLI option,
[ This would be a general solution that would allow extensions in
the future, but complicates the interface. ]
c) add an explicit --for-build mode which would turn the output into
the *_FOR_BUILD features,
[ This is a targeted straightforward change, and a clear interface,
but it involves running the tool twice to get all the info. ]
d) or use different field keys for the paragraphs.
[ This is rather meh. ]

None of which seem ideal (although some seem less desirable) TBH.

* Need to honor build features, either from DEB_BUILD_OPTIONS or a
new envvar (and finding a non-confusing name!), but probably only for
a few of the features where it might really make sense (such as abi,
qa, hardening.pie) and which might affect the buildability or
runability of the artifacts, as anything else seems unnecessary.

I'm not certain about the envvar part, I think the last part is in
theory straightforward, the code just needs to duplicate the checks
per the various arches, and keep track of the various features per arch
internally, which might require breaking changes in the interfaces,
but I think the perl modules are not public interfaces anyway.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

[ See <https://lists.debian.org/debian-dpkg/2023/11/msg00031.html>. ]

This problem is related to the current (dpkg) policy where flags target
the current default compiler and version for the specific vendor, but
some of the flags emitted are toolchain specific and can break when
used with another toolchain (say gcc vs clang for example).

It should be possible to tell the buildflags machinery what toolchain
to use and get appropriate build flags for that (some example requests
for this could be #782039, #900830 or #1055605, and the effort to build
Debian with clang).

I guess the options are either introducing a global toolchain setting,
for example a new CLI option and say global DEB_BUILD_TOOLCHAIN or
similar, or one per flag type, for example DEB_CFLAGS_TOOLCHAIN or DEB_FCFLAGS_FOR_BUILD_TOOLCHAIN. But this would need to contain a
generic term such as «gcc» or «clang», instead of a specific command
to use such as «g++-13», or «clang-17».

The problem with a global selector is that maintainers (or tooling
such as debhelper) might need to call dpkg-buildflags multiple times
in case the build system uses different toolchains, or each different
flag might in fact use a different toolchain by default already. So
I'm inclined to go with a per flag selector.

(For reference rpm has a similar toolchain selector concept, but AFAICT
it uses a global selector.)

This also ties with the variables for the actual toolchain and specific compiler tools being used (such as CC or CXX), where there is currently
only partial support for this in the dpkg Makefile fragments, but not
in dpkg-buildpackage or dpkg-buildflags proper. Ideally these would be
set by dpkg-buildpackage so the tools and flags would be coherent, and
as a side effect its arch detection from the toolchain would then be
also consistent with the arch specified (#644664), but this cannot be
done as debian/rules is still the supported entry point for builds.
Setting this by dpkg-buildflags would be an option, and it is heavily
related to its purpose, but it's a bit off relative to its name, but
oh well. So I'm pondering to add support for setting compiler variables
too (such as CC or CXX) in dpkg-buildflags, although that would only
fix one aspect of the overall tool vs flags coherence.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Nov 30, 2023 at 01:35:17PM +0100, Guillem Jover wrote:

This problem is related to the current (dpkg) policy where flags target
the current default compiler and version for the specific vendor, but
some of the flags emitted are toolchain specific and can break when
used with another toolchain (say gcc vs clang for example).

As far as I can see, there currently is no support for toolchain
selection nor toolchain-dependent compiler flags. To put this another
way: Anyone who wants to build with a non-default toolchain not only has
to supply CC and friens but also CFLAGS and friends. While this is very inconvenient, it readily works today.

I guess the options are either introducing a global toolchain setting,
for example a new CLI option and say global DEB_BUILD_TOOLCHAIN or
similar, or one per flag type, for example DEB_CFLAGS_TOOLCHAIN or DEB_FCFLAGS_FOR_BUILD_TOOLCHAIN. But this would need to contain a
generic term such as «gcc» or «clang», instead of a specific command
to use such as «g++-13», or «clang-17».

I'm not sure whether per-setting toolchain actually is that useful in
the end. I would expect most users of this to want to change them all at
once. For instance, changing CC without to clang without also updating
CFLAGS accordingly is doomed to failure. The global setting very much
makes sense to me.

However, I think that both users building a package and maintainers of a package may want to express a preference here much like compiler flags
have a "MAINT" variant that can be used from debian/rules without
overriding a user choice.

The problem with a global selector is that maintainers (or tooling
such as debhelper) might need to call dpkg-buildflags multiple times
in case the build system uses different toolchains, or each different
flag might in fact use a different toolchain by default already. So
I'm inclined to go with a per flag selector.

Given that the use case of multiple toolchains is rare, I see little
problems with the requirement of calling dpkg-buildflags multiple times.
This doesn't seem like a use case we want to optimize for.

This also ties with the variables for the actual toolchain and specific compiler tools being used (such as CC or CXX), where there is currently
only partial support for this in the dpkg Makefile fragments, but not
in dpkg-buildpackage or dpkg-buildflags proper. Ideally these would be
set by dpkg-buildpackage so the tools and flags would be coherent, and
as a side effect its arch detection from the toolchain would then be
also consistent with the arch specified (#644664), but this cannot be
done as debian/rules is still the supported entry point for builds.

Can we eventually challenge debian/rules being a supported entry point? Practically, nobody does this.

Setting this by dpkg-buildflags would be an option, and it is heavily
related to its purpose, but it's a bit off relative to its name, but
oh well. So I'm pondering to add support for setting compiler variables
too (such as CC or CXX) in dpkg-buildflags, although that would only
fix one aspect of the overall tool vs flags coherence.

This comes with another wrinkle. What do you set CC to when your
toolchain is clang and your host architecture differs from your build architecture? I think the only correct way is to include the -target
option in the CC variable and that's going to trip up a lot of uses.
Until clang provides <triplet>-clang symlinks (which clang already
interprets correctly as far as I know), I don't clang as a supportable toolchain given our interface. Also clang supporting pkgconf seems like
a requirement to me to become a first-class toolchain in Debian. Fixing
its many Multi-Arch: same violations also kinda feels like a requirement
to me. So I think, clang very much is not up to the task and needs
significant improvement in order to be considered. The current state is
so broken that it only works in exceptional circumstances by chance. I'd
object to it becoming easily selectable by maintainers given the current problems.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Guillem,

On Thu, Nov 30, 2023 at 01:17:37PM +0100, Guillem Jover wrote:

[ See <https://lists.debian.org/debian-dpkg/2023/11/msg00031.html>. ]

This problem is related to the *_FOR_BUILD support (to specify flags for
the build instead of host system during cross-building), which got
recently partially improved (in dpkg 1.22.1), but there are still things
not covered.

These are at least two of the sub-problems that need to be tackled:

I am not sure whether these problems actually need solving, but saying
this I may curse myself in some years. ;)

* Need to make features per-arch in output (such as --query), but that
might break users not expecting repeated entries. I guess the options
here might be to either:

a) try to look for users of the interfaces and make them cope (but
that would miss internal/private usage),
[ If starting from scratch I'd rather do this, but it has now
the potential for breakage. ]
b) version the interface via a CLI option,
[ This would be a general solution that would allow extensions in
the future, but complicates the interface. ]
c) add an explicit --for-build mode which would turn the output into
the *_FOR_BUILD features,
[ This is a targeted straightforward change, and a clear interface,
but it involves running the tool twice to get all the info. ]
d) or use different field keys for the paragraphs.
[ This is rather meh. ]

e) dpkg-architecture -a${DEB_BUILD_ARCH} -f -c dpkg-buildflags --query-feature $SOMEFEATURE

This is roughly the same as c) and works today.

None of which seem ideal (although some seem less desirable) TBH.

* Need to honor build features, either from DEB_BUILD_OPTIONS or a
new envvar (and finding a non-confusing name!), but probably only for
a few of the features where it might really make sense (such as abi,
qa, hardening.pie) and which might affect the buildability or
runability of the artifacts, as anything else seems unnecessary.

I'm not certain about the envvar part, I think the last part is in
theory straightforward, the code just needs to duplicate the checks
per the various arches, and keep track of the various features per arch
internally, which might require breaking changes in the interfaces,
but I think the perl modules are not public interfaces anyway.

In general, using a different variable than DEB_BUILD_OPTIONS seems a
difficult sell to me. For instance, DEB_BUILD_OPTIONS=noopt affects
build and host and to me that seems vaguely sane. Beyond that, I concur
with there being little need for configurability here.

Maybe this is just me being unimaginative.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-11-30 21:10 +0100, Helmut Grohne wrote:

[lots of sensible things]

I'm not sure whether per-setting toolchain actually is that useful in
the end. I would expect most users of this to want to change them all at once. For instance, changing CC to clang without also updating
CFLAGS accordingly is doomed to failure. The global setting very much
makes sense to me.

agreed

Can we eventually challenge debian/rules being a supported entry point? Practically, nobody does this.

That's not entirely true. I still do 'debian/rules clean' a lot and 'debian/rules binary' quite often. But I agree that if we need to run dpkg-buildpackage to set up the right environment, then declaring that
to be the suppported interface is reasonable at this point. I do
recall 'debian/rules binary' being more reliable than
'dpkg-buildpackage -nc -T binary' in the past, but I forget why.

This comes with another wrinkle. What do you set CC to when your
toolchain is clang and your host architecture differs from your build architecture? I think the only correct way is to include the -target
option in the CC variable and that's going to trip up a lot of uses.
Until clang provides <triplet>-clang symlinks (which clang already
interprets correctly as far as I know), I don't class clang as a supportable toolchain given our interface.

Yes this has been a problem for many years and no-one has shown much
interest in fixing it over the last decade or so. Providing the
symlinks at least shouldn't be very hard?


Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEER4nvI8Pe/wVWh5yq+4YyUahvnkcFAmVqma8ACgkQ+4YyUahv nkeDvg//cjQhiyLMr4Dpa+Nx854FHczeVpQTDdihkjTzw/azLjI5ppD/jwP2x7ZO IbMqK3lkuByNB63x2VfyCPS2jnGmJJsVW3ra03pjixZtsHrosr/+CmeLyxDLm2lz +oZU5kR0Syf4HVnvf3zVsIJwfcYHI6jkq3Ftb2g5dRn5TOprbDyJuWyGUbE6Ns60 HZIcad73agTDNeuRBja2X7oBBB7amUForXr6Mfhr9llGuVZ7QqFXG1mOpe/S9rdm fVY+NX9NgMbFgV372QisUxBZzSuTKeeNqOrYSG3ott1JynMRbxKG16zZKoBRhCnF P0AotzgtF2abG5aXK5il3VhFZ64xYYCBpZFpAjxWW7H0adxrK9YBiH338pMknkbz +sCF2kiP16q/6MFRh0UF3Lnu/EFx6aPfKS20rrTJPZ2U7VRanlbb3AmZw9AkD1B+ hLt49TxAezXKY8WaW+qJFBbHq+xrIY2TMAqsE2gU8efofoDVJ/gUURtr/0zmQw9B w5vF3Fcq4q55xnF/bSf/bCPbsPVCsjEr9S5m37W3ENFkuMf5p391Q/PQ7jCgVWpV rVT/A41rq8SjT6EB9UIUOXgWwj92BGFFpXCUhCDWsiTaW7jKyNOVek5Mbp/hzLPB ZxuQg6zG0FjLlDBwEwkeHb+SiIx+aprJr5Rf0F3XTIey4U2RqlQ=
=MKJF
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)