"Fabian" == Fabian Grünbichler <debian@fabian.gruenbichler.email> writes:

Fabian> On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:
>> Hi,
>>
>> I'm writing this mail in order to get further input from
>> knowledgeable, but not directly involved DDs - mostly those
>> involved with cross-building and multi-arch matters.

Fabian> sorry for the noise - now with correct CC of
Fabian> debian-cross@lists.debian.org, instead of the mis-pasted
Fabian> d-cross@ ..

A couple of questions that aren't obvious to me:

1) Is my understanding generally correct that it mostly doesn't matter
if you get the wrong architecture's package for a rust library package,
since it's the same source code in both cases?

2) When does it matter which architecture's packages you get for rust
library packages?

3) When do rust library packages have dependencies on other kinds of packages--packages that have more than source code?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

--7a3e4de7bfe14198999917a6b9310bae
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:

Hi,

I'm writing this mail in order to get further input from knowledgeable, but not directly involved DDs - mostly those involved with cross-building and multi-arch matters.

sorry for the noise - now with correct CC of debian-cross@lists.debian.org, instead of the mis-pasted d-cross@ ..

Some background for those not familiar with rust packaging in Debian, skip below for the actual examples and question.

The debian-rust team uses a rather unified workflow to build a large amount of rust library crates and a not-quite-as-large amount of binary crates/applications. Each source package (rust-foo) corresponds to a single crate (rust package) released on

crates.io, building one or more librust-foo*-dev binary packages that actually contain the crate's (patched) source code, and in case of a "bin" crate shipping an actual program/application, a binary package containing the compiled executable + support
files (which can have an arbitrary name - usually just that of the program in question). The reason that the library packages ship source code in a binary package is that rust doesn't (properly) support rust->rust dynamic linking (yet), so each rust
executable is statically linking all it's rust dependencies. This has the side-effect that what are actually (transitive) build-dependencies are encoded as a mix of build-dependencies (direct) and regular dependencies (transitive dependencies of direct
build-dependencies).

E.g., a crate foo that depends on a crate bar which in turn depends on a crate baz will have the foo->bar relationship encoded as build-dependency (rust-foo -> librust-bar-dev) and dependency (librust-foo-dev -> librust-bar-dev, for packages [build-]

depending on crate foo), while the bar->baz dependency will *only* be encoded as regular dependency (librust-bar-dev -> librust-baz-dev) in rust-foo's dependency tree.

debcargo, the tool used by the Debian rust team to streamline the work of transforming upstream crates into Debian source packages generates d/control entries for librust-* binary packages qualified as arch:any and M-A:same, in order to support cross

compilation. This solution was suggested by dpkg developers (Guillem?) according to the following comment left in the debcargo source code:

// This is the best but not ideal option for us.
//
// Currently Debian M-A spec has a deficiency where a package X that
// build-depends on a (M-A:foreign+arch:all) package that itself
// depends on an arch:any package Z, will pick up the BUILD_ARCH of
// package Z instead of the HOST_ARCH. This is because we currently
// have no way of telling dpkg to use HOST_ARCH when checking that the
// dependencies of Y are satisfied, which is done at install-time
// without any knowledge that we're about to do a cross-compile. It
// is also problematic to tell dpkg to "accept any arch" because of
// the presence of non-M-A:same packages in the archive, that are not
// co-installable - different arches of Z might be depended-upon by
// two conflicting chains. (dpkg has so far chosen not to add an
// exception for the case where package Z is M-A:same co-installable).
//
// The recommended work-around for now from the dpkg developers is to
// make our packages arch:any M-A:same even though this results in
// duplicate packages in the Debian archive. For very large crates we
// will eventually want to make debcargo generate -data packages that
// are arch:all and have the arch:any -dev packages depend on it.

https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342

Some time ago, Jonas (CCed) started packaging rust crates in the same "namespace" (src:rust-foo building bin:librust-foo-*) using a slightly different approach, but in a *mostly* compatible fashion. More recently, Jonas started switching over his rust

packages from arch:any, M-A:same (like the rust team's) to arch:all, M-A:foreign. There was no agreement between the Debian rust team and Jonas whether this change is problematic and should be reverted (or not), after a bit of discussion we agreed on
getting feedback from people more involved with cross-building efforts (hence this mail).

One example of a working in practice (as in, the build doesn't fail), but technically wrong (cross-)build (pulls in dependencies from wrong architecture) can be found here:

https://paste.debian.net/1270516/

The chain is

rust-lscolors (arch:any) --BD--> librust-tempfile-dev (arch:any) librust-tempfile-dev (arch:any) --D--> librust-remove-dir-all-0.7+default-dev (arch:any) --D--> librust-log-0.4+default-dev (arch:any) --D--> librust-value-bag-1.0.0+serde-dev (arch:any) --D--> librust-serde-fmt-1+default-dev (arch:all) --D--> librust-

serde-dev (arch:any, but wrongly resolved!)

with librust-serde-fmt-dev being switched to arch:all, it will resolve to :amd64 instead of :i386 and in turn pull in its own dependencies librust-serde-1+std-dev & librust-serde-1-dev (both provided by librust-serde-dev) from amd64 (in addition to

librust-serde-dev:i386, which is also part of the arch:any (build-)dependency tree of rust-lscolors itself, without the "hop" over librust-serde-fmt-dev).

src:rust-lscolors was taken as minimal example bin crate here - many rust binary crates are affected by this issue since a few crates which are part of common dependency chains are maintained by Jonas and recently got switched to arch:all (serde-fmt

and ahash are the two most prominent). Note that much of the rust ecosystem relies on a crate called bindgen to generate rust bindings for C libraries, which in turn leverages clang under the hood, which unfortunately makes those packages currently not
cross-compilable because libclang-common-14-dev is not co-installable on multiple architectures. Many/most rust-*-sys packages and crates depending on those are practically not cross-compilable for this reason (in addition to pulling in packages from the
wrong architecture via arch:all, as described above).

Another example would be src:rust-hashbrown, which directly build-depends on librust-ahash-0.7-dev which got switched to arch:all, cross building it on an amd64 machine for i386 with

DEB_BUILD_OPTIONS='' sbuild -c debian-unstable-amd64-sbuild -d unstable --host i386 --profiles cross rust-hashbrown_0.12.3-1.dsc

will pull in all librust-* packages from the wrong architecture (am64 instead of i386):

$ grep -E '^Get.*librust-serde' rust-hashbrown_0.12.3-1_i386.build

Get:144 http://deb.debian.org/debian unstable/main amd64 librust-cfg-if-dev amd64 1.0.0-1 [10.4 kB]
Get:145 http://deb.debian.org/debian unstable/main amd64 librust-libc-dev amd64 0.2.139-1 [290 kB]
[ .. snip lots of librust-*-dev amd64 downloads ..]
Get:170 http://deb.debian.org/debian unstable/main amd64 librust-version-check-dev amd64 0.9.4-1 [15.9 kB]
Get:171 http://deb.debian.org/debian unstable/main amd64 librust-ahash-0.7-dev all 0.7.6-11 [477 kB]

Reverting librust-ahash-0.7-dev to arch:any, M-A:same (and injecting both the resulting amd64 and i386 package into the build env, to override the arch:all one from the archive) makes the cross-build correctly only pull in librust-*-dev packages from

i386.

TL;DR: Is the switch to arch:all one that should be reverted in the face of it apparently breaking cross builds? Or is there another alternative (nowadays) that makes the "workaround" employed by debcargo no longer needed?

Depending on feedback, the rust team would either ask Jonas to switch back his packages to arch:any, M-A:same (probably after bookworm, to prevent further fallout/need for RMs/.. during the freeze), or will evaluate whether switching to arch:all is an

option for debcargo-managed packages as well, and which changes on the team tooling side are needed to avoid losing test coverage or increasing friction.

Thanks for reading and any informed input,
Fabian

--7a3e4de7bfe14198999917a6b9310bae
Content-Type: text/html;charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><head><title></title><style type="text/css"> p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div>On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:<br></div><blockquote type="cite" id="qt" style=""><div>Hi,<br></div><div><br></div><div>I'm writing this mail in order to get
further input from knowledgeable, but not directly involved DDs - mostly those involved with cross-building and multi-arch matters.<br></div></blockquote><div><br></div><div>sorry for the noise - now with correct CC of <a href="mailto:debian-cross@lists.
debian.org">debian-cross@lists.debian.org</a>, instead of the mis-pasted d-cross@ ..<br></div><div><br></div><blockquote type="cite" id="qt" style=""><div><br></div><div>Some background for those not familiar with rust packaging in Debian, skip below for
the actual examples and question.<br></div><div><br></div><div>The debian-rust team uses a rather unified workflow to build a large amount of rust library crates and a not-quite-as-large amount of binary crates/applications. Each source package (rust-foo)
corresponds to a single crate (rust package) released on crates.io, building one or more librust-foo*-dev binary packages that actually contain the crate's (patched) source code, and in case of a "bin" crate shipping an actual program/application, a
binary package containing the compiled executable + support files (which can have an arbitrary name - usually just that of the program in question). The reason that the library packages ship source code in a binary package is that rust doesn't (properly)
support rust-&gt;rust dynamic linking (yet), so each rust executable is statically linking all it's rust dependencies. This has the side-effect that what are actually (transitive) build-dependencies are encoded as a mix of build-dependencies (direct) and
regular dependencies (transitive dependencies of direct build-dependencies).<br></div><div><br></div><div>E.g., a crate foo that depends on a crate bar which in turn depends on a crate baz will have the foo-&gt;bar relationship encoded as build-
dependency (rust-foo -&gt; librust-bar-dev) and dependency (librust-foo-dev -&gt; librust-bar-dev, for packages [build-]depending on crate foo), while the bar-&gt;baz dependency will *only* be encoded as regular dependency (librust-bar-dev -&gt; librust-
baz-dev) in rust-foo's dependency tree.<br></div><div><br></div><div>debcargo, the tool used by the Debian rust team to streamline the work of transforming upstream crates into Debian source packages generates d/control entries for librust-* binary
packages qualified as arch:any and M-A:same, in order to support cross compilation. This solution was suggested by dpkg developers (Guillem?) according to the following comment left in the debcargo source code:<br></div><div><br></div><pre class="qt-code
qt-highlight" lang="rust"><span id="qt-LC343" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// This is the best but not ideal option for us.</span></span></span>
<span id="qt-LC344" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment">//</span></span>
<span id="qt-LC345" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// Currently Debian M-A spec has a deficiency where a package X that</span></span></span>
<span id="qt-LC346" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// build-depends on a (M-A:foreign+arch:all) package that itself</span></span></span>
<span id="qt-LC347" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment">// depends on an arch:any package Z, will pick up the BUILD_ARCH of</span></span>
<span id="qt-LC348" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// package Z instead of the HOST_ARCH. This is because we currently</span></span></span>
<span id="qt-LC349" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// have no way of telling dpkg to use HOST_ARCH when checking that the</span></span></span>
<span id="qt-LC350" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// dependencies of Y are satisfied, which is done at install-time</span></span></span>
<span id="qt-LC351" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// without any knowledge that we're about to do a cross-compile. It</span></span></span>
<span id="qt-LC352" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// is also problematic to tell dpkg to "accept any arch" because of</span></span></span>
<span id="qt-LC353" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// the presence of non-M-A:same packages in the archive, that are not</span></span></span>
<span id="qt-LC354" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment">// co-installable - different arches of Z might be depended-upon by</span></span>
<span id="qt-LC355" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// two conflicting chains. (dpkg has so far chosen not to add an</span></span></span>
<span id="qt-LC356" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// exception for the case where package Z is M-A:same co-installable).</span></span></span>
<span id="qt-LC357" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment">//</span></span>
<span id="qt-LC358" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// The recommended work-around for now from the dpkg developers is to</span></span></span>
<span id="qt-LC359" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// make our packages arch:any M-A:same even though this results in</span></span></span>
<span id="qt-LC360" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// duplicate packages in the Debian archive. For very large crates we</span></span></span>
<span id="qt-LC361" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// will eventually want to make debcargo generate -data packages that</span></span></span>
<span id="qt-LC362" class="qt-line" lang="rust"><span class="qt-"> </span><span class="qt-hljs-comment"><span class="qt-hljs-comment">// are arch:all and have the arch:any -dev packages depend on it.</span></span></span><br></pre><div><br></

<div><a href="https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342">https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342</a><br></div><div><br></div><div>Some time ago, Jonas (CCed)

started packaging rust crates in the same "namespace" (src:rust-foo building bin:librust-foo-*) using a slightly different approach, but in a *mostly* compatible fashion. More recently, Jonas started switching over his rust packages from arch:any, M-A:
same (like the rust team's) to arch:all, M-A:foreign. There was no agreement between the Debian rust team and Jonas whether this change is problematic and should be reverted (or not), after a bit of discussion we agreed on getting feedback from people
more involved with cross-building efforts (hence this mail).<br></div><div><br></div><div>One example of a working in practice (as in, the build doesn't fail), but technically wrong (cross-)build (pulls in dependencies from wrong architecture) can be
found here:<br></div><div><br></div><div><a href="https://paste.debian.net/1270516/">https://paste.debian.net/1270516/</a><br></div><div><br></div><div>The chain is<br></div><div><br></div><div>rust-lscolors (arch:any) --BD--&gt; librust-tempfile-dev (
arch:any)<br></div><div>librust-tempfile-dev (arch:any) --D--&gt; librust-remove-dir-all-0.7+default-dev (arch:any) --D--&gt; librust-log-0.4+default-dev (arch:any)&nbsp;--D--&gt; librust-value-bag-1.0.0+serde-dev (arch:any)&nbsp;--D--&gt; librust-serde-
fmt-1+default-dev (arch:all) --D--&gt; librust-serde-dev (arch:any, but wrongly resolved!)<br></div><div><br></div><div>with librust-serde-fmt-dev being switched to arch:all, it will resolve to :amd64 instead of :i386 and in turn pull in its own
dependencies librust-serde-1+std-dev &amp; librust-serde-1-dev (both provided by librust-serde-dev) from amd64 (in addition to librust-serde-dev:i386, which is also part of the arch:any (build-)dependency tree of rust-lscolors itself, without the "hop"
over librust-serde-fmt-dev).<br></div><div><br></div><div>src:rust-lscolors was taken as minimal example bin crate here - many rust binary crates are affected by this issue since a few crates which are part of common dependency chains are maintained by
Jonas and recently got switched to arch:all (serde-fmt and ahash are the two most prominent). Note that much of the rust ecosystem relies on a crate called bindgen to generate rust bindings for C libraries, which in turn leverages clang under the hood,
which unfortunately makes those packages currently not cross-compilable because libclang-common-14-dev is not co-installable on multiple architectures. Many/most rust-*-sys packages and crates depending on those are practically not cross-compilable for
this reason (in addition to pulling in packages from the wrong architecture via arch:all, as described above).<br></div><div><br></div><div>Another example would be src:rust-hashbrown, which directly build-depends on librust-ahash-0.7-dev which got
switched to arch:all, cross building it on an amd64 machine for i386 with<br></div><div><br></div><div>DEB_BUILD_OPTIONS='' sbuild -c debian-unstable-amd64-sbuild -d unstable --host i386 --profiles cross rust-hashbrown_0.12.3-1.dsc<br></div><div><br></

<div>will pull in all librust-* packages from the wrong architecture (am64 instead of i386):<br></div><div><br></div><div>$ grep -E '^Get.*librust-serde' rust-hashbrown_0.12.3-1_i386.build<br></div><div><br></div><div>Get:144 <a href="http://deb.

debian.org/debian">http://deb.debian.org/debian</a> unstable/main amd64 librust-cfg-if-dev amd64 1.0.0-1 [10.4 kB]<br></div><div>Get:145 <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> unstable/main amd64 librust-libc-dev amd64 0.
2.139-1 [290 kB]<br></div><div>[ .. snip lots of librust-*-dev amd64 downloads ..]<br></div><div>Get:170 <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> unstable/main amd64 librust-version-check-dev amd64 0.9.4-1 [15.9 kB]<br></

<div>Get:171 <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> unstable/main amd64 librust-ahash-0.7-dev all 0.7.6-11 [477 kB]<br></div><div><br></div><div>Reverting librust-ahash-0.7-dev to arch:any, M-A:same (and injecting

both the resulting amd64 and i386 package into the build env, to override the arch:all one from the archive) makes the cross-build correctly only pull in librust-*-dev packages from i386.<br></div><div><br></div><div>TL;DR: Is the switch to arch:all one
that should be reverted in the face of it apparently breaking cross builds? Or is there another alternative (nowadays) that makes the "workaround" employed by debcargo no longer needed?<br></div><div><br></div><div>Depending on feedback, the rust team
would either ask Jonas to switch back his packages to arch:any, M-A:same (probably after bookworm, to prevent further fallout/need for RMs/.. during the freeze), or will evaluate whether switching to arch:all is an option for debcargo-managed packages as
well, and which changes on the team tooling side are needed to avoid losing test coverage or increasing friction.<br></div><div><br></div><div>Thanks for reading and any informed input,<br></div><div>Fabian<br></div></blockquote><div><br></div></body></
html>
--7a3e4de7bfe14198999917a6b9310bae--

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On February 15, 2023 8:57:12 PM GMT+01:00, Sam Hartman <hartmans@debian.org> wrote:

"Fabian" == Fabian Grünbichler <debian@fabian.gruenbichler.email> writes:

Fabian> On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:
>> Hi,
>>
>> I'm writing this mail in order to get further input from
>> knowledgeable, but not directly involved DDs - mostly those
>> involved with cross-building and multi-arch matters.

Fabian> sorry for the noise - now with correct CC of
Fabian> debian-cross@lists.debian.org, instead of the mis-pasted
Fabian> d-cross@ ..

A couple of questions that aren't obvious to me:

1) Is my understanding generally correct that it mostly doesn't matter
if you get the wrong architecture's package for a rust library package,
since it's the same source code in both cases?

For librust-*, yes, unless something requires special treatment they'll be identical across the architectures, and such special circumstances are usually limited to bin crates.

2) When does it matter which architecture's packages you get for rust
library packages?

Same as above - unless the package in question has some custom rules/install files, it never should, except for transitive effects on stuff outside of librust-*.

3) When do rust library packages have dependencies on other kinds of >packages--packages that have more than source code?

This mainly happens with crates (dynamically) linking to C libraries (or C bindings to libraries written in other languages). By convention these end in -sys, and when packaged for Debian, will mostly generate the needed Rust binding code at build time
using the bindgen crate/tool - but build time here means both building of the rust-foo-sys crate itself, as well as any (transitive!) reverse dependency (again, because the Rust part will be statically linked).

Like I said in the original mail, bindgen is currently not working in a cross compilation scenario because the required clang packages don't seem to be co-installable (rustc which is the cross compiler in that case itself also depends on libllvm).

There are other cases as well, like -sys crates with manually written bindings that don't use bindgen, or other code that uses arch-specific non Rust things that might not be there/in the wrong path/have the wrong content when a package from the wrong
arch is pulled in. E.g. besides the clang-related failures there's also one involving cython that I haven't fully analyzed yet. And of course, there might be things that compile/build but are in fact broken.

All in all it seems to me like the problem currently is more a theoretical one - it doesn't seem to cause (much, if at all) additional breakage on top of stuff that is already not cross compilable at the moment for other reasons. It does seem like a step
in the wrong direction though.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Fabian" == Fabian Grünbichler <debian@fabian.gruenbichler.email> writes:

Fabian> All in all it seems to me like the problem currently is more
Fabian> a theoretical one - it doesn't seem to cause (much, if at
Fabian> all) additional breakage on top of stuff that is already not
Fabian> cross compilable at the moment for other reasons. It does
Fabian> seem like a step in the wrong direction though.

It seems like if a librust-* package depended on some system library it
had a binding for, that's going to be a problem once the binding
generation works cross-compiled. Because if I'm understanding correctly
once you manage to get onto the wrong architecture's dependencies, you
will continue to use the wrong architecture.

So, I'd like to be able to say use arch all packages except for the
things that will break if they are not arch all.
But it sounds like you'd need to make the transitive **reverse
dependencies** arch any as well, and that's a complete mess.

So, it seems like options are:

1) Decide that we'll wait until dpkg to improve for rust to be cross
compilable

or

2) decide that arch any is needed.

However, adding a question to the mix, why does arch all work for go
packages?
Do they not care about cross compilation, or are concerns somehow
different there?

--Sam

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

On Wed, Feb 15, 2023, at 7:51 PM, Fabian Grünbichler wrote:

debcargo, the tool used by the Debian rust team to streamline the work
of transforming upstream crates into Debian source packages generates d/control entries for librust-* binary packages qualified as arch:any
and M-A:same, in order to support cross compilation. This solution was suggested by dpkg developers (Guillem?) according to the following
comment left in the debcargo source code:

Hmm, I had no recollection of having had such conversation or given
that recommendation. Checking the history for that comment, and the
IRC logs for #debian-dpkg around the same time, seems like that was
discussed there and apparently resolved, and I guess I mostly skimmed afterwards over the conversation and mentioned I was glad it had been
resolved (but where I don't recall having spent time pondering about
it :).

// This is the best but not ideal option for us.
//
// Currently Debian M-A spec has a deficiency where a package X that
// build-depends on a (M-A:foreign+arch:all) package that itself
// depends on an arch:any package Z, will pick up the BUILD_ARCH of
// package Z instead of the HOST_ARCH. This is because we currently
// have no way of telling dpkg to use HOST_ARCH when checking that the
// dependencies of Y are satisfied, which is done at install-time
// without any knowledge that we're about to do a cross-compile. It
// is also problematic to tell dpkg to "accept any arch" because of
// the presence of non-M-A:same packages in the archive, that are not
// co-installable - different arches of Z might be depended-upon by
// two conflicting chains. (dpkg has so far chosen not to add an
// exception for the case where package Z is M-A:same co-installable).
//
// The recommended work-around for now from the dpkg developers is to
// make our packages arch:any M-A:same even though this results in
// duplicate packages in the Debian archive. For very large crates we
// will eventually want to make debcargo generate -data packages that
// are arch:all and have the arch:any -dev packages depend on it.

https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342

I'm afraid after a quick read, I'm failing to understand what this
comment is trying to convey. Some of the things it's saying seem like
would apply to frontends driving dpkg for example, others I'm not sure
what they refer to.

From my reread of the IRC logs, it seems like the thing this was trying
to avoid was the "multiarch interpreter problem", but then w/o having
really put much thought at all into this right now (so take this with
a huge grain of salt), this does not feel to me like the same issue
when it comes to arch:all packages shipping sources to be used to
build the actual binaries. But again take this more like me having
currently no opinion on it right now w/o investing some time to think
this thoroughly, than anything else.

I think Helmut is probably at a better position to at least give the
rationale he had at the time for this.

Thanks,
Guillem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-02-15 15:37 -0700, Sam Hartman wrote:

"Fabian" == Fabian Gr�nbichler <debian@fabian.gruenbichler.email> writes:

Fabian> All in all it seems to me like the problem currently is more
Fabian> a theoretical one - it doesn't seem to cause (much, if at
Fabian> all) additional breakage on top of stuff that is already not
Fabian> cross compilable at the moment for other reasons.

I admit to not having studied this properly, but I think this is the
(farily) well-known 'multiarch interpreter' problem, when any arch-all
package in the dependency chain 'loses' the inherited 'host-arch'
value so all dependency resolution further down the chain is wrong.

This doesn't matter if everything further down the chain _is_
arch-independent, but as soon as you hit something where the arch
matters (e.g. a package which links to a C-library), stuff breaks.

It seems like if a librust-* package depended on some system library it
had a binding for, that's going to be a problem once the binding
generation works cross-compiled. Because if I'm understanding correctly
once you manage to get onto the wrong architecture's dependencies, you
will continue to use the wrong architecture.

Correct.

However, adding a question to the mix, why does arch all work for go packages?

Good question. I don't know anything about go, but I assume it has the
same problem as rust here, because it's also compiled and statically
linked. Maybe we package it defferently so there is never an arch-all dep above an arch-any dep in the dependency chain?

But in general this problem applies equally to any language where a dependency tree can have
arch-any
arch-all
arch-any
in the stack (and the bottom one needs to be the host arch)

The rust form of this is slightly different because of the unusual 'source-included' scheme that rust's inability to dynamic-link has
forced upon us. But (unless I misunderstand) the fundamental issue is
the same.

The ultimate problem here is that multiarch was designed with a 'C'
mindset, so the assumption that architecture-dependencies would
propagate down the chain (or things were arch-independent all the way
further down) was implicit. The
actually-quite-common-in-other-languages pattern where things were
_mostly_ arch-independent (due to being an interpreted language, for
example), but some random library in the dependency-chain (at least 2
levels down) might actually be a wrapper for an arch-specific
C-library, was not considered in the design.

The 'fix' (a workaround) is to make any packages which appear as the
arch-all intermediaries in dep-chains to be made arch-any so that
chain is not broken. We can either do that for all packages in an
ecosystem, or just for ones that actually exhibit this problem. Doing
it for all is safer.

HTH

Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEER4nvI8Pe/wVWh5yq+4YyUahvnkcFAmPtaBgACgkQ+4YyUahv nkdGvQ//epgTpT876v0kMUupWpdDmh9kNXq4QvP47hSziDgw3HqQBSAC/f4gLKHj YDcvDqMG3lZp/DVyYpvryw4vJBz6XYAGlBQ1PrSrptYz3P9B3sNhvN9LlA2xqwnT ZKB3qNlEtOQyWdHWlOYF06/pY6D9Hftgf/Pa6pJC7pvyJRP9yACU7vFWqvChxCHH BAgqzmdSq+LYQATtqvXxy1dWuPbbyy4D0cEkZiwHbBBIUGhoScHcAb49DDSDHIm7 dYeGuEUcdipAZXzRcsRih+GsTK8rk9fRQgHRtlWoJo1nR/i1M0F22uEJRe28V01p t491KuXgSZTEE8V5Y12oVQk2XbeO89gUcZeTf6mP4RG1h3qxNX3SxOMwamQJpVal 2nniux2UKohq56f9QBq3jgxx1i0/C+KfXy297t2CJmWYwm7eBVmoB4UjG90tyRE+ G8J2VItvEdn70YneKs58N7lof/ukBu6xyvUtx/VYuQJgcFYVf4KFr7isR+DCBrV2 j+uOa6X7lxALtxWMz0K7yRZnr2Jlgn085yv921Ny4iG7clNFkXF/LpyZsxS2/Awp 8IadqJ70boStq/xJrtHpjxDrRpXRICqpJYSILUwaDvFx4TLDoEq68GO7VDNJ6ieW /B1MvieDLgNxrW3yTnuQK+q+ha9EtO8g2n1ZmU6Twm86zePX+lc=
=oiid
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Sam Hartman <hartmans@debian.org> 于 2023年2月16日周四 06:37写道：

However, adding a question to the mix, why does arch all work for go packages?
Do they not care about cross compilation, or are concerns somehow
different there?

Go doesn't work. I guess we started from the common sense that
arch-indepent files should be arch:all. Or at that time Multi-Arch is not widely recognized.

(There is also a cross compile related patch in golang-defaults that
haven't been merged yet for some years.)

(Sent from my mobile device)

<div dir="auto"><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Sam Hartman &lt;<a href="mailto:hartmans@debian.org">hartmans@debian.org</a>&gt; 于 2023年2月16日周四 06:37写道：</div><blockquote class="gmail_quote" style="margin:
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
However, adding a question to the mix, why does arch all work for go<br> packages?<br>
Do they not care about cross compilation, or are concerns somehow<br>
different there?</blockquote></div></div><div dir="auto"><br></div><div dir="auto">Go doesn&#39;t work. I guess we started from the common sense that arch-indepent files should be arch:all. Or at that time Multi-Arch is not widely recognized.</div><div
dir="auto"><br></div><div dir="auto">(There is also a cross compile related patch in golang-defaults that haven&#39;t been merged yet for some years.)</div><div dir="auto"><br></div><div dir="auto">(Sent from my mobile device)</div></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Shengjing Zhu <zhsj@debian.org> 于 2023年2月16日周四 09:16写道：

Sam Hartman <hartmans@debian.org> 于 2023年2月16日周四 06:37写道：

However, adding a question to the mix, why does arch all work for go
packages?
Do they not care about cross compilation, or are concerns somehow
different there?

Go doesn't work. I guess we started from the common sense that
arch-indepent files should be arch:all. Or at that time Multi-Arch is not widely recognized.

I should say Go doesn't work when an arch:any (usually a C library) is involved. But well most Go programs are pure Go which are not affected by
this all-any trap.

(Sent from my mobile device)

<div dir="auto"><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">Shengjing Zhu &lt;<a href="mailto:zhsj@debian.org">zhsj@debian.org</a>&gt; 于 2023年2月16日周四 09:16写道：<br></div><blockquote class="gmail_quote" style="
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Sam Hartman &lt;<a href="mailto:hartmans@debian.org" target="_blank" rel="noreferrer">hartmans@debian.org</a>&
gt; 于 2023年2月16日周四 06:37写道：</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
However, adding a question to the mix, why does arch all work for go<br> packages?<br>
Do they not care about cross compilation, or are concerns somehow<br>
different there?</blockquote></div></div><div dir="auto"><br></div><div dir="auto">Go doesn&#39;t work. I guess we started from the common sense that arch-indepent files should be arch:all. Or at that time Multi-Arch is not widely recognized.</div></div><
/blockquote></div><div dir="auto"><br></div><div dir="auto">I should say Go doesn&#39;t work when an arch:any (usually a C library) is involved. But well most Go programs are pure Go which are not affected by this all-any trap.</div><div dir="auto"><br></

<div dir="auto">(Sent from my mobile device)</div><div class="gmail_quote" dir="auto"></div></div>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Quoting Helmut Grohne (2023-02-20 18:12:14)

On Wed, Feb 15, 2023 at 08:14:17PM +0100, Fabian Grünbichler wrote:

TL;DR: Is the switch to arch:all one that should be reverted in the face of it apparently breaking cross builds? Or is there another alternative (nowadays) that makes the "workaround" employed by debcargo no longer needed?

Yes, the switch to arch:all needs to be reverted at least partially.

Thanks, Fabian, for bringing this up, and thanks Helmut for the
clarification.

Could it be that Jonas picked up such hints and thus added m-a:foreign?
Then later a C binding was added to the mix, which broke the hint, but
the conversion wasn't reverted.

No, I was not guided by m-a hinter rules (assuming you mean the kind
of hints that appear in the "action needed" section of e.g. https://tracker.debian.org/pkg/rust-test-case ) when I did the switch,
so I am not aware that that hinting are broken.

This suggests that the hinter should gain another condition to never
issue arch:all+m-a:foreign conversion hints for librust-*-dev packages.
Do you agree?

I don't see a need for such general rule for all librust-*-dev packages.

E.g. librust-futures-timer-dev and librust-event-listener-dev has no dependencies so should be safe to distribute as arch-all and marked as m-a:foreign, right?

Sure, there is a theoretical risk that some day in the future those
packages grow dependencies that require changing that, but isn't that
the case generally for m-a hinting?

I guess it makes sense instead to have a lintian warning for packages
marked m-a:foreign that all its dependencies are equally marked (or if
that's unreasonably expensive then check only one level deep). That
should help flag misuses of m-a:foreign including this one of mine (for
most but not all the librust-*-dev packages that I maintain).

- Jonas

--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/
* Sponsorship: https://ko-fi.com/drjones

[x] quote me freely [ ] ask before reusing [ ] keep private --==============w47395366962028281=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEn+Ppw2aRpp/1PMaELHwxRsGgASEFAmPzsXQACgkQLHwxRsGg ASELpQ//TWjGLSDZOq6hNLJ/Ul4KwI7iuRemavu4M3LvPee3By4cPXeFvC7y5Goh diHS4FatoqWALLD7kyyrmvqbUx2SK917DACYt6PqPHb1mj+TJRlGc2NfeoH9mWCw c/Pj3TUVSvJsx/42DlHnEAP9efBzID+DSeO9AAh6t9elnedAnNaHNuWdCoIVlGv1 sO6g5CCKWwVw7H9UHYrNyt9S/aWSwYyUvC9aoCXYlNpRkfsMhJIYdbX3ZjW+p3gL LLsvZDradReXB16CKjvm/q3pN9X8DI3GvVunpgJyyBg8U9TBYkjsMzz5r9dSapHg eakmaCqcXgL9HnXQ3A3WkLJWo78Mfj5Ghl5cbr4q7EsAAE9NSvjvtI9ZtDSL+9Sw BXBvq0XL6Za6B7cMW9UjVpRQfHwrAAUXeOt6R1L/YS5gNSZiOo16nH79e677rqoG G+CvrI+zzJzdKHuIF4YMvKCZKfGhWgLM3UAjWpWg

Hi Fabian,

On Wed, Feb 15, 2023 at 08:14:17PM +0100, Fabian Gr�nbichler wrote:

// This is the best but not ideal option for us.
//
// Currently Debian M-A spec has a deficiency where a package X that
// build-depends on a (M-A:foreign+arch:all) package that itself
// depends on an arch:any package Z, will pick up the BUILD_ARCH of
// package Z instead of the HOST_ARCH. This is because we currently
// have no way of telling dpkg to use HOST_ARCH when checking that the
// dependencies of Y are satisfied, which is done at install-time
// without any knowledge that we're about to do a cross-compile. It
// is also problematic to tell dpkg to "accept any arch" because of
// the presence of non-M-A:same packages in the archive, that are not
// co-installable - different arches of Z might be depended-upon by
// two conflicting chains. (dpkg has so far chosen not to add an
// exception for the case where package Z is M-A:same co-installable).
//
// The recommended work-around for now from the dpkg developers is to
// make our packages arch:any M-A:same even though this results in
// duplicate packages in the Debian archive. For very large crates we
// will eventually want to make debcargo generate -data packages that
// are arch:all and have the arch:any -dev packages depend on it.

https://salsa.debian.org/rust-team/debcargo/-/blob/master/src/debian/control.rs#L342

I guess that I was the one giving the advice that was recorded here
although I agree with Guillem that it is not recorded in a very
consumable way.

The chain is

rust-lscolors (arch:any) --BD--> librust-tempfile-dev (arch:any) librust-tempfile-dev (arch:any) --D--> librust-remove-dir-all-0.7+default-dev (arch:any) --D--> librust-log-0.4+default-dev (arch:any) --D--> librust-value-bag-1.0.0+serde-dev (arch:any) --D--> librust-serde-fmt-1+default-dev (arch:all) --D-->

librust-serde-dev (arch:any, but wrongly resolved!)

with librust-serde-fmt-dev being switched to arch:all, it will resolve to :amd64 instead of :i386 and in turn pull in its own dependencies librust-serde-1+std-dev & librust-serde-1-dev (both provided by librust-serde-dev) from amd64 (in addition to

librust-serde-dev:i386, which is also part of the arch:any (build-)dependency tree of rust-lscolors itself, without the "hop" over librust-serde-fmt-dev).

The analysis is well done. Thank you. This is a real example of why we
do this annoying arch:any+m-a:same dance.

TL;DR: Is the switch to arch:all one that should be reverted in the face of it apparently breaking cross builds? Or is there another alternative (nowadays) that makes the "workaround" employed by debcargo no longer needed?

Yes, the switch to arch:all needs to be reverted at least partially.

No, there is no better way to achieve this. We extensively discussed alternatives in Vaumarcus and reached the conclusion that no possible
interface was acceptable to dpkg. In essence, dpkg could automatically
track the architectures of packages (which would incur a recursive
dependency resolver onto dpkg) or architecture tracking could be a
manual thing, which would be far too annoying to end users. Therefore,
there is no sane alternative to the workaround.

Depending on feedback, the rust team would either ask Jonas to switch back his packages to arch:any, M-A:same (probably after bookworm, to prevent further fallout/need for RMs/.. during the freeze), or will evaluate whether switching to arch:all is

an option for debcargo-managed packages as well, and which changes on the team tooling side are needed to avoid losing test coverage or increasing friction.

Technically, not all packages need to become arch:any+m-a:same. As you correctly observed, the problem arises from non-rust dependencies in the dependency tree. For instance, librust-typenum-dev has no dependencies.
As such, it could be changed to arch:all+m-a:foreign without incurring
problems to cross compilation. Likewise, librust-bitmaps-dev only
depends on librust-typenum-1+default-dev, which is provided by librust-typenum-dev. Consequently, it could be converted to arch:all+m-a:foreign as well. However, as soon as you reach C bindings,
such as librust-typenum-dev, the workaround to use arch:any+m-a:same is
needed. Then any library that transitively depends on any C binding also
needs the workaround. The problem here is that lower layers can
introduce bindings at a later time and thus break cross compilation
involving a higher level library. Whether a library can safely use arch:all+m-a:foreign is dependent on its entire dependency tree. For
that reason, using arch:all+m-a:foreign is quite a good way to shoot
yourself in the foot.

Let me try to distill a rule of thumb from this: Whenever you require a
package to transfer an architecture constraint to downstream
dependencies (i.e. one of them likely is a C binding), arch:all package
cannot be used.

This actually is one of the m-a hinter rules. It suggests a conversion
from arch:any to arch:all+m-a:foreign whenever three conditions are met:
* The contents are equal on all release architectures.
* There are no maintainer scripts.
* All of the dependencies are marked m-a:foreign or annotated :any.

Could it be that Jonas picked up such hints and thus added m-a:foreign?
Then later a C binding was added to the mix, which broke the hint, but
the conversion wasn't reverted.

This suggests that the hinter should gain another condition to never
issue arch:all+m-a:foreign conversion hints for librust-*-dev packages.
Do you agree?

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jonas,

On Mon, Feb 20, 2023 at 06:44:23PM +0100, Jonas Smedegaard wrote:

I don't see a need for such general rule for all librust-*-dev packages.

This definitely is a trade-off. Hence I asked about what we should do
about it.

E.g. librust-futures-timer-dev and librust-event-listener-dev has no dependencies so should be safe to distribute as arch-all and marked as m-a:foreign, right?

Yeah, I mentioned librust-typenum-dev earlier and you found more. I
expect that there are probably hundreds of pure rust libraries without dependencies. Really, the only reason to not have them arch:all is
mitigating the risk of changing dependency trees.

Sure, there is a theoretical risk that some day in the future those
packages grow dependencies that require changing that, but isn't that
the case generally for m-a hinting?

You are entirely correct about this. It definitely is a heuristic.

I guess it makes sense instead to have a lintian warning for packages
marked m-a:foreign that all its dependencies are equally marked (or if
that's unreasonably expensive then check only one level deep). That
should help flag misuses of m-a:foreign including this one of mine (for
most but not all the librust-*-dev packages that I maintain).

lintian does not have the full picture. It looks at one source package
plus all of its binary packages in isolation. It cannot see the
multi-arch annotations of dependencies. Hence the multi-arch hinter was
added as an extra component outside lintian. On the flip side, the
hinter could do the flagging that you suggest here. In essence, it could
flag all packages satisfying all of the following conditions:
* Architecture: all
* Multi-Arch: foreign
* Package: librust*-dev
* Depends: some package that is not m-a:foreign

We can actually express this in SQL:

SELECT name
FROM indeppackage AS i
WHERE name LIKE 'librust%-dev'
AND multiarch = 'foreign'
AND EXISTS (SELECT 1 FROM archdepcandidate AS c WHERE c.dependerid = i.id);

I guess you're also interested in the output of the query.

librust-crdts-dev
librust-ahash-dev
librust-assert-json-diff-dev
librust-async-attributes-dev
librust-async-channel-dev
librust-async-executor-dev
librust-async-fs-dev
librust-async-net-dev
librust-async-process-dev
librust-async-std-dev
librust-blocking-dev
librust-femme-dev
librust-flume-dev
librust-hyper-rustls-dev
librust-kv-log-macro-dev
librust-lazy-regex-dev
librust-leptonica-plumbing-dev
librust-leptonica-sys-dev
librust-nom-locate-dev
librust-oxilangtag-dev
librust-resource-proof-dev
librust-rustls-native-certs-dev
librust-rusty-pool-dev
librust-serde-fmt-dev
librust-smol-dev
librust-socks-dev
librust-tesseract-plumbing-dev
librust-tesseract-sys-dev
librust-test-case-dev
librust-version-sync-dev
librust-criterion-dev
librust-oxhttp-dev
librust-oxiri-dev
librust-retry-dev
librust-rustls-dev
librust-xor-name-dev
librust-nanorand-dev
librust-threadfin-dev
librust-ahash-0.7-dev
librust-criterion-0.3-dev
librust-isahc-dev
librust-ureq-dev

As a confirmation of Fabian's observation, this list includes librust-serde-fmt-dev. Note that this is not transitive yet. After
fixing these, some reverse dependencies will be added to the list. Also
note that running this query on delfin.d.o cannot be easily done by DDs
yet, because the underlying sqlite database uses wal mode. Sorry about
that.

Should the hinter generate reverse hints for these and thus issue action
items on tracker.d.o?

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)