Several Cargo-based ebuilds cannot use cargo_src_install for various
reasons and manually install binaries from within the target directory
instead. It is common to see `target/$(usex debug debug release)`, but
this lacks the target ABI when cross-compiling, so provide a helper
function.

There are some multilib Cargo-based ebuilds that always set the target
ABI, even when not cross-compiling. It would be simpler to do this in
general, so once ebuilds have been updated to use this new helper, I
might change the eclass again accordingly.

Signed-off-by: James Le Cuirot <chewi@gentoo.org>
---
eclass/cargo.eclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 40d98211ce7f9..2036fb635d569 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -319,6 +319,16 @@ _cargo_gen_git_config() {
fi
}

+# @FUNCTION: cargo_target_dir
+# @DESCRIPTION:
+# Return the directory within target that contains the build, e.g.
+# target/aarch64-unknown-linux-gnu/release.
+cargo_target_dir() {
+ local abi
+ tc-is-cross-compiler && abi=$(rust_abi)
+ echo "${CARGO_TARGET_DIR:-target}${abi+/}${abi}/$(usex debug debug release)"
+}
+
# @FUNCTION: cargo_src_unpack
# @DESCRIPTION:
# Unpacks the package and the cargo registry.
--
2.45.1

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Jun 13, 2024 at 04:03:44PM +0100, James Le Cuirot wrote:

Several Cargo-based ebuilds cannot use cargo_src_install for various
reasons and manually install binaries from within the target directory instead. It is common to see `target/$(usex debug debug release)`, but
this lacks the target ABI when cross-compiling, so provide a helper
function.

There are some multilib Cargo-based ebuilds that always set the target
ABI, even when not cross-compiling. It would be simpler to do this in general, so once ebuilds have been updated to use this new helper, I
might change the eclass again accordingly.

Signed-off-by: James Le Cuirot <chewi@gentoo.org>
---
eclass/cargo.eclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 40d98211ce7f9..2036fb635d569 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -319,6 +319,16 @@ _cargo_gen_git_config() {
fi
}

+# @FUNCTION: cargo_target_dir
+# @DESCRIPTION:
+# Return the directory within target that contains the build, e.g.
+# target/aarch64-unknown-linux-gnu/release.
+cargo_target_dir() {
+ local abi
+ tc-is-cross-compiler && abi=$(rust_abi)
+ echo "${CARGO_TARGET_DIR:-target}${abi+/}${abi}/$(usex debug debug release)"
+}
+

How about giving this function an optional parameter that gets passed to rust_abi so that you can get the build directory for targets other than
CHOST?

What initially motivated this suggestion is a quirk in app-misc/anki's
build system together with your recent change to cargo.eclass 375da3684,
which enables cross-compilation by setting environment variables.
Here, I first build the 'build system runner' for CBUILD, which then
compiles the actual package for CHOST. I think I can avoid explicitly
setting --target $(rust ${CBUILD}) for the runner for now, but it would
be handy to easily get the build directory for CBUILD in the future!

--
Lucio Sauer

# @FUNCTION: cargo_src_unpack
# @DESCRIPTION:
# Unpacks the package and the cargo registry.
--
2.45.1

-----BEGIN PGP SIGNATURE-----

iQGTBAABCgB9FiEElFlueg0TS/aEyc5qVWii3aYMopcFAmZrZX9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk0 NTk2RTdBMEQxMzRCRjY4NEM5Q0U2QTU1NjhBMkREQTYwQ0EyOTcACgkQVWii3aYM opc9zAf8Ckn8Uy9NJRj5hXDq+dk6xGYg1QL5N2cEDi5fu0hlcUBDtdOfSbJIqQ8B PxpkOj+1/gKtL+YvcDjJ32vT/Xz9QIgVrJRCTrc25RqYHUf6LF8/pSvTHLpfLQWS CDcYZr5xfNA0hSigoRGxdWTCs0bHyUIcoUsJUB85FO8dRWS7LdCOP37E/mHpkZCv WO1CUptt8mzmwt6R79euRSkfMoa4ug9I/nKy0CYwhfEKLH25jDSeO7yP9tUhG+3G bQNFdzoBE19fhFBaCNswL3eFl62tICuRq5VLbYYv9wNr8hUyACk/XxL2JWXJIvWc apMzfT3PrXZrnnSWO/xxJldDyNQCgA==
=pngx
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 2024-06-13 at 21:32 +0000, Lucio Sauer wrote:

On Thu, Jun 13, 2024 at 04:03:44PM +0100, James Le Cuirot wrote:

Several Cargo-based ebuilds cannot use cargo_src_install for various reasons and manually install binaries from within the target directory instead. It is common to see `target/$(usex debug debug release)`, but
this lacks the target ABI when cross-compiling, so provide a helper function.

There are some multilib Cargo-based ebuilds that always set the target
ABI, even when not cross-compiling. It would be simpler to do this in general, so once ebuilds have been updated to use this new helper, I
might change the eclass again accordingly.

Signed-off-by: James Le Cuirot <chewi@gentoo.org>
---
eclass/cargo.eclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 40d98211ce7f9..2036fb635d569 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -319,6 +319,16 @@ _cargo_gen_git_config() {
fi
}

+# @FUNCTION: cargo_target_dir
+# @DESCRIPTION:
+# Return the directory within target that contains the build, e.g.
+# target/aarch64-unknown-linux-gnu/release.
+cargo_target_dir() {
+ local abi
+ tc-is-cross-compiler && abi=$(rust_abi)
+ echo "${CARGO_TARGET_DIR:-target}${abi+/}${abi}/$(usex debug debug release)"
+}
+

How about giving this function an optional parameter that gets passed to rust_abi so that you can get the build directory for targets other than CHOST?

What initially motivated this suggestion is a quirk in app-misc/anki's
build system together with your recent change to cargo.eclass 375da3684, which enables cross-compilation by setting environment variables.
Here, I first build the 'build system runner' for CBUILD, which then
compiles the actual package for CHOST. I think I can avoid explicitly
setting --target $(rust ${CBUILD}) for the runner for now, but it would
be handy to easily get the build directory for CBUILD in the future!

I did consider that, but for the sake of simplicity, I think I'd prefer just overriding CHOST when you need to do this. There are already similar examples of this.

I had a look at app-misc/anki. I think you do need to give --target to the runner right now if that's building something, otherwise how would it know
what the target is? Perhaps the runner respects CARGO_BUILD_TARGET? If so, I was thinking of making a cargo_env wrapper helper that would set that and
other variables in the context of the given command. This would be used in cargo_src_compile, as well as cargo_src_install and cargo_src_test in case any rebuilding occurs. It could also be used for other custom scenarios like this one. What do you think?

diff --git a/app-misc/anki/anki-24.04.1.ebuild b/app-misc/anki/anki-24.04.1.ebuild
index 54dfbf94f31b2..2764659ccbf51 100644
--- a/app-misc/anki/anki-24.04.1.ebuild
+++ b/app-misc/anki/anki-24.04.1.ebuild
@@ -871,9 +871,9 @@ src_compile() {
# * generate the ninja file and run ninja afterwards
# * create the Python wheel files in "${S}"/out/wheels

- cargo build --release --package runner || die
+ tc-env_build cargo_src_compile --package runner
if use gui; then
- out/rust/release/runner build -- $(get_NINJAOPTS) wheels || die
+ cargo_env "$(CHOST=${CBUILD} cargo_target_dir)/runner" build -- $(get_NINJAOPTS) wheels || die
else
cargo_src_compile --package anki-sync-server
fi
@@ -901,7 +901,7 @@ src_test() {
"${S}"/build/ninja_gen/src/cargo.rs || die

for runner in pytest rust_test jest; do
- out/rust/release/runner build -- $(get_NINJAOPTS) check:$runner || \
+ cargo_env "$(CHOST=${CBUILD} cargo_target_dir)/runner" build -- $(get_NINJAOPTS) check:$runner || \

On Fri, Jun 14, 2024 at 02:54:47PM +0100, James Le Cuirot wrote:

On Thu, 2024-06-13 at 21:32 +0000, Lucio Sauer wrote:

On Thu, Jun 13, 2024 at 04:03:44PM +0100, James Le Cuirot wrote:

Several Cargo-based ebuilds cannot use cargo_src_install for various reasons and manually install binaries from within the target directory instead. It is common to see `target/$(usex debug debug release)`, but this lacks the target ABI when cross-compiling, so provide a helper function.

There are some multilib Cargo-based ebuilds that always set the target ABI, even when not cross-compiling. It would be simpler to do this in general, so once ebuilds have been updated to use this new helper, I might change the eclass again accordingly.

Signed-off-by: James Le Cuirot <chewi@gentoo.org>
---
eclass/cargo.eclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 40d98211ce7f9..2036fb635d569 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -319,6 +319,16 @@ _cargo_gen_git_config() {
fi
}

+# @FUNCTION: cargo_target_dir
+# @DESCRIPTION:
+# Return the directory within target that contains the build, e.g.
+# target/aarch64-unknown-linux-gnu/release.
+cargo_target_dir() {
+ local abi
+ tc-is-cross-compiler && abi=$(rust_abi)
+ echo "${CARGO_TARGET_DIR:-target}${abi+/}${abi}/$(usex debug debug release)"
+}
+

How about giving this function an optional parameter that gets passed to rust_abi so that you can get the build directory for targets other than CHOST?

What initially motivated this suggestion is a quirk in app-misc/anki's build system together with your recent change to cargo.eclass 375da3684, which enables cross-compilation by setting environment variables.
Here, I first build the 'build system runner' for CBUILD, which then compiles the actual package for CHOST. I think I can avoid explicitly setting --target $(rust ${CBUILD}) for the runner for now, but it would
be handy to easily get the build directory for CBUILD in the future!

I did consider that, but for the sake of simplicity, I think I'd prefer just overriding CHOST when you need to do this. There are already similar examples of this.

I had a look at app-misc/anki. I think you do need to give --target to the runner right now if that's building something, otherwise how would it know what the target is?

Cargo builds for the "host architecture" (CBUILD, right?) if no target
is supplied. Since I currently compile the runner before ever calling cargo_src_compile, CARGO_BUILD_TARGET="$(rust_abi) has not been exported
yet and Cargo falls back to compiling for CBUILD. Because this is not future-proof and my current approach is lacking, I'll switch to cargo_src_compile with the upcoming PR.

Perhaps the runner respects CARGO_BUILD_TARGET?

Yes, it does.

If so, I
was thinking of making a cargo_env wrapper helper that would set that and other variables in the context of the given command. This would be used in cargo_src_compile, as well as cargo_src_install and cargo_src_test in case any
rebuilding occurs. It could also be used for other custom scenarios like this one. What do you think?

Sounds like a good idea. Ebuild writers will be able to specify the
platform that cargo_env is supposed to set environment variables for,
right?

diff --git a/app-misc/anki/anki-24.04.1.ebuild b/app-misc/anki/anki-24.04.1.ebuild
index 54dfbf94f31b2..2764659ccbf51 100644
--- a/app-misc/anki/anki-24.04.1.ebuild
+++ b/app-misc/anki/anki-24.04.1.ebuild
@@ -871,9 +871,9 @@ src_compile() {
# * generate the ninja file and run ninja afterwards
# * create the Python wheel files in "${S}"/out/wheels

- cargo build --release --package runner || die
+ tc-env_build cargo_src_compile --package runner

Thank you for bringing this to my attention. I forgot that I need to set
cross vars for cc-rs for the runner to compile in a cross-compilation
scenario.

--
Lucio Sauer

if use gui; then
- out/rust/release/runner build -- $(get_NINJAOPTS) wheels || die
+ cargo_env "$(CHOST=${CBUILD} cargo_target_dir)/runner" build -- $(get_NINJAOPTS) wheels || die
else
cargo_src_compile --package anki-sync-server
fi
@@ -901,7 +901,7 @@ src_test() {
"${S}"/build/ninja_gen/src/cargo.rs || die

for runner in pytest rust_test jest; do
- out/rust/release/runner build -- $(get_NINJAOPTS) check:$runner || \
+ cargo_env "$(CHOST=${CBUILD} cargo_target_dir)/runner" build -- $(get_NINJAOPTS) check:$runner || \
die "check:$runner failed!"
done
}

-----BEGIN PGP SIGNATURE-----

iQGTBAABCgB9FiEElFlueg0TS/aEyc5qVWii3aYMopcFAmZt80NfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk0 NTk2RTdBMEQxMzRCRjY4NEM5Q0U2QTU1NjhBMkREQTYwQ0EyOTcACgkQVWii3aYM opeo9QgAukowBzmHG6oVDh

On Sat, Jun 15, 2024 at 08:14:34PM +0200, Florian Schmaus wrote:

On 13/06/2024 17.03, James Le Cuirot wrote:

Several Cargo-based ebuilds cannot use cargo_src_install for various reasons and manually install binaries from within the target directory instead. It is common to see `target/$(usex debug debug release)`, but
this lacks the target ABI when cross-compiling, so provide a helper function.

Glad that someone is working on better cross-compilation support for
rust. :)

That said, many rust ebuilds generate shell completions by invoking the
just compiled binary in src_install(). This does not work out of the box when cross-compiling.

Is there anything we can or should do about it (besides binfmt_misc)?
For example, asking the according projects to implement a mechanism to extract the completions files from the binary without running it, or
have the build system generate the completion files as result of the build?

All I can think of without changing entirely how they're generated is:
1. (worst) either make a 2nd build using CBUILD (yeah no), or BDEPEND
on same-version self with a USE only when cross compiling
2. (lazy) skip completions w/ ewarn if tc-is-cross-compiler, just so
it at least won't break cross over completion files that the user
may not even care much about
3. generate them yourself with each releases and ship in a tarball,
works out for users but extra burden for maintainer over little
4. ask upstream to provide proper release tarballs if not already and
include pre-generated completions/docs/etc... so it's otherwise
only a problem with arbitrary git checkouts
(3+4 may possibly be more complicated if cargo features / USE change
cli options and thus completion files)

For one of my ebuilds I went with better-than-nothing #2.
--
ionen

-----BEGIN PGP SIGNATURE-----

iQEzBAABCAAdFiEEx3SLh1HBoPy/yLVYskQGsLCsQzQFAmZuDfIACgkQskQGsLCs QzTc9Qf9FitG4Vsi+I57anG6f2ySvqk5kmjDg9Z/1VqE0GPaBw7/nbrnftPtbwbw 5YlTM4SVxKB4+x7SnK1j4mEuG/+P1/RboFX5t4cqLkJ8g5UfcQqTTwyJD28/Bdbh pKiYlo8gcX0YF3PK0eXGpZ7BT1W+iIQZzKndXQq59sw+nIPGUsJdfzQT7DQINstp 8ky79qacjU/Qv08kKVBONuP7FH6TuhtcFUkPjcGJyM+5hlYBtG8KCaWjsCo9gHVo TjltDyKHv4TCYgXnLXWZyrUeKl4Z1nAngBWE7zTa2ciIMiV5fC/0aqZG0T99g/iX ZlsYVOBLwSYzEss+sSiFbaCxfz77VQ==
=+Hwq
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 2024-06-15 at 17:56 -0400, Ionen Wolkens wrote:

On Sat, Jun 15, 2024 at 08:14:34PM +0200, Florian Schmaus wrote:

On 13/06/2024 17.03, James Le Cuirot wrote:

Several Cargo-based ebuilds cannot use cargo_src_install for various reasons and manually install binaries from within the target directory instead. It is common to see `target/$(usex debug debug release)`, but this lacks the target ABI when cross-compiling, so provide a helper function.

Glad that someone is working on better cross-compilation support for
rust. :)

That said, many rust ebuilds generate shell completions by invoking the just compiled binary in src_install(). This does not work out of the box when cross-compiling.

Is there anything we can or should do about it (besides binfmt_misc)?
For example, asking the according projects to implement a mechanism to extract the completions files from the binary without running it, or
have the build system generate the completion files as result of the build?

All I can think of without changing entirely how they're generated is:
1. (worst) either make a 2nd build using CBUILD (yeah no), or BDEPEND
on same-version self with a USE only when cross compiling
2. (lazy) skip completions w/ ewarn if tc-is-cross-compiler, just so
it at least won't break cross over completion files that the user
may not even care much about
3. generate them yourself with each releases and ship in a tarball,
works out for users but extra burden for maintainer over little
4. ask upstream to provide proper release tarballs if not already and
include pre-generated completions/docs/etc... so it's otherwise
only a problem with arbitrary git checkouts
(3+4 may possibly be more complicated if cargo features / USE change
cli options and thus completion files)

For one of my ebuilds I went with better-than-nothing #2.

I've been doing this work for Flatcar, which doesn't use any of the packages affected by this, but it does still concern me a little. Unfortunately, none
of the options are great.

I've seen #2 in one or two cases, and thanks for doing that.

For #1, the BDEPEND approach isn't possible with current Portage/PMS. The experimental EAPI 5-hdepend, which led to BDEPEND in EAPI 7, had a USE flag called targetroot (terrible name!) for this purpose. It wasn't based on cross-compiling, because Portage/PMS don't actually have any concept of that. It applied when ROOT!=/. We decided not to keep it though. Chances are that / won't already have an identical or at least compatible version already installed. Doing another temporary build was therefore deemed preferable. On the plus side, that's quite easy with Cargo. On the minus side, almost all
Rust builds take ages. It hardly seems worth it for shell completions.

I really don't see #4 happening, given that most end-user applications are normally installed using cargo install.

Regarding binfmt_misc, you could explicitly invoke via qemu-user instead, making it less dependent on a particular setup. I have hacked together some helpers to do stuff like this because we really need it for Perl. In Perl's case, it would be mandatory when cross-compiling because there's really no other way. I wouldn't want it to be mandatory in this case though.

-----BEGIN PGP SIGNATURE-----

iQJFBAABCAAvFiEEPxcZ3tkwcedKm2a8EiZBXQDdMTcFAmZuFBIRHGNoZXdpQGdl bnRvby5vcmcACgkQEiZBXQDdMTeN2g//XLXajdR3FIS/huXD9rfGDJV/SSGk1nwQ z34vxUED0JXyWWhIoLO0PtJRp2N9CLjSMxOx3ZEgqUnV9ZBHUXP+QJH2lFS1QqFi oXgbnb95wMQg5pGoBAiR7lTiigWzltG1JdJXVyvdCmzVAg0Yuw5Ku2EZPWblmTtW DnOf5eZyaIuS68h0PO3V4TuJN9MJn0Cg4XlrumTClnw79HfaV+S3usoWU/AHDnUH XcNL5j+3GG/9zGlDhdW8IneiwnPPE96prV5k9aVrLWshRXNejinXwhJjsupJ+hEG qMcau6Fx/Vz7QywymFTnFQ5+fPCnIU7YNYVeb6ibPOZifnL8IdEHpt7D8Nt4OXB6 ECltSCO9oWBZxX5vhZ2ouGmFzgzTlfY2DTA8Y2uWeXR1EuHPJyg0Hh5iPuGfN/GL ZOLxMDPGiRHZczTyLOX5lKJ8ATxDxVpM2YvSQuD+1PsbBytS3WKDY0canZjOidpF ESG/aYwh5ZrSci0Gtbsbnx6W6A3xHfLIHbs0CLMVenlb/28mZ5NaswBkAA/rbsDC JR8i0gVquEqwwsr3+mI8NOx274hwhJMwKOCdBHc73HdAVWHmA3eFMkamDj4xZGNQ pAwWSbpzMz8/0iqfdpueFmn/MboDrdrbSXBxuSQZUF7X92ApZBPWUeYCn8ZsAu6G
LmvPqMTkYfc=
=UvOb
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------yNmsimlAKUnxlbAOYzvgdhC0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 7/6/24 5:04 AM, Florian Schmaus wrote:

FYI, it turns out that at least clap-rs has support to generate the completion files as part of the build process:

https://docs.rs/clap_complete/latest/clap_complete/generator/fn.generate_to.html

Maybe we could encourage projects to simply adjust their build.rs to
generate the completion files. This would avoid inconsistent (Gentoo)
package contents when cross compiling.

Cargo is NOT a real build system, as partially evidenced by the fact
that it does not possess an install system either. build.rs is an
extremely crude hack around this and doesn't work well for that use
case. The main consequence is that you have to manually find a
checksummed build artifact, since that's all build.rs can create.

It reduces the temptation of projects to support this at all, in favor
of just providing an argument to generate it which they figure is the
only thing people can practically use. So, you can try to convince
upstreams to care, but I am cynical.



ripgrep, which I suspect due to recent conversations is the reason
you're mentioning this, *used* to do exactly what you suggest. Recent
versions have moved on to rolling their own argument parser and manpage generator, which they then use to produce prebuilt binary packages via,
well, running the compiled binary with --generate. I get the feeling
they aren't worried about cross compiling.


--
Eli Schwartz


--------------yNmsimlAKUnxlbAOYzvgdhC0--

-----BEGIN PGP SIGNATURE-----

wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZooNPwUDAAAAAAAKCRCEp9ErcA0vV+Wk AQCIXjVyWvF2fSqMbkP6fsbGb20baHgEmscs/9hG0uRNxQD/b85918diD4cgEyDjMBuhWHZRfm8c mbq7UbibIFyezA4=
=iIPK
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)