--2yB5+CA2g0IhBuTl
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

This mail is about whether, and how best, to enable Pointer
Authentication and Branch Target Identification (PAC+BTI) on arm64 in
Debian.

Summary:

Pointer Authentication and Branch Target Identification are
significant new security features in ARMv8.3 and ARMv8.5 respectively
arm64 hardware. They are present in new (debian-relevant) hardware,
starting with the Graviton 3. It is both straightforward and
reasonably safe to enable these features by default now so that they
can be reasonably well-tested in time for Bookworm. There is a kernel
option to turn them off at runtime should hardware be found where this
is a problem, and of course a compiler option to disable them at build
time. They are important security enhancements, with a very small
overhead, which can only work if enabled at build-time, so adding -mbranch-protection=standard to the default build options seems like
the right thing to do.

Discussion:

See below for the details of these options, what they do, how they
work, etc, and the argument for why they should be enabled by default.
My main question at this point is about exactly how this should be
dealt with in dpkg and/or (Debian) gcc.

The -mbranch-protection=standard option could be set either by dpkg or
by gcc. I'm not quite sure how we decide which is most appropriate?
They could be an unconditional architecture default, or could be part
of the dpkg hardening flags. It could be one hardening feature 'PACBTI',
or separate 'PAC' and 'BTI' features (with corresponding logic to set
the right flags).

Attached is my simple-minded example patch to dpkg to set it
unconditionally (for arm64). This is not intended to be final, but
illustrates what I have tried so far.

A notable aspect of this flag is that it is arch-specific. It should
not be issued when targetting arches other than arm64 as it is not a
valid flag there. dpkg-buildflags gets this right with below patch.
The -fcf-protection option on amd64 has the same
characteristic and indeed is pretty-much the same functionality (as
BTI). (It might have been better if the gcc developers had had one
option that added this type of branch protection on both
architectures, and ignored it on other arches, but they didn't.) That
option appears to have been set by default in Ubuntu 19.10 gcc
targetting aarch64 (although it's a bit hard to tell as gcc dumpspecs
is pretty cryptic, and I could be wrong.

The hardening features seem to assume that they are all
implemented/available on all architectures? I'm not sure if extra work
would be needed for a hardening feature that only existed on one arch
(so far) (PAC). (BTI exists on two arches). Perhaps all this is an
argument for just turning it on by default in gcc instead?

Currently the small number of packages that fail to build with -mbranch-protection=standard are almost all because the build does a cross-build of some sort, and the non-arm64 toolchain barfs on the
unrecognised option. A simple example whch dies is nmap (running i686-w64-mingw32-gcc ).

Given that dpkg-buildflags does this right (with my patch):
DEB_HOST_ARCH=amd64 dpkg-buildflags --get CFLAGS
-g -O2 -ffile-prefix-map=/home/wookey/packages/dpkg-1.21.8=. -fstack-protector-strong -Wformat -Werror=format-security

DEB_HOST_ARCH=arm64 dpkg-buildflags --get CFLAGS
-g -O2 -mbranch-protection=standard -ffile-prefix-map=/home/wookey/packages/dpkg-1.21.8=.
-fstack-protector-strong -Wformat -Werror=format-security

I presume these are bugs in the packaging to do with not setting the
HOST_ARCH correctly, or assuming that flags do not
vary by architecture. This needs further investigation.

Hardening flags are not set by default but are strongly recomended for
package builds, and flagged by bldc if not present, so whilst setting
this option in there will not give as complete coverage as setting it by default, it
should end up being applied to most builds.

As I said, I'm not sure what our policy is on what goes in gcc default
flags, dpkg default flags or dpkg feature flags. So I'm open to
suggestions on the best/right way to implement this, then will prepare patches/file bugs.

Details on PAC and BTI:

An important security feature on arm64 hardware that supports it
(ARMv8.3 or later), is Pointer Authentication ('PAC'). This allows
pointers to be tagged (signed) and checked, mitigating some types of
attack that corrupt and/or replace pointers ('Return Oriented
Programming' (ROP) attacks). Details on how it actually works can be
found here (2019 Suse Conf talk:
https://www.youtube.com/watch?v=iW3mXDSijSQ). ARM developer blog: https://developer.arm.com/documentation/102433/0100/Return-oriented-programming?lang=en

There are both user and kernel aspects to the support. Both need to be
enabled to get the functionality. The user aspect is simple to enable
and not intrusive so we think it should be enabled by default on arm64
builds. This is done by setting a suitable -mbranch-protection=
option. ('standard', 'pac-ret' or 'pac-ret+leaf', see below).
Functionality was implemented in gcc7 but 10.2 is recommended.

There is also the related functionality Branch Target Identification
(BTI) (available in ARMv8.5 onwards) which creates defined 'landing
pads' as the only places where indirect jumps are allowed to land.

This protects against 'Jump Oriented Programming' (JOP) attacks. https://developer.arm.com/documentation/102433/0100/Jump-oriented-programming?lang=en

This functionality (emitting these instructions) is enabled with -mbranch-protection=bti which has been available since gcc9.1 (and
binutils 2.32) (but gcc10.2 or later is recommended)

The new instructions which actually generate and check these tagged
pointers and 'landing pads' are implemented in the NOP space of
ARMv8.0/8.1/8.2 so that they have no effect at all on older hardware
which does not have this functionality. This means that the features
should (by design) be safe to enable by default so that they work on
hardware with support, but do nothing on hardware which does not have
it. In principle there is a small overhead to fetching (and
not-executing) these instructions on earlier hardware, but due to
packing rules and pipelining most of the time it won't actually make
any difference. The overhead that does exist (either as NOPs on older
hardware or actual instructions on newer hardware) is considered
worthwhile for the improved security, and enabling these by default
fit's with debian's general approach of taking security seriously,
even if that adds some small overhead.

In practice, because these features have complimentary
characteristics, and are safe on older instruction set/hardware
versions in the same way, it is recommended to enable both together
using the GCC option -mbranch-protection=standard, which enables both
PAC and BTI instructions to be emitted.

The only known compatibility risk is running modern binaries (built
with PAC enabled) on modern hardware (ARMv8.3 or newer) on binaries
(e.g. in an old chroot) built with gcc older than 7 (so that the
tagged pointers are not recognised and properly masked), which might
cause hangs/faults during exception unwinding, should some
exception-causing error occur. I think gcc7 was the default compiler
in Stretch/oldoldstable? The number of people running oldoldstable chroots/containers on very new hardware should be quite small, so I
think we can live with this. The alternative is to wait for another
release cycle.

Obviously this is a potentially intrusive change, despite the backwards-compatible design, so we have done a mass rebuild of the
archive with this option set to see if there were any problems. Of
14371 packages there were 12 packages with build issues, 4 of those
were trying to use -mbranch-protection=standard with an x86 compiler
due to the simplistic way the flags were set for the test: echo -e
"APPEND CFLAGS -mbranch-protection=standard\nAPPEND CXXFLAGS -mbranch-protection=standard"> /etc/dpkg/buildflags.conf (which
applies to all arches, not just arm64)

Logs are here: http://qa-logs.debian.net/2021/11/18/

So that leaves a few packages (<8) that do appear to need
investigation. I will file bugs for any changes needed.

General info on arm hardware changes+compiler/kernel support: https://en.opensuse.org/Arm_architecture_support

I have implemented this in the Debian Vendor options, but actually it
should probably be turned on everywhere unless some distro has a good
reason not to. IIUC the debian settings are inherited unless overridden?

Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

--2yB5+CA2g0IhBuTl
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="dpkg-1.21.7-PAC-BTI-arch.patch" Content-Transfer-Encoding: quoted-printable

--- scripts/Dpkg/Vendor/Debian.pm.orig 2022-03-26 17:17:59.000000000 +0000
+++ scripts/Dpkg/Vendor/Debian.pm 2022-05-29 01:20:36.368000000 +0000
@@ -185,6 +185,15 @@
$flags->append($_, $default_flags) foreach @compile_flags;
$flags->append('DFLAGS', $default_d_flags);

+ ## Area: arch-specific
+
+ if ($arch eq 'arm64') {
+ my $flag = '-mbranch-protection=standard';
+ $flags->append('CFLAGS', $flag);
+ $flags->append('CXXFLAGS', $flag);
+ }
+
+
## Area: future

if ($use_feature{future}{lfs}) {

--2yB5+CA2g0IhBuTl--

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEER4nvI8Pe/wVWh5yq+4YyUahvnkcFAmKTcsAACgkQ+4YyUahv nkdUKRAAysEiKDyap/vAIMA34igxHj7C+gZdFxpdGhYymggXppvlI/3AcMbjzXZA
/ZRV06cv

Hi!

On Sun, 2022-05-29 at 14:19:05 +0100, Wookey wrote:

Discussion:

The -mbranch-protection=standard option could be set either by dpkg or
by gcc. I'm not quite sure how we decide which is most appropriate?

I think if the option has potential for breakage or for a performance
hit, then it tends to be better to enable them via dpkg hardening
flags, otherwise people building in Debian systems that are not
interested in the packaging defaults might get surprised.

They could be an unconditional architecture default, or could be part
of the dpkg hardening flags. It could be one hardening feature 'PACBTI',
or separate 'PAC' and 'BTI' features (with corresponding logic to set
the right flags).

A notable aspect of this flag is that it is arch-specific. It should
not be issued when targetting arches other than arm64 as it is not a
valid flag there. dpkg-buildflags gets this right with below patch.
The -fcf-protection option on amd64 has the same
characteristic and indeed is pretty-much the same functionality (as
BTI). (It might have been better if the gcc developers had had one
option that added this type of branch protection on both
architectures, and ignored it on other arches, but they didn't.) That
option appears to have been set by default in Ubuntu 19.10 gcc
targetting aarch64 (although it's a bit hard to tell as gcc dumpspecs
is pretty cryptic, and I could be wrong.

If there's potential for needing to disable them (for whatever reason),
then adding them into their own feature would be better. But having a
feature that is arch-specific (not just that it does not currently
work on some arches), seems not ideal.

I think I'd probably want a new feature that can potentially be used
on other arches such as with -fcf-protection on amd64, yes.

Say hardening=+branch or perhaps branchprotection or similar.

The hardening features seem to assume that they are all
implemented/available on all architectures? I'm not sure if extra work
would be needed for a hardening feature that only existed on one arch
(so far) (PAC). (BTI exists on two arches). Perhaps all this is an
argument for just turning it on by default in gcc instead?

The features can be easily shadowed for specific arches, if for
example they currently have no proper support for the flags. But
I'd rather abstract this than have a new feature per option denoting
similar protections for each arch.

As I said, I'm not sure what our policy is on what goes in gcc default
flags, dpkg default flags or dpkg feature flags. So I'm open to
suggestions on the best/right way to implement this, then will prepare patches/file bugs.

I think the attached patch might do (it's still missing man page
update), assuming that the flags work on all compiler variables, and
the amd64 case could be disabled for now if there are concerns of
enabling both at the same time (it could be brought up later).

The main issue is that because most packages just do hardening=+all,
even adding such feature disabled by default, would imply pretty much
enabling it immediately, so we might as well just enable it by
default. But to do that, the usual step after having done an archive
rebuild (if it seemed appropriate) and which you already did, would be
to bring this up on the debian-devel mailing list. If there are no
objections after a bit, I'm fine adding the support on the next dpkg
release after that time.

The only known compatibility risk is running modern binaries (built
with PAC enabled) on modern hardware (ARMv8.3 or newer) on binaries
(e.g. in an old chroot) built with gcc older than 7 (so that the
tagged pointers are not recognised and properly masked), which might
cause hangs/faults during exception unwinding, should some
exception-causing error occur.

I'm not sure I understand this sentence above about “running modern
binaries … on binaries”? Did you mean run-time linked, or is this
related to the kernel? Or something else?

Obviously this is a potentially intrusive change, despite the backwards-compatible design, so we have done a mass rebuild of the
archive with this option set to see if there were any problems. Of
14371 packages there were 12 packages with build issues, 4 of those
were trying to use -mbranch-protection=standard with an x86 compiler
due to the simplistic way the flags were set for the test: echo -e
"APPEND CFLAGS -mbranch-protection=standard\nAPPEND CXXFLAGS -mbranch-protection=standard"> /etc/dpkg/buildflags.conf (which
applies to all arches, not just arm64)

That result looks pretty good TBH.

I have implemented this in the Debian Vendor options, but actually it
should probably be turned on everywhere unless some distro has a good
reason not to. IIUC the debian settings are inherited unless overridden?

Currently the default settings cannot be modified by derivatives, but
I've got some code adding support for that. In any case see above
about making it possible to disable, and the «+all» case anyway.

Thanks,
Guillem

diff --git i/scripts/Dpkg/Vendor/Debian.pm w/scripts/Dpkg/Vendor/Debian.pm index c293a99d6..7dcaab718 100644
--- i/scripts/Dpkg/Vendor/Debian.pm
+++ w/scripts/Dpkg/Vendor/Debian.pm
@@ -129,6 +129,7 @@ sub _add_build_flags {
format => 1,
relro => 1,
bindnow => 0,
+ branch => 1,
},
);

@@ -364,6 +365,11 @@ sub _add_build_flags {
# relro not implemented on ia64, hppa, avr32.
$use_feature{hardening}{relro} = 0;
}
+ if ($cpu !~ /^(?:amd64|arm64)$/) {
+ # On amd64 use -fcf-protection.
+ # On arm64 use -mbranch-protection=standard.
+ $use_feature{hardening}{branch} = 0;
+ }

# Mask features that might be influenced by other flags.
if ($opts_build->has('noopt')) {
@@ -430,6 +436,17 @@ sub _add_build_flags {
$flags->append('LDFLAGS', '-Wl,-z,now');
}

+ # Branch protection
+ if ($use_feature{hardening}{branch}) {
+ my $flag;
+ if ($cpu eq 'arm64') {
+ $flag = '-mbranch-protection=standard';
+ } elsif ($cpu eq 'amd64') {
+ $flag =

On 2022-06-04 06:25 +0200, Guillem Jover wrote:

On Sun, 2022-05-29 at 14:19:05 +0100, Wookey wrote:

Discussion:

The -mbranch-protection=standard option could be set either by dpkg or
by gcc. I'm not quite sure how we decide which is most appropriate?

I think if the option has potential for breakage or for a performance
hit, then it tends to be better to enable them via dpkg hardening
flags, otherwise people building in Debian systems that are not
interested in the packaging defaults might get surprised.

Makes sense. On the other hand ARM people would like this to be an
achitecture default everywhere so that the protection afforded
actually gets used. Because this hardware is not widespread yet beyond
phone hardware (and Apple M1) Linux hasn't been run much on it yet so
it is early days and we don't know if there will be significant issues
that might mke turning it on everywhere a bad plan.

It's easy enough to make it a gcc default later, after initial
implementaiton as a hardening flag.

They could be an unconditional architecture default, or could be part
of the dpkg hardening flags. It could be one hardening feature 'PACBTI',
or separate 'PAC' and 'BTI' features (with corresponding logic to set
the right flags).

A notable aspect of this flag is that it is arch-specific. It should
not be issued when targetting arches other than arm64 as it is not a
valid flag there. dpkg-buildflags gets this right with below patch.
The -fcf-protection option on amd64 has the same
characteristic and indeed is pretty-much the same functionality (as
BTI).

If there's potential for needing to disable them (for whatever reason),
then adding them into their own feature would be better. But having a
feature that is arch-specific (not just that it does not currently
work on some arches), seems not ideal.

I think I'd probably want a new feature that can potentially be used
on other arches such as with -fcf-protection on amd64, yes.

Say hardening=+branch or perhaps branchprotection or similar.

OK, that sounds sensible to me. I agree that a generic name makes
sense. We are quite likely to get something like PAC on x86 too at
some point and then that can be added easily in this framework. Does 'jumpprotection' (or shorter 'jumpprot' or even 'jump') work any
better given that 'returns' are not normally described as 'branches'
(and PAC protects against 'return-oriented-programming' attacks and
BTI aginst 'jump-oriented-programming' attacks)?

But the name is bikeshedding. 'branch' is succinct and fine with me.

The hardening features seem to assume that they are all implemented/available on all architectures? I'm not sure if extra work would be needed for a hardening feature that only existed on one arch
(so far) (PAC). (BTI exists on two arches). Perhaps all this is an
argument for just turning it on by default in gcc instead?

The features can be easily shadowed for specific arches, if for
example they currently have no proper support for the flags. But
I'd rather abstract this than have a new feature per option denoting
similar protections for each arch.

Agreed.

As I said, I'm not sure what our policy is on what goes in gcc default flags, dpkg default flags or dpkg feature flags. So I'm open to
suggestions on the best/right way to implement this, then will prepare patches/file bugs.

I think the attached patch might do (it's still missing man page
update), assuming that the flags work on all compiler variables, and
the amd64 case could be disabled for now if there are concerns of
enabling both at the same time (it could be brought up later).

cf-protection has been enabled by default on redhat since 2018 and
Suse gcc since oct 2021. Only Suse has also enabled
branch-protection=standard on arm64 (in Tumbleweed since nov 2020) -
Rhel has not done it yet, but is trying to. So the x86 version of
this has has a good level of testing without causing any trouble I am
aware of, and the arm64 has had reasonable expsure in Suse for a while
too.

The main issue is that because most packages just do hardening=+all,
even adding such feature disabled by default, would imply pretty much enabling it immediately, so we might as well just enable it by
default. But to do that, the usual step after having done an archive
rebuild (if it seemed appropriate) and which you already did, would be
to bring this up on the debian-devel mailing list. If there are no
objections after a bit, I'm fine adding the support on the next dpkg
release after that time.

The only known compatibility risk is running modern binaries (built
with PAC enabled) on modern hardware (ARMv8.3 or newer) on binaries
(e.g. in an old chroot) built with gcc older than 7 (so that the
tagged pointers are not recognised and properly masked), which might
cause hangs/faults during exception unwinding, should some exception-causing error occur.

I'm not sure I understand this sentence above about “running modern binaries … on binaries”? Did you mean run-time linked, or is this
related to the kernel? Or something else?

Sorry. I'm a bit vague about the details, but the issue as I
understand it is to do with exception handling/stack unwinding, which
is handled by a low-level gcc library, so if a binary built with PAC
enabled (so it has PAC instructions) is run on new hardware (so it has
tagged pointers) but in an old enough rootfs/container (so the gcc is
<7), then the bit of libgcc/libunwind(?) responsible for unwinding the
function stack for error reporting will be confused by the extra tag bits/instructions in NOP space. That's not a very likely combination,
but could happen if someone puts an old enough chroot on new (8.3)
hardware and then puts some newer-than-now binaries into it. GCC 7 was
mid 2017.

In a debian context this means running a Stretch (debian 9) chroot on a graviton3/M1 or newer and putting unstable (from mid 2022) binaries
onto it (and wanting functioning exception handling, but not disabling
PAC in the kernel). This sort of package mixing (stretch+bookworm) is discouraged, so the issue would be most likely to arise if running new 3rd-party binaries from elsewhere (and they crash).

The right thing to do in this case would be setting arm64.nopauth on the
kernel command line, then everything would work fine.

I'm not sure if it will always fail or just might - I'll have to find
out. Overall my feeling is that this is not a large enough
problem/risk to delay enabling 'branch' hardening for another release,
but I should try it to see exactly how it manifests.

<snip patch>

Cheers for that. I'll test it, but it looks fine. And I should try and
add some man-page words.


Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEER4nvI8Pe/wVWh5yq+4YyUahvnkcFAmKhFQ8ACgkQ+4YyUahv nkc3Eg/9EfvyyiUwaxnN927+CVXEUbOr0j1S/jZ4o3GZr0cWk3faVBeqLgt4/KcC V9PZzJJt+csLnhA1rZyUuC2xVpI6LMPE8L+WewWrWI+i8tpYwN/A5ZkzWLSDrQ5w afzl6Aemwpryn8SnI2kFOhDWQT1424GlWF7TA3Mz29Nc+LNlgNmnkDUXNAO5GotR o9euAD86sia88cTmAPi3HIvLkQNpo82NfxvnlTxmiVLJAz2Vr+lM5KN+rRSgoJCd 77ZkCKLu6mZij3VGhEU2NjN8FlM9urPosn+hcS4kkLuws11drSlLEN113dVFo7da bNs/q7xGWW2dV9WbVSmZdOhq66riwJ1b5Sr6qx5LYTsTuwVuZzYWa87nNIoYKs4Q 35zR4xead5+dY3T9CfeU19R3CuytVpaEL4iOJIddXLmBtyslxUg/3YLEvzJwwwbu AzLb9kK0HiFytNuG+PGk+usxOyddR5ldn54RAH0VkS8/x3/MSewLgR7UDAHBG6nR CS+v96O4p+NJPbCp8zYbcvYd6XA7GENat0yo70qZHHS8ojBPrbXRpyf9Czkp6hzG 0l+MlTX7hbSbtKOnpkgRy/W649PF3s5xRnSi6/AALSnA6QRqq62cFjEI/CAWgHRg 8lpJ0LHVJZ3tRcPBTSKVdz77Vf7FeFwE0FMUrbiDFnyt6Fs5DII=
=ePmV
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)