Greetings! There seems to be a subarchitecture within the current 32bit
Debian arm universes and buildds. armv8 processors will leave the C
stack start at 0xffffffff even when personality ADDR_LIMIT_3GB is set,
whereas on armv7 the address starts at 0xbfffffff, as on other 32bit
linux machines.

Is this a bug? Are there other (perhaps gcc?) switches which can match
the armv7 behavior on armv8 machines (e.g. running a armel or armhf
chroot?)

Right now gcl binaries compiled on armv7 will not run on armv8 in a
32bit chroot because of this. Only workaround I see at the moment is to cripple all gcl 32bit arm binaries and remove immediate fixnum addresses
which should go above 0xc0000000.

Thanks so much,
--
Camm Maguire camm@maguirefamily.org ==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Oct 4, 2021 at 7:38 PM Camm Maguire <camm@maguirefamily.org> wrote:

Greetings! There seems to be a subarchitecture within the current 32bit Debian arm universes and buildds. armv8 processors will leave the C
stack start at 0xffffffff even when personality ADDR_LIMIT_3GB is set, whereas on armv7 the address starts at 0xbfffffff, as on other 32bit
linux machines.

Is this a bug? Are there other (perhaps gcc?) switches which can match
the armv7 behavior on armv8 machines (e.g. running a armel or armhf
chroot?)

Right now gcl binaries compiled on armv7 will not run on armv8 in a
32bit chroot because of this. Only workaround I see at the moment is to cripple all gcl 32bit arm binaries and remove immediate fixnum addresses which should go above 0xc0000000.

Hi Camm,

This is a complex topic, a few notes on this:

- Just to clarify, this is unrelated to armv7 vs armv8 processors, only the
kernel running on them. Running a 32-bit kernel on an armv8 CPU
has the same limit as on an armv7 CPU. The generally recommendation
however is to run 64-bit kernels on these CPUs anyway, so this is
indeed a problem in compat mode.

- PER_LINUX32_3GB is technically an x86-only feature, none of the
other architectures with compat mode support (powerpc, s390, sparc,
mips, parisc and arm64) implement this. Unfortunately, there is no
check for invalid arguments to the personality() syscall, so passing this
flag on architectures other than x86 will just silently ignore it. Most of
these will use a 4GB address space here.

- There is ongoing work by Linus Walleij and others to extend 32-bit
arm kernels to also have a wider address space, allowing up to
3.75GB of user space addresses (and also up to 3.75GB of
lowmem). When this is implemented, the application will stop
working on native 32-bit kernels as well.

- It would probably be possible to implement PER_LINUX32_3GB on
all architectures, including native 32-bit ones, if there is demand.
More generally, we have previously discussed adding a kernel
interface to pick an arbitrary maximum user space address limit
per process, which would solve other problems as well. I don't
think anyone is working on that at the moment.

Arnd

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Greetings, and thank you so much for your very detailed, clear, and comprehensive reply!

PER_LINUX_32GB and/or a userspace interface to set the address space
layout would be nice, but my chief concern is that whatever the kernel
provides to userspace be the same on all machines purporting to be of
the same 'architecture'. On further investigation, it appears that
32bit arm(el)(hf) kernels have a 3GB address space, starting the stack
at 0xbfffffff, regardless of the PER_LINUX_32GB personality setting, and
that 32bit compatibility mode on a 64bit kernel provides a 4GB address
space, starting the stack at 0xffffffff, again regardless of
personality, as you state. To me it seems that the 64bit kernel, if it
offers a compatibility mode, should match whatever the contemporaneous
32bit kernel behavior is, making this a bug in the compatibility mode.

Even if this is not deemed a bug, 32bit chroot under 64bit is
effectively a different architecture at present. I suggest that
arm(el)(hf) refer to a genuine 32bit kernel, whatever address space it
chooses to provide, and that therefore as long as this difference is outstanding, some method in buildd be provided to allow packages to
specify that they should only be built in a 'true' environment. Might
this be possible?

I can put a runtime check in gcl to detect such a mismatch and exit with
an explanatory message for the (hopefully) unlikely case that some non-developer user wants to run these 32bit binaries under a 64bit
kernel in chroot.

If/when the 32bit address space goes to 3.75GB, that will necessitate a recompile, but as long as consistency is maintained 32bit binaries can
be distributed and expected to run on machines of the same
'architecture'.

Your thoughts?

Thanks again!

On Tue, Oct 5, 2021 at 11:50:58 +0200 Arnd Bergmann <arnd@arndb.de> wrote:

On Mon, Oct 4, 2021 at 7:38 PM Camm Maguire <camm@maguirefamily.org> wrote:

Greetings! There seems to be a subarchitecture within the current 32bit Debian arm universes and buildds. armv8 processors will leave the C
stack start at 0xffffffff even when personality ADDR_LIMIT_3GB is set, whereas on armv7 the address starts at 0xbfffffff, as on other 32bit
linux machines.

Is this a bug? Are there other (perhaps gcc?) switches which can match
the armv7 behavior on armv8 machines (e.g. running a armel or armhf chroot?)

Right now gcl binaries compiled on armv7 will not run on armv8 in a
32bit chroot because of this. Only workaround I see at the moment is to cripple all gcl 32bit arm binaries and remove immediate fixnum addresses which should go above 0xc0000000.

Hi Camm,

This is a complex topic, a few notes on this:

- Just to clarify, this is unrelated to armv7 vs armv8 processors, only the
kernel running on them. Running a 32-bit kernel on an armv8 CPU
has the same limit as on an armv7 CPU. The generally recommendation
however is to run 64-bit kernels on these CPUs anyway, so this is
indeed a problem in compat mode.

- PER_LINUX32_3GB is technically an x86-only feature, none of the
other architectures with compat mode support (powerpc, s390, sparc,
mips, parisc and arm64) implement this. Unfortunately, there is no
check for invalid arguments to the personality() syscall, so passing this
flag on architectures other than x86 will just silently ignore it. Most of
these will use a 4GB address space here.

- There is ongoing work by Linus Walleij and others to extend 32-bit
arm kernels to also have a wider address space, allowing up to
3.75GB of user space addresses (and also up to 3.75GB of
lowmem). When this is implemented, the application will stop
working on native 32-bit kernels as well.

- It would probably be possible to implement PER_LINUX32_3GB on
all architectures, including native 32-bit ones, if there is demand.
More generally, we have previously discussed adding a kernel
interface to pick an arbitrary maximum user space address limit
per process, which would solve other problems as well. I don't
think anyone is working on that at the moment.

Arnd

--
Camm Maguire camm@maguirefamily.org ==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 05, 2021 at 01:37:49PM -0400, Camm Maguire wrote:

Greetings, and thank you so much for your very detailed, clear, and comprehensive reply!

PER_LINUX_32GB and/or a userspace interface to set the address space
layout would be nice, but my chief concern is that whatever the kernel provides to userspace be the same on all machines purporting to be of
the same 'architecture'. On further investigation, it appears that
32bit arm(el)(hf) kernels have a 3GB address space, starting the stack
at 0xbfffffff, regardless of the PER_LINUX_32GB personality setting, and
that 32bit compatibility mode on a 64bit kernel provides a 4GB address
space, starting the stack at 0xffffffff, again regardless of
personality, as you state. To me it seems that the 64bit kernel, if it offers a compatibility mode, should match whatever the contemporaneous
32bit kernel behavior is, making this a bug in the compatibility mode.

Even if this is not deemed a bug, 32bit chroot under 64bit is
effectively a different architecture at present. I suggest that
arm(el)(hf) refer to a genuine 32bit kernel, whatever address space it chooses to provide, and that therefore as long as this difference is outstanding, some method in buildd be provided to allow packages to
specify that they should only be built in a 'true' environment. Might
this be possible?

I can put a runtime check in gcl to detect such a mismatch and exit with
an explanatory message for the (hopefully) unlikely case that some non-developer user wants to run these 32bit binaries under a 64bit
kernel in chroot.

If/when the 32bit address space goes to 3.75GB, that will necessitate a recompile, but as long as consistency is maintained 32bit binaries can
be distributed and expected to run on machines of the same
'architecture'.

Your thoughts?

Seems to me that gcl is making assumptions that are invalid. Like
assuming the stack start location at compile time will forever be valid.
Or assuming that there is any address space free above the stack.
I don't think any linux architecture ABI has ever made such a promise.

So it seems to me that gcl is simply wrong and should be rewritten
properly. Looks like a potentially large job.

From what I could find, some programs allocate their own stack early in execution and can hence put it where they want which I guess would free

up some known address space range. Potentially. With address space randomization, can you really ever be sure that there will be a part of
address space that isn't used? Code really should stop trying to abuse
bits in pointers for their own purposes. It always ends up breaking no
matter how clever they people that came up with it thought they were at
the time.

--
Len Sorensen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 05, 2021 at 01:37:49PM -0400, Camm Maguire wrote:

...
To me it seems that the 64bit kernel, if it
offers a compatibility mode, should match whatever the contemporaneous
32bit kernel behavior is, making this a bug in the compatibility mode.
...

You are expecting compatibility with something that is changable even
in 32bit kernels.

For both 32bit x86 and 32bit arm, the amount of address space provided
to userspace is a compile option for the kernel: https://sources.debian.org/src/linux/5.14.9-2/arch/arm/Kconfig/#L1250 https://sources.debian.org/src/linux/5.14.9-2/arch/x86/Kconfig/#L1412

cu
Adrian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 05, 2021 at 03:23:53PM -0400, Camm Maguire wrote:

Greetings!

Fair enough, but *Debian* ships a given compiled kernel fixing this parameter, no? That is the target for the distribution and
apps/packages. Users compiling their own kernel can expect incompatibilities.

Debian is not just expected to be compatible with packages from the
same release.

This fixnum idea in gcl is broken. It must go away. Pointers are for addresses and nothing else.

--
Len Sorensen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Greetings!

Fair enough, but *Debian* ships a given compiled kernel fixing this
parameter, no? That is the target for the distribution and
apps/packages. Users compiling their own kernel can expect
incompatibilities.

Take care,

Adrian Bunk <bunk@debian.org> writes:

On Tue, Oct 05, 2021 at 01:37:49PM -0400, Camm Maguire wrote:

...
To me it seems that the 64bit kernel, if it
offers a compatibility mode, should match whatever the contemporaneous
32bit kernel behavior is, making this a bug in the compatibility mode.
...

You are expecting compatibility with something that is changable even
in 32bit kernels.

For both 32bit x86 and 32bit arm, the amount of address space provided
to userspace is a compile option for the kernel: https://sources.debian.org/src/linux/5.14.9-2/arch/arm/Kconfig/#L1250 https://sources.debian.org/src/linux/5.14.9-2/arch/x86/Kconfig/#L1412

cu
Adrian

--
Camm Maguire camm@maguirefamily.org ==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 05, 2021 at 04:17:51PM -0400, Jeffrey Walton wrote:

On Tue, Oct 5, 2021 at 4:00 PM Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:

...
This fixnum idea in gcl is broken. It must go away. Pointers are for addresses and nothing else.

+1. Tagged pointers caused a lot of problems porting some packages to Aarch64. Tagged pointers were blocking a number of web related
packages. It also caused a number of CVEs, like CVE-2020-9391.

And I found this post:

https://lore.kernel.org/lkml/20081006132651.GG3180@one.firstfloor.org/

where Andi Kleen calls the need for ADDR_LIMIT_3GB "a kludge for
bug-to-bug compatibility with old binaries (that is where the 3GB
personality came from to work around bugs in some old JVMs that could
not deal with a full 4GB address space), it shouldn't be really used
for anything new." And that was 13 years ago. Seems some code still
isn't fixed. Not working with a full 4GB address space is considered
a bug and should be treated as such.

--
Len Sorensen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 5, 2021 at 4:00 PM Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:

...
This fixnum idea in gcl is broken. It must go away. Pointers are for addresses and nothing else.

+1. Tagged pointers caused a lot of problems porting some packages to
Aarch64. Tagged pointers were blocking a number of web related
packages. It also caused a number of CVEs, like CVE-2020-9391.

Jeff

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 5, 2021 at 7:37 PM Camm Maguire <camm@maguirefamily.org> wrote:

Greetings, and thank you so much for your very detailed, clear, and comprehensive reply!

PER_LINUX_32GB and/or a userspace interface to set the address space
layout would be nice, but my chief concern is that whatever the kernel provides to userspace be the same on all machines purporting to be of
the same 'architecture'. On further investigation, it appears that
32bit arm(el)(hf) kernels have a 3GB address space, starting the stack
at 0xbfffffff, regardless of the PER_LINUX_32GB personality setting, and
that 32bit compatibility mode on a 64bit kernel provides a 4GB address
space, starting the stack at 0xffffffff, again regardless of
personality, as you state. To me it seems that the 64bit kernel, if it offers a compatibility mode, should match whatever the contemporaneous
32bit kernel behavior is, making this a bug in the compatibility mode.

Even if this is not deemed a bug, 32bit chroot under 64bit is
effectively a different architecture at present. I suggest that
arm(el)(hf) refer to a genuine 32bit kernel, whatever address space it chooses to provide, and that therefore as long as this difference is outstanding, some method in buildd be provided to allow packages to
specify that they should only be built in a 'true' environment. Might
this be possible?

There are a number of differences between running 32-bit and 64-bit
kernels, and we try to limit those, but I wouldn't consider this
as one I'm worried about.

Most of the applications that abuse pointers were fixed long ago,
and there are a number of applications that actually fail on 32-bit
kernels now because they require the larger virtual address space.

I can put a runtime check in gcl to detect such a mismatch and exit with
an explanatory message for the (hopefully) unlikely case that some non-developer user wants to run these 32bit binaries under a 64bit
kernel in chroot.

If/when the 32bit address space goes to 3.75GB, that will necessitate a recompile, but as long as consistency is maintained 32bit binaries can
be distributed and expected to run on machines of the same
'architecture'.

Your thoughts?

Recompiling an application to match a kernel's implementation detail
is really not how this should work. Possibly workable solutions I can
think of would include:

- coming up with a runtime detection that works across architectures
to figure out which pointer representation to use.
- reserve the upper part of the memory using a MAP_FIXED
mmap()
- drop the pointer hack altogether.

Arnd

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Oct 05, 2021 at 03:23:53PM -0400, Camm Maguire wrote:

...
Fair enough, but *Debian* ships a given compiled kernel fixing this parameter, no? That is the target for the distribution and
apps/packages.

Most x86 users use our kernel, but not on arm.

On x86 nearly all hardware support is upstream, and the Debian kernel
works fine with most hardware.

On arm it is an exception to have complete support for a device in
the upstream kernel.
The single-board computer or reference platform you buy comes with a
vendor kernel that is often > 100k LOC away from the upstream kernel.
There are special cases where the Debian kernel works fine on arm,
but the typical Debian user on real hardware does not have the option
to use our kernel.

There is even an external project Armbian[1] that is combining Debian
with kernels for over 100 boards.

Users compiling their own kernel can expect incompatibilities.

Using own kernels is supposed to work.

Userspace code expecting one specific address space size is something
I'd consider a bug.

I cannot judge whether this is fixable with reasonable effort in your
package, or whether it's a bug that has to stay unfixed.

Take care,

cu
Adrian

[1] https://www.armbian.com/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Greetings, and thanks for your reply

On Tue, 5 Oct 2021 15:01:59 -0400, Len Sorensen wrote:

From what I could find, some programs allocate their own stack early in >execution and can hence put it where they want which I guess would free

up some known address space range. Potentially. With address space

Might you please provide an explicit reference for this?

Thanks so much!
--
Camm Maguire camm@maguirefamily.org ==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Oct 06, 2021 at 02:08:49PM -0400, Camm Maguire wrote:

Greetings, and thanks for your reply

On Tue, 5 Oct 2021 15:01:59 -0400, Len Sorensen wrote:

From what I could find, some programs allocate their own stack early in >execution and can hence put it where they want which I guess would free

up some known address space range. Potentially. With address space

Might you please provide an explicit reference for this?

I can't seem to find the page I read, but it seems that you might be
able to use clone() to create a new process with a stack you allocated.
So if you can somehow allocate a stack at a lower address and then
clone() to switch to a process with that stack, and reserve the memory
range you want to abuse, and check that it was able to (after all with
address space randomization it doesn't have to work if you ask for a
specific address), then you could probably get away with things after
that by using that reserved range for the fixnum stuff. But it would
probably need different handling for 32 and 64 bit machines, and would
have to do this all at runtime of course, since there is no promise of
what size of address space the kernel will give you. I see arm has 1G,
2G, 3G and possibly others. What you build on is certainly not any
indication of what you might run on. If I recall correctly the page I
read was someone trying to do this in virgil.

There really doesn't seem to be any way to ensure you can abuse a
certain range of the address space, beacuse well, you are abusing it.
It's not proper coding and not something linux ever promised you could do.
x86 being one of the first 32/64 bit dual architectures added the 3GB workaround to help out code while it got fixed up. But it seems no other architectures later did the same since it wasn't meant to be a solution,
just a workaround.

I also get the impression in my searching for stuff about this problem,
that 3/4 of the things I find about wanting to do this are gcl related.
I guess it may be the only major offender really left that anyone uses
at all. Seems everything else either gave up, or actually fixed their
code.

--
Len Sorensen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Oct 7, 2021 at 10:28 PM Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:

On Wed, Oct 06, 2021 at 02:08:49PM -0400, Camm Maguire wrote:

I also get the impression in my searching for stuff about this problem,
that 3/4 of the things I find about wanting to do this are gcl related.
I guess it may be the only major offender really left that anyone uses
at all. Seems everything else either gave up, or actually fixed their
code.

Looking at https://codesearch.debian.net/search?q=ADDR_LIMIT_3GB supports
this. While a number of packages enumerate all the possible personality
flags, gcl is the only one that actually passes this one.

I also looked at users of "setarch --32bit --3gb", which brings up a
ruby package,
but this does it only for x86.

Arnd

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)