How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port a jonesforth variant for the heck of it, I did notice that Apple seems to disallow having pages with rwx permissions. You either can write to a page, or you can execute it, but not
both.

Executing machine code created at runtime raises segfaults/gives bus errors.

For a FORTH that assembles code into a common segment, that poses a real quandary. I don't know how to separate the 'already written and immutable' and 'modifiable'/'new' parts of the code in FORTH - it's seems antithetical to FORTH's philosophy, if
there is such a thing.

See:
[1] https://stackoverflow.com/questions/74132150/forth-implementation-with-jit-write-protection
[2] https://stackoverflow.com/questions/74124485/mmap-rwx-page-on-macos-arm64-architecture

Maybe you are seeing the same thing. If gforth has figured this out, I'd be super happy to learn how.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 20/10/2022 11:11, Ron AARON wrote:

On 20/10/2022 10:08, Anton Ertl wrote:

Klapaucius Klapaucius <klapaucius@gmail.com> writes:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to
port a =
jonesforth variant for the heck of it, I did notice that Apple seems
to dis=
allow having pages with rwx permissions.

Yes, my (unfinished) attempts at porting Gforth to MacOS on ARM64
revealed that to me.

Maybe you are seeing the same thing. If gforth has figured this out,
I'd be=
super happy to learn how.

For now Gforth does not work on MacOS on ARM64.  My plan is to disable
native code generation and fall back to threaded code (so Gforth on
MacOS will be slower than Gforth on Linux on the same hardware).  If I
was much more motivated, I would try to map the same memory twice,
once RW and once RX, and use one mapping for writing and one for
executing.  I don't know if that works on MacOS on ARM64, though.
Another alternative is to change the page(s) you want to write to RW
before writing and back to RX afterwards, using mprotect(); note that
the code that is executed between these two mprotect()s has to be
executable all the time.

- anton

8th's M1 port uses "pthread_jit_write_protect_np()" as appropriate
before and after writing native code.

And, I almost forgot:

void __clear_cache(void *start, void *end)
{
sys_icache_invalidate(start, (char *)end-(char *)start);
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 20/10/2022 10:08, Anton Ertl wrote:

Klapaucius Klapaucius <klapaucius@gmail.com> writes:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port a = >> jonesforth variant for the heck of it, I did notice that Apple seems to dis= >> allow having pages with rwx permissions.

Yes, my (unfinished) attempts at porting Gforth to MacOS on ARM64
revealed that to me.

Maybe you are seeing the same thing. If gforth has figured this out, I'd be= >> super happy to learn how.

For now Gforth does not work on MacOS on ARM64. My plan is to disable
native code generation and fall back to threaded code (so Gforth on
MacOS will be slower than Gforth on Linux on the same hardware). If I
was much more motivated, I would try to map the same memory twice,
once RW and once RX, and use one mapping for writing and one for
executing. I don't know if that works on MacOS on ARM64, though.
Another alternative is to change the page(s) you want to write to RW
before writing and back to RX afterwards, using mprotect(); note that
the code that is executed between these two mprotect()s has to be
executable all the time.

- anton

8th's M1 port uses "pthread_jit_write_protect_np()" as appropriate
before and after writing native code.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Klapaucius Klapaucius <klapaucius@gmail.com> writes:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port a = >jonesforth variant for the heck of it, I did notice that Apple seems to dis= >allow having pages with rwx permissions.

Yes, my (unfinished) attempts at porting Gforth to MacOS on ARM64
revealed that to me.

Maybe you are seeing the same thing. If gforth has figured this out, I'd be=
super happy to learn how.

For now Gforth does not work on MacOS on ARM64. My plan is to disable
native code generation and fall back to threaded code (so Gforth on
MacOS will be slower than Gforth on Linux on the same hardware). If I
was much more motivated, I would try to map the same memory twice,
once RW and once RX, and use one mapping for writing and one for
executing. I don't know if that works on MacOS on ARM64, though.
Another alternative is to change the page(s) you want to write to RW
before writing and back to RX afterwards, using mprotect(); note that
the code that is executed between these two mprotect()s has to be
executable all the time.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: https://forth-standard.org/
EuroForth 2022: https://euro.theforth.net

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Ron AARON schrieb am Donnerstag, 20. Oktober 2022 um 10:11:45 UTC+2:

On 20/10/2022 10:08, Anton Ertl wrote:

Klapaucius Klapaucius <klapa...@gmail.com> writes:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port a =
jonesforth variant for the heck of it, I did notice that Apple seems to dis=
allow having pages with rwx permissions.

Yes, my (unfinished) attempts at porting Gforth to MacOS on ARM64
revealed that to me.

Maybe you are seeing the same thing. If gforth has figured this out, I'd be=
super happy to learn how.

For now Gforth does not work on MacOS on ARM64. My plan is to disable native code generation and fall back to threaded code (so Gforth on
MacOS will be slower than Gforth on Linux on the same hardware). If I
was much more motivated, I would try to map the same memory twice,
once RW and once RX, and use one mapping for writing and one for
executing. I don't know if that works on MacOS on ARM64, though.
Another alternative is to change the page(s) you want to write to RW
before writing and back to RX afterwards, using mprotect(); note that
the code that is executed between these two mprotect()s has to be executable all the time.

- anton

8th's M1 port uses "pthread_jit_write_protect_np()" as appropriate
before and after writing native code.

Yep. See for instance: https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <48865615-4d33-400a-975f-aa6b530dd74an@googlegroups.com>,
Klapaucius Klapaucius <klapaucius@gmail.com> wrote:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port
a jonesforth variant for the heck of it, I did notice that Apple seems
to disallow having pages with rwx permissions. You either can write to a >page, or you can execute it, but not both.

Executing machine code created at runtime raises segfaults/gives bus errors.

For a FORTH that assembles code into a common segment, that poses a real >quandary. I don't know how to separate the 'already written and
immutable' and 'modifiable'/'new' parts of the code in FORTH - it's
seems antithetical to FORTH's philosophy, if there is such a thing.

See:
[1] >https://stackoverflow.com/questions/74132150/forth-implementation-with-jit-write-protection
[2] >https://stackoverflow.com/questions/74124485/mmap-rwx-page-on-macos-arm64-architecture

Maybe you are seeing the same thing. If gforth has figured this out, I'd
be super happy to learn how.

jonesforth is loosely based on ciforth. https://github.com/albertvanderhorst/ciforth
A guy name Spykerman has figured this out a long time ago for me
long ago for the Apple.

ciforth comes in a number of thoroughly tested versions under which
xina for the apple.
See ciforth.html on my site below.
The exercise to convert a ciforth to a jonesforth is trivial.
I don't like this much.
- ciforth is as simple as jonesforth
- ciforth is solid ISO94
- ciforth is carefully designed to allow extensions that
you can, but need not, use.
(fp, multitasking,decorators, scripting, native executables,
32/64 bit assembler, you name it.)

I have made a competing pedagogical effort to jonesforth: https://github.com/albertvanderhorst/yourforth
that has at least no gratitious deviations from ISO94.
Throwing out doubles, the <# # #S #> has to be redefined as
<% % %S %> .

Groetjes Albert
--
"in our communism country Viet Nam, people are forced to be
alive and in the western country like US, people are free to
die from Covid 19 lol" duc ha
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I don't think that's enough. For a traditional java JIT etc, the code that generates the new code lives in a separate space, presumably. This is possible because the jit compiler itself is immutable.

In FORTH, the compiler itself can be and often is (re)written at runtime, in the same segments non-compiler stuff is.

I believe one would have to somehow separate out the 'compiler' code from 'non-compiler' code - then you could at least create native non-compiler code your way.

It seems like a lot of work for not that much benefit, at the cost of significantly complicating the FORTH language model.

It's annoying that Apple seems to have made a policy choice to intentionally cripple their machines without an out (e.g. remove sip protection). Sigh.


On Wednesday, October 19, 2022 at 10:24:09 PM UTC-10, minf...@arcor.de wrote:

Ron AARON schrieb am Donnerstag, 20. Oktober 2022 um 10:11:45 UTC+2:

On 20/10/2022 10:08, Anton Ertl wrote:

Klapaucius Klapaucius <klapa...@gmail.com> writes:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port a =
jonesforth variant for the heck of it, I did notice that Apple seems to dis=
allow having pages with rwx permissions.

Yes, my (unfinished) attempts at porting Gforth to MacOS on ARM64 revealed that to me.

Maybe you are seeing the same thing. If gforth has figured this out, I'd be=
super happy to learn how.

For now Gforth does not work on MacOS on ARM64. My plan is to disable native code generation and fall back to threaded code (so Gforth on
MacOS will be slower than Gforth on Linux on the same hardware). If I
was much more motivated, I would try to map the same memory twice,
once RW and once RX, and use one mapping for writing and one for executing. I don't know if that works on MacOS on ARM64, though.
Another alternative is to change the page(s) you want to write to RW before writing and back to RX afterwards, using mprotect(); note that
the code that is executed between these two mprotect()s has to be executable all the time.

- anton

8th's M1 port uses "pthread_jit_write_protect_np()" as appropriate
before and after writing native code.

Yep. See for instance: https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Very cool, ciforth!

I chose jonesforth mostly because I already had ported it to macOS Catalina on x64, and because I found a linux implemenation for ARM64 that I hoped would be trivial to port:

https://github.com/narenratan/jonesforth_arm64_apl/blob/master/jonesforth.S

It's pretty fun with all the APL syntax ;) I don't mind that it's not standards compatible, I'd care more if I were to write larger systems with it. It's more like a toy for me right now, and to learn a bit of ARM assembly code - porting a forth is a fun
way of doing that ;)

If I had more time, I'd create a usable system for ARM/RISC SOC boards, like micropython. But my RTOS-fu is not up to the task and I don't have the time to level up ...
When I do, I keep ciforth in mind!

Have a great day!
K
On Thursday, October 20, 2022 at 1:47:47 AM UTC-10, none albert wrote:

In article <48865615-4d33-400a...@googlegroups.com>,
Klapaucius Klapaucius <klapa...@gmail.com> wrote:

How does gforth assemble new machine code at runtime ?

I'm not a FORTH expert, barely a neophyte. However, while trying to port
a jonesforth variant for the heck of it, I did notice that Apple seems
to disallow having pages with rwx permissions. You either can write to a >page, or you can execute it, but not both.

Executing machine code created at runtime raises segfaults/gives bus errors.

For a FORTH that assembles code into a common segment, that poses a real >quandary. I don't know how to separate the 'already written and
immutable' and 'modifiable'/'new' parts of the code in FORTH - it's
seems antithetical to FORTH's philosophy, if there is such a thing.

See:
[1] >https://stackoverflow.com/questions/74132150/forth-implementation-with-jit-write-protection
[2] >https://stackoverflow.com/questions/74124485/mmap-rwx-page-on-macos-arm64-architecture

Maybe you are seeing the same thing. If gforth has figured this out, I'd >be super happy to learn how.

jonesforth is loosely based on ciforth. https://github.com/albertvanderhorst/ciforth
A guy name Spykerman has figured this out a long time ago for me
long ago for the Apple.

ciforth comes in a number of thoroughly tested versions under which
xina for the apple.
See ciforth.html on my site below.
The exercise to convert a ciforth to a jonesforth is trivial.
I don't like this much.
- ciforth is as simple as jonesforth
- ciforth is solid ISO94
- ciforth is carefully designed to allow extensions that
you can, but need not, use.
(fp, multitasking,decorators, scripting, native executables,
32/64 bit assembler, you name it.)

I have made a competing pedagogical effort to jonesforth: https://github.com/albertvanderhorst/yourforth
that has at least no gratitious deviations from ISO94.
Throwing out doubles, the <# # #S #> has to be redefined as
<% % %S %> .

Groetjes Albert
--
"in our communism country Viet Nam, people are forced to be
alive and in the western country like US, people are free to
die from Covid 19 lol" duc ha
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)