doing so on MS-DOS. The former was much slower than I had expected.
The only bottle-neck I could see was READ-LINE. The solution was to
read the input file in one go (or at least in large chunks) and parse
out the lines, joining partial ones when they occurred. The code makes
use of library functions. I've attempted to explain these for anyone >thinking of porting it.
https://pastebin.com/XpBmTFXW
In article <ua35nq$2kjcn$1...@dont-email.me>, dxforth <dxf...@gmail.com> wrote:
This was prompted after trying to run an app on CP/M after successfully >doing so on MS-DOS. The former was much slower than I had expected.
The only bottle-neck I could see was READ-LINE. The solution was to
read the input file in one go (or at least in large chunks) and parse
out the lines, joining partial ones when they occurred. The code makes
use of library functions. I've attempted to explain these for anyone >thinking of porting it.
https://pastebin.com/XpBmTFXWAs long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
I called this (ACCEPT). This does away with the need to provide
a buffer. You must not of course change the string constant
returned.
(ACCEPT)
STACKEFFECT: -- sc
Accept characters from the terminal, until a RET is received and
return the result as a constant string sc. It doesn't contain any line ending, but the buffer still does and after 1+ the string ends in a
LF. The editing functions are the same as with ACCEPT . This is
lighter on the system and sometimes easier to use than ACCEPT This
input remains valid until the next time that the console buffer is
refilled.
...
As long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
The main issue in my case seems to be the file-repositioning that is
the basis of most READ-LINE implementations. Under CP/M it hits hard
for some reason.
...
I.e., this just measures reading a 43MB file with 998170 lines with
repeated READ-LINEs. Gforth uses buffered I/O (with the buffering implemented by glibc), while SwiftForth uses unbuffered I/O and REPOSITION-FILE. As a result, SwiftForth spends more than 7 times
more cycles on this benchmark than Gforth. I would be surprised if
the strategy used by SwiftForth fared better than buffering on other
OSs.
...
As long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
AFAICS that's only feasable if the whole file fits in memory. I don't
have that option.
The main issue in my case seems to be the file-repositioning that is
the basis of most READ-LINE implementations. Under CP/M it hits hard
for some reason. I should first verify the algorithm I used is as
efficient as can be. (It came from a C compiler, so how could it not? :)
dxforth <dxforth@gmail.com> writes:
The main issue in my case seems to be the file-repositioning that is
the basis of most READ-LINE implementations. Under CP/M it hits hard
for some reason.
Not just on CP/M. E.g., on Linux I reported ><2021Oct19.095538@mips.complang.tuwien.ac.at>:
|perf stat -e cycles:u -e instructions:u ~/gforth/gforth -e >'"count-unique.in" r/o open-file throw constant f create buf 256 allot :
foo 0 begin buf 256 f read-line throw nip while 1+ repeat ; foo . cr
bye'
|
|count-unique.in is the input for Ben Hoyt's count-unique example and >|consists of 10 bibles, 998170 lines, or 43MB. On a 4GHz Skylake >|gforth-fast takes 802M cycles or 0.201s user time for this (and 31M
|cycles or 0.011s without the call to FOO).
|
|[...]
|I have also measured SwiftForth 3.11.0 with the
|same benchmark:
|
|[...]
| 5,945,307,655 cycles
| 768,834,475 cycles:u
| 5,131,957,800 cycles:k
| 733,948,005 instructions:u # 0.95 insn per cycle >|
| 1.488727585 seconds time elapsed
|
| 0.916473000 seconds user
| 0.572295000 seconds sys
I.e., this just measures reading a 43MB file with 998170 lines with
repeated READ-LINEs. Gforth uses buffered I/O (with the buffering >implemented by glibc), while SwiftForth uses unbuffered I/O and >REPOSITION-FILE. As a result, SwiftForth spends more than 7 times
more cycles on this benchmark than Gforth. I would be surprised if
the strategy used by SwiftForth fared better than buffering on other
OSs.
- anton--
In article <ua4id9$2s1s9$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
On 30/07/2023 9:11 am, albert wrote:
...
As long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
AFAICS that's only feasable if the whole file fits in memory. I don't
have that option.
I invented (ACCEPT) that exactly for the situation reading from a stream.
(I never ever use READ-LINE for a file that fits in memory.)
It goes as follows:
1.
If there is a ret after the pointer in the input buffer,
return the part between the pointer and the ret
advance pointer. you're done
2.
copy the remainder to the start of the buffer
read from the sream to fill the buffer
continue at step 1
The main issue in my case seems to be the file-repositioning that is
the basis of most READ-LINE implementations. Under CP/M it hits hard
for some reason. I should first verify the algorithm I used is as
efficient as can be. (It came from a C compiler, so how could it not? :)
Under CP/M file segments are visible. I would read only whole segments
which made the above algorithm even simpler. The copying step
is superfluous.
In article <ua4id9$2s1s9$1...@dont-email.me>, dxforth <dxf...@gmail.com> wrote:
On 30/07/2023 9:11 am, albert wrote:
...
As long as you are buffering you can deal out the lines without
copying as a string constant (addr n).
AFAICS that's only feasable if the whole file fits in memory. I don'tI invented (ACCEPT) that exactly for the situation reading from a stream.
have that option.
(I never ever use READ-LINE for a file that fits in memory.)
It goes as follows:
1.
If there is a ret after the pointer in the input buffer,
return the part between the pointer and the ret
advance pointer. you're done
2.
copy the remainder to the start of the buffer
read from the sream to fill the buffer
continue at step 1
And for Rick : RET is trimmed off, so why don you care?
On Sunday, July 30, 2023 at 6:26:56 AM UTC-5, dxforth wrote:
I do the same as Albert, read in a buffer then point to a
line and work down the buffer till no more line feed then
move the residue up and do again.
Back in DOS days had heard that 4096 byte buffer was
optimum for the read so that was what I used but I also
used much smaller buffer without problem.
...
Further experiments with CP/M showed I didn't need to read all or most
of the file to gain a substantial speed increase. A 512 byte buffer
was sufficient to gain a 10x improvement over READ-LINE. Raising it
50KB produced no noticeable benefit. /EOL was written in assembler.
...
Further experiments with CP/M showed I didn't need to read all or most
of the file to gain a substantial speed increase. A 512 byte buffer
was sufficient to gain a 10x improvement over READ-LINE. Raising it
50KB produced no noticeable benefit. /EOL was written in assembler.
Unfortunately it has a bug. If the last byte in the buffer is the
first char of a CRLF it will result in an additional empty line.
Initial tests with a large buffer didn't reveal the problem. The EOL
scanner itself is fine. It simply wasn't intended for use in a stream
which doesn't backtrack.
In article <uaa03r$3ijne$1@dont-email.me>, dxforth <dxforth@gmail.com> wrote:
On 30/07/2023 5:07 pm, dxforth wrote:Use a circular buffer array that corresponds to disk sectors.
...
Further experiments with CP/M showed I didn't need to read all or most
of the file to gain a substantial speed increase. A 512 byte buffer
was sufficient to gain a 10x improvement over READ-LINE. Raising it
50KB produced no noticeable benefit. /EOL was written in assembler.
Unfortunately it has a bug. If the last byte in the buffer is the
first char of a CRLF it will result in an additional empty line.
Initial tests with a large buffer didn't reveal the problem. The EOL
scanner itself is fine. It simply wasn't intended for use in a stream
which doesn't backtrack.
If you discover that you can't do READ-LINE because the line endings
are not present in the buffers, read the buffers that have become
empty. Then try again.
How long the line endings are doesn't matter.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 300 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 52:52:50 |
| Calls: | 6,712 |
| Files: | 12,243 |
| Messages: | 5,355,184 |