Hello World!
just want to create a topic for my repo. maybe anyone finds it useful.
https://github.com/go-while/nntp-overview
it is running great so far and connected to a usenet server i'm writing.
feeding mbox files from archive.org as hard as i can and the
nntp-storage module (not released yet) in combo with nntp-overview
writes articles faster than i can extract them from mbox files and feed
them via nntp :=D
why i'm doing this?
i tried importing few tb from archive.org and did not wanna wait for
month. inn2 is too slow. limited to single core and gets slower and
slower with every more incoming articles it has to check vs history.
diablo is hell to compile... config... documentation?
any (performant) alternatives do(n't) exist?
my code works without a history file. it can write one but i see no need
why. the overview files represent a perfect history as long as the app
not crashes and crushes the memory mapped overview files xD
i'm simply storing articles with their messageid as sha256 in flat file structure. checking if an article exists is easy, stat the filesystem
for the head file (and body if you want).
storage engine splits head and body in different files and stores them
in different parent dirs, 3 levels deep [a-f0-9] so heads and body can
go to different hdd/ssd/nfs storage for example.
good old reiserfs on zfs is quite fast with enough disks and ram :D
still not sure about recordsize, need more benchmarking. bigger
recordsize gives better compression but write amplification gets nasty.
set to 32K at the moment but got a broken disk and waiting for
resilvering to finish. almost done my playground! :D :=)
state: DEGRADED
status: One or more devices is currently being resilvered.
The pool will continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 12 16:32:32 2023
39.2T scanned at 1.66G/s, 38.0T issued at 1.61G/s, 39.2T total
1.79T resilvered, 96.95% done, 00:12:42 to go
redis is also already included in the storage engine to map group/msgnum
to storage/msgidhash or/and it creates softlinks, both from overview
xref information. that's already working and you can set as many workers
as you need =) even more workers that can push to redis but limit
syncwrites for softlinks ;) maybe use redis, mongo or rocksdb as storage
too or invent cyclic buffers in go but i only have 2 hands and a wife
and i love both more than anything, my wife and go, gogo power rangers!
what do you think?
does the world need another usenet server written in go?
do it anyways, as far as i can ;)
i'll update when new server arrives and public access to the still
importing archive is possible, maybe this weekend.
a patch-1 is queued and i hope to find some beta tester here.
go-while, over and out.
https://github.com/go-while/nntp-overview
nntp-overview
nntp-overview generates .overview files per group from incoming usenet
headers (POST, IHAVE, TAKETHIS).
Generation is done in a concurrent way and files are mmap'ed while open.
Overview file content is human readable based on RFC overview.FMT
OV_Handler
OV_Handler processes MMAP open/retrieve/park/close requests and
schedules workers for writing overview data.
The system keeps track of last message number per group when adding new overview to group.
When integrated into a usenet server: works as a central message
numbering station per group.
Example integration in repo:
https://github.com/go-while/nntp-overview_test
License
MIT
Author
go-while
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)