I want all Fidonet echomail messages ever created.
I want all Fidonet echomail messages ever created.
LOL! I don't think you understand what you are asking for.
I have every message created in this echo since I was elected
moderator in 2002. It takes up 268MB of space here, meaning that it's probably more than a quarter of a million messages. This is from
almost ten years after Fidonet's peak -- there's probably another
quarter of a million messages from the years before that.
Can your BBS handle even this one echo? It's not even the biggest
one...
I want all Fidonet echomail messages ever created.
you can forget that... there's no such archive... there was an attempt
at one time but it was shelved after a decade or so... it didn't have everything, either...
it should also be pointed out that the FTN MSGID standard limits
MSGIDs to only 3 years before the serial numbers may be repeated...
dupe detection will eliminate many...
then there's the problem of how will you store them... you're
attempting to work with wildcat bbs software which has a proprietary format and most certainly has limitations that will prevent it from containing all the messages if they could all be found... every message base format used in FTN has limitations that will prevent this task...
not to mention that it is about 30 years too late getting started...
not to mention that it is about 30 years too late getting started...
It is not too late. Try telling Jason Scott that it is too late.
it should also be pointed out that the FTN MSGID standard limits MSGIDs to only 3 years before the serial numbers may be repeated... dupe detection will eliminate many...
Then disable dupe detection or limit each segment to 3 years. It is not necessary to have 1 monolith message base containing data for all years at once. The message base can be broken up into static sections. Or somebody c
not to mention that it is about 30 years too late getting started...
It is not too late. Try telling Jason Scott that it is too late.
yes, it is too late because the messages are simply gone... deleted
during system maint as they were purged out due to the operators'
message retention settings... we won't even mention all the ones that
were lost when a message base overflowed and corrupted itself requiring deletion of the message base data files...
Nevermind that lazy bullshit that is MSGID... its really just for
reply linking which most messages from the 80's and 90's didn't have.
The only "challenge" is deciding on a storage method for containing
large amounts of messages.
Nothing is ever purged here. I have message bases going back to....
early 2000's I think? Would love to help out or help track down older stuff.
Nick
Nevermind that lazy bullshit that is MSGID... its really just for reply linking which most messages from the 80's and 90's didn't have. The only "challenge" is deciding on a storage method for containing large amounts of messages.
Any modern database will handle it.
You can setup multiple D'Bridge nodes to process gigabytes of mail 24/7. Ha a web status page or echo announcing statistics (eg. volume, messages processed, number of [new] contributors, echo size etc).
Someone would have to write a tosser for the database.
People could still submit what they have. This is a history
preservation project.
Somebody should set up a Fidonet Archive mailer to process and
dedupe all historical mail.
All messages get tossed into a database. SysOps can then scan out
all their existing message bases, including older ones from backups.
It is that easy.
Not sure if anyone ever attempted to write a tosser for SQL... it
would be an interesting thing to try.
Quoting Mark Lewis to Marceline Jones <=-
it should also be said that there was never any consideration for archiving messages like that... being able to retain messages for
several months was a huge step forward at the time but no operator
took the time to move their message storage from floppy to some fancy
new HD and properitary message base format... the conversion tools
simply did not exist...
Not sure if anyone ever attempted to write a tosser for SQL... it
would be an interesting thing to try.
If this idea is to have some sort of Fido history website... what
would be kindof cool is to take that idea further with messages in a database that can be rescanned. Have it so that any Sysop can visit
that site, they specify a Fido address, echoes and quantity to scan.
The website builds an archive of packets with the desired address that
the Sysop can toss back to their BBS running whatever and thus having whatever backlog of mail they wanted.
Nick
People could still submit what they have. This is a history
preservation project.
preservation with massive holes is not really preservation...
especially in communications where one lost letter completely changes
what someone has said/written... we see this with quote systems that
are line oriented and chop off data when quote attributions are added
to the beginning of the line...
Somebody should set up a Fidonet Archive mailer to process and
dedupe all historical mail.
as noted previously, this was done and eventually abandoned...
All messages get tossed into a database. SysOps can then scan out
all their existing message bases, including older ones from backups.
you're not listening... i've been in fidonet since before Policy 4 was introduced for review before being presented for voting... my system
has run on four different physical machines... there is no way to have backed up and saved all those messages without dupes or loss...
especially when a message base corrupts itself due to growing oversize
or disk corruption...
it should also be said that there was never any consideration for archiving messages like that... being able to retain messages for
several months was a huge step forward at the time but no operator
took the time to move their message storage from floppy to some fancy
new HD and properitary message base format... the conversion tools
simply did not exist...
It is that easy.
practical and pragmatic say differently...
My initial thoughts were for a downloadable database like Wikipedia that SysOps can use as a datasource for whatever they want. They can build servi around it, write more import/export scripts, use it to catchup missing mail performance and stress test tools etc. The fun lies in the system design an implementation. Maybe it can run on block chain.
Not sure if anyone ever attempted to write a tosser for SQL... it
would be an interesting thing to try.
someone did... a couple of them, AIR... mostly to import/export FTN messages in/out of phpBBS forums... while they worked, kinda, there are/were still problems and some data loss similar to what QWK
exhibits... we won't even mention the tossing speed...
Sounds like they were not using threads.
Sounds like they were not using threads.
threads are only good for non-serial processes, really... tossing and scanning FTN mail is pretty much a serial process no matter what the storage format is...
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 296 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 60:10:33 |
| Calls: | 6,654 |
| Calls today: | 6 |
| Files: | 12,200 |
| Messages: | 5,331,392 |