• Fidonet archive

    From Marceline Jones@1:103/705 to ALL on Thu Feb 25 18:07:00 2021
    I want all Fidonet echomail messages ever created.

    Is there such an archive ? is it on "textfiles.com", or did some precient coordinator backup all their uncompressed packets somewhere ?

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From mark lewis@1:3634/12 to Marceline Jones on Thu Feb 25 04:00:12 2021
    Re: Fidonet archive
    By: Marceline Jones to ALL on Thu Feb 25 2021 18:07:00


    I want all Fidonet echomail messages ever created.

    you can forget that... there's no such archive... there was an attempt at one time but it was shelved after a decade or so... it didn't have everything, either...

    it should also be pointed out that the FTN MSGID standard limits MSGIDs to only 3 years before the serial numbers may be repeated... dupe detection will eliminate many...

    then there's the problem of how will you store them... you're attempting to work with wildcat bbs software which has a proprietary format and most certainly has limitations that will prevent it from containing all the messages
    if they could all be found... every message base format used in FTN has limitations that will prevent this task... not to mention that it is about 30 years too late getting started...


    )\/(ark
    --- SBBSecho 3.11-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Björn Felten@2:203/2 to Marceline Jones on Thu Feb 25 16:48:24 2021
    I want all Fidonet echomail messages ever created.

    LOL! I don't think you understand what you are asking for.

    I have every message created in this echo since I was elected moderator in 2002. It takes up 268MB of space here, meaning that it's probably more than a quarter of a million messages. This is from almost ten years after Fidonet's peak -- there's probably another quarter of a million messages from the years before that.

    Can your BBS handle even this one echo? It's not even the biggest one...



    ..

    --- Mozilla/5.0 (Windows; U; Windows NT 5.1; sv-SE; rv:1.9.1.16) Gecko/20101125
    * Origin: news://eljaco.se (2:203/2)
  • From Marceline Jones@1:103/705 to BJöRN FELTEN on Sun Feb 28 16:47:00 2021
    LOL! I don't think you understand what you are asking for.

    I want all the data.

    I have every message created in this echo since I was elected
    moderator in 2002. It takes up 268MB of space here, meaning that it's probably more than a quarter of a million messages. This is from
    almost ten years after Fidonet's peak -- there's probably another
    quarter of a million messages from the years before that.

    I expect there to be millions of messages ever created in FN_SYSOP. Is that a problem ?

    Can your BBS handle even this one echo? It's not even the biggest
    one...

    Why not. I can make it work.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Marceline Jones@1:103/705 to MARK LEWIS on Sun Feb 28 16:48:00 2021
    I want all Fidonet echomail messages ever created.

    you can forget that... there's no such archive... there was an attempt
    at one time but it was shelved after a decade or so... it didn't have everything, either...

    People should bring the attempt back to life.

    There are a lot of projects like that:

    - archive.org
    - bbsdocumentary
    - textfiles.com
    - wayback machine
    - winworld
    - vetusware

    just to name a few.

    it should also be pointed out that the FTN MSGID standard limits
    MSGIDs to only 3 years before the serial numbers may be repeated...
    dupe detection will eliminate many...

    Then disable dupe detection or limit each segment to 3 years. It is not necessary to have 1 monolith message base containing data for all years at once. The message base can be broken up into static sections. Or somebody can write a new message base format that can handle it all. Maybe put it in PostgreSQL.

    then there's the problem of how will you store them... you're
    attempting to work with wildcat bbs software which has a proprietary format and most certainly has limitations that will prevent it from containing all the messages if they could all be found... every message base format used in FTN has limitations that will prevent this task...

    Just work around the limitations.

    not to mention that it is about 30 years too late getting started...

    It is not too late. Try telling Jason Scott that it is too late.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From mark lewis@1:3634/12 to Marceline Jones on Sun Feb 28 04:51:43 2021
    Re: Fidonet archive
    By: Marceline Jones to MARK LEWIS on Sun Feb 28 2021 16:48:00


    not to mention that it is about 30 years too late getting started...

    It is not too late. Try telling Jason Scott that it is too late.

    yes, it is too late because the messages are simply gone... deleted during system maint as they were purged out due to the operators' message retention settings... we won't even mention all the ones that were lost when a
    message base overflowed and corrupted itself requiring deletion of the message base data files...


    )\/(ark
    --- SBBSecho 3.11-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Nick Andre@1:229/426 to Marceline Jones on Sun Feb 28 04:54:26 2021
    On 28 Feb 21 16:48:00, Marceline Jones said the following to Mark Lewis:

    it should also be pointed out that the FTN MSGID standard limits MSGIDs to only 3 years before the serial numbers may be repeated... dupe detection will eliminate many...

    Then disable dupe detection or limit each segment to 3 years. It is not necessary to have 1 monolith message base containing data for all years at once. The message base can be broken up into static sections. Or somebody c

    Nevermind that lazy bullshit that is MSGID... its really just for reply linking which most messages from the 80's and 90's didn't have. The only "challenge" is deciding on a storage method for containing large amounts of messages.

    Nothing is ever purged here. I have message bases going back to.... early 2000's I think? Would love to help out or help track down older stuff.

    Nick

    --- Renegade vY2Ka2
    * Origin: Joey, do you like movies about gladiators? (1:229/426)
  • From Marceline Jones@1:103/705 to MARK LEWIS on Sun Mar 7 17:53:00 2021
    not to mention that it is about 30 years too late getting started...

    It is not too late. Try telling Jason Scott that it is too late.

    yes, it is too late because the messages are simply gone... deleted
    during system maint as they were purged out due to the operators'
    message retention settings... we won't even mention all the ones that
    were lost when a message base overflowed and corrupted itself requiring deletion of the message base data files...

    People could still submit what they have. This is a history preservation project.

    Somebody should set up a Fidonet Archive mailer to process and dedupe all historical mail. All messages get tossed into a database. SysOps can then scan out all their existing message bases, including older ones from backups. It is that easy.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Marceline Jones@1:103/705 to NICK ANDRE on Sun Mar 7 17:54:00 2021
    Nevermind that lazy bullshit that is MSGID... its really just for
    reply linking which most messages from the 80's and 90's didn't have.
    The only "challenge" is deciding on a storage method for containing
    large amounts of messages.

    Any modern database will handle it.

    Nothing is ever purged here. I have message bases going back to....
    early 2000's I think? Would love to help out or help track down older stuff.
    Nick

    You can setup multiple D'Bridge nodes to process gigabytes of mail 24/7. Have a web status page or echo announcing statistics (eg. volume, messages processed, number of [new] contributors, echo size etc).

    Someone would have to write a tosser for the database.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nick Andre@1:229/426 to Marceline Jones on Sun Mar 7 00:57:01 2021
    On 07 Mar 21 17:54:00, Marceline Jones said the following to Nick Andre:

    Nevermind that lazy bullshit that is MSGID... its really just for reply linking which most messages from the 80's and 90's didn't have. The only "challenge" is deciding on a storage method for containing large amounts of messages.

    Any modern database will handle it.

    I know. I put the the word challenge in quotes because:

    You can setup multiple D'Bridge nodes to process gigabytes of mail 24/7. Ha a web status page or echo announcing statistics (eg. volume, messages processed, number of [new] contributors, echo size etc).

    Someone would have to write a tosser for the database.

    Not sure if anyone ever attempted to write a tosser for SQL... it would be
    an interesting thing to try.

    If this idea is to have some sort of Fido history website... what would be kindof cool is to take that idea further with messages in a database that can be rescanned. Have it so that any Sysop can visit that site, they specify
    a Fido address, echoes and quantity to scan. The website builds an archive of packets with the desired address that the Sysop can toss back to their BBS running whatever and thus having whatever backlog of mail they wanted.

    Nick

    --- Renegade vY2Ka2
    * Origin: Joey, do you like movies about gladiators? (1:229/426)
  • From mark lewis@1:3634/12 to Marceline Jones on Sun Mar 7 06:53:37 2021
    Re: Fidonet archive
    By: Marceline Jones to MARK LEWIS on Sun Mar 07 2021 17:53:00


    People could still submit what they have. This is a history
    preservation project.

    preservation with massive holes is not really preservation... especially in communications where one lost letter completely changes what someone has said/written... we see this with quote systems that are line oriented and
    chop off data when quote attributions are added to the beginning of the line...

    Somebody should set up a Fidonet Archive mailer to process and
    dedupe all historical mail.

    as noted previously, this was done and eventually abandoned...

    All messages get tossed into a database. SysOps can then scan out
    all their existing message bases, including older ones from backups.

    you're not listening... i've been in fidonet since before Policy 4 was introduced for review before being presented for voting... my system has run on four different physical machines... there is no way to have backed up and
    saved all those messages without dupes or loss... especially when a message base corrupts itself due to growing oversize or disk corruption...

    it should also be said that there was never any consideration for archiving messages like that... being able to retain messages for several months was a huge step forward at the time but no operator took the time to move their
    message storage from floppy to some fancy new HD and properitary message base format... the conversion tools simply did not exist...

    It is that easy.

    practical and pragmatic say differently...


    )\/(ark
    --- SBBSecho 3.11-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From mark lewis@1:3634/12 to Nick Andre on Sun Mar 7 07:16:54 2021
    Re: Re: Fidonet archive
    By: Nick Andre to Marceline Jones on Sun Mar 07 2021 00:57:01


    Not sure if anyone ever attempted to write a tosser for SQL... it
    would be an interesting thing to try.

    someone did... a couple of them, AIR... mostly to import/export FTN messages in/out of phpBBS forums... while they worked, kinda, there are/were still problems and some data loss similar to what QWK exhibits... we won't even
    mention the tossing speed...


    )\/(ark
    --- SBBSecho 3.11-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Mauro Veiga@4:801/194 to MARK LEWIS on Mon Mar 8 11:42:00 2021
    Quoting Mark Lewis to Marceline Jones <=-

    it should also be said that there was never any consideration for archiving messages like that... being able to retain messages for
    several months was a huge step forward at the time but no operator
    took the time to move their message storage from floppy to some fancy
    new HD and properitary message base format... the conversion tools
    simply did not exist...

    The NNTP standard always has the hability to retain an enormous
    amount of messages. Today, FTN - NNTP integration is much easier.
    A large NNTP server may be able to store a Fido history if someone
    wishes. Could be one way.



    []'s

    Mauro R. Veiga abutre.no-ip.org:2323 *


    MeSSaGe SiTTeR 1.00 - Full Version
    Live Long and Prosper

    ... * <- Tribble # <- Tribble After Borg Assimilation
    ___ Blue Wave/DOS v2.30
    --- SBBSecho 3.06-Win32
    * Origin: Ninho do Abutre 2 - Rio de Janeiro - Brasil * (4:801/194)
  • From Marceline Jones@1:103/705 to NICK ANDRE on Sun Mar 14 16:45:00 2021
    Not sure if anyone ever attempted to write a tosser for SQL... it
    would be an interesting thing to try.

    We can call the tosser "Dan Clough" (because, reasons):

    ==================================================
    Dan Clough v0.0[izza] Professional Tool

    Usage:

    DC <commands> [switches]

    Commands:

    Toss Garbage in
    Scan Garbage out
    ==================================================

    If this idea is to have some sort of Fido history website... what
    would be kindof cool is to take that idea further with messages in a database that can be rescanned. Have it so that any Sysop can visit
    that site, they specify a Fido address, echoes and quantity to scan.
    The website builds an archive of packets with the desired address that
    the Sysop can toss back to their BBS running whatever and thus having whatever backlog of mail they wanted.
    Nick

    My initial thoughts were for a downloadable database like Wikipedia that SysOps can use as a datasource for whatever they want. They can build services around it, write more import/export scripts, use it to catchup missing mail, performance and stress test tools etc. The fun lies in the system design and implementation. Maybe it can run on block chain.

    Imagine the glory of being the one who pulls it off. Everybody will be using it. It will be more popular than Synchronet and Husky combined because message areas are common to all BBS's. Maybe gamify it to show who has contributed the most.
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Marceline Jones@1:103/705 to MARK LEWIS on Sun Mar 14 16:58:00 2021
    People could still submit what they have. This is a history
    preservation project.

    preservation with massive holes is not really preservation...
    especially in communications where one lost letter completely changes
    what someone has said/written... we see this with quote systems that
    are line oriented and chop off data when quote attributions are added
    to the beginning of the line...

    That does not prevent collecting what is available.

    Somebody should set up a Fidonet Archive mailer to process and
    dedupe all historical mail.

    as noted previously, this was done and eventually abandoned...

    The longer you do nothing, the more you will lose.

    All messages get tossed into a database. SysOps can then scan out
    all their existing message bases, including older ones from backups.

    you're not listening... i've been in fidonet since before Policy 4 was introduced for review before being presented for voting... my system
    has run on four different physical machines... there is no way to have backed up and saved all those messages without dupes or loss...
    especially when a message base corrupts itself due to growing oversize
    or disk corruption...

    On some disk somewhere somebody has some message that can be restored.

    it should also be said that there was never any consideration for archiving messages like that... being able to retain messages for
    several months was a huge step forward at the time but no operator
    took the time to move their message storage from floppy to some fancy
    new HD and properitary message base format... the conversion tools
    simply did not exist...
    It is that easy.

    practical and pragmatic say differently...

    You will not get it done with that defeatist attitude.

    If you build it, they will come.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nick Andre@1:229/426 to Marceline Jones on Sun Mar 14 08:54:13 2021
    On 14 Mar 21 16:45:00, Marceline Jones said the following to Nick Andre:

    My initial thoughts were for a downloadable database like Wikipedia that SysOps can use as a datasource for whatever they want. They can build servi around it, write more import/export scripts, use it to catchup missing mail performance and stress test tools etc. The fun lies in the system design an implementation. Maybe it can run on block chain.

    I would be supportive of this; mostly for the rescan-ability for missing mail.

    Nick

    --- Renegade vY2Ka2
    * Origin: Joey, do you like movies about gladiators? (1:229/426)
  • From Marceline Jones@1:103/705 to MARK LEWIS on Mon Mar 15 21:15:00 2021
    Not sure if anyone ever attempted to write a tosser for SQL... it
    would be an interesting thing to try.

    someone did... a couple of them, AIR... mostly to import/export FTN messages in/out of phpBBS forums... while they worked, kinda, there are/were still problems and some data loss similar to what QWK
    exhibits... we won't even mention the tossing speed...

    Sounds like they were not using threads.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.13-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From mark lewis@1:3634/12 to Marceline Jones on Mon Mar 15 08:51:15 2021
    Re: Re: Fidonet archive
    By: Marceline Jones to MARK LEWIS on Mon Mar 15 2021 21:15:00


    Sounds like they were not using threads.

    threads are only good for non-serial processes, really... tossing and scanning FTN mail is pretty much a serial process no matter what the storage format is...


    )\/(ark
    --- SBBSecho 3.11-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Marceline Jones@1:103/705 to MARK LEWIS on Sun Mar 21 10:30:00 2021
    Sounds like they were not using threads.

    threads are only good for non-serial processes, really... tossing and scanning FTN mail is pretty much a serial process no matter what the storage format is...

    Tossing and scanning are atomic operations (ie. processing one message does not depend on another) and can be run in multiple threads without synchronisation or interference. Any need to serialise reads and writes sounds like a limitation of the BBS message base format because moderm databases like PostgreSQL, Oracle, MySQL, SQL Server etc are designed for concurrency.

    ___ Blue Wave/386 v2.30
    --- SBBSecho 3.14-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)