• Re: Critical issue with maintenance PurgeUser/PackUser stanzas

    From Avon@21:1/101 to All on Sun Jan 5 12:39:42 2020
    On 02 Jan 2020 at 04:45p, Netsurge pondered and said...

    When my "PurgeUserBase" and "PackUserBase" stanzas ran, it deleted around 15 users from my user base (very active users). The problem is these users were not flagged for deletion. Since I backup my Mystic d files every hour so I restored the userbase prior to the maintenance confirmed they were not marked for deletion.

    Funny you mention that, it also happened to me. It deleted about 20 of my users including my account, the sysop account. I wonder if it has something to do with the year becoming 2020.

    Thanks both for this report... I am going to place a warning on the Wiki not
    to use this function until g00r00 resurfaces and can look into a fix. It's
    not one I have been running at Agency on a daily basis and right now I'm
    kinda glad I have not been. Yikes!

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Avon@21:1/101 to StackFault on Sun Jan 5 12:49:06 2020
    On 02 Jan 2020 at 10:44a, StackFault pondered and said...

    When my "PurgeUserBase" and "PackUserBase" stanzas ran, it deleted
    around 15 users from my user base (very active users). The problem is,

    Just so I'm clear this issue occurs if either function is run?

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From StackFault@21:1/172 to Avon on Sat Jan 4 19:59:44 2020
    When my "PurgeUserBase" and "PackUserBase" stanzas ran, it deleted around 15 users from my user base (very active users). The problem is

    Just so I'm clear this issue occurs if either function is run?

    According to logs, it deleted the users on the Purge stanza, then deleted all messages from the users in ALL bases on the Pack stanza.

    Can't really try to reproduce at the moment and it may also be related to the year change as Netsurge suggested and would be harder to fully reproduce.

    Cheers!

    |15 ß Þ |15StackFault |08<|03.|11.|15P|11h|03EN|11o|15M|11.|03.|08>
    |11 Ý ß |11The Bottomless Abyss BBS
    |03 ß Ýß |03ssh|08.|072222 |08/ |03telnet|08.|072023 |08/ |03https
    |08 ÜþÞ |08bbs|07.|08bottomlessabyss|07.|08net

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: The Bottomless Abyss BBS * bbs.bottomlessabyss.net (21:1/172)
  • From Avon@21:1/101 to StackFault on Sun Jan 5 14:21:04 2020
    On 04 Jan 2020 at 07:59p, StackFault pondered and said...

    According to logs, it deleted the users on the Purge stanza, then
    deleted all messages from the users in ALL bases on the Pack stanza.

    Yep removing all messages from all bases (echo and private) ..if this is what you're seeing... is not good. When g00r00 moved the function to MUTIL it was designed to just remove the private messages on the base (as I understood it
    to be) Looking back the notes show

    [snip]

    MUTIL has a new "PackUserBase" function which will physically delete user
    records, private messages to and from them, their lastread pointers in
    message bases, their scan settings in message and file bases. This can
    only be executed when there are no users online. Default mutil.ini has
    been updated.

    [snip]

    I wonder if the old MBBSUTIL would behave in the same fashion? But then again it's been some time since 1.12 A37 saw it's removal (1 Jan 2018) and when
    1.12 39 came out he made quite a few record data changes to the user data
    base. Hmmm..

    OK thanks for the info, appreciated :)

    Best, Paul

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From ryan@21:1/168 to Avon on Sat Jan 4 17:51:34 2020
    Thanks both for this report... I am going to place a warning on the Wiki not to use this function until g00r00 resurfaces and can look into a
    fix. It's not one I have been running at Agency on a daily basis and
    right now I'm kinda glad I have not been. Yikes!

    Hey Avon / Netsurge - someone mentioned in the general fsxNet sub that there are weird date issues, his echomail subs set the date to 1970. Something strange is afoot here, for sure.

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: monterey bbs (21:1/168)
  • From Avon@21:1/101 to ryan on Sun Jan 5 14:56:00 2020
    On 04 Jan 2020 at 05:51p, ryan pondered and said...

    Hey Avon / Netsurge - someone mentioned in the general fsxNet sub that there are weird date issues, his echomail subs set the date to 1970. Something strange is afoot here, for sure.

    MMmmm... yeah I am just trying to play catch up on several days of echomail right now.. with mixed success I may add :)

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Netsurge@21:4/154 to Avon on Sat Jan 4 22:28:50 2020
    Yep removing all messages from all bases (echo and private) ..if this is what you're seeing... is not good. When g00r00 moved the function to
    MUTIL it was designed to just remove the private messages on the base
    (as I understood it to be) Looking back the notes show

    The user gets deleted with the Purge Function, more than likely due to the 20 year issue. The Pack feature is just doing it's job once the user is deleted.
    I don't think there is anything wrong with that function.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://scinet-ftn.org

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (21:4/154)
  • From Netsurge@21:4/154 to ryan on Sat Jan 4 22:30:54 2020
    Hey Avon / Netsurge - someone mentioned in the general fsxNet sub that there are weird date issues, his echomail subs set the date to 1970. Something strange is afoot here, for sure.

    I was doing some reading on Pascal compilers at two digit year issues and
    sure enough, some of them are afflicted by the year 2020 in which they think it's 1920.

    Not knowing what the source looks like and what compiler g00r00 uses, I can only speculate at this point.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://scinet-ftn.org

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (21:4/154)
  • From Avon@21:1/101 to Netsurge on Sun Jan 5 19:09:46 2020
    On 04 Jan 2020 at 10:28p, Netsurge pondered and said...

    The user gets deleted with the Purge Function, more than likely due to
    the 20 year issue. The Pack feature is just doing it's job once the user is deleted. I don't think there is anything wrong with that function.

    Thanks, this is what I had figured also... but had not tested... the bit that is also being reported that seems a bit strange relates to reports of
    echomail being removed during the pack function... but I may have misunderstood. I think for now just flagging the purge function on the Wiki should cover it.

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From StackFault@21:1/172 to Netsurge on Sun Jan 5 01:34:10 2020
    The user gets deleted with the Purge Function, more than likely due to
    the 20 year issue. The Pack feature is just doing it's job once the user is deleted. I don't think there is anything wrong with that function.

    Well, I think it deletes more than it's supposed to when a user gets deleted. Local public messages and echomail messages should not be deleted from that function.

    Cheers!

    |15 ß Þ |15StackFault |08<|03.|11.|15P|11h|03EN|11o|15M|11.|03.|08>
    |11 Ý ß |11The Bottomless Abyss BBS
    |03 ß Ýß |03ssh|08.|072222 |08/ |03telnet|08.|072023 |08/ |03https
    |08 ÜþÞ |08bbs|07.|08bottomlessabyss|07.|08net

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: The Bottomless Abyss BBS * bbs.bottomlessabyss.net (21:1/172)
  • From StackFault@21:1/172 to Avon on Sun Jan 5 01:44:56 2020
    Thanks, this is what I had figured also... but had not tested... the bit that is also being reported that seems a bit strange relates to reports
    of echomail being removed during the pack function... but I may have misunderstood. I think for now just flagging the purge function on the Wiki should cover it.

    Ok, I've checked the logs closer and while I may have initially thought it
    was related, it appears two functions might be somewhat independently
    affected. The User cleanup and the Messages cleanup.

    Not too sure why it was not all the users or all the messages however...

    Here is a log snippet I had:

    + Jan 02 04:30:10 Purge by age: 365
    ! Jan 02 04:30:10 Purge #82 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #83 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #84 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #85 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #86 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #87 date 01/01/20 days 36526
    + Jan 02 04:30:10 Purge by msgs: Max 500 Cur 81
    + Jan 02 04:30:10 Purged 6

    Here we see it purges messages posted on Jan 1st 2020 thinking it was posted 36526 days from then. And looking at the date it was 36526 days, we get to January 1920, which tend to confirm the possible issue behing related to 1920.

    Looks like we're back to Y2K code issues :)

    Cheers!

    |15 ß Þ |15StackFault |08<|03.|11.|15P|11h|03EN|11o|15M|11.|03.|08>
    |11 Ý ß |11The Bottomless Abyss BBS
    |03 ß Ýß |03ssh|08.|072222 |08/ |03telnet|08.|072023 |08/ |03https
    |08 ÜþÞ |08bbs|07.|08bottomlessabyss|07.|08net

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: The Bottomless Abyss BBS * bbs.bottomlessabyss.net (21:1/172)
  • From Netsurge@21:4/154 to StackFault on Sun Jan 5 02:03:38 2020
    Well, I think it deletes more than it's supposed to when a user gets deleted. Local public messages and echomail messages should not be
    deleted from that function.

    I've replicated the users who have a last login date of 2020 getting deleted
    5 times using different users but none of their messages are deleted. As soon as I restore the users.dat file everything is as it was, with the exception
    of their message read pointers.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://scinet-ftn.org

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (21:4/154)
  • From Nodoka Hanamura@21:2/106 to Avon on Sun Jan 5 02:34:10 2020
    On 05 Jan 2020, Avon said the following...
    Thanks both for this report... I am going to place a warning on the Wiki not to use this function until g00r00 resurfaces and can look into a
    fix. It's not one I have been running at Agency on a daily basis and
    right now I'm kinda glad I have not been. Yikes!


    Second the Notion. I had to go check NCBBS to verify if any of my maint files were running those with those flags on. Thankfully it wasn't.

    Born too late to experience the scene.
    Born just in time to see it come back.
    Nodoka Hanamura - NeoCincinnati BBS SYSOP - neocinci.bbs.io

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/32)
    * Origin: NeoCincinnati BBS - neocinci.bbs.io:23 (21:2/106)
  • From Zip@21:1/202 to StackFault on Sun Jan 5 09:15:44 2020
    Hello StackFault!

    On 05 Jan 2020, StackFault said the following...
    Here we see it purges messages posted on Jan 1st 2020 thinking it was posted 36526 days from then. And looking at the date it was 36526 days,
    we get to January 1920, which tend to confirm the possible issue behing related to 1920.

    Looks like we're back to Y2K code issues :)

    Who would have thought that 20 years ago... :-O

    It appears Mystic BBS 1.12A43 is compiled using FreePascal:

    $ strings mutil | grep FPC
    FPC 3.0.2 [2017/02/05] for x86_64 - Linux

    Best regards
    Zip

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Star Collision BBS, Uppsala, Sweden (21:1/202)
  • From Avon@21:1/101 to StackFault on Sun Jan 5 22:45:06 2020
    On 05 Jan 2020 at 01:44a, StackFault pondered and said...

    Ok, I've checked the logs closer and while I may have initially thought
    it was related, it appears two functions might be somewhat independently affected. The User cleanup and the Messages cleanup.

    Not too sure why it was not all the users or all the messages however...

    Here is a log snippet I had:

    + Jan 02 04:30:10 Purge by age: 365
    ! Jan 02 04:30:10 Purge #82 date 01/01/20 days 36526
    ! Jan 02 04:30:10 Purge #83 date 01/01/20 days 36526

    Which function is being run above SF? Is this the PurgeMessageBases function? Just trying to be clear about what functions in MUTIL are being run that causes these probs :)

    I'm picking we're talking now about PurgeMessageBases and PurgeUserBase both being caught up with this bug. Sound about right?

    Here we see it purges messages posted on Jan 1st 2020 thinking it was posted 36526 days from then. And looking at the date it was 36526 days,
    we get to January 1920, which tend to confirm the possible issue behing related to 1920.
    Looks like we're back to Y2K code issues :)

    I agree - yikes!

    Best, Paul

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Bucko@21:4/131 to Avon on Sun Jan 5 09:23:32 2020
    On 05 Jan 2020, Avon said the following...


    Which function is being run above SF? Is this the PurgeMessageBases function? Just trying to be clear about what functions in MUTIL are
    being run that causes these probs :)

    I'm picking we're talking now about PurgeMessageBases and PurgeUserBase both being caught up with this bug. Sound about right?

    Here we see it purges messages posted on Jan 1st 2020 thinking it was posted 36526 days from then. And looking at the date it was 36526 day we get to January 1920, which tend to confirm the possible issue behi

    Definitly something going on.. I am 2 new messages in fsx this morning, I am showing 5 new messages in Fido.. Last post dates in most of my message bases
    is 1/1/70.. I believe my workaround will be just setting the last scanned
    date to yesterday and all of the new messages will pop up..I have since
    turned off all maint since the majority of it is packing the userbase and purging and packing message and file bases.. Hopefully, g00r00 will toss out
    a quick fix..

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: The Wrong Number Family Of BBS' - Wrong Number ][ (21:4/131)
  • From Bucko@21:4/131 to Avon on Sun Jan 5 09:28:26 2020
    On 05 Jan 2020, Avon said the following...


    Thanks, this is what I had figured also... but had not tested... the bit that is also being reported that seems a bit strange relates to reports
    of echomail being removed during the pack function... but I may have misunderstood. I think for now just flagging the purge function on the Wiki should cover it.


    Echomail is definitly an issue. I posted in another post what I am seeing the last post date is getting set to 1/1/1970.. There was 1 new post for me this morning in fsx that was my logging on and sending the post for the last 10 callers. I have since reset the last scanned date to yesterday and I have new posts showing..

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: The Wrong Number Family Of BBS' - Wrong Number ][ (21:4/131)
  • From Netsurge@21:4/154 to StackFault on Sun Jan 5 14:06:50 2020
    Well, I think it deletes more than it's supposed to when a user gets deleted. Local public messages and echomail messages should not be
    deleted from that function.

    I don't use Mystic for binkp. echomail processing or message base maint; I
    use the husky stuff for all of that. That might be why I haven't experienced all of the message weirdness that you have.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://scinet-ftn.org

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (21:4/154)
  • From Avon@21:1/101 to Netsurge on Tue Jan 7 10:52:24 2020
    On 04 Jan 2020 at 10:30p, Netsurge pondered and said...

    I was doing some reading on Pascal compilers at two digit year issues and sure enough, some of them are afflicted by the year 2020 in which they think it's 1920.

    Not knowing what the source looks like and what compiler g00r00 uses, I can only speculate at this point.

    I've looked at some of the source code and found the issue (I think).

    In short there's a couple of functions that convert dates to Julian and DOS formats... in both you see this


    If Year < 20 Then
    Inc(Year, 2000)
    Else
    Inc(Year, 1900);

    and


    If Dt.Year < 80 Then
    Inc(DT.Year, 2000)
    Else
    Inc(DT.Year, 1900);

    There's a our problem...

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From Netsurge@21:4/154 to Avon on Mon Jan 6 21:16:56 2020
    If Year < 20 Then
    Inc(Year, 2000)
    Else
    Inc(Year, 1900);

    That will definitely do it.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://scinet-ftn.org

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (21:4/154)
  • From Avon@21:1/101 to Netsurge on Wed Jan 8 13:53:06 2020
    On 06 Jan 2020 at 09:16p, Netsurge pondered and said...

    That will definitely do it.

    Yep Yep... wish I could fix it.

    --- Mystic BBS v1.12 A43 2019/03/03 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)