• HPT and dupe detection

    From Dan Richter@1:317/3 to All on Sun Jul 14 16:51:04 2019
    Hi All,

    I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding message areas, but found out that Mystic didn't do a very good job with dupes with different paths.

    I'm a little leery to set any echos up with multiple feeds until I know HPT
    is up to the task... :)


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From mark lewis@1:3634/12.73 to Dan Richter on Sun Jul 14 21:01:06 2019

    On 2019 Jul 14 16:51:04, you wrote to All:

    I was just wondering how well the dupe detection works with HPT.

    it seems to work pretty well here on this point system... granted, it is fed from a synchronet system that is configured to pass on all mail even if it is a dupe... i have my HPT set to retain hashes for just over 3 years... i've not seen any dupes get by and some have been caught that really were dupes or at least using recycled serial numbers...

    i say go for it :)

    )\/(ark

    And to this end they built themselves a stupendous super-computer which was
    so amazingly intelligent that even before its data banks had been connected
    up it had started from "I think therefore I am" and got as far as deducing
    the existence of rice pudding and income tax before anyone managed to turn
    it off.
    ... I just invested half my life savings in yeast!
    ---
    * Origin: (1:3634/12.73)
  • From Alan Ianson@1:153/757 to Dan Richter on Sun Jul 14 18:00:16 2019
    Hello Dan,

    I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding
    message areas, but found out that Mystic didn't do a very good job
    with dupes with different paths.

    I don't link with multiple feeds for the same area but some of my links do so I get dupes here. HPT handles that without any problems that I have seen.

    I'm a little leery to set any echos up with multiple feeds until I
    know HPT is up to the task... :)

    I don't think you'll have any problems... :)

    Ttyl :-),
    Al

    ... A program is used to turn data into error messages.
    --- GoldED+/LNX 1.1.5-b20180707
    * Origin: The Rusty MailBox - Penticton, BC Canada (1:153/757)
  • From Dan Richter@1:317/3 to Alan Ianson on Sun Jul 14 19:30:24 2019
    On 14 Jul 2019, Alan Ianson said the following...

    I was just wondering how well the dupe detection works with HPT. I originally had a fidoweb setup here, with multiple hubs feeding

    I don't link with multiple feeds for the same area but some of my links
    do so I get dupes here. HPT handles that without any problems that I
    have seen.

    That's what I wanted to hear. :) I have only seen a few messages get caught
    as dupes so far with HPT, so I wasn't sure if there just wasn't many coming through, or if HPT wasn't catching them...


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Dan Richter@1:317/3 to mark lewis on Sun Jul 14 19:31:46 2019
    On 14 Jul 2019, mark lewis said the following...

    I was just wondering how well the dupe detection works with HPT.

    i say go for it :)

    Alright. I'll turn on a few echos and see what happens. If it doesn't work, I'll just say that mark told me to... ;)


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Frank Linhares@1:229/101 to Dan Richter on Sun Jul 14 23:11:24 2019
    I'm a little leery to set any echos up with multiple feeds until I know HPT is up to the task... :)

    If you mean multiple feeds feeding the same echo, then it isn't dupe
    detection that will keep things from being dupes but seen-bys.

    For example, I also cross hub with other major fidonet hubs. If I send a message in this echo it will send it to all the other hubs and since I am sending it to them it will put their node numbers in the seen-by. When they receive it they can see that the other hubs already have it and they won't
    send it to them.

    Clear as mud?

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)
  • From Dan Richter@1:317/3 to Frank Linhares on Sun Jul 14 21:43:36 2019
    On 14 Jul 2019, Frank Linhares said the following...

    If you mean multiple feeds feeding the same echo, then it isn't dupe detection that will keep things from being dupes but seen-bys.

    That's true. But, it would still cause dupes... :)

    For example, I also cross hub with other major fidonet hubs. If I send a message in this echo it will send it to all the other hubs and since I am sending it to them it will put their node numbers in the seen-by. When they receive it they can see that the other hubs already have it and
    they won't send it to them.

    That's how we have fsxNet hubs set up, and it seems to work fine. I just
    didn't want to cause a bunch dupes in any fidonet echos... Right now, I do
    have this echo set up with multiple feeds. I just want to make sure... :)


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Frank Linhares@1:229/101 to Dan Richter on Mon Jul 15 00:31:20 2019
    That's true. But, it would still cause dupes... :)

    How would it cause dupes? If one of my peer hubs sends me a message that has another peer hubs node number in the seen-by then my system won't export it
    to theres, plain and simple.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)
  • From Dan Richter@1:317/3 to Frank Linhares on Sun Jul 14 22:57:20 2019
    On 15 Jul 2019, Frank Linhares said the following...

    That's true. But, it would still cause dupes... :)

    How would it cause dupes? If one of my peer hubs sends me a message that has another peer hubs node number in the seen-by then my system won't export it to theres, plain and simple.

    If the seen-bys are read and processed correctly, there wouldn't be any
    dupes. But, not all systems do that, as I found out previously with Mystic. I was receiving messages from one hub, they would get sent out to my other hub, and caused a mess in those echos.


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Frank Linhares@1:229/101 to Dan Richter on Mon Jul 15 01:06:00 2019
    If the seen-bys are read and processed correctly, there wouldn't be any dupes. But, not all systems do that, as I found out previously with Mystic. I was receiving messages from one hub, they would get sent out
    to my other hub, and caused a mess in those echos.

    I wasn't referencing Mystic. I was talking about Husky.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)
  • From Richard Menedetter@2:310/31 to Dan Richter on Mon Jul 15 09:10:52 2019
    Hi Dan!

    14 Jul 2019 16:51, from Dan Richter -> All:

    I'm a little leery to set any echos up with multiple feeds until I
    know HPT is up to the task... :)

    Many HPT users, including me, do that.
    I have not heared (or seen) any major issues.

    You should be good to go.
    I have these configs:
    # The type of dupebase used. See docs. # dupeBaseType <TextDupes | HashDupes | HashDupesWMsgId | CommonDupeBase> DupeBaseType HashDupesWMsgId
    # Maximum days for storing you hashes in CommonDupeBase. # For other dupebase types see -DupeHistory option in EchoArea statement. AreasMaxDupeAge 1100

    CU, Ricsi

    ... Character is what you are. Reputation is what people think you are.
    --- GoldED+/LNX
    * Origin: I always use the goodest English. (2:310/31)
  • From Dan Richter@1:317/3 to Richard Menedetter on Mon Jul 15 01:45:50 2019
    On 15 Jul 2019, Richard Menedetter said the following...

    Many HPT users, including me, do that.
    I have not heared (or seen) any major issues.

    Well, I have this echo set to receive from two hubs as a test.

    # For other dupebase ty pes see -DupeHistory option in EchoArea
    statement. AreasMaxDupeAge 1100

    1100?! Wow! I thought I was doing good when I had it set to 31... :) I did change it to 1100, just to be safe.

    # The type of dupebase used. See docs. # dupeBaseType <TextDupes | HashDupes | H ashDupesWMsgId | CommonDupeBase> DupeBaseType HashDupesWMsgId # Maximum days for storing you hashes in CommonDupeBase.

    I didn't have the dupeBaseType set, but I did set it now to HashDupesWMsgId.

    Thanks


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Richard Menedetter@2:310/31 to Dan Richter on Mon Jul 15 12:58:22 2019
    Hi Dan!

    15 Jul 2019 01:45, from Dan Richter -> Richard Menedetter:

    # For other dupebase ty pes see -DupeHistory option in EchoArea
    statement. AreasMaxDupeAge 1100
    1100?! Wow! I thought I was doing good when I had it set to 31... :)
    I did change it to 1100, just to be safe.

    3 years
    Why not ;)

    # The type of dupebase used. See docs. # dupeBaseType <TextDupes
    | HashDupes | H ashDupesWMsgId | CommonDupeBase> DupeBaseType
    HashDupesWMsgId # Maximum days for storing you hashes in
    CommonDupeBase.
    I didn't have the dupeBaseType set, but I did set it now to HashDupesWMsgId.

    I have forgotten what the config default is.
    When I had a look I concluded for myself that that is the best type.
    I hope I was right ;)

    CU, Ricsi

    ... If you do a favor, forget it. If you receive a favor, remember it.
    --- GoldED+/LNX
    * Origin: Windows NT? New Technology? I don't think so... (2:310/31)
  • From mark lewis@1:3634/12.73 to Dan Richter on Mon Jul 15 09:31:22 2019

    On 2019 Jul 14 19:31:46, you wrote to me:

    I was just wondering how well the dupe detection works with HPT.

    i say go for it :)

    Alright. I'll turn on a few echos and see what happens. If it doesn't
    work,
    I'll just say that mark told me to... ;)

    hahahahaha... :thumbsup:

    )\/(ark

    And to this end they built themselves a stupendous super-computer which was
    so amazingly intelligent that even before its data banks had been connected
    up it had started from "I think therefore I am" and got as far as deducing
    the existence of rice pudding and income tax before anyone managed to turn
    it off.
    ... Money Talks! Mine always says goodbye!
    ---
    * Origin: (1:3634/12.73)
  • From mark lewis@1:3634/12.73 to Dan Richter on Mon Jul 15 09:33:14 2019

    On 2019 Jul 14 22:57:20, you wrote to Frank Linhares:

    That's true. But, it would still cause dupes... :)

    How would it cause dupes? If one of my peer hubs sends me a message
    that has another peer hubs node number in the seen-by then my system
    won't export it to theres, plain and simple.

    If the seen-bys are read and processed correctly, there wouldn't be
    any dupes.

    ummm... yes, there could be... but that depends on what is defined as a dupe... is it duplicated MSGID? is it duplicate message body even with a different MSGID? is it a duplicate message body that has been reformatted in transit (which should never be done)??

    But, not all systems do that, as I found out previously with Mystic. I
    was receiving messages from one hub, they would get sent out to my
    other hub, and caused a mess in those echos.

    mystic is known to have problems in a couple of areas... some versions are known to reformat messages that should not be reformatted... that reformatting will absolutely throw off tossers that hash the message bodies in addition to tracking MSGID control lines and headers... we've seen messages passing through mystic systems coming out the other side with EOLs added where there were none before, characters stripped that should not have been and similar problems... this stuff was all working perfectly in v1.11 when i was using mystic and providing feedback but something has changed that should not have and well... it just ain't right...

    )\/(ark

    And to this end they built themselves a stupendous super-computer which was
    so amazingly intelligent that even before its data banks had been connected
    up it had started from "I think therefore I am" and got as far as deducing
    the existence of rice pudding and income tax before anyone managed to turn
    it off.
    ... No woman ever shot a man while he was doing the dishes.
    ---
    * Origin: (1:3634/12.73)
  • From Tommi Koivula@2:221/6 to Dan Richter on Mon Jul 15 20:25:36 2019
    Hi Dan.

    15 Jul 19 09:10, Richard Menedetter wrote to you:

    Hi Dan!

    14 Jul 2019 16:51, from Dan Richter -> All:

    I'm a little leery to set any echos up with multiple feeds until I
    know HPT is up to the task... :)

    Many HPT users, including me, do that.
    I have not heared (or seen) any major issues.

    I second this. I've run hpt for a long time as a fidoweb hub. The only bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

    Do you want another feed? ;-)

    'Tommi

    --- GoldED+/LNX 1.1.5-b20180707
    * Origin: nntps://news.fidonet.fi (2:221/6)
  • From Dan Richter@1:317/3 to Richard Menedetter on Mon Jul 15 13:14:42 2019
    On 15 Jul 2019, Richard Menedetter said the following...

    1100?! Wow! I thought I was doing good when I had it set to 31... :) I did change it to 1100, just to be safe.

    3 years
    Why not ;)

    Good point. Why not. ;) I think the default is 10...

    I have forgotten what the config default is.
    When I had a look I concluded for myself that that is the best type.
    I hope I was right ;)

    Between you and me, we'll find out... :)


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From mark lewis@1:3634/12.73 to Dan Richter on Mon Jul 15 16:08:14 2019

    On 2019 Jul 15 13:14:42, you wrote to Richard Menedetter:

    I did change it to 1100, just to be safe.

    3 years
    Why not ;)

    Good point. Why not. ;) I think the default is 10...

    do i see some binary humor there? -=B-)

    )\/(ark

    And to this end they built themselves a stupendous super-computer which was
    so amazingly intelligent that even before its data banks had been connected
    up it had started from "I think therefore I am" and got as far as deducing
    the existence of rice pudding and income tax before anyone managed to turn
    it off.
    ... A wise man who knows proverbs, reconciles difficulties -Yoruba
    ---
    * Origin: (1:3634/12.73)
  • From Dan Richter@1:317/3 to mark lewis on Mon Jul 15 13:19:32 2019
    On 15 Jul 2019, mark lewis said the following...

    If the seen-bys are read and processed correctly, there wouldn't be any dupes.

    ummm... yes, there could be... but that depends on what is defined as a dupe... is it duplicated MSGID? is it duplicate message body even with a different MSGID ? is it a duplicate message body that has been
    reformatted in transit (which sho uld never be done)??

    In this case, it would be the same message and MSGID, but with different PATH and SEEN-BY kludge lines.

    mystic is known to have problems in a couple of areas... some versions
    are known to reformat messages that should not be reformatted... that reformatting will a bsolutely throw off tossers that hash the message bodies in addition to tracking MSGID control lines and headers... we've

    Now, I don't want to bash Mystic, as I run it here, and love it. There are
    some quirks when trying to run it as a hub, that I've seen. That's part of
    the reason I switched over to Husky projects. :)


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Dan Richter@1:317/3 to Tommi Koivula on Mon Jul 15 13:25:44 2019
    On 15 Jul 2019, Tommi Koivula said the following...

    I second this. I've run hpt for a long time as a fidoweb hub. The only
    bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

    So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do you have new messages sent to both feeds for that echo?

    I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other systems.

    Do you want another feed? ;-)

    I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From mark lewis@1:3634/12 to Dan Richter on Mon Jul 15 17:57:24 2019
    Re: Re: HPT and dupe detection
    By: Dan Richter to Tommi Koivula on Mon Jul 15 2019 13:25:44

    So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do
    you have new messages sent to both feeds for that echo?

    yes, absolutely... otherwise the multiple connections to ensure that mail traverses all systems won't work...

    some of my echos have five or more links that are also interlinked either between themselves or even with other systems between them... it all flows fine... the main thing is the throwing away of all the dupes... it is quite a mindset change from traditional FTN echomail distribution...

    i say that as one of the Z1 backbone star hubs, too ;)

    )\/(ark
    --- SBBSecho 3.07-Linux
    * Origin: SouthEast Star Mail HUB - SESTAR (1:3634/12)
  • From Frank Linhares@1:229/101 to Dan Richter on Mon Jul 15 21:24:58 2019
    So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do you have new messages sent to both feeds for that echo?

    I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other
    systems.

    Yes. If the message comes from your system then both of those feeds would be added to the seen-by. This will prevent them from sending it to each other.

    I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other
    systems.

    If you want to put your mind at ease, feel free to connect to FIDOSOFT.HUSKY
    on my system and test it.

    |15frank |08// |15netsurge
    |07disksh0p|08!|07bbs |08% |07bbs.diskshop.ca |08% |07mystic goodness |11SciNet |03ftn hq |08% |07https://diskshop.ca/scinet

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: % disksh0p!bbs % bbs.diskshop.ca % SciNet ftn hq % (1:229/101)
  • From Tommi Koivula@2:221/6 to Dan Richter on Tue Jul 16 08:01:50 2019
    On 15.7.2019 19.25, Dan Richter wrote:

    I second this. I've run hpt for a long time as a fidoweb hub. The only bug found was that messages without MSGID were not detected as dupes because they arrived via different paths. It was fixed. Works nice now.

    So, if you have multiple feeds for an echo, let's say FIDOSOFT.HUSKY. Do
    you have new messages sent to both feeds for that echo?

    Oh yes. :)

    I have two feeds set for this echo now, but I did set one of them to WRITE-ONLY. I guess I just don't want to cause problems for other systems.

    Sending to both does not normally cause problems. Properly behaving
    tossers check seenbys and paths to _prevent_ dupes. In fidoweb style
    echomail transportation there are always dupes when one message goes
    different ways from a node to another.

    Do you want another feed? ;-)

    I've got that covered already. Right now, I've got access to 5 fido
    hubs. :) Thank you though.

    You could have 6. :D

    'Tommi

    --- Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.
    * Origin: smapinntpd/linux @ nntps://news.fidonet.fi (2:221/6.0)
  • From Dan Richter@1:317/3 to Tommi Koivula on Mon Jul 15 23:26:50 2019
    On 16 Jul 2019, Tommi Koivula said the following...

    Sending to both does not normally cause problems. Properly behaving tossers check seenbys and paths to _prevent_ dupes. In fidoweb style echomail transportation there are always dupes when one message goes different ways from a node to another.

    Ok, I've got the messages being sent to both hubs. It's always interesting to see which one delivers the new messages first... :)

    I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.

    You could have 6. :D

    Do you have any English echos that aren't available from the other hubs? ;)
    I'm connected to 1:229/426, 1:261/38, 1:15/0, 2:250/1, and 3:770/1.


    ---

    |03B|09lack |03P|09anther|03(|09RCS|03)|07

    --- Mystic BBS v1.12 A43 2019/03/02 (Linux/64)
    * Origin: Castle Rock BBS - bbs.castlerockbbs.com (1:317/3)
  • From Tommi Koivula@2:221/6 to Dan Richter on Tue Jul 16 18:03:58 2019
    On 16.07.2019 5:26, Dan Richter -> Tommi Koivula :

    I've got that covered already. Right now, I've got access to 5 fido hubs. :) Thank you though.

    You could have 6. :D

    Do you have any English echos that aren't available from the other hubs? ;) I'm connected to 1:229/426, 1:261/38, 1:15/0, 2:250/1, and 3:770/1.

    I have no link with these, so I can't tell... :) But "full" fidoweb feed
    plus some selected usenet groups.

    'Tommi

    --- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.
    * Origin: smapinntpd/linux @ nntps://news.fidonet.fi (2:221/6.0)