* Originally in SYNCHRONET
* Crossposted in NET_DEV
Hi mark,
On 2021-04-01 08:33:26, you wrote to me:
in any case, dupe checking in FTN is not done my /just/ detecting duplicate MSGIDs and rejecting the others... the header, including the time stamp, as well as the message body should be taken into
account...
That's how FMail does it.
it should also be said that CRC16/CRC32 on the message bodies is also
not sifficient... even with filtering out white space and the various EoLs... this because, and most programmers know this, there's a
limited supply of CRC values in the tables and it is all too easy to
find "hash clashes"... CRC16 has only 65536 values... CRC32 has only 4294967296 values... the "Birthday Problem" also comes into play...
Indeed. That's why it's on my todo list to go from 32 bit to 64 bit hash values in FMail. That would be more than enough to keep a (very) big dupe "database", and still have a very small probability for hash collisions.
these days, MD5 and SHA1 are also out due to defects in them...
That's because of their security aspect. That's not an issue when you would be using them as hash's for dupe detection.
SHA256 would be the first really useful algorithm or SHA512...
Using a 256 or even 512 bit secure hash value would be overkill for dupe detection. And would be using way too much resources to calculate and check them.
64 Bit is enough, and it doesn't have to be secure. My prime candidate for FMail is this one:
https://github.com/Cyan4973/xxHash
but the real key is to filter out the stuff that can change and hash
only that which won't...
Indeed.
anyway, i'm done with this topic in this area... the discussion really belongs elsewhere for those that are truely interested in implementing proper duplicate detection in FTNs...
(I've crossposted to NET_DEV.)
Bye, Wilfred.
--- FMail-lnx64 2.1.0.18-B20170815
* Origin: FMail development HQ (2:280/464)