Maybe just change:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;
to:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
I recently switched to Diablo on the transit layer, but am observing it is rejecting messages that contain PGP as binary articles. This is preventing valid messages, such as NoCeM notices and messages that aren't encrypted but contain a PGP Signature, to be rejected in some cases, or not propagated to peers (due to "arttypes !binary" in the dnewsfeed definition).
In lib/arttype.c, I see there is a specific check to identify an article with an article type of pgp, but this only checks for "-----BEGIN PGP MESSAGE-----".
Looking at the contents of articles I am rejecting or not propagating, I believe the check should also look the following:
"-----BEGIN PGP SIGNED MESSAGE-----"
and
"-----BEGIN PGP SIGNATURE-----"
but I am not a developer and can't quite cobble anything together from the existing code that works to add these additional values.
If anyone out there is willing to help me with a patch, I will be forever grateful!
This is the portion of lib/arttype.c that is causing the misidentification:
/*
* We look for binary formats first, since these eat
* up CPU like there is no tomorrow. We may miss
* some flags if we detect a binary file, the
* alternative is to remove the return(Type)'s and
* eat CPU for the entire binary.
*/
linetype = classifyLineAsTypes(ptr, len);
if (linetype & UUE) {
Base64 = 0;
BinHex = 0;
UUencode++;
if (UUencode > 8) {
Type |= ARTTYPE_UUENCODE | ARTTYPE_BINARY;
return(Type);
}
} else if (linetype & BHX) {
UUencode = 0;
Base64 = 0;
BinHex++;
if (BinHex > 8) {
Type |= ARTTYPE_BINHEX | ARTTYPE_BINARY;
return(Type);
}
} else if (linetype & B64) {
UUencode = 0;
BinHex = 0;
Base64++;
if (Base64 > 8) {
Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
return(Type);
}
} else {
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;
UUencode = 0;
Base64 = 0;
BinHex = 0;
}
}
On Mar 19, 2023 at 8:30:32 PM CDT, "Retro Guy" <Retro Guy> wrote:
Maybe just change:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;
to:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
Thanks for that, not being a C developer I didn't even put together that "27" was the number of characters in the string, didn't know how to account for that, and now feel like a n00b.
I will recompile and give this a shot, I really appreciate it!
-Jesse
On Mar 19, 2023 at 9:05:25 PM CDT, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:
On Mar 19, 2023 at 8:30:32 PM CDT, "Retro Guy" <Retro Guy> wrote:
Maybe just change:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP MESSAGE-----", 27) == 0)
Type |= ARTTYPE_PGPMESSAGE;
to:
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
Thanks for that, not being a C developer I didn't even put together that "27"
was the number of characters in the string, didn't know how to account for >> that, and now feel like a n00b.
I will recompile and give this a shot, I really appreciate it!
-Jesse
Hrm, I guess another portion of the code higher up is taking precedence as this didn't change the behavior with messages like: <tv8i7j$q8m$1@rasp.pasdenom.info>
On Mar 19, 2023 at 10:25:12 PM CDT, "Jesse Rehmer"
Hrm, I guess another portion of the code higher up is taking precedence as >> this didn't change the behavior with messages like:
<tv8i7j$q8m$1@rasp.pasdenom.info>
I'm thinking it is the portion of code right above the PGP stuff:
} else if (linetype & B64) {
UUencode = 0;
BinHex = 0;
Base64++;
if (Base64 > 8) {
Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
return(Type);
}
I'm assuming this is counting any article with more than 8 lines of Base64 encoded text as binary?
On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
On Mar 19, 2023 at 10:25:12 PM CDT, "Jesse Rehmer"
Hrm, I guess another portion of the code higher up is taking precedence as >>> this didn't change the behavior with messages like:
<tv8i7j$q8m$1@rasp.pasdenom.info>
I'm thinking it is the portion of code right above the PGP stuff:
} else if (linetype & B64) {
UUencode = 0;
BinHex = 0;
Base64++;
if (Base64 > 8) {
Type |= ARTTYPE_BASE64 | ARTTYPE_BINARY;
return(Type);
}
I'm assuming this is counting any article with more than 8 lines of Base64 >> encoded text as binary?
That certainly seems to be the case. I'd be tempted to switch the order
of this whole 'if-else' sequence so the check for "-----BEGIN PGP ..."
comes before the binary tests.
Cheers,
Tom
I'm a bit confused by it's intended behavior, as I thought it was only for rejecting misplaced binaries, but looking at the logs I see it is
identifying these particular messages as some other type, but according to what is in lib/defs.h, I can't quite make heads or tails of how these translate.
Take these two entries for example:
2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642 020b00 IncomingFilter
2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656 020b00 IncomingFilter
The article type logged is "020b00", which that string does not appear anywhere in the source code. In lib/defs.h there is the following block:
/*
* Article types that diablo can determine when receiving an article
*/
#define ARTTYPE_NONE 0x000000
#define ARTTYPE_DEFAULT 0x000001
#define ARTTYPE_CONTROL 0x000002
#define ARTTYPE_CANCEL 0x000004
#define ARTTYPE_MIME 0x000100
#define ARTTYPE_BINARY 0x000200
#define ARTTYPE_UUENCODE 0x000400
#define ARTTYPE_BASE64 0x000800
#define ARTTYPE_MULTIPART 0x001000
#define ARTTYPE_HTML 0x002000
#define ARTTYPE_PS 0x004000
#define ARTTYPE_BINHEX 0x008000
#define ARTTYPE_PARTIAL 0x010000
#define ARTTYPE_PGPMESSAGE 0x020000
#define ARTTYPE_YENC 0x040000
#define ARTTYPE_BOMMANEWS 0x080000
#define ARTTYPE_UNIDATA 0x100000
#define ARTTYPE_ALL 0xFFFFFF
Other articles that are misplaced binaries show up in the logs like this:
2023-03-20 02:50:35.991 somehost - <001d0260a48f41ff95a5dd1ff1830a4d@lfxlxgf.ytv> 1737 050200 IncomingFilter
Where the article type is logged as "050200" but again, I'm having a hard time understanding what this is truly supposed to indicated as its
detected type.
On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
I'm a bit confused by it's intended behavior, as I thought it was only for >> rejecting misplaced binaries, but looking at the logs I see it is
identifying these particular messages as some other type, but according to >> what is in lib/defs.h, I can't quite make heads or tails of how these
translate.
Take these two entries for example:
2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642
020b00 IncomingFilter
2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656
020b00 IncomingFilter
The article type logged is "020b00", which that string does not appear
anywhere in the source code. In lib/defs.h there is the following block:
/*
* Article types that diablo can determine when receiving an article
*/
#define ARTTYPE_NONE 0x000000
#define ARTTYPE_DEFAULT 0x000001
#define ARTTYPE_CONTROL 0x000002
#define ARTTYPE_CANCEL 0x000004
#define ARTTYPE_MIME 0x000100
#define ARTTYPE_BINARY 0x000200
#define ARTTYPE_UUENCODE 0x000400
#define ARTTYPE_BASE64 0x000800
#define ARTTYPE_MULTIPART 0x001000
#define ARTTYPE_HTML 0x002000
#define ARTTYPE_PS 0x004000
#define ARTTYPE_BINHEX 0x008000
#define ARTTYPE_PARTIAL 0x010000
#define ARTTYPE_PGPMESSAGE 0x020000
#define ARTTYPE_YENC 0x040000
#define ARTTYPE_BOMMANEWS 0x080000
#define ARTTYPE_UNIDATA 0x100000
#define ARTTYPE_ALL 0xFFFFFF
This looks like a bitfield map, so the 020b00 you see above equates to a sum of PGPMESSAGE + BASE64 + BINARY + MIME. At least it's recognised the PGP element. The size of the signature (greater than the 8 line limit in the earlier check) seems to be what triggers the filter to flag the article as binary.
On Mar 20, 2023 at 11:37:46 AM CDT, "Tom Furie" <tom@furie.org.uk> wrote:
On 2023-03-20, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
I'm a bit confused by it's intended behavior, as I thought it was only for >>> rejecting misplaced binaries, but looking at the logs I see it is
identifying these particular messages as some other type, but according to >>> what is in lib/defs.h, I can't quite make heads or tails of how these
translate.
Take these two entries for example:
2023-03-20 01:00:03.737 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2642 >>> 020b00 IncomingFilter
2023-03-20 01:00:03.738 somehost - <tv8sp3$fql$1@rasp.pasdenom.info> 2656 >>> 020b00 IncomingFilter
The article type logged is "020b00", which that string does not appear
anywhere in the source code. In lib/defs.h there is the following block: >>>
/*
* Article types that diablo can determine when receiving an article
*/
#define ARTTYPE_NONE 0x000000
#define ARTTYPE_DEFAULT 0x000001
#define ARTTYPE_CONTROL 0x000002
#define ARTTYPE_CANCEL 0x000004
#define ARTTYPE_MIME 0x000100
#define ARTTYPE_BINARY 0x000200
#define ARTTYPE_UUENCODE 0x000400
#define ARTTYPE_BASE64 0x000800
#define ARTTYPE_MULTIPART 0x001000
#define ARTTYPE_HTML 0x002000
#define ARTTYPE_PS 0x004000
#define ARTTYPE_BINHEX 0x008000
#define ARTTYPE_PARTIAL 0x010000
#define ARTTYPE_PGPMESSAGE 0x020000
#define ARTTYPE_YENC 0x040000
#define ARTTYPE_BOMMANEWS 0x080000
#define ARTTYPE_UNIDATA 0x100000
#define ARTTYPE_ALL 0xFFFFFF
This looks like a bitfield map, so the 020b00 you see above equates to a sum >> of PGPMESSAGE + BASE64 + BINARY + MIME. At least it's recognised the PGP
element. The size of the signature (greater than the 8 line limit in the
earlier check) seems to be what triggers the filter to flag the article as >> binary.
Ahhh thank you, this makes sense.
I did move the PGP related if statements above the others as you had suggested
earlier. So far, so good, but may take a while to know for sure. The changes didn't break the binary detection, so there's that.
I'm curious if there's any known reason why the developers chose 8 lines for the checks? I assume this is somewhat arbitrary, but not certain, and thinking
if I'm still having trouble raising that value may be a better solution than trying to make the code do things I don't fully understand.
linetype = classifyLineAsTypes(ptr, len);
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
return(Type);
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:
linetype = classifyLineAsTypes(ptr, len);
if (*ptr == '-' &&
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;
return(Type);
Just FYI, by doing it this way you are allowing people to send you arbitrarily large binaries (up to your article size limit) by just adding
a line starting with "-----BEGIN PGP " before the encoded binary. This is likely why the code didn't short-circuit analysis of the rest of the
article in that case and still flagged messages as binary if there were
more than 8 lines of encoded text. (No PGP *signature* should be that
long.)
Back in the day, if any change like this were used on a bunch of servers,
the people trading binaries that for legal reasons you really do not want
on your server would probably quickly start doing that. You're only
making this change on one server, and hopefully there isn't as much abuse
of binaries on Usenet these days, but you may still want to keep that in mind.
if (*ptr == '-' &&{
strncmp(ptr, "-----BEGIN PGP ", 15) == 0)
Type |= ARTTYPE_PGPMESSAGE;}
return(Type);
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 300 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 74:49:52 |
| Calls: | 6,715 |
| Calls today: | 3 |
| Files: | 12,246 |
| Messages: | 5,357,277 |