I'm developing some python scripts to implement an NNTP/WordPress
gateway, and I ran into unexpected behavior (and a less than helpful
error message) from inews. INN is from the inn2-2.6.0-2 package through
apt on linux mint.

Python's email.message module creates messages that (by default? I
wasn't able to find a setting for this) terminate lines with \r\n
(CRLF). Feeding those messages to 'inews -h -O' produces the error
message (from memory) "No space after colon in <the first line of the
article body>".

Digging into the source code (latest version, online), it seems that
inews is treating the \r character as folding whitespace? Converting
the \r\n to \n in the message solved the problem, but:

Is this really the correct behavior? I'm looking at RFCs 5322 and 5536
here...

Also, a more specific/helpfull error message would have saved me
considerable debugging time. INN devs, is this possible?

Thanks!

-mg

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <s1715o$e99$1@dont-email.me>, mg <mglover@pobox.com> wrote:

I'm developing some python scripts to implement an NNTP/WordPress
gateway, and I ran into unexpected behavior (and a less than helpful
error message) from inews. INN is from the inn2-2.6.0-2 package through
apt on linux mint.

Python's email.message module creates messages that (by default? I
wasn't able to find a setting for this) terminate lines with \r\n
(CRLF). Feeding those messages to 'inews -h -O' produces the error
message (from memory) "No space after colon in <the first line of the
article body>".

inews is not nntp just as sendmail is not smtp. Both run inside your Unix system and use the Unix \n line terminator. If you were opening a TCP connection to port 119 and issuing nntp commands, then you would use \r\n
as RFC 3877 says.

In python3, email.message normally uses \n line terminators unless you
tell it to use the email.policy.SMTP or email.policy.SMTPUTF8 policy
option. So don't do that. If you're still using python2, this would
be a good time to upgrade.





--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

John Levine <johnl@taugh.com> wrote:

inews is not nntp just as sendmail is not smtp. Both run inside your Unix system and use the Unix \n line terminator. If you were opening a TCP connection to port 119 and issuing nntp commands, then you would use \r\n
as RFC 3877 says.

Point taken.

In python3, email.message normally uses \n line terminators unless you
tell it to use the email.policy.SMTP or email.policy.SMTPUTF8 policy
option. So don't do that. If you're still using python2, this would
be a good time to upgrade.

It's python3, and I *was* using email.policy.SMTP. Removing that and my .replace('\r\n', '\n') kludge is working. Thanks.

But, still, what's the benefit to treating \r as folding whitespace in
this case?

-mg

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Russ Allbery <eagle@eyrie.org> writes:

It may make sense to check if the first line of the input ends in \r\n
and, if so, call wire_to_native on the input buffer before processing
it. inews isn't really designed to accept arbitrary line endings, but
it looks easy enough to make it compatible with CRLF line endings.

Oh, wait, no, this doesn't work because wire_to_native expects
dot-stuffing and a terminating dot. Bleh. We'd have to actually teach
inews about \r\n as a line ending.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Mike Glover <mglover@pobox.com> writes:

But, still, what's the benefit to treating \r as folding whitespace in
this case?

It's not. It uses ISWHITE for folding whitespace, which only accepts \t
and space.

The problem is that it's not recognizing a line consisting of \r\n as a
blank line because of this check at the end of StripOffHeaders:

/* Get start of next header; if it's a blank line, we hit the end. */
p = NextHeader(p);
if (*p == '\n')
break;

*p is \r here, so it thinks there's another header. I think it then tries
to parse \r\n as a header line and then errors out because there's no
colon. It was clearly never tested with CRLF line endings.

It may make sense to check if the first line of the input ends in \r\n
and, if so, call wire_to_native on the input buffer before processing it.
inews isn't really designed to accept arbitrary line endings, but it looks
easy enough to make it compatible with CRLF line endings.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In article <87im6eh7vb.fsf@hope.eyrie.org>,
Russ Allbery <eagle@eyrie.org> wrote:

It may make sense to check if the first line of the input ends in \r\n
and, if so, call wire_to_native on the input buffer before processing it. >inews isn't really designed to accept arbitrary line endings, but it looks >easy enough to make it compatible with CRLF line endings.

On the other hand, INN is 30 years old and I don't recall this particular question coming up before, so perhaps it doesn't need solving.

--
Regards,
John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

John Levine <johnl@taugh.com> writes:

Russ Allbery <eagle@eyrie.org> wrote:

It may make sense to check if the first line of the input ends in \r\n
and, if so, call wire_to_native on the input buffer before processing
it. inews isn't really designed to accept arbitrary line endings, but
it looks easy enough to make it compatible with CRLF line endings.

On the other hand, INN is 30 years old and I don't recall this
particular question coming up before, so perhaps it doesn't need
solving.

The error message being extremely unhelpful is a fair point, though, and probably easier to fix.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)