XPost: news.software.nntp

Julien wrote:

The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

However, RFC 3977 also states in Section 8.4.2 that "for compatibility
with existing implementations, the last two lines MAY instead be:

Bytes:
Lines:

even though they refer to metadata, not headers."

Are you aware of news readers that send LIST OVERVIEW.FMT and would
break if they encountered ":bytes" or ":lines"? I'm wondering whether
it is safe nowadays to advertise the expected strings.

Also, are there useful other header fields that news readers are already >searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news >servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would make my life easier, certainly. I also have kill file
entries based on Path.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

It is the whole point of having them in the overview database. Yet, I
don't know whether current versions of news clients would directly grab
them from there. (It may be an interesting feature to implement in news clients if they aren't already doing that.)

JFTR, tin should (for Path)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Julien ÉLIE wrote:

[...]
Also, are there useful other header fields that news readers are already searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would be nice to have for scoring.

Currently the OVER command must be disabled for this to work in flnews.
Quoted from man page:
|
| o nntp_no_over (Integer) Disable usage of overview (OVER command) for
| NNTP protocol
|
| Setting this to a nonzero value disables usage of overview and makes
| the whole newsgroup list usable for scoring rules.
| [...]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Julien ÉLIE wrote:

[...]
The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

RFC 3977 says in Section 8.4.2:
|
| This command MAY generate different results if it is used more than
| once in a session.

A newsreader needs to know when to refresh the information in this case.
Or is it intended to be repeated before every OVER command?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Michael Bäuerle <michael.baeuerle@stz-e.de> writes:

Julien ÉLIE wrote:

[...]
The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

RFC 3977 says in Section 8.4.2:

| This command MAY generate different results if it is used more than
| once in a session.

A newsreader needs to know when to refresh the information in this case.
Or is it intended to be repeated before every OVER command?

That is just a cover-your-ass provision for the news server that allows
the admin to reconfigure overview storage dynamically without having to
take special precautions to prevent the list from changing. I wouldn't
expect a newsreader to refresh its understanding of the valid fields
except when it reconnects.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Michael Bäuerle <michael.baeuerle@stz-e.de> wrote:

I've also added Path, as Urs has a special handling of it in tin.

JFTR: works like expected in tin; no extra HDR, XHDR or XPAT cmds. issued
when Path data is available in NOV and used for scoring.

My plan is to simply skip the first 7 fields of the response.
Different order or meaning is not allowed by RFC 3977. And this

newsoverview(5) from Geoff Collyer (some time in 1992) did specify an order (but did't mention LIST OVERVIEW.FMT as that was invented AFAIK ~1 year
later).

RFC 2980 (from 10.2000) didn't specify an order.

RFC 3977 (from 10.2006) does specify an order.

IIRC I've seen non RFC 3977 compliant setups in the wild (reversed order, missing mandatory fields, duplicate fields, ...).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.nntp

Julien ÉLIE wrote:

[...]
In case you wish to test a possible use of the Newsgroups field if
present in overview data, I've added it in my news server (news.trigofacile.com):

LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
.
[...]

I have implemented support for LIST OVERVIEW.FMT in the snapshot flnews 1.2.0pre1. According to the comment from Russ, the command is executed
only once per session.

Protocol log: ------------------------------------------------------------------------
[Using NNTP protocol driver]

Connect to news.trigofacile.com:nntps]

[Established encrypted connection using TLSv1.3 protocol with cipher suite TLS_AES_256_GCM_SHA384]
[<= Expect X509 certificate]
Certificate:
[...]
[Server certificate verification successful]
[<=] 200 news.trigofacile.com InterNetNews NNRP server INN 2.8.0 (20220801 prerelease) ready (no posting)

] CAPABILITIES

[<=] 101 Capability list:
[<= Expect multiline data block]
VERSION 2
IMPLEMENTATION INN 2.8.0 (20220801 prerelease)
AUTHINFO USER SASL
COMPRESS DEFLATE
HDR
LIST ACTIVE ACTIVE.TIMES COUNTS DISTRIB.PATS DISTRIBUTIONS HEADERS MODERATORS MOTD NEWSGROUPS OVERVIEW.FMT SUBSCRIPTIONS
OVER
READER
SASL GS2-IAKERB GS2-KRB5 SCRAM-SHA-1 SCRAM-SHA-256 GSS-SPNEGO GSSAPI DIGEST-MD5 OTP NTLM CRAM-MD5 LOGIN PLAIN
XPAT
[Switch to NNTP V2 (RFC 3977) mode]

[Client authentication]
[Compression]

] LIST OVERVIEW.FMT

[<=] 215 Order of fields in overview database
[<= Expect multiline data block]
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
[Newsgroups header field available from overview]
[...]
------------------------------------------------------------------------

Urs noted that some server implementations are broken.
The mandatory fields are now checked for presence and order.

The Newsgroups header field, if present, is copied from the overview.
Scoring rules of type "group" are applied to all groups in this case.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)