REQUEST FOR DISCUSSION (RFD)
moderated group news.groups.proposals
This is a formal Request For Discussion (RFD) to suspend the charter and >moderation policy of the Usenet newsgroup news.groups.proposals.
RATIONALE
Since 2006, the Big-8 hierarchies have undergone an overall reduction in >their active user base and article traffic. The news.groups newsgroup
has followed this general trend; the past few years have seen some
measure of spam and other off-topic messages, but little of the
acrimonious content that was the main impetus behind the creation of >news.groups.proposals. There is therefore reason to believe that
news.groups could once again function as "a healthy environment" for the >discussion of RFDs.
By contrast, in the past few years news.groups.proposals has had
problems of its own, mostly stemming from its convoluted and antiquated >moderation system. Many submissions have gone missing or unnoticed by
the moderators due to breakdowns in the submission pipeline. While the >current Board members have been working to streamline and modernize the >moderation system they inherited, and to put better fault detection and >prevention measures in place, there is always the risk of further
unexpected technical issues.
Would recommend that Big-8 continue their work on streamlining and modernizing the moderation system.
Finally, I wonder if there are enough users posting to news.groups and news.groups.proposals to generate any meaningful discussion on what
course of action should be taken: new groups, removing old groups, etc.
Changing how moderation works might require a change to the nntp standard
and changes to the software usenet providers use for their servers.
What we need more than updating moderation is modern moderation software.
That's why this post was made. Does moderating group-related discussions
help in 2022 or is it just more work? Those of us on the board can see
things both ways and that's why we're opening this up for discussion.
On Sun, 22 May 2022 04:54:33 CST, D Finnigan wrote:I made my statement of recommendation in response to the RFD author's
Would recommend that Big-8 continue their work on streamlining and
modernizing the moderation system.
There's not much that the board can do when it comes to modernizing moderation.
We've had people ask why we continue to use the moderated group whenthere
is so little traffic on news.groups that could be considered to benecessary
actually trolling (besides to obnoxious drug spam). Is it really
or just causing more work for the board?
Jason Evans <jsevans@mailfence.com> wrote:
Changing how moderation works might require a change to the nntp standard
and changes to the software usenet providers use for their servers.
What we need more than updating moderation is modern moderation software.
I wonder what the blockers on that are, and whether the Board and others might encourage somebody to do it.
It seems to me to not be super complicated:
- a program that receives emails of submissions
- applies group-specific filter rules (eg 'all messages containing XXX are banned')
- offer a web/other interface for moderators to approve/reject messages (similar to mailing list moderation)
- sign the message and post it to an NNTP server
- handle moderator user accounts / password resets / etc etc
I'm not the person to do it, but I imagine writing something like that in a modern language with decent libraries to handle the mail/web/NNTP side of things (eg Python) wouldn't be a lot of work.
If any of the Board members had the time, we'd love to dive in and write
a brand-new moderation system from the ground up, using today's state-of-the-art libraries and best practices in security, UI design,
etc. But we've all got day jobs, and other real-life commitments, and plenty of other mundane Board-related administration work to do, and so
at the moment the best we're able to manage is to apply the occasional enhancement or bugfix to STUMP to keep it limping along on modern OSes.
Still, we've had some offers to help with moderation software in the
past, and some of these people have actually followed through, so a new moderation system might indeed happen one day. (It might even happen without us, though as far as I know we're the only organized group
actively maintaining moderation software nowadays.)
None of the infrastructure used to moderate Usenet newsgroups has
changed in decades.
Greetings.
On 28/05/2022 00.00, Theo wrote:
Jason Evans <jsevans@mailfence.com> wrote:
Changing how moderation works might require a change to the nntp standard >> and changes to the software usenet providers use for their servers.
What we need more than updating moderation is modern moderation software.
I wonder what the blockers on that are, and whether the Board and others might encourage somebody to do it.
It seems to me to not be super complicated:
It may not seem so, but there are countless details and special (albeit
not entirely uncommon) cases that need to be accounted for and
documented. Some examples:
- a program that receives emails of submissions
Receives them how? Do we build the software in a way that the moderated group must have a dedicated e-mail account? Or is it sufficient for
there to simply be a dedicated alias that auto-forwards to a real e-mail account, which is possibly used for other purposes? Or is there no requirement at all that the "To:" address be unique for a given group?
How is the e-mail account to be accessed? POP? IMAP? Unix mail spool?
Local mbox/maildir folder? In the case where the account is shared,
what software is responsible for ensuring that article submissions get
passed to the moderation software? Is it the moderation software itself which examines all incoming messages, or do they get pre-filtered by an
MDA? If the latter, which MDA? Previous moderation packages have
relied on procmail, and usually include documentation that instructs
users on how to interface with it, but many users don't want to use
procmail nowadays due to its arcane, practically unmaintainable syntax
and its history of upstream maintenance problems.
- applies group-specific filter rules (eg 'all messages containing XXX are banned')
Before you can even start filtering, you need to get the submission in a format that can be processed by a filter. What happens if you receive a MIME-encoded message? What if it's multi-part MIME? You may need to separate the message into parts, find the plain-text one, and decode it
if necessary.
And before you start applying group-specific filters, you need to apply
ones that are universal to Usenet. For this you need a good knowledge
of the relevant RFCs. For example, RFC 5536 and 5537 prescribe special handling of the Injection-Date, Injection-Info, and Xref headers. If
the submission contains these headers (and it probably does), and if you simply pass them on as-is, your NNTP posting agent is likely to reject
the submission. You should probably also make sure that all mandatory headers (Newsgroups, Subject, Date, From, etc.) are present and
well-formed, and that certain special optional headers (Approved,
Control, etc.) are absent. If not, you need to implement and document
some way of dealing with such cases.
You will probably also want to implement some default, common-sense
filters that nearly all moderators will want, such as stripping MIME attachments (possibly except for things like OpenPGP signatures),
stripping non-plain-text "alternative" versions of multi-part MIME
messages, and wrapping long lines (or simply detecting these cases and rejecting the message with an appropriate message to the submitter).
- offer a web/other interface for moderators to approve/reject messages (similar to mailing list moderation)
How should the software authenticate a moderator? Does it need to
implement some sort of access control, or does it delegate this to the
web server (as with HTTP authentication)? How should the authentication state be preserved -- via cookies or an HTTP request method?
- sign the message and post it to an NNTP server
Sign it how? Does the software depend on a library for OpenPGP
signatures, or does it make a system call to GnuPG/PGP or the like?
Does the software assume that it has unrestricted access to the private
keys or does it require the user to decrypt them with a passphrase for
each signature?
- handle moderator user accounts / password resets / etc etc
This goes hand-in-hand with the authentication issues above. Probably nowadays there are libraries that securely handle authentication and credentials for web accounts. (About 25 years ago, when STUMP and other moderation packages were being developed, there weren't.)
I'm not the person to do it, but I imagine writing something like that in a modern language with decent libraries to handle the mail/web/NNTP side of things (eg Python) wouldn't be a lot of work.
Well, none of the issues I've listed above are insurmountable -- in
fact, they're not even technically difficult ones. But they do require getting acquainted with the relevant RFCs and mail/web/security/NNTP frameworks, making a lot of well-informed design decisions, and
documenting everything in sufficient detail to let (a)
technically-inclined folk install and configure the system, (b) non-technically-inclined moderators use the system, and (c) current and future programmers further develop or maintain the system.
If any of the Board members had the time, we'd love to dive in and write
a brand-new moderation system from the ground up, using today's state-of-the-art libraries and best practices in security, UI design,
etc. But we've all got day jobs, and other real-life commitments, and
plenty of other mundane Board-related administration work to do, and so
at the moment the best we're able to manage is to apply the occasional enhancement or bugfix to STUMP to keep it limping along on modern OSes.
Still, we've had some offers to help with moderation software in the
past, and some of these people have actually followed through, so a new moderation system might indeed happen one day. (It might even happen
without us, though as far as I know we're the only organized group
actively maintaining moderation software nowadays.)
How easy is it to get STUMP+WebSTUMP up and running today, on a fresh server? (I see some instructions from 1990s, but not sure if there are up to date)
Tristan Miller <tmiller@big-8.org> wrote:
Greetings.
On 28/05/2022 00.00, Theo wrote:
Jason Evans <jsevans@mailfence.com> wrote:
Changing how moderation works might require a change to the nntp standard >> >> and changes to the software usenet providers use for their servers.I wonder what the blockers on that are, and whether the Board and others >> > might encourage somebody to do it.
What we need more than updating moderation is modern moderation software. >> >
It seems to me to not be super complicated:
It may not seem so, but there are countless details and special (albeit
not entirely uncommon) cases that need to be accounted for and
documented. Some examples:
Some high level sketches below. I'm assuming it's a standalone server (or Docker container / whatever) and naming some Python libraries that look plausible to do the heavy lifting. No doubt there are many issues and awkwardnesses I haven't thought of.
- a program that receives emails of submissions
Receives them how? Do we build the software in a way that the moderated
group must have a dedicated e-mail account? Or is it sufficient for
there to simply be a dedicated alias that auto-forwards to a real e-mail
account, which is possibly used for other purposes? Or is there no
requirement at all that the "To:" address be unique for a given group?
I'm assuming that the moderation server operates its own MDA which receives messages addressed to news.groups-submission@mod.example.com
or similar addresses. This is a list maintained by the mod software (ie add a new group, get a new mailbox in /var/spool/mail/news.groups-submission).
Said MDA delivers messages into a mailspool that the mod software picks up. Library for that:
https://docs.python.org/3/library/mailbox.html
How is the e-mail account to be accessed? POP? IMAP? Unix mail spool?
Local mbox/maildir folder? In the case where the account is shared,
what software is responsible for ensuring that article submissions get
passed to the moderation software? Is it the moderation software itself
which examines all incoming messages, or do they get pre-filtered by an
MDA? If the latter, which MDA? Previous moderation packages have
relied on procmail, and usually include documentation that instructs
users on how to interface with it, but many users don't want to use
procmail nowadays due to its arcane, practically unmaintainable syntax
and its history of upstream maintenance problems.
In general, the fewer external tools the less there is to break. So minimal bits of shell, glue and string. Config of necessary external tools (eg MDA) would be managed by the mod software (ie the aliases above).
- applies group-specific filter rules (eg 'all messages containing XXX are >> > banned')
Before you can even start filtering, you need to get the submission in a
format that can be processed by a filter. What happens if you receive a
MIME-encoded message? What if it's multi-part MIME? You may need to
separate the message into parts, find the plain-text one, and decode it
if necessary.
https://docs.python.org/3/library/email.html
And before you start applying group-specific filters, you need to apply
ones that are universal to Usenet. For this you need a good knowledge
of the relevant RFCs. For example, RFC 5536 and 5537 prescribe special
handling of the Injection-Date, Injection-Info, and Xref headers. If
the submission contains these headers (and it probably does), and if you
simply pass them on as-is, your NNTP posting agent is likely to reject
the submission. You should probably also make sure that all mandatory
headers (Newsgroups, Subject, Date, From, etc.) are present and
well-formed, and that certain special optional headers (Approved,
Control, etc.) are absent. If not, you need to implement and document
some way of dealing with such cases.
https://docs.python.org/3/library/email.parser.html
Is there anything about incoming messages that are different to regular
email MIME such that parsing Usenet messages as email won't work?
(eg apart from different headers, eg differences in message formatting)
You will probably also want to implement some default, common-sense
filters that nearly all moderators will want, such as stripping MIME
attachments (possibly except for things like OpenPGP signatures),
stripping non-plain-text "alternative" versions of multi-part MIME
messages, and wrapping long lines (or simply detecting these cases and
rejecting the message with an appropriate message to the submitter).
https://docs.python.org/3/library/email.contentmanager.html
- offer a web/other interface for moderators to approve/reject messages
(similar to mailing list moderation)
How should the software authenticate a moderator? Does it need to
implement some sort of access control, or does it delegate this to the
web server (as with HTTP authentication)? How should the authentication
state be preserved -- via cookies or an HTTP request method?
Here diverges a bit depending on what kind of web framework you're using,
but essentially:
Authenticate users via OAuth:
https://oauth.net/code/python/
(using their password on another service, eg Github / Google / Facebook / Microsoft / Apple / whatever they prefer)
Authentication is via a cookie set in their browser, like every other
service these days. The web framework knows who you are. (eg the Django framework might be a fit for Python, although I'm not familiar with it)
Someone from the Board maintains the list of who moderates which groups, via their login.
Users are able to login and see the groups they moderate.
- sign the message and post it to an NNTP server
Sign it how? Does the software depend on a library for OpenPGP
signatures, or does it make a system call to GnuPG/PGP or the like?
Does the software assume that it has unrestricted access to the private
keys or does it require the user to decrypt them with a passphrase for
each signature?
I don't have a strong opinion either way - an integrated library
would be easier, but using a distro-provided GnuPG might be better maintained. Some examples of both:
https://pypi.org/project/py-pgp/
Key management is an interesting question, but in general I'd expect that users with moderation rights also have posting rights. You wouldn't need to unlock the keys by hand each time.
It is possible the keys could be held in some more secure way (secure enclave, SGX?) but I'm not au fait with the current state of software for that. It would add another dependency which might risk longevity.
- handle moderator user accounts / password resets / etc etc
This goes hand-in-hand with the authentication issues above. Probably
nowadays there are libraries that securely handle authentication and
credentials for web accounts. (About 25 years ago, when STUMP and other
moderation packages were being developed, there weren't.)
OAuth is common these days - you don't need to manage usernames and passwords, you just ask Google 'can you confirm this really is
xyz@gmail.com' (or whoever), and once that's confirmed they are logged in. Once you are logged in, the web pages you see are generated by software that knows who you are and customises your view of the content.
Basically you could sketch it as composed of several parts:
1) A web site with login, like every other web site in the world today
2) A part to handle incoming email traffic
3) A part to apply rules to messages
4) A part to interact with moderators what to do with messages
5) A part to inject messages into NNTP and handle PGP
6) A part to record activity and display status
Many of those features are shared with existing email mailing list software, and that may be a suitable starting point. Apart from 5) and maybe 6), most of this seems similar.
I'm not the person to do it, but I imagine writing something like that in a
modern language with decent libraries to handle the mail/web/NNTP side of >> > things (eg Python) wouldn't be a lot of work.
Well, none of the issues I've listed above are insurmountable -- in
fact, they're not even technically difficult ones. But they do require
getting acquainted with the relevant RFCs and mail/web/security/NNTP
frameworks, making a lot of well-informed design decisions, and
documenting everything in sufficient detail to let (a)
technically-inclined folk install and configure the system, (b)
non-technically-inclined moderators use the system, and (c) current and
future programmers further develop or maintain the system.
I agree, nailing down the requirements is a key part of the brief, as is documentation and making the thing easy to run. There's some useful documentation here:
https://www.algebra.com/~ichudov/stump/spec.html
(the comment 'Maybe I'll do it after getting 32 meg of memory and installing Postgres.' suggests this text is very old)
If any of the Board members had the time, we'd love to dive in and write
a brand-new moderation system from the ground up, using today's
state-of-the-art libraries and best practices in security, UI design,
etc. But we've all got day jobs, and other real-life commitments, and
plenty of other mundane Board-related administration work to do, and so
at the moment the best we're able to manage is to apply the occasional
enhancement or bugfix to STUMP to keep it limping along on modern OSes.
Understood... I am not volunteering either, for the above reasons.
(this post is by no means intended as complaining, btw. Just trying to better understand some of those requirements)
Am I right in thinking one of the issues with STUMP is it's really a collection of Perl and shell, calling out to a lot of external tools. And
so it's difficult to install because any time one of those dependencies changes, it breaks something?
How easy is it to get STUMP+WebSTUMP up and running today, on a fresh server? (I see some instructions from 1990s, but not sure if there are up to date)
Still, we've had some offers to help with moderation software in the
past, and some of these people have actually followed through, so a new
moderation system might indeed happen one day. (It might even happen
without us, though as far as I know we're the only organized group
actively maintaining moderation software nowadays.)
I hope so too.
Theo
These is a group around that has the STUMP code and run by Big8
https://github.com/UsenetBig8/STUMP
I applied for membership in February but haven't heard anything.
You can find the latest code here:
https://savannah.gnu.org/projects/stump/
Is the ReadyStump service described on this site still offered /
supported?
I'm assuming that the moderation server operates its own MDA which receives >messages addressed to news.groups-submission@mod.example.com
or similar addresses. This is a list maintained by the mod software (ie add >a new group, get a new mailbox in /var/spool/mail/news.groups-submission).
On Wed, 8 Jun 2022 09:12:58 -0500, Steve Bonine wrote:
Is the ReadyStump service described on this site still offered /
supported?
No, Igor Chudov stopped offering that service years ago.
Greetings.
On 03/06/2022 15.07, Steve Bonine wrote:
None of the infrastructure used to moderate Usenet newsgroups has
changed in decades.
Sure it has:
On 2022-06-06, Theo <theom+news@chiark.greenend.org.uk> wrote:
Tristan Miller <tmiller@big-8.org> wrote:
In general, the fewer external tools the less there is to break. So minimal
bits of shell, glue and string. Config of necessary external tools (eg MDA)
would be managed by the mod software (ie the aliases above).
I think it's "safe" to say at this point that most mail is stored in
Mbox or Maildir formats, so picking Maildir probably makes sense
here. Though if I think about it from a dependency perspective, both
Mbox and Maildir rely on having a local mailbox of sorts.
The above pieces of hygeine that Tristan discussed are still needed
but all are simple tweaks.
Here diverges a bit depending on what kind of web framework you're using, but essentially:
Authenticate users via OAuth:
https://oauth.net/code/python/
(using their password on another service, eg Github / Google / Facebook / Microsoft / Apple / whatever they prefer)
I think the folks using Usenet may not like the identity of many big
OAuth providers but this is immaterial to me.
Would the Web portion be centrally run moderation software then?
What happens if GPG versions rev? Seems like there's anxiety in this
group about GPG key types. FWIW I don't have much GPG experience.
As I mentioned above, I'd be interested in writing moderation software
but before that I think we'd need to decide on a few things:
1. The technical details as discussed in this thread
2. A timeframe to begin implementation so I can actually block time
out to work on it.
3. What the future of maintaining the software would look like.
I'm both familiar with and willing to implement this in Python or Go,
but would depend on the above points first.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 293 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 232:29:12 |
| Calls: | 6,624 |
| Files: | 12,171 |
| Messages: | 5,319,537 |