• Usenet Hierarchy Administration FAQ

    From Russ Allbery@21:1/5 to All on Wed Jan 15 08:01:05 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sat Feb 15 08:01:06 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sun Mar 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Wed Apr 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Fri May 15 07:01:07 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Mon Jun 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Wed Jul 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sat Aug 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Tue Sep 15 07:01:03 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Thu Oct 15 07:01:04 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sun Nov 15 08:01:07 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Tue Dec 15 08:01:07 2020
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Fri Jan 15 08:01:03 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Mon Feb 15 08:01:04 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Mon Mar 15 07:01:04 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sat May 15 07:01:02 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Tue Jun 15 07:01:03 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Thu Jul 15 07:01:02 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sun Aug 15 07:01:04 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Wed Sep 15 07:01:03 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Fri Oct 15 07:01:05 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Mon Nov 15 08:01:02 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Wed Dec 15 08:01:02 2021
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Sat Jan 15 08:01:03 2022
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Tue Feb 15 08:01:03 2022
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Tue Mar 15 07:01:03 2022
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, or if you ever have to change this, it will be a major hassle and it will take a very long time
    to catch all the stragglers.

    What many hierarchy administrators do is combine these two approaches.
    They set up their own forwarding system that they have direct control
    over, so that they can change the moderators for an existing group without having to contact moderators-request@isc.org, and they ask the moderators.isc.org sites to just forward to the appropriate address at
    that host. Then, the only things they have to let moderators-request know about are new moderated groups or removal of moderated groups. This does
    mean that posts to moderated groups go through two hops before reaching
    the moderator instead of just one, though, which can make problems
    slightly more difficult to diagnose.

    Unfortunately, due to the way that they're distributed and used, the alias lists for moderators.isc.org cannot contain wildcard entries, so you
    cannot ask them to just forward all posts to any moderated example.* group
    to example.org. Each group still has to be configured separately and each
    new moderated group still has to be sent to moderators-request@isc.org for manual processing. Hopefully in the future there will be some more easily automated system for handling this.

    ------------------------------

    Subject: 5. About the ftp.isc.org Newsgroup Lists

    Maintained at:

    <ftp://ftp.isc.org/pub/usenet/CONFIG/>

    is a copy of a control.ctl file with as many public hierarchies as
    possible with active maintainers listed in it, along with lists of
    newsgroups in active file and checkgroups format maintained by processing control messages using those control.ctl rules. Also maintained from the
    same information is a collection of hierarchy public keys for news administrators to get from one source, in:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    A unified view of this information, including logs of recent changes, is
    at:

    <http://usenet.trigofacile.com/hierarchies/>

    These can be valuable resources for you as a hierarchy administrator in a couple of different ways.

    First, some users and some news sites use these lists of newsgroups to determine what newsgroups to carry, either using the list verbatim,
    checking the list to see if a newsgroup is listed before being willing to
    add it, or synchronizing particular hierarchies against that list. Having
    your hierarchy there may therefore increase the number of sites that carry
    your hierarchy and may make it easier for interested users to get the
    groups added at their local site.

    Second, particularly if you're using PGP-signed control messages, this
    list can serve as a check that your control messages are working properly.
    The list updates once an hour, so if you send a control message and in a
    few hours the result isn't reflected in the ftp.isc.org lists, something
    may have gone wrong (maybe you forgot an Approved header, or the PGP
    signature wasn't valid for some reason).

    Finally, many news administrators, even if they don't use the newsgroup
    lists, use the control.ctl file and PGPKEYS file to configure their own
    news servers. This control.ctl file is also included in INN releases.

    To get your hierarchy listed in these files, send mail to usenet-config@isc.org. Please include your control.ctl entry, as
    described above, along with your PGP public key or a URL from which it can
    be obtained (preferrably the latter). Also provide a pointer to a current
    list of newsgroups in checkgroups format so that they can be added to the
    list at the same time. usenet-config@isc.org sometimes has a bit of a
    backlog, so please allow a couple of weeks for a response.

    ------------------------------

    Subject: 6. Other Resources

    Here are some other resources to be aware of if you're maintaining a
    hierarchy:

    news.admin.hierarchies
    Most discussion of new hierarchies and of hierarchy administration in
    general takes place here. There isn't a lot of discussion normally,
    just hierarchy FAQs, but a lot of people read the group and can
    provide help and suggestions if you post.

    news.software.nntp
    The technical discussion group for news software and the news
    protocols, the readers of this group may be able to help if you have
    questions about news server configuration or about the details of the
    PGP signature system for control messages.

    Control message archive <ftp://ftp.isc.org/pub/usenet/control/>
    An archive of all newgroup and rmgroup control messages, used by some
    sites to check to see whether a requested group had a valid control
    message and useful as a check to be sure that your control messages
    are getting out.

    Newsgroup list archive <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    A collected control.ctl file (which is also included in INN) and lists
    of newsgroups maintained by those rules. Also includes a file
    describing the checkgroups format, providing some guidelines for
    newsgroup naming, and describing how to write newsgroup descriptions.

    Usenet hierarchy information <http://usenet.trigofacile.com/hierarchies/>
    A merged view of the information maintained at ftp.isc.org and the
    easiest way to view the information together and check it for
    accuracy.

    pgpcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    Mentioned many times above, this is the central site for the reference
    implementations of the PGP control message signing protocol. It also
    has a list of public keys of hierarchy administrators and information
    about how to enable PGP verification of control messages.

    news-admin.org <http://www.news-admin.org/>
    A hosting service for hierarchy administrators and Usenet hierarchies
    that you may want to consider using, as well as a good collection of
    examples of web sites for various hierarchies.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to All on Fri Apr 15 07:01:02 2022
    Last-modified: 2018-07-16
    Posted-by: postfaq 1.17 (Perl 5.28.1)
    Archive-name: usenet/hierarchy-admin
    URL: https://www.eyrie.org/~eagle/faqs/usenet-hier.html
    Posting-frequency: monthly

    This FAQ attempts to provide help to Usenet hierarchy administrators, the people who try to maintain the canonical lists of newsgroups in managed hierarchies. It is aimed at the hierarchy administrators rather than at
    news admins and tries to summarize the issues to consider in making it
    easy for news admins to carry the hierarchy.

    If you're reading this on Usenet, this FAQ is formatted as a minimal
    digest, so if your news or mail reader has digest handling capabilities
    you can use them to navigate between sections. In rn variants, you can
    use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
    each section into a separate article.

    Please send any comments, suggestions, or updates to eagle@eyrie.org.
    Bear in mind when sending me e-mail that I receive upwards of 800 mail
    messages a day and sometimes have a large backlog of personal e-mail.

    This FAQ is posted monthly to news.admin.hierarchies, and is available on
    the web at <https://www.eyrie.org/~eagle/faqs/usenet-hier.html>.

    ------------------------------

    Subject: Contents

    1. Introduction and Terms
    2. Basic Hierarchy Administration
    3. PGP-Signing Control Messages
    4. Maintaining Moderated Groups
    5. About the ftp.isc.org Newsgroup Lists
    6. Other Resources

    ------------------------------

    Subject: 1. Introduction and Terms

    This FAQ assumes a basic familiarity with Usenet (there are other
    documents that explain the fundamentals better), but there are a few
    additional concepts that are specifically important for Usenet hierarchy administrators.

    A Usenet hierarchy is, reduced to its essence, a set of Usenet newsgroups
    that share a common naming prefix, such as all groups starting with
    "comp." or all groups starting with "de.". The names of Usenet newsgroups define a hierarchy of names, with "." used as the separator between the
    levels of the hierarchy, like host names. Unlike host names, the most significant part of the name is given first. The first component of the
    name is special and more significant than the rest of the name, since it defines the top-level Usenet hierarchy to which that group belongs.

    Generally, every top-level hierarchy is completely independent of the
    others (although there are a few exceptions where multiple hierarchies
    share the same management procedures). How the list of newsgroups in that hierarchy should be maintained varies very widely between hierarchies,
    from the complete anarchy of alt.* to the highly formal system used by
    comp.*, or simply by fiat of the organization running the hierarchy as
    with microsoft.*. Which maintenance methods you should use for your
    hierarchy is out of scope for this FAQ; this document is about how to
    publish the results of those methods, assuming that you want to have a
    single canonical list of newsgroups that everyone carrying that hierarchy
    can agree on. If you don't want that (if your hierarchy is like alt.*,
    for example), most of this document will not apply.

    Usenet newsgroups are created and removed via specially formatted messages called control messages that tell news servers to do something. A
    hierarchy should have a hierarchy administrator who is responsible for following whatever procedures were agreed on for changing the list of
    groups in that hierarchy and then publishing the results using control messages. The structure of control messages is explained in the Usenet
    news standards and in many FAQs and web pages, so it won't be explained
    here. (Some sites don't use control messages for various reasons, and
    it's therefore best to also publish the results via other methods, as will
    be explained.)

    Besides newgroup (create a newsgroup) and rmgroup (remove a newsgroup)
    control messages, there is also a control message called a checkgroups
    which provides a complete list of newsgroups in a hierarchy complete with
    short descriptions. A "checkgroups" is therefore also used to mean a
    complete list of newsgroups in a hierarchy. The checkgroups format is a
    list of groups, one per line, with the name of the group, a tab, and a
    short description. If the newsgroup is moderated, the description must
    end in the literal text " (Moderated)". (Sadly, you can't translate the
    word to another language. It's an ugly wart on the protocol.)

    Originally, control messages were authenticated only by the (easily
    forged) address of the sender of the message, which worked when Usenet was small but broke down badly as it got larger. As a result, most
    hierarchies now sign their control messages using PGP, an open standard
    for public key cryptography, allowing receiving sites to verify that the control message was issued by the person it claims to be from.

    ------------------------------

    Subject: 2. Basic Hierarchy Administration

    A hierarchy administrator has three separate audiences who should be kept
    in mind when publishing information about a Usenet hierarchy: users who
    may be interested in reading or posting to the hierarchy, news
    administrators who are not currently carrying the hierarchy on their
    servers and want to, and news administrators who are already carrying the hierarchy and want to keep current with changes to it.

    The most important audience is probably the users, but that's also the
    audience that's the hardest to make general statements about, since how
    you communicate with potential users varies quite a bit by hierarchy. The users will primarily be interested in a description of what the hierarchy
    is for, a list of groups in the hierarchy, any hierarchy-wide policies,
    the charters of the newsgroups, and the newsgroup creation procedure.
    They'll generally speak the same language as the hierarchy (since if they don't, they probably won't be interested in reading it).

    News administrators who aren't currently carrying the hierarchy likely
    won't be as interested in detail like particuliar policies or group
    charters, but will also be interested in the overall purpose for the
    hierarchy and any policies that specifically affect sites carrying the
    groups. They will specifically need the list of newsgroups in the
    hierarchy in checkgroups format, however, and preferrably as a plain-text
    file that they can download and feed into their news software. They'll
    also need instructions for processing control messages to pick up changes
    to the hierarchy, generally as an INN control.ctl fragment. Finally, news administrators may not speak the language of your hierarchy (they may be running a news server for a large international ISP, for example), so you
    may want to provide instructions specifically for news administrators in English if the language of your hierarchy isn't English.

    News administrators who are already carrying the hierarchy are mostly interested in being notified of changes to it (usually via control
    messages). They also want ways of checking their group list against the current one so that they can get back into sync if they've missed some
    updates.

    So, given that, here are a few specific recommendations:

    * Have a web site. Try to make sure that the URL for your hierarchy web
    site is stable and doesn't change, since the URL makes it into various
    FAQs and configuration files that live for years. Put all of the
    hierarchy information on the web site, and make sure that the web site
    stays up to date. <http://www.news-admin.org/> may be available as a
    hosting site for your web site.

    * Every time you create a group, remove a group, rename a group (which
    can be done with a creation and removal), or change the moderation
    status of a group, send the appropriate control message. By looking
    for it in <ftp://ftp.isc.org/pub/usenet/control/> under your hierarchy,
    you can check that the control message has propagated. Control
    messages should have, in the Newsgroups header, the group being created
    or removed (even for newgroup messages; they'll propagate correctly
    even though the group hasn't been created yet because newgroup messages
    have special propagation rules). The exception is checkgroups
    messages, which should normally be posted to the administrative group
    of your hierarchy and possibly crossposted to news.admin.hierarchies.
    Don't put a Distribution header in your control messages unless you
    really intend to limit the availability of the groups to sites
    configured to accept that distribution (which mostly doesn't work
    anyway).

    * Send a checkgroups control message periodically. If you do this, you
    don't really need to send duplicate control messages for changes; if
    someone misses a change, they'll catch it with the next checkgroups
    message. In the checkgroups control message (and ideally in your other
    control messages as well), include an X-* header pointing to your
    hierarchy web site so that people can get more information about the
    hierarchy. Do not put a Supersedes header on your checkgroups
    messages; some sites will filter out any message that contains both a
    Control header and a Supersedes header.

    * Put the *current* checkgroups on your web site as a plain-text file
    (ending the file name with .txt will give this hint to most web
    servers) so that it can be easily downloaded by news administrators.
    Make sure that this file is kept current. Having a separate pretty
    HTMLified list of groups is often useful for users, but please be sure
    to provide the plain text checkgroups message as well since the HTML
    version is nearly useless for news administrators trying to add your
    hierarchy.

    * Write a guide for news administrators on how to carry the hierarchy,
    including an overall description. Make sure that it clearly indicates
    where to find the checkgroups for the hierarchy. Put this guide on
    your web site, and it doesn't hurt to put a URL for it in the headers
    of your control messages as well. If your hierarchy uses a language
    other than English, write this guide both in that language and in
    English, since English seems to be the de facto international language
    for news administration right now. (At the least, try to use the
    literal word "checkgroups" on the web site on the top page for a link
    to the current checkgroups; that's the main thing that people will be
    looking for, and that way news admins can find it even if they can't
    read the rest of the site.)

    * Write a control.ctl entry for your hierarchy and publish it on your web
    site in the section for news admins and in the guide. For a hierarchy
    that doesn't PGP-sign control messages, this will look like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    checkgroups:news@example.com:example.*:doit
    newgroup:news@example.com:example.*:doit
    rmgroup:news@example.com:example.*:doit

    The first line contains your hierarchy name in all caps and then in
    parens a description of your hierarchy, including the applicable
    location if the hierarchy is regional. Then comes information about
    who to contact about the hierarchy and the URL for your web site.
    Finally, the uncommented lines specify the type of control message, the
    sender (which should match the From and Sender headers of your control
    message), the pattern matching the groups in your hierarchy, and the
    word "doit" saying to act on those control messages.

    News administrators can then just cut and paste this entry into their
    news server configuration files and start honoring your control
    messages. If you're using PGP, see the section on that below.

    * Use PGP to sign all your control messages. This isn't necessary for
    small hierarchies, but if there end up being any fights over your
    hierarchy or it draws hostile attention for some reason, people can
    easily cause a lot of confusion if your control messages aren't signed.
    There's more information later about this.

    * Make sure the information on your web site stays up to date! In
    particular, be *sure* that the checkgroups on the web site is accurate,
    since that's the list most people will start with. It's also sometimes
    more convenient for someone to process a checkgroups from a web site
    than process a control message (one can use tools like actsync to do
    that, for example), so if it's not accurate, a news administrator may
    undo all your control messages by getting information from the web site
    instead.

    * Check <http://usenet.trigofacile.com/hierarchies/> for your hierarchy
    and make sure the information is accurate. The information here is
    based on the information kept at <ftp://ftp.isc.org/pub/usenet/CONFIG/>
    and <ftp://ftp.isc.org/pub/usenet/control/> and provides a quick way to
    check on recent changes to your hierarchy and ensure that nothing is
    out of date.

    Some hierarchies also have a publically available news server that carries
    that hierarchy (this is particularly common for hierarchies intended for support of the products of a particular company). If you do, make sure to point both users and news administrators at it. Users can use that server
    if their local server doesn't carry your hierarchy, and news
    administrators can use that server to get an up-to-date list of newsgroups
    in the hierarchy using tools like actsync. If you do have such a server,
    add the line:

    # Syncable server: news.example.org

    pointing to that server to your control.ctl entry.

    Be careful about running open servers like this, as they have frequently
    been abused to damage the rest of Usenet. At the least, it's best to not
    allow users to crosspost to groups not carried by the server or add
    Supersedes or Control headers, and ideally you should have some sort of
    spam filtering or posting rate limits in place.

    ------------------------------

    Subject: 3. PGP-Signing Control Messages

    Most hierarchies now PGP-sign all control messages. The PGP signature is visible in the X-PGP-Sig header of the control message. Background
    information and (now somewhat outdated) instructions are at:

    <ftp://ftp.isc.org/pub/pgpcontrol/README.html>

    and the exact format of the signature is at:

    <ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>

    You don't need to understand the information in the last link, though, at
    least if you're using Unix and can use the existing tools for generating
    signed control messages.

    In order to sign control messages, you'll need three things: A PGP key
    pair that you'll use for signing and verifying the messages, a PGP implementation you can use, and some software to generate the right type
    of signatures for Usenet control messages.

    If you're not already familiar with public key cryptography, here's a
    brief primer: A PGP key pair consists of two keys, a public one and a
    private one. You sign messages with the private key, which you have to
    keep secret and protect since anyone with possession of it can pretend to
    be the hierarchy administrator. You give out the public key to anyone and everyone, and anyone with the public key can check a signature and verify
    that it was signed by the private key (without being able to sign messages themselves).

    Most Usenet news sites that honor control messages are set up to verify messages signed with an algorithm called RSA, which was the algorithm used
    by the original PGP implementation. Unfortunately, this is now fairly obsolete. Current PGP implementations use a newer, more secure algorithm
    for generating signatures (although the additional security is probably overkill for Usenet control messages, at least for right now). While this doesn't pose a problem for signing messages (current PGP implementations
    can still use old RSA keys to sign things), it does cause problems if
    you're starting fresh, since the keys generated by current implementations
    will not work with old versions of PGP.

    What all this means is that you have a few hard choices when it comes to choosing a PGP implementation and generating your initial key pair. You
    can use GnuPG <http://www.gnupg.org/>, which is probably the best
    available PGP implementation, and not bother with a RSA key at all. This
    will mean, unfortunately, that only sites that are also using GnuPG or
    another current PGP implementation will be able to verify your control messages. Or you can go to <http://www.pgpi.org/>, download an older
    version of PGP (something of the 2.6 vintage), and use it to sign your
    control messages, which will work with all versions of PGP but may be more
    of a pain. (You can also use an old version of PGP only to generate the initial key, and then import it into GnuPG and use GnuPG to sign control messages, but this is complex and not recommended for people who have
    never touched PGP before. There are some instructions on the GnuPG web
    site, though.)

    Whatever choice you make, follow the documentation of your PGP
    implementation to generate a key pair. Pay careful attention to the user
    ID that you put on your key. In order to work with Usenet, that user ID
    must not contain any spaces. The two most common key IDs used for signing Usenet control messages are the name of the administrative group in the hierarchy (like example.config) or the sender of the control message (like news@example.org). The latter is better practice and is recommended for
    new hierarchies, although make sure that e-mail address is stable and is
    one that you will be able to use for decades to come.

    If you're using GnuPG, in order to not get any spaces into the user ID,
    you need to use gpg --gen-key --allow-freeform-uid, enter the desired user
    ID as the name, and then press Enter when asked for an e-mail address or comment. The recommended user ID is the e-mail address of the sender, but
    it has to be entered as the name or GnuPG will not generate the right user
    ID. A later version of pgpverify will hopefully make this unnecessary,
    but older versions will be around for quite some time.

    Resist the temptation to put any additional user IDs on your key. Your
    key should only have one user ID, the one that will be used in control.ctl entries. If it has any additional user IDs, this can confuse the
    pgpverify process with some PGP implementations and cause your control
    messages to be ignored.

    If you're using a modern PGP implementation, it will automatically sign
    the public key (this is called a self-signature). If you're using an
    older PGP implementation, make sure that you do this, following the instructions for your software. Keys that aren't self-signed can be
    tampered with in various ways, and modern PGP implementations will refuse
    to import or honor them.

    Now, you're ready to create and sign a control message. I'm aware of
    several major implementations of the glue software to do the signing:

    signcontrol <ftp://ftp.isc.org/pub/pgpcontrol/>
    This is the original implementation by David Lawrence of the signed
    control message protocol. The signcontrol script is a Perl script
    that works with pgp. It will require some editing to set information
    about your hierarchy. There is also a shell implementation in the
    same directory.

    signcontrol <http://www.trigofacile.com/divers/usenet/clefs/signcontrol.htm>
    A different implementation in Python that may be somewhat easier to
    use. It similarly will require some editing to add information for
    your particular hierarchy and requires that GnuPG be available.

    News::Article <http://search.cpan.org/~agierth/>
    A Perl module, and therefore mostly useful if you're writing your own
    software for your hierarchy in Perl. It makes fewer checks than
    signcontrol and in general does less (just the signing) which can be
    more convenient. It requires the PGP::Sign module from CPAN; see your
    Perl documentation for how to install it.

    cmsg-tools <http://www.linux.it/~md/software/cmsg-tools.tgz>
    A Perl script which uses News::Article and GnuPG (but not PGP::Sign)
    to automatically generate control messages. It can be used to manage
    multiple hierarchies.

    Pointers to additional implementations, particularly instructions for
    Windows users, are very welcome.

    Once you have a signed control message, verify it to make sure that it
    verifies properly. You can do that with the pgpverify program that comes
    with INN or is available in the same directory as signcontrol above.

    If you got all of that working, you should put the PGP public key into
    your guide for news administrators. To see examples of PGP public keys
    for various hierarchies, see:

    <ftp://ftp.isc.org/pub/pgpcontrol/PGPKEYS>

    You should also make the public key available for download on your web
    site, preferrably as a plain text file (ending the file name in .txt will
    give this hint to most web servers) that can be easily downloaded. It's
    best to also submit it to the keyservers (see <http://pgp.mit.edu/>).
    Also change your control.ctl entry that you have on your web site to look
    like:

    ## EXAMPLE (Example Hierarchy, Olympus Mons)
    # Contact: news@example.org
    # URL: http://www.usenet.example.org/
    # Key URL: http://www.usenet.example.org/pgpkey.txt
    # Key fingerprint = G7 11 96 E8 34 32 7E 78 01 0D 69 99 A3 8F 34 B8
    # *PGP* See comment at top of file.
    checkgroups:news@example.com:example.*:verify-news@example.org
    newgroup:news@example.com:example.*:verify-news@example.org
    rmgroup:news@example.com:example.*:verify-news@example.org

    The key URL field is the URL of the plain-text PGP public key available
    from your web site. The fingerprint is the output of gpg --fingerprint or
    pgp -kvc (depending on what version of PGP that you're using) and is used
    as a check to be sure that the key downloaded is the right one. The *PGP* comment makes sense in the context of the standard control.ctl file, which
    has information about PGP-signed control messages at the top. And note
    that the "doit" string to act on the control messages has been replaced by "verify-" followed by the user ID of your PGP key (whatever that may be).

    Now, just use your new signing procedure whenever you send a control
    message for your hierarchy. Oh, and if you're changing to signing control messages from not signing them, be sure to announce that in news.admin.hierarchies.

    ------------------------------

    Subject: 4. Maintaining Moderated Groups

    If there are moderated newsgroups in your hierarchy, this involves a bit
    of additional hassle over unmoderated groups. Rather than just checking whether the newsgroup is there are not, news administrators also have to
    make sure that the moderation status is set correctly so that users can
    post, and news servers need to know to which e-mail address to send posts
    so that they'll reach the moderator.

    When managing moderated groups, be sure to always include the correct moderation flag in your newgroup control messages, and make sure that your checkgroups lines for moderated groups end in " (Moderated)". (Yes,
    literally. It can't be translated; news software actually looks for
    this.) It's sometimes useful to send a few duplicate control messages if
    you ever convert an unmoderated group to a moderated group or vice versa,
    since it's harder to get this change to take than a simple newsgroup
    creation. Expect to have a few frustrated users who's news providers just cannot get this right. You may want to put together a form message to
    send to those providers explaining who you are and that they have groups
    in your hierarchy configured incorrectly.

    Finally, you have to deal with getting the posts to the moderator. There
    are two main ways of handling this, which can be combined. One is to keep
    the central moderators.isc.org relay systems up to date, and the other is
    to set up your own relay system that accepts mail addressed to the
    newsgroup name with all periods replaced by hyphens @ some system you
    control and then forwards that mail along to the real moderator.

    Nearly all news servers that try to handle moderated groups default to forwarding any message posted to a moderated group to the name of the
    group with periods replaced by hyphens @moderators.isc.org. The relay
    systems behind that DNS record then forward those messages along to the
    real moderator. To have one of those addresses created or to change where
    it forwards to, mail moderators-request@isc.org. These changes have to be
    made manually and there's often a backlog; to save time, please identify yourself as the hierarchy administrator for the hierarchy and preferrably
    send the mail from your hierarchy contact address (and you may also want
    to PGP-sign the message if you can do that easily). Even still, be
    prepared for this to sometimes take a while.

    It's possible to configure news servers to forward mail somewhere other
    than to moderators.isc.org, and easiest to do that as wildcard entries for hierarchies that forward all posts to moderated groups in that hierarchy
    to a particular host instead of moderators.isc.org (again with the period
    to hyphen change). Some hierarchies do this instead so that they don't
    have to wait for moderators.isc.org changes, but it makes things harder
    for news administrators who have to keep track of those special cases. If
    you do do this, you have to tell all news administrators to add a line
    like:

    example.*:%s@example.org

    to their moderators file, where the first part is the pattern that matches groups in your hierarchy and the second part is the e-mail address to send postings to (%s will be replaced by the group name with periods changed to hyphens). If that forwarding host ever goes away, o