Last-modified: 2021-01-20
Posted-by: postfaq 1.17 (Perl 5.28.1)
Archive-name: usenet/software/inn2-faq
URL:
https://www.eyrie.org/~eagle/faqs/inn.html
Posting-frequency: monthly
This FAQ is intended to answer frequently asked questions concerning the current versions of INN (INN 2.x and later) seen on news.software.nntp.
It should be referred to in preference to the old INN FAQ, which only
documents versions up to 1.7. It mostly covers INN 2.3 and later; earlier versions of INN may behave differently or use different configuration
files.
If you're reading this on Usenet, this FAQ is formatted as a minimal
digest, so if your news or mail reader has digest handling capabilities
you can use them to navigate between sections. In rn variants, you can
use Ctrl-G to skip to the next section; in Gnus, press Ctrl-D to break
each section into a separate article.
Please send any comments, suggestions, or updates to <
eagle@eyrie.org>.
Bear in mind when sending me e-mail that I receive upwards of 800 mail
messages a day and have unanswered personal e-mail dating back six months
or more, so please don't expect an immediate response. You may receive
quicker responses by posting to news.software.nntp (even, due to the
quirky way in which I read mail and news, from me).
This FAQ is posted monthly to news.software.nntp, and is available on the
web at <
https://www.eyrie.org/~eagle/faqs/inn.html>.
------------------------------
Subject: Contents
1. General Questions
1.1. What is INN?
1.2. What is the current version?
1.3. Where can I get INN?
1.4. Where can I find documentation?
1.5. What newsgroups are there for INN?
1.6. What mailing lists are there for INN?
1.7. How can I support INN development?
1.8. How can I contribute to INN?
2. Terms
2.1. What is tradspool (traditional spool)?
2.2. What is CNFS?
2.3. What are timehash and timecaf?
2.4. What is overview?
2.5. What are deferrals (NNTP code 431)?
3. Specific Problems
3.1. INN won't start after a new installation
3.3. The news server isn't keeping up with incoming news
3.4. news.notice is empty and the nightly report is missing things
3.5. INN is running out of file descriptors
3.6. Can't get debugging information out of INN
3.7. Articles aren't being sent to remote peers
3.8. sendmail isn't installed
4. Error Messages
4.1. innd: SERVER cant store article
4.2. innd: SERVER internal no control and/or junk group
4.3. Modification of read-only value attempted (Cleanfeed)
4.4. tradspool: could not open ... File exists
4.5. Binary posting to non-binary group (Cleanfeed)
5. Problems on Specific Systems
5.1. INN won't compile on SCO OpenServer / UnixWare / OpenUNIX
5.2. Using raw devices on Solaris destroys the partition table
5.3. Will INN run on Windows?
5.4. Why aren't INN's files where the documentation says they are?
5.5. Running INN on macOS
6. How Do I...
6.1. Set up a server with no external feeds, just local groups
6.2. Process a single control message
6.4. Feed all articles on a server to another server
6.5. Rename a newsgroup
6.6. Change the domain used for message IDs
6.7. Use INN without a direct news feed
6.8. Generate MRTG graphs for INN
6.9. Hide the junk and control groups from users
6.10. Modify the body of posts made through my server
6.11. Hide the Injection-Info header
6.12. Run innd and nnrpd on separate ports
6.13. Back up and restore an INN installation
(Note that some numbers have been skipped. When questions are removed,
the remaining questions are not renumbered to avoid breaking links in
Usenet and mailing list archives.)
------------------------------
Subject: 1. General Questions
Contained in this section are general questions about INN, where to find
it, and things of that sort. It is aimed at the person who is not yet
running INN, or who has general questions about how it works.
------------------------------
Subject: 1.1. What is INN?
The README that comes with INN has this to say (in part):
INN (InterNetNews), originally written by Rich Salz, is an extremely
flexible and configurable Usenet / Netnews news server. For a complete
description of the protocols behind Usenet and Netnews, see RFC 3977
(NNTP), RFC 4642 updated by RFC 8143 (TLS/NNTP), RFC 4643 (NNTP
authentication), RFC 4644 (streaming NNTP feeds), RFC 5536 (USEFOR),
RFC 5537 (USEPRO), RFC 6048 (NNTP LIST additions) and RFC 8054 (NNTP
compression) or their replacements.
In brief, Netnews is a set of protocols for exchanging messages between
a decentralized network of news servers. News articles are organized
into newsgroups, which are themselves organized into hierarchies.
Each individual news server stores locally all articles it has received
for a given newsgroup, making access to stored articles extremely fast.
Netnews does not require any central server; instead, each news server
passes along articles it receives to all of the news servers it peers
with, those servers pass the articles along to their peers, and so on,
resulting in "flood fill" propagation of news articles.
INN is free software, supported by Internet Systems Consortium and
volunteers around the world.
For a more complete answer, see that file. A full description of what
Usenet and Netnews are is beyond the scope of this document; for a
beginner's introduction, see the news.newusers.questions home page at <
http://www.tokak.us/nnq/>.
------------------------------
Subject: 1.2. What is the current version?
The most recently released version of INN is 2.6.3.
INN development proceeds in two branches, as with many other free software projects. The STABLE branch is maintenance of the most recently released stable version, and only bug fixes are added to it. The CURRENT branch is
the development version of the next release of INN.
As mentioned in the next section, when installing a new INN server, you
may wish to download the latest snapshot of the STABLE branch rather than
the current full release.
Note that the previous STABLE series for INN 2.5 terminated in the release
of INN 2.5.5 and current STABLE snapshots are based on INN 2.6. You
should therefore read the upgrade instructions in NEWS when upgrading from
a STABLE snapshot before September 12th, 2015 to one dated after that.
------------------------------
Subject: 1.3. Where can I get INN?
The download site for INN is <
https://ftp.isc.org/isc/inn/>. In that
directory are the various releases of INN, some additional documentation (particularly of security holes), the original INN Usenix paper.
There is also a snapshots subdirectory, in which you will find daily
snapshots of the INN Subversion repository for the last seven days. The snapshots with STABLE in the name are the latest versions of the STABLE
branch and may have some additional bug fixes over the current released version. The snapshots with CURRENT in the name are of the current
development version.
Please note: There is no guarantee that a snapshot will even compile, let alone function well as a news server. In particular, the CURRENT branch
is under active development, and all sorts of things may be broken at any
given point in time. Use snapshots with caution, and don't use snapshots
from the CURRENT branch on any production system unless you're prepared to debug the inevitable problems in code that's actively changing and not yet thoroughly tested. (The STABLE snapshots should be fairly reliable,
however.)
------------------------------
Subject: 1.4. Where can I find documentation?
INN comes with extensive documentation. See the files INSTALL and README
at the top level of the source tree, for starters. In addition, nearly
every program and configuration file has its own Unix man page. The best
place to start is by reading the entire INSTALL file and then from there discovering which configuration files and programs do what you want to do
and reading their individual man pages.
There are HTML conversions of the documentation that comes with recent
versions of INN available at:
<
https://www.eyrie.org/~eagle/software/inn/>
For additional documentation beyond what is distributed with INN, follow
the links suggested in the above page.
The documentation that comes with INN is fairly technical in nature and
lacking in some more general details on configuring news servers. Some of
the links off of the INN home page have additional overview documentation
or documentation on how to set up servers for specific roles.
Another good resource is the newsgroup news.software.nntp (and the Google archives thereof) and the archive of the inn-workers mailing list. A link
to the latter is off the INN page referenced above.
Finally, the following additional links may be useful:
<
http://aplawrence.com/Unixart/newsserver.html>
A tutorial on setting up INN aimed at beginners using SCO Unix. While
it's mostly focused on SCO, it may be useful for any beginner to INN
and news servers.
<
http://www.kozubik.com/published/inn_tutorial.txt>
A tutorial on setting up INN on FreeBSD. Contains a lot of
information focused on FreeBSD and its preferred file layout, so may
be easier to follow than the generic instructions on that platform.
------------------------------
Subject: 1.5. What newsgroups are there for INN?
news.software.nntp discusses all NNTP-based news servers, including INN.
It's the best newsgroup for technical questions and discussion. The
newsgroup news.software.b is also chartered for such discussion, but it's essentially dead now. General news administration questions are also
on-topic in news.admin.technical (moderated) and news.admin.misc
(unmoderated).
news.admin.hierarchies covers questions of general hierarchy configuration
and is where announcements of new news hierarchies are generally posted. news.admin.net-abuse.* covers the topic of network abuse and prevention (including spam), but is not for the faint of heart; it is extremely noisy
to the point of being essentially unreadable without a lot of time and
patience (and a good killfile).
------------------------------
Subject: 1.6. What mailing lists are there for INN?
There are several INN-related mailing lists:
inn-announce@lists.isc.org
Where announcements about INN are set (no posting allowed).
inn-workers@lists.isc.org
Discussion of INN development. It is also where to send bug reports
and patches for consideration for inclusion into INN (postings by
members only). If you're an INN expert and have the time to help out
other users, we encourage you to join this mailing list to answer
questions. (You may also want to read the newsgroup
news.software.nntp, which gets a lot of INN-related questions.)
inn-committers@lists.isc.org
Subversion commit messages for INN are sent to this list (only the
automated messages are sent here, no regular posting).
inn-bugs@lists.isc.org
Trac tickets for INN are sent to this list (only the automated
messages are sent here, no regular posting). Bug reports should be
sent to the inn-workers mailing list.
To join these lists, send a subscription request to the `-request'
address. The addresses for the above lists are:
inn-announce-request@lists.isc.org
inn-workers-request@lists.isc.org
inn-committers-request@lists.isc.org
inn-bugs-request@lists.isc.org
inn-workers tends to be moderate volume (3-5 messages a day, but varying a
lot depending on what's being discussed). inn-bugs and inn-committers are occasionally higher volume but entirely automatically generated Trac
or Subversion notifications. inn-announce is a low-volume moderated list containing only major announcements.
------------------------------
Subject: 1.7. How can I support INN development?
There are four major ways. First, like with any other free software
project, a great way to support INN development is to join in yourself.
If you know how to program and have an interest in working on a widely
deployed and fairly intricate news server, we'd love to have your help.
See the next question for more details.
Second, even if you don't have the time or expertise to write much code,
any contributions of documentation are *greatly* appreciated. There's
always documentation work to be done, from maintenance of INN's technical documentation to tutorials and overviews for the new user or the user who
wants to do something specific. Listen on news.software.nntp for what
people are looking for, or ask on inn-workers. Similarly, beta testers
are always welcome; if you have a test news server and some knowledge of
how to diagnose server problems and want to try out the current
development code and report any bugs you run into, that helps the
developers immensely.
Third, there are always more questions from new INN users to answer. news.software.nntp gets a regular stream of them, and it's a great way to
help out intermittantly when you have a few moments to read news. If you
can identify general solutions to frequent problems and pass them along to
the INN maintainers in the form of documentation or suggestions, even
better.
Fourth, from the README:
Note that INN is supported by Internet Systems Consortium, and
although it is free for use and redistribution and incorporation into
vendor products and export and anything else you can think of, it
costs money to produce. That money comes from ISP's, hardware and
software vendors, companies who make extensive use of the software,
and generally kind hearted folk such as yourself.
Internet Systems Consortium has also commissioned a DHCP server
implementation and handles the official support/release of BIND. You
can learn more about the ISC's goals and accomplishments from the web
page at <
https://www.isc.org/>.
The ISC provides ftp and web space and mailing lists and archives.
Donations to help support all of that are greatly appreciated.
------------------------------
Subject: 1.8. How can I contribute to INN?
First, join inn-workers, since that's where all the development discussion takes place. The traffic isn't that high.
Next, download a snapshot of the INN CURRENT branch as described above so
that you have a relatively current source base to work from. You may want
to check out the current source from Subversion; just point a Subversion
client at:
https://inn.eyrie.org/svn/
This repository is a read-only mirror that can lag up to an hour behind
the working repository. Read the HACKING file at the top of the INN
source tree for some general information and tips for working on INN.
Then pick something that looks interesting to you, mention what you're
doing on inn-workers if it's likely to affect other parts of the
development, and have at it! The TODO file in the CURRENT tree has a
pretty comprehensive list of things that could be done. Best to start
with something small (getting INN to work correctly on a platform where it doesn't currently and which you have available is often a great start, or working on one of the supporting programs or scripts that's a bit easier
to wrap one's mind around than the core INN daemons). Patches to INN
should be sent to
inn-workers@lists.isc.org, or put on an ftp or web site somewhere and the URL sent to inn-workers if they're extremely large.
------------------------------
Subject: 2. Terms
Here are definitions of some commonly used terms related to INN. (More definitions are welcome; this section is extremely incomplete at the
moment and the FAQ maintainer tends not to recognize terms that need a definition for people unfamiliar with INN.)
------------------------------
Subject: 2.1. What is tradspool (traditional spool)?
Traditional spool is called that because it's the way that all news
servers used to store articles. A traditional news spool is a tree of directories matching the hierarchical structure of newsgroups. For
example, the newsgroup news.software.nntp would be stored in a directory news/software/nntp under the root of the news spool, and next to the
"nntp" directory in news/software would be a "readers" directory for the
group news.software.readers.
As of INN 2.3, traditional spool is completely integrated into the storage
API as the tradspool storage method and use the same overview mechanisms
as the rest of INN.
Storing articles in the traditional spool format is slow relative to other storage mechanisms. It's probably nearly impossible to keep up with a
full Usenet feed using pure traditional spool. It is, however, the
recommended storage method for low-traffic local newsgroups and any
newsgroups that you want to back up.
For more details, see the INSTALL file.
------------------------------
Subject: 2.2. What is CNFS?
CNFS is the Cyclic News File System, written by Scott Fritchie. It is a high-performance method of storing news articles, designed to avoid the
high overhead involved in interacting with the file system when storing articles in individual files. CNFS stores articles sequentially in pre-configured buffer files. When the end of the buffer is reached, new articles are stored from the beginning of the buffer, overwriting older articles.
It's the fastest article storage method in terms of write performance, and
is recommended for storing binaries.
For more details, see the INSTALL file.
------------------------------
Subject: 2.3. What are timehash and timecaf?
These are two less-used storage mechanisms available under the INN storage
API (similar in that respect to CNFS). Both can usefully be thought of as compromises between the write speed of CNFS and the fine-grained control
over article expiration. INSTALL says for timehash:
Articles are stored as individual files as in tradspool, but are
divided into directories based on the arrival time to ensure that no
single directory contains so many files as to cause a bottleneck.
and for timecaf:
Similar to timehash, articles are stored by arrival time, but instead
of writing a separate file for each article, multiple articles are put
in the same file.
timecaf was new in INN 2.3.
------------------------------
Subject: 2.4. What is overview?
Overview is summary information about articles in a newsgroup that is
returned to news reading clients as a response to the OVER command. It's
a very common extension to the NNTP protocol that allows readers to review summary information about articles before taking the time (and bandwidth)
to download the entire article.
The canonical items of information included in an overview record are the Subject, From, Date, References, and Message-ID headers of the article,
the byte count of the article, and the line count of the article. Nearly
every server now also returns the Xref header (a list of the newsgroups
carried by the server to which the article was posted and the article
number in each of those newsgroups) as an additional field.
Note that with the References and Message-ID headers, the overview record contains enough information to do article threading. It also contains all
of the fields normally keyed on for client-side filtering (killfiles and
the like).
Generating overview information for a newsgroup on the fly would be prohibitively expensive, particularly for large groups, since the server
daemon would have to find all of those articles and scan them to build the information. It would also be inefficient, since the overview information
for a particular group will generally be requested many times by different clients.
Any INN server that supports readers must therefore have an overview
method configured. There are four different methods to choose from:
- buffindexed, which stores overview data and index information
into preconfigured large files like CNFS. Fast at writing, the
buffindexed overview storage method can keep up with a large feed
more easily and never consumes additional disk space beyond that
allocated to these buffers. The downside is that these buffers
are hard to recover in case of corruption and somewhat slower for
readers and the expiry process;
- ovdb, which stores overview information into a Berkeley DB database,
whose development pace has stalled these last years. This method
is fast and very robust, but may require more disk space, unless
compression is enabled. Overview data is fetched one article at a
time, which makes this method a bit slower than ovsqlite for readers;
- ovsqlite, which stores overview information into an SQLite database,
known for its long-term stability and compatibility. Robust and
faster than ovdb at reading ranges of overview data (since overview
data is transferred in 128-kilobyte chunks between ovsqlite-server
and nnrpd) but somewhat slower at writing, this method may require
more disk space, unless compression is enabled;
- tradindexed, which uses two files per newsgroup, one containing
the overview data and one containing the index. Fast for readers,
but slow to write to because it has to update two files for each
incoming article. Its main advantage is to be the best tested,
the most reliable and the method with the best recovery tools.
Here are a few elements that can be helpful in choosing the right
overview method for your needs and estimating the associated storage
size. In 2020, the volume for a full-text Usenet feed is about 18,000
articles per day, with peaks to 1,200 articles per hour. Article storage
size is about 65 MB per day.
As for overview storage size, if you have 5 million articles, you'll
need at least 3.25 GB of disk space for buffindexed, 5.5 GB for ovdb
(4.5 GB if compressed), 4.65 GB for ovsqlite (2 GB if compressed),
and 3.10 GB for tradindexed.
If you store more header fields in overview data than the standard
ones, the space needed to store overview data will be superior than
these estimates. (This is configured in the extraoverviewadvertised
and extraoverviewhidden inn.conf parameters.)
------------------------------
Subject: 2.5. What are deferrals (NNTP code 431)?
Consider the following situation. You have two incoming peers, both of
which are getting ready to offer you an article in streaming mode. The
first sends you a CHECK <message-id> message, to which you respond affirmatively (i.e., you don't already have the article). Then, before
that peer sends you the article with TAKETHIS, you receive a CHECK
<message-id> from the second peer for the same message. What response
does INN send to the second peer?
If deferrals are enabled (noresendid == false in incoming.conf for that
peer, the default), INN will send a 431 deferral telling that peer that
you may or may not want the article; try again later. Chances are that
when it retries, you will have received the article from the first peer
and you'll just refuse it. But if the first peer dies before it ever
sends you the article, this way you can still get it from the second peer.
If deferrals are disabled, INN will refuse the article from the second
peer, which means there's a possibility you'll lose news if the first peer
dies before sending you the article.
As a side note, some older versions of Diablo, upon receiving a deferral,
turn around and immediately send the article via TAKETHIS, which is
basically exactly what you don't want. (Chances are extremely high in
practice that the first peer will come through with the article.)
------------------------------
Subject: 3. Specific Problems
This section contains specific problems that are frequently reported when
using INN, and includes fixes or suggestions for fixes. Candidates for inclusion in this section are any problems reported frequently on news.software.nntp or
inn-workers@lists.isc.org. Contributions, including fixes, are very welcome.
------------------------------
Subject: 3.1. INN won't start after a new installation
The most common cause of this problem is that inndstart isn't setuid root (please note that it only affects versions prior to INN 2.5.0 because
inndstart was removed in INN 2.5.0). inndstart must be installed owned
by root and group news, mode 4550. The ls -l output for inndstart should
look something like:
-r-sr-x--- 1 root news 53768 Jan 8 00:47 inndstart*
inndstart will automatically be installed with the right permissions if
you run make install as root. If inndstart isn't setuid root, it will log errors to syslog when it tries to start and cannot. If you aren't seeing
those error messages in syslog either, you probably haven't set up syslog properly (see 3.4).
The other most frequent cause of this problem is not correctly following
the instructions in INSTALL on how to set up the initial history database.
If you run makedbz without the -o flag, the initial history database files
will have names starting with history.n. These files must be renamed to
remove the ".n" before innd will start.
------------------------------
Subject: 3.3. The news server isn't keeping up with incoming news
Start by looking for the profile information in your nightly report. That
will tell you where the news server is spending most of its time and may identify the exact nature of the problem.
This problem is quite frequently due to using the traditional spool
storage format for news articles. This storage method is now too slow to
be able to handle a full Usenet news feed (although with a more limited selection of groups it can still do just fine). If your server is
spending a lot of time writing articles and you're using traditional
spool, this is probably the problem.
One possible solution would be to switch to CNFS as a storage mechanism.
You can do this simply by configuring CNFS (see INSTALL for details),
changing storage.conf to direct some or all of the incoming traffic to
CNFS buffers, and then restarting INN. Older articles will continue to be stored in tradspool until they expire, but new articles will go into CNFS.
------------------------------
Subject: 3.4. news.notice is empty and the nightly report is missing things
You have syslog set up incorrectly.
INN logs nearly everything except article trace information via syslog.
It expects syslog to write its log messages into particular files under ~news/log, unless you gave it a different path at configure time (see
the pathlog parameter in inn.conf). You'll need to set up logging of INN-related log messages in your system /etc/syslog.conf. See the
"Configuring syslog" section in INSTALL.
Note that you don't have to worry about rotating these log files;
news.daily (which should be run nightly from cron) will take care of that
and innreport generates a daily summary report from them.
------------------------------
Subject: 3.5. INN is running out of file descriptors
You may need to increase your system file descriptor limits. See the
"File Descriptor Limits" section of INSTALL for more details. This is particularly a concern on Solaris systems, since Solaris by default has an exceptionally low file descriptor limit.
------------------------------
Subject: 3.6. Can't get debugging information out of INN
The INN startup process is quite complicated, involving the rc.news shell script (and the setuid inndstart wrapper for versions of INN prior to 2.5.0). This can make it rather difficult to get enough debugging information out
of it to determine what's going wrong if it's crashing immediately after startup or otherwise having serious difficulties.
One approach is to run innd by hand directly, giving it the -d option.
This requires setting up a configuration where innd doesn't need to bind
to privileged ports, however.
Another, sometimes better option, is move the innd binary to another name,
like innd-real, and put a shell script in its place. Here's an example,
from Kai Henningsen:
#! /bin/bash
# allow core dumps
ulimit -c unlimited
# save any output
exec > /tmp/innd.log 2>&1
# who are we running as, anyway?
id
# show exported environment
export
# start innd (don't forget the arguments, or it will complain)
exec strace -o /tmp/innd.strace -f -F /path/to/innd-orig "$@"
This starts innd under strace, suitable for debugging startup core dumps
and the like. You can use this as a general model for a variety of
debugging; for example, you could replace the strace invocation with an invocation of gdb and then start innd from inside gdb with the -d option.
------------------------------
Subject: 3.7. Articles aren't being sent to remote peers
(This entry is based on a post by Jeffrey M. Vinocur.)
Here's how to trace through INN's logs to figure out what's happened to a particular article. This should help you discover where the process of
feeding an article to a peer broke down.
1. First you look in the $pathlog/news file. There should be one line per
article (search for the Message-ID, or they're in order by time of
arrival if you know that).
If you don't see a line for the article in question, your innd has
never seen it. For articles being fed remotely, this means your peer
didn't feed it to you. For articles being posted to your server, this
generally indicates some sort of problem in nnrpd.
(The only other time you wouldn't see a line for the article in
question is if your innd has seen it in the past, and is considering
this attempt a "duplicate".)
2. Next, look at the second field of the line you've found in
$pathlog/news.
If it's "-", then your innd rejected the article. The reason should be
at the end of that line.
3. At this point, you should be looking at a line with "+" in the second
field. The article should be on your server at this point.
If it's not, either it's been cancelled, or has already expired.
4. You're now interested in whether the article was sent to your peers.
At the end of the same line in $pathlog/news, innd puts all of the
peers it thinks should receive this article.
If you don't see a peer you expect there, it indicates that
$pathetc/newsfeeds is not configured in the way you think it is.
5. If a peer is listed at the end of the line, the article should have
been fed to that peer.
If a peer doesn't have that article, it's possible that the article is
spooled on your system somewhere. Check $pathoutgoing, or the
innfeed spool if the peer is configured to use innfeed. (It's probably
easier to look for error messages in $pathlog/news.notice than to
actually wade around in $pathspool/innfeed.)
6. If you're sure the article isn't spooled, and it doesn't show up on the
peer, you have to consider the possibility that the peer has rejected
the article. Alternatively, it's possible that the peer has some
misconfiguration like the ones described above.
In either case, if you're sure that the article was offered to the peer
and not spooled, you will need the assistance of the peer's admin to
investigate further. INN does not generally log enough information
about outgoing articles to be able to tell more from your server alone.
It may be possible to get a slight bit of information from the remote
server by connecting with telnet (usually to port 119) and issuing
"IHAVE <message-id>". The peer may respond with something like "435
Duplicate" which means that the problem is not likely to be with your
server (it may be still a problem with the article itself). If the
peer responds with something like "335", your server probably did not
offer the article after all.
If you really are at a dead end and need to get more information about
[continued in next message]
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)