Some of the headers that I pasted in my original message didn't make it through to the posted message. I may be celebrating too hard while computing this evening. ;)
Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018 alt.test:8601
On 2022-07-04, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
Some of the headers that I pasted in my original message didn't make it
through to the posted message. I may be celebrating too hard while computing >> this evening. ;)
Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
alt.test:8601
Hmmm or maybe my news client was stripping them, trying from slrn this
time:
Newsgroups: rec.arts.disney.parks,alt.test
Date: Sat, 11 Apr 2009 00:53:23 GMT
Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018 alt.test:8601
Newsgroups: rec.arts.disney.parks,alt.test
Date: Sat, 11 Apr 2009 00:53:23 GMT
Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
alt.test:8601
Also, shouldn't I be getting more output?
[news@spool1 ~]$ expire -p -v 5
Article lines processed 32971875
Articles retained 32966677
Entries expired 5198
Hi Jesse,
What's the posting date INN has for this article?
If you have its Message-ID, look at the last number in the result of grephistory:
% grephistory -l '<9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>' [A6E3D4975ED59BD167F5380850829D76] 1656898303~-~1656898302 @020162C242FF001200000000000000000000@
% convdate -dc 1656898302
Mon, 4 Jul 2022 01:31:42 -0000 (UTC)
Also, shouldn't I be getting more output?
[news@spool1 ~]$ expire -p -v 5
Article lines processed 32971875
Articles retained 32966677
Entries expired 5198
"-v 5" should definitely be far more verbose, printing something for
each article line processed...
I ran expire several times with '-v 2' through '-v 5' and never get more output than the standard three lines. I'm running INN 2.7.0rc1 with a rather large tradspool for most articles and a single CNFS buffer for binary articles.
I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
a test but same results.
Hi Jesse,
I ran expire several times with '-v 2' through '-v 5' and never get more
output than the standard three lines. I'm running INN 2.7.0rc1 with a rather >> large tradspool for most articles and a single CNFS buffer for binary
articles.
I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
a test but same results.
I've tested a bit.
In fact, the flags are correctly seen. The -v flag works fine but the
-p flag does not seem to do what is expected, and also maybe without -p
(the part of the code where expire removes articles does not look right
to me, using conditions related to groupbaseexpiry and whether the
storage method self expires).
I have to look at it deeper.
I ran expire several times with '-v 2' through '-v 5' and never get more >>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
large tradspool for most articles and a single CNFS buffer for binary
articles.
If there is anything I can do to help assist, let me know.
If there is anything I can do to help assist, let me know. I do notice odd behavior, such as running expire back to back results in a few thousand articles being expired each time. I wouldn't expect so many articles to expire
in such a short time, but since I cannot get more output I don't know what it is doing exactly.
What conditions did you run expire where the -v flag was honored?
I want MORE output, not less. With any -v flag above '1' I only get three
lines of output, the same three lines from '-v 1'.
Perhaps I am not articulating this well, but I expect more output from these commands:
$ expire -v 3 -n -t
Expire tracing: history files not updated.
Article lines processed 206532
Articles retained 111477
Entries expired 95055
The above is from a different INN 2.7.0rc1 box than I've been having issues with, just to test that it wasn't a single server issue.
I want MORE output, not less. With any -v flag above '1' I only get three lines of output, the same three lines from '-v 1'.
You can use "-x -t" for a dry-run of expire.If you run that without the patch I gave (which changes a boolean in a
expire -x -t -v4 > exp.log
Simply use "-v 0" and you'll no longer see any output :-)
I've investigated more today, and I believe there's still a bit of documentation to highlight more.
I discovered a few things (I still had not had a chance to have a closer
look at how expire really works).
I've played a bit with expire, adding a few lines of code and trying a
few things. I obtained errors of class definition missings (which are
the classes numbers in storage.conf).
And then I found out that expire does not do much work when
groupbaseexpiry is true. It does not even parse expire.ctl when using "<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
is false).
And there's a note in the code saying:
/* don't process this line -- per-group expiry is done by expireover */
Ah ah, it becomes clearer now.
From inn.conf man page:
groupbaseexpiry
Whether to enable newsgroup-based expiry. If set to false, article
expiry is done based on storage class of storing method. If set to
true (and overview information is available), expiry is done by
newsgroup name.
Note the "and overview information is available".
So basically, if you want the behaviour you explained in the first
message of this thread:
* either you need to set groupbaseexpiry to false, rewrite expire.ctl accordingly, and then "expire -p -v 3" will work as expected.
It would mean that you have storage classes definitions suiting your
needs of expiration (as it will be by class), so in fact the classes
should be done with a sorting by newsgroups: one class for "*.jobs*",
another class for "news.lists.filters", another for "*.test", etc.
* either you keep groupbaseexpiry to true, and then it means overview
data is needed.
In your previous article, you mentioned:
news.daily noexpireover flags="-v5 -p"
Does it mean you do not have overview data?
If no, then you need creating it or switching to groupbaseexpiry set to false...
And if you have overview data, the Key is to run:
expireover -e -N -p
-e: Remove articles from the news spool and all overview databases as
soon as they expire out of any newsgroup to which they are posted
-N: Apply expire.ctl rules to expire articles even from storage methods
that have self-expire functionality
-p: Expiration decisions are based on the article posting date
and update your cron line:
news.daily expireover expireoverflags='-e -N -p' flags='-N -v1'
I agree that the expire man page is clearly misleading about that, and
should clearly explain that. I'll have a look at where to put such information (several man pages are concerned).
If someone has a different understanding of how expire/expireover work
with regards to withgroupbaseexpiry, do not hesitate to react!
I may as well be misunderstanding something.
I'm going to have to take more time later to digest this further.
I do have
overview data, but I don't run expireover very often because on my spool it takes more than a day and causes the locking problems I brought up recently
but I was attempting to see how much space I could reclaim by getting rid of stuff cross posted to *.test using expire.
Hi Jesse,
I ran expire several times with '-v 2' through '-v 5' and never get more >>>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
large tradspool for most articles and a single CNFS buffer for binary
articles.
If there is anything I can do to help assist, let me know.
"expireover -e -N -p" should do what you want (see my previous response).
I've a bit improved expire for logging and, above all, with a few comments.
I don't know whether you could try the following patch easily (you will have to rebuild INN and run "make update") but in case you want to test it, feel free :-)
I'm going to do testing with a few groups as you outlined in a previous response and report back my findings.
"expireover -e -N -p" should do what you want (see my previous response).
That expireover command should do what I'm after without having to change groupbaseexpiry to false?
Or do I still need to set groupbaseexpiry to false,
setup the various storage classes, and change expire.ctl accordingly?
Hi Jesse,
"expireover -e -N -p" should do what you want (see my previous response). >>That expireover command should do what I'm after without having to change
groupbaseexpiry to false?
Exactly!
And you can even try it on 1 newsgroup (or more) with the example of
echo or grep active piped to it.
Or do I still need to set groupbaseexpiry to false,
setup the various storage classes, and change expire.ctl accordingly?
No need to do change anything; groupbaseexpiry set to false was to make expire work as you wanted (without expireover).
Hi Jesse,
"expireover -e -N -p" should do what you want (see my previous response). >>That expireover command should do what I'm after without having to change
groupbaseexpiry to false?
Exactly!
And you can even try it on 1 newsgroup (or more) with the example of
echo or grep active piped to it.
An expiration policy is applied to every article in a newsgroup it
matches. There is no way to set an expiration policy for articles crossposted to groups you don't carry that's different than other
articles in the same group. Normally, articles are not completely
deleted until they expire out of every group to which they were posted,
but if an article is expired following a rule where <flag> contains
"X", it is deleted out of all newsgroups to which it was posted
immediately.
And then I found out that expire does not do much work when
groupbaseexpiry is true. It does not even parse expire.ctl when using "<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
is false).
On Jul 7, 2022 at 3:32:03 PM CDT, "Jesse Rehmer" in <2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com> wrote:
I'm going to do testing with a few groups as you outlined in a previous
response and report back my findings.
This is interesting:
$ time echo "rec.arts.tv" | expireover -f - -Z /usr/local/news/tmp/lowmark Article lines processed 1371021
Articles dropped 0
Overview index dropped 0
real 8m4.764s
user 0m22.424s
sys 1m3.622s
So for the largest group I currently have, it only took 8 minutes. I wonder why it is taking so long on the entire spool when that group represents something like 2-3% of total articles. It did cause the locking issue for a few minutes during, but I think I can live with that if I can get things staggered a bit.
"expireover -e -N -p" should do what you want (see my previous response).
Can I inquire the purpose of the "X" flag in expire.ctl, specifically if expireover also needs to be called with '-e' to get the same behavior as the "X" flag as described in the manpage for expire.ctl
My assumption from reading the expire.ctl manpage was that my "AX" flag in expire.ctl would accomplish the same thing as what 'expireover -e' does.
It would seem based on the observation you made (quoted below) that the X flag
in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
the '-e' flag. Is that correct, or am I still missing something?
Hi Jesse,
"expireover -e -N -p" should do what you want (see my previous response). >>Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
expireover also needs to be called with '-e' to get the same behavior as the >> "X" flag as described in the manpage for expire.ctl
Running "expireover -e" is tantamount to adding "X" to every expiry rule
(and in that case, you are not obliged to add these "X").
Oh, I see that you did not add "X" to news.lists.filters so in fact you should not use the "-e" flag.
"expireover -N -p" is the command to run in your case.
I don't think you lost useful articles with your run with "-e",
according to your expire.ctl rules.
My assumption from reading the expire.ctl manpage was that my "AX" flag in >> expire.ctl would accomplish the same thing as what 'expireover -e' does.
Yes, and "-e" will force "X" in all rules.
It would seem based on the observation you made (quoted below) that the X flag
in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
the '-e' flag. Is that correct, or am I still missing something?
You're not obliged to add "-e" to expireover.
Got it, thank you very much for the clear explanation. I think I am
still mixing/confusing the operations of expireover and expire, where
when running expire along it wasn't doing what I expected (which now I
also understand thanks to your help).
Got it, thank you very much for the clear explanation. I think I am
still mixing/confusing the operations of expireover and expire, where
when running expire along it wasn't doing what I expected (which now I
also understand thanks to your help).
It doesn't help that this whole thing grew organically over many years and involves multiple programs with multiple somewhat confusing modes of operation and one really intrusive configuration option (groupbaseexpiry) that basically changes how everything works. This thread makes it clear
that this is all more complicated than it probably should be, but of
course making it less complicated without breaking anything is also very complicated!
It doesn't help that this whole thing grew organically over many years and involves multiple programs with multiple somewhat confusing modes of operation and one really intrusive configuration option (groupbaseexpiry) that basically changes how everything works. This thread makes it clear
that this is all more complicated than it probably should be, but of
course making it less complicated without breaking anything is also very complicated!
Hi Jesse and Russ,
Per-group expiry, that is to say when groupbaseexpiry is true, is
mostly done by expireover(8): news articles are removed from the news
spool by expireover, and then expire purges the history file.
Regardless the setting of groupbaseexpiry, expireover should be run
along with expire, usually via news.daily out of cron.
Note that expire never purges articles which do not match any entry in
expire.ctl.
Now that expire's documentation is fixed and "expire -p" is in fact working as it does, let's release INN 2.7.0 :-)
I'll generate the release this week-end.
Thanks again Jesse for all your questions on your fresh INN installation which
permitted to make it better, and especially its documentation!
Now that expire's documentation is fixed and "expire -p" is in fact working >> as it does, let's release INN 2.7.0 :-)
I'll generate the release this week-end.
No rc2? I'm still pouring over everything, might find more... ;) (Joking - I say go for it, I'll gladly update my boxes)
Per-group expiry, that is to say when groupbaseexpiry is true, is
mostly done by expireover(8): news articles are removed from the
news spool by expireover, and then expire purges the history
file.
This is getting at something that if I remember correctly is true, but
which I'm not sure we say anywhere. (But don't take my word for it since it's been a very long time since I looked at this code!) Specifically, I believe it's correct to say something like this:
When groupbaseexpiry is true, article expiration is primarily done by
expireover based on the expiration rules that match each newsgroup.
expire then does some additional cleanup to remove old history
database entries.
When groupbaseexpiry is false, article expiration is primarily done by
expire based on the expiration rules that match the storage class of
each article. expire removes the articles and the history entries,
and then expireover does the additional cleanup of removing the
overview database entry.
This is the thing that always confused me about groupbaseexpiry: depending
on what it's set to, which program does the actual work of deleting
articles changes.
Hmm, if a news admin ever configures his server with groupbaseexpiry to
true and *without* overview, he won't be able to expire articles from
his spool, will he? (something maybe worth mentioning!)
This is getting at something that if I remember correctly is true, but
which I'm not sure we say anywhere.
When groupbaseexpiry is true, article expiration is primarily done by
expireover based on the expiration rules that match each newsgroup.
expire then does some additional cleanup to remove old history
database entries.
When groupbaseexpiry is false, article expiration is primarily done by
expire based on the expiration rules that match the storage class of
each article. expire removes the articles and the history entries,
and then expireover does the additional cleanup of removing the
overview database entry.
This is the thing that always confused me about groupbaseexpiry: depending
on what it's set to, which program does the actual work of deleting
articles changes.
When groupbaseexpiry is set to true, expire just does a probe whether the article is still in the spool.
I would tend to think the 4 entries expired are cancelled articles whose tokens are removed from the history file. (More than 4 entries would
have expired for all test groups...)
Hmm, if a news admin ever configures his server with groupbaseexpiry to
true and *without* overview, he won't be able to expire articles from
his spool, will he? (something maybe worth mentioning!)
This may be helpful, my feeder has no overview and only a CNFS buffer:
$ expire -v 3
Article lines processed 124432
Articles retained 124428
Entries expired 4
My expire.ctl entries (for this test):
*:A:1:90:never
*.test:AX:1:1:1
Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> writes:
When groupbaseexpiry is set to true, expire just does a probe whether the
article is still in the spool.
I would tend to think the 4 entries expired are cancelled articles whose
tokens are removed from the history file. (More than 4 entries would
have expired for all test groups...)
The other common case is articles with Expires headers, which is still sometimes done for the few FAQs that are autoposted.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 300 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 60:17:15 |
| Calls: | 6,712 |
| Files: | 12,243 |
| Messages: | 5,355,756 |