This morning I added nocem-fr@alphanet.ch's key to the keyring and bit to the nocem.ctl file. I tried to reprocess some of the recent NoCeMs but found that
a good portion of them don't do anything. I'm not sure if this is something
I'm misunderstanding, or if the notices aren't being generated properly, or if there is an issue with perl-nocem?

For example, when I process this one cancels are generated as expected:

grephistory '<nocem.ALPHANET.fr.20230902123509.18620.584@nocem.alphanet.ch>' | perl-nocem

However, many other NoCeMs from the same issuer do nothing:

grephistory '<nocem.ALPHANET.fr.20230902045542.24754.171@nocem.alphanet.ch>' | perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125503.23507.509@nocem.alphanet.ch>' | perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125530.23877.24@nocem.alphanet.ch>' | perl-nocem

The only thing I see in the logs related to those is the following:

nocem[25391]: starting up

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 2, 2023 at 7:12:41 AM CDT, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:

This morning I added nocem-fr@alphanet.ch's key to the keyring and bit to the nocem.ctl file. I tried to reprocess some of the recent NoCeMs but found that a good portion of them don't do anything. I'm not sure if this is something I'm misunderstanding, or if the notices aren't being generated properly, or if
there is an issue with perl-nocem?

For example, when I process this one cancels are generated as expected:

grephistory '<nocem.ALPHANET.fr.20230902123509.18620.584@nocem.alphanet.ch>' |
perl-nocem

However, many other NoCeMs from the same issuer do nothing:

grephistory '<nocem.ALPHANET.fr.20230902045542.24754.171@nocem.alphanet.ch>' |
perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125503.23507.509@nocem.alphanet.ch>' |
perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125530.23877.24@nocem.alphanet.ch>' | perl-nocem

The only thing I see in the logs related to those is the following:

nocem[25391]: starting up

Determined this is because the issuer uses 'Types' that are not "spam", so I had to adjust my nocem.ctl to include those types:

nocem-fr@alphanet.ch:spam,cleanfeed-emp,cleanfeed-too-many-newsgroups

Next question, is there an easier way for me to process the notices I have in news.lists.filters without grabbing Message-IDs one-by-one?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 2, 2023 at 7:28:10 AM CDT, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:

On Sep 2, 2023 at 7:12:41 AM CDT, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:

This morning I added nocem-fr@alphanet.ch's key to the keyring and bit to the
nocem.ctl file. I tried to reprocess some of the recent NoCeMs but found that
a good portion of them don't do anything. I'm not sure if this is something >> I'm misunderstanding, or if the notices aren't being generated properly, or if
there is an issue with perl-nocem?

For example, when I process this one cancels are generated as expected:

grephistory '<nocem.ALPHANET.fr.20230902123509.18620.584@nocem.alphanet.ch>' |
perl-nocem

However, many other NoCeMs from the same issuer do nothing:

grephistory '<nocem.ALPHANET.fr.20230902045542.24754.171@nocem.alphanet.ch>' |
perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125503.23507.509@nocem.alphanet.ch>' |
perl-nocem
grephistory '<nocem.ALPHANET.fr.20230902125530.23877.24@nocem.alphanet.ch>' |
perl-nocem

The only thing I see in the logs related to those is the following:

nocem[25391]: starting up

Determined this is because the issuer uses 'Types' that are not "spam", so I had to adjust my nocem.ctl to include those types:

nocem-fr@alphanet.ch:spam,cleanfeed-emp,cleanfeed-too-many-newsgroups

Next question, is there an easier way for me to process the notices I have in news.lists.filters without grabbing Message-IDs one-by-one?

Also, shouldn't perl-nocem be logging when it receives a notice for which one does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in my logs even though I was receiving messages for which were not in my keyring.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 02/09/2023 14:47, Jesse Rehmer wrote:

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

This is a script I use when I want to re-run the notice again:

http://paganini.bofh.team/ru_nocem_again.pl

Just adapt them to your system

Also, shouldn't perl-nocem be logging when it receives a notice for which one does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in my logs even though I was receiving messages for which were not in my keyring.

Almost prob. you have same my problem, the "debug" doent's work. Just
edit your perl-nocem script, and change all log "debug" with "err" or
"notice", and u find all on news.err or news.notice. Reload the config
on inn with cltinnd reload


Have fun.


Sincerely

--
Ivo Gandolfo

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 2, 2023 at 8:31:05 AM CDT, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:

On Sep 2, 2023 at 8:14:32 AM CDT, "Ivo Gandolfo" <usenet@bofh.team> wrote:

On 02/09/2023 14:47, Jesse Rehmer wrote:

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

This is a script I use when I want to re-run the notice again:

http://paganini.bofh.team/ru_nocem_again.pl

Just adapt them to your system

This will work, thank you!

Also, shouldn't perl-nocem be logging when it receives a notice for which one
does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in
my logs even though I was receiving messages for which were not in my keyring.

Almost prob. you have same my problem, the "debug" doent's work. Just
edit your perl-nocem script, and change all log "debug" with "err" or
"notice", and u find all on news.err or news.notice. Reload the config
on inn with cltinnd reload

I see what you mean about some logging set to 'debug', but this portion of the
code that should log the one I mention doesn't specify a level:
<SNIP>

I see they do end up in debug, if I add news.debug to my syslog configuration
I do see the messages in that log.

When I run your script I notice a lot of occurrences of this error from perl-nocem, but things seem to be working.

Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 44.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 45.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 46.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 2, 2023 at 8:14:32 AM CDT, "Ivo Gandolfo" <usenet@bofh.team> wrote:

On 02/09/2023 14:47, Jesse Rehmer wrote:

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

This is a script I use when I want to re-run the notice again:

http://paganini.bofh.team/ru_nocem_again.pl

Just adapt them to your system

This will work, thank you!

Also, shouldn't perl-nocem be logging when it receives a notice for which one
does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in
my logs even though I was receiving messages for which were not in my keyring.

Almost prob. you have same my problem, the "debug" doent's work. Just
edit your perl-nocem script, and change all log "debug" with "err" or "notice", and u find all on news.err or news.notice. Reload the config
on inn with cltinnd reload

I see what you mean about some logging set to 'debug', but this portion of the code that should log the one I mention doesn't specify a level:

if (/^\[GNUPG:\]\s+GOODSIG\s+\S+\s+(.*)/m) {
return 1 if $1 =~ /\Q$issuer\E/;
logmsg("Article $msgid: signed by $1 instead of $issuer");
} elsif (/^\[GNUPG:\]\s+NO_PUBKEY\s+(\S+)/m) {
logmsg("Article $msgid: $issuer (ID $1) not in keyring");
} elsif (/^\[GNUPG:\]\s+BADSIG\s+\S+\s+(.*)/m) {
logmsg("Article $msgid: bad signature from $1");
} elsif (/^\[GNUPG:\]\s+BADARMOR/m or /^\[GNUPG:\]\s+UNEXPECTED/m) {
logmsg("Article $msgid: malformed signature");
} elsif (/^\[GNUPG:\]\s+ERRSIG\s+(\S+)/m) {
# safety net: we get there if we don't know about some token
logmsg("Article $msgid: unknown error (ID $1)");
} else {
# some other error we don't know about happened.
# 126 is returned by the child if exec fails.
s/ at \S+ line \d+\.\n$//;
s/\n/_/;
if ($INN::Config::gpg) {
logmsg(
"Article $msgid: $INN::Config::gpg exited "
. (($status == 126) ? "($_)" : "with status $status"),
'err'
);
} else {
logmsg(
"Article $msgid: $INN::Config::gpgv exited "
. (($status == 126) ? "($_)" : "with status $status"),
'err'
);
}
}
return 0;
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 2, 2023 at 9:18:58 AM CDT, "Ivo Gandolfo" <usenet@bofh.team> wrote:

On 02/09/2023 15:47, Jesse Rehmer wrote:
[cut]

Just add where you want the log(, "err") for example:

if (/^\[GNUPG:\]\s+GOODSIG\s+\S+\s+(.*)/m) {
return 1 if $1 =~ /\Q$issuer\E/;
logmsg("Article $msgid: signed by $1 instead of $issuer",
"notice");
} elsif (/^\[GNUPG:\]\s+NO_PUBKEY\s+(\S+)/m) {
logmsg("Article $msgid: $issuer (ID $1) not in keyring", "err");

etc :)

When I run your script I notice a lot of occurrences of this error from
perl-nocem, but things seem to be working.

Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line >> 183, <ART> line 44.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line >> 183, <ART> line 45.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line >> 183, <ART> line 46.

You can safe ignore them.

Sincerely

Thank you, Ivo!

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 02/09/2023 15:47, Jesse Rehmer wrote:
[cut]

Just add where you want the log(, "err") for example:

if (/^\[GNUPG:\]\s+GOODSIG\s+\S+\s+(.*)/m) {
return 1 if $1 =~ /\Q$issuer\E/;
logmsg("Article $msgid: signed by $1 instead of $issuer",
"notice");
} elsif (/^\[GNUPG:\]\s+NO_PUBKEY\s+(\S+)/m) {
logmsg("Article $msgid: $issuer (ID $1) not in keyring", "err");

etc :)

When I run your script I notice a lot of occurrences of this error from perl-nocem, but things seem to be working.

Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 44.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 45.
Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 46.

You can safe ignore them.


Sincerely

--
Ivo Gandolfo

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jesse,

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

Besides Ivo's script working for the tradspool storage method, there is
also a straight-forward way to do that depending on the overview method
you're using.

With ovsqlite, you obtain all these storage tokens with:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' '

See tdx-util and ovdb_stat for a similar dump with tradindexed and ovdb.
I can add a paragraph about that in the perl-nocem documentation as it
is an interesting use case to document.

Use of uninitialized value $key in lc at /usr/local/news/bin/perl-nocem line 183, <ART> line 44.

Noted. I'll fix that warning in the next release.

Also, shouldn't perl-nocem be logging when it receives a notice for which one
does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in
my logs even though I was receiving messages for which were not in my keyring.

The "not in keyring" log should really be in your news.notice file as
the default log level for the logmsg() function in perl-nocem is notice:
syslog($lvl || 'notice', '%s', $msg);

Are you sure the nocem-fr@alphanet.ch entry was in nocem.ctl when you
did that test?
I think the log you expected was another one, for a issuer not present
in nocem.ctl:

logmsg(
"Article $msgid: unwanted ($hdrs{issuer}/$hdrs{type})",
'debug'
);

In that case, yes, this log is forced at the debug level. I believe the rationale is for not repeating every couple of minutes that you don't
process the notices or types from one or several issuers. It's not
necessary when you already know that.
Yet, I understand it may be useful to be aware of new issuers or a new
type sent by a known issuer.
Should this very log be changed to be at the notice level? I'm OK with
that move unless you see any drawback. It seems more helpful for admins.

--
Julien ÉLIE

« On ne peut jamais être neutre. Le silence est une opinion. » (Henri
Moret)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 3, 2023 at 4:36:15 PM CDT, "Julien ÉLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Jesse,

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

Besides Ivo's script working for the tradspool storage method, there is
also a straight-forward way to do that depending on the overview method you're using.

With ovsqlite, you obtain all these storage tokens with:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' '

See tdx-util and ovdb_stat for a similar dump with tradindexed and ovdb.
I can add a paragraph about that in the perl-nocem documentation as it
is an interesting use case to document.

If it isn't overly complicated to provide an example for tdx-util that would
be helpful to me and I'm sure others. I'll get around to figuring it out at some point. I'm currently using tradspool so Ivo's script works fine for now.

Also, shouldn't perl-nocem be logging when it receives a notice for which one
does not have a key?

I see the logging in the perl-nocem code, but I never get "not in keyring" in
my logs even though I was receiving messages for which were not in my keyring.

The "not in keyring" log should really be in your news.notice file as
the default log level for the logmsg() function in perl-nocem is notice:
syslog($lvl || 'notice', '%s', $msg);

Are you sure the nocem-fr@alphanet.ch entry was in nocem.ctl when you
did that test?
I think the log you expected was another one, for a issuer not present
in nocem.ctl:

logmsg(
"Article $msgid: unwanted ($hdrs{issuer}/$hdrs{type})",
'debug'
);

In that case, yes, this log is forced at the debug level. I believe the rationale is for not repeating every couple of minutes that you don't
process the notices or types from one or several issuers. It's not
necessary when you already know that.
Yet, I understand it may be useful to be aware of new issuers or a new
type sent by a known issuer.
Should this very log be changed to be at the notice level? I'm OK with
that move unless you see any drawback. It seems more helpful for admins.

You are correct, what I was looking for then would have been related to
missing issuer in nocem.ctl. I don't think changing it would spam logs too much?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 05/10/2023 17:52, Jesse Rehmer wrote:

You are correct, what I was looking for then would have been related to missing issuer in nocem.ctl. I don't think changing it would spam logs too much?

I have areally modified, and this is a 30 day result (I log everything,
and passed 30 day's the old log was moved to a backup server for history
and legal reason):

root@paganini:/var/log# du -hs news/
123M news/
root@paganini:/var/log#


Sincerely

--
Ivo Gandolfo

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jesse,

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

Besides Ivo's script working for the tradspool storage method, there is
also a straight-forward way to do that depending on the overview method
you're using.

With ovsqlite, you obtain all these storage tokens with:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' '

See tdx-util and ovdb_stat for a similar dump with tradindexed and ovdb.
I can add a paragraph about that in the perl-nocem documentation as it
is an interesting use case to document.

If it isn't overly complicated to provide an example for tdx-util that would be helpful to me and I'm sure others.

Sure. As for tdx-util, the storage token is the 6th field in the output:

tdx-util -g -n news.lists.filters | cut -f 6 -d' '

Piping that into perl-nocem will re-process all the notices in this
newsgroup.


With the Berkeley DB database, it is less straight-forward as the
ovdb_stat program works differently. We have to retrieve the overview
data ("-r 1-" for all the articles, or "-r low-high" for a specific
range of articles), find the Message-ID in the 5th field, and convert
them to storage tokens.

ovdb_stat -r 1- trigofacile.test | cut -f 5 | grephistory -s



I'll add these commands in the perl-nocem documentation as they seem to
prove useful.

--
Julien ÉLIE

« Le vrai danger, ce n'est pas quand les ordinateurs penseront comme les
hommes, c'est quand les hommes penseront comme les ordinateurs. »

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Adam W. <gof-cut-this-news@cut-this-chmurka.net.invalid> wrote:

for f in $(cat spool/overview/n/l/f/news.lists.filters.DAT |awk -F'\t' '{print $5}' ); do bin/grephistory $f | bin/perl-nocem; done

Sorry, old version. It's better this way (runs perl-nocem only once,
not for every message):

for f in $(cat spool/overview/n/l/f/news.lists.filters.DAT |awk -F'\t' '{print $5}' ); do bin/grephistory $f; done | bin/perl-nocem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

Next question, is there an easier way for me to process the notices I have in news.lists.filters without grabbing Message-IDs one-by-one?

I see the date of the post and you probably already solved it (and forgot
about it), but here's my solution. Works for traindexed overview.

for f in $(cat spool/overview/n/l/f/news.lists.filters.DAT |awk -F'\t' '{print $5}' ); do bin/grephistory $f | bin/perl-nocem; done

It will re-feed the full content of news.lists.filters to perl-nocem.
Quite resource intensive.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sep 3, 2023 at 4:36:15 PM CDT, "Julien ÉLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Jesse,

Next question, is there an easier way for me to process the notices I have in
news.lists.filters without grabbing Message-IDs one-by-one?

Besides Ivo's script working for the tradspool storage method, there is
also a straight-forward way to do that depending on the overview method you're using.

With ovsqlite, you obtain all these storage tokens with:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' '

Hi Julien,

Finally got my history and overview rebuilt with ovsqlite...

When I use the command above piped to perl-nocem, innd grinds to a halt. It seems to process a few nocem articles and then everything stops. No articles processed from peers and incoming connections hang, nothing reported in the logs except perl-nocem starting. I don't see anything with truss against innd or perl-nocem, and as soon as I cancel the command everything springs back to life.

If I take the output of the ovqlite-util command to a file and do the
following it seems to run fine:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' ' > list
perl-nocem < list

When I run it like this everything hangs:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' ' | perl-nocem

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jesse,

With ovsqlite, you obtain all these storage tokens with:

ovsqlite-util -g -n news.lists.filters | cut -f 5 -d' '

Finally got my history and overview rebuilt with ovsqlite...

Good news :)

When I use the command above piped to perl-nocem, innd grinds to a halt. It seems to process a few nocem articles and then everything stops. No articles processed from peers and incoming connections hang, nothing reported in the logs except perl-nocem starting. I don't see anything with truss against innd or perl-nocem, and as soon as I cancel the command everything springs back to life.

I also see the same behaviour, indeed, when thousands of articles are processed.
After investigation, perl-nocem hangs at the receival of a response from
innd:

foreach (@$ids) {
print NNTP "$_\r\n";
if (($r = <NNTP>) !~ /^289/) {
$r =~ s/\r\n$//;
logmsg("cannot cancel $_: $r", 'err');
goto ERR;
}
}

<NNTP> waits for more bytes.

Yet, innd has properly seen the article, as enabling traces (ctlinnd
trace innd y) shows at debug level:

Nov 4 11:23:53 news innd[1620966]: localhost:46 NCproc state=15 next "<O9KdnVDIDbin9V"
Nov 4 11:23:53 news innd[1620966]: localhost:46 < <O9KdnVDIDbin9VP_nZ2dnUU7_83NnZ2d@giganews.com>

and innd logs a response only when I stop perl-nocem:

Nov 4 11:24:01 news innd[1620966]: localhost:46 NCwritereply
26=write(46, "289 Article can", 26)
Nov 4 11:24:01 news innd[1620966]: localhost:46 > 289 Article cancelled OK



As you noted, the behaviour is different when running "perl-nocem <
file" instead of piping the ovsqlite-util result into perl-nocem.
There's no hang in that case.

I am unsure what should be fixed there; innd shouldn't really be
indefinitely blocked in reading in the Unix-domain socket opened by
perl-nocem. Seems like after NCwritedone(), the channel is added to be
read (RCHANadd) but nothing is to be read as perl-nocem did not receive
any response for whatever reason.

--
Julien ÉLIE

« There's a way of transferring funds that is even faster than
electronic banking. It's called marriage. » (Sam Kinison)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)