• expireover issue after move to ovsqlite

    From Jesse Rehmer@21:1/5 to All on Wed Dec 6 22:23:35 2023
    After switching to ovsqlite, I get the following error for one group anytime I run expireover, whether targeting directly or from news.daily, the only output is "can't expire fr.soc.politique":

    $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
    expireover: can't expire fr.soc.politique
    Article lines processed 4119153
    Articles dropped 0
    Overview index dropped 0

    Is there a way to get more output?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Julien_=C3=89LIE?=@21:1/5 to All on Sun Dec 10 10:24:02 2023
    Hi Jesse,

    After switching to ovsqlite, I get the following error for one group anytime I
    run expireover, whether targeting directly or from news.daily, the only output
    is "can't expire fr.soc.politique":

    $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
    expireover: can't expire fr.soc.politique
    Article lines processed 4119153
    Articles dropped 0
    Overview index dropped 0

    Is there a way to get more output?

    There's unfortunately no more output to get, and I do not know why
    you're facing that error only with one newsgroup.

    Is there a matching rule in expire.ctl?
    Nothing weird when you dump its overview?

    ovsqlite-util -g -n fr.soc.politique

    --
    Julien ÉLIE

    « Tout est dans tout, et réciproquement. » (Pierre Dac)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jesse Rehmer@21:1/5 to iulius@nom-de-mon-site.com.invalid on Sun Dec 10 21:19:07 2023
    On Dec 10, 2023 at 3:24:02 AM CST, "Julien ÉLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

    Hi Jesse,

    After switching to ovsqlite, I get the following error for one group anytime I
    run expireover, whether targeting directly or from news.daily, the only output
    is "can't expire fr.soc.politique":

    $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
    expireover: can't expire fr.soc.politique
    Article lines processed 4119153
    Articles dropped 0
    Overview index dropped 0

    Is there a way to get more output?

    There's unfortunately no more output to get, and I do not know why
    you're facing that error only with one newsgroup.

    Is there a matching rule in expire.ctl?
    Nothing weird when you dump its overview?

    ovsqlite-util -g -n fr.soc.politique

    The only matching rule is the default one I have not to expire anything.

    *:A:never:never:never

    I don't need to expire this group, but found it odd that the error was
    produced from the daily expireover, but only for this one group.

    I can dump the contents with ovsqlite-util and, while I didn't examine all 4+ million lines, everything looks normal.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Julien_=C3=89LIE?=@21:1/5 to All on Mon Dec 11 21:50:49 2023
    Hi Jesse,

    $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
    expireover: can't expire fr.soc.politique

    I don't need to expire this group, but found it odd that the error was produced from the daily expireover, but only for this one group.

    I can dump the contents with ovsqlite-util and, while I didn't examine all 4+ million lines, everything looks normal.

    That's indeed very odd, either a bug or a corruption of some data.

    This error appears when the expire_one() function in storage/ovsqlite/ovsqlite.c returns false.

    If you wish and have a bit of time to investigate, is it possible to
    rebuild INN with additional logs at each "return false"?
    For instance, changing:

    if (!write_request())
    return false;

    to:

    if (!write_request()) {
    syswarn("first write_request failed");
    return false;
    }

    and so on...
    There are 20 occurrences.

    In the for loop, you may want to add the processed article number:

    if (!unpack_now(response, &expires, sizeof expires)) {
    syswarn("unpack_now expires failed for %lu", artnum);
    return false;
    }


    Once the culprit is found, additional investigations will have to be
    done, with again additional more precise logs, so it will take more time.
    Tell me if you want to.
    Also, for the changes of the 20 occurrences, do you want to do that
    yourself or should I send you the modified ovsqlite.c file to your
    e-mail address?

    --
    Julien ÉLIE

    « Mon grand-père buvait un litre de whisky par jour et il est mort à 103
    ans. À l'incinération, on a cru que le feu ne s'arrêterait jamais. »
    (Sloppy White)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jesse Rehmer@21:1/5 to iulius@nom-de-mon-site.com.invalid on Mon Dec 11 22:05:22 2023
    On Dec 11, 2023 at 2:50:49 PM CST, "Julien ÉLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

    Hi Jesse,

    $ echo fr.soc.politique | expireover -f - -Z tmp/lowmark
    expireover: can't expire fr.soc.politique

    I don't need to expire this group, but found it odd that the error was
    produced from the daily expireover, but only for this one group.

    I can dump the contents with ovsqlite-util and, while I didn't examine all 4+
    million lines, everything looks normal.

    That's indeed very odd, either a bug or a corruption of some data.

    This error appears when the expire_one() function in storage/ovsqlite/ovsqlite.c returns false.

    If you wish and have a bit of time to investigate, is it possible to
    rebuild INN with additional logs at each "return false"?
    For instance, changing:

    if (!write_request())
    return false;

    to:

    if (!write_request()) {
    syswarn("first write_request failed");
    return false;
    }

    and so on...
    There are 20 occurrences.

    In the for loop, you may want to add the processed article number:

    if (!unpack_now(response, &expires, sizeof expires)) {
    syswarn("unpack_now expires failed for %lu", artnum);
    return false;
    }


    Once the culprit is found, additional investigations will have to be
    done, with again additional more precise logs, so it will take more time. Tell me if you want to.
    Also, for the changes of the 20 occurrences, do you want to do that
    yourself or should I send you the modified ovsqlite.c file to your
    e-mail address?

    I'm always open to troubleshooting, if you can send the modified file so I don't accidentally muck it up, I would appreciate it. I will recompile and provide new output. If it outputs the offending article number I can provide the contents/output from sm, etc.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Julien_=C3=89LIE?=@21:1/5 to All on Sat Dec 16 16:23:16 2023
    Hi all,

    expireover: can't expire fr.soc.politique

    I'm always open to troubleshooting, if you can send the modified file so I don't accidentally muck it up, I would appreciate it.

    After investigation (thanks again Jesse!), the problem comes from an
    article (spam) with an insanely long Subject header field:

    <rvcdut$26a$10@dont-email.me>


    http://al.howardknight.net/?STYPE=msgid&MSGI=%3Crvcdut%2426a%2410%40dont-email.me%3E

    The *headers* of that article have a size of 112 kilobytes, and both
    ovdb and ovsqlite consider overview data of a single article as
    corrupted if it exceeds 100 kb.
    As discussed with Bo Lindbergh, a check is currently missing: also only allowing overview data of that size at addition time. Otherwise,
    overview data of any size is added, and then no longer considered valid
    when accessing it. Only spam or buggy articles are likely to have such
    an insane size for headers, so we won't really miss articles with that
    check.

    --
    Julien ÉLIE

    « Affirmanti incumbit probatio. »

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)