• Re: [gentoo-user] Bug in run-crons?

    From Rich Freeman@21:1/5 to Warp_7@gmx.de on Sun Dec 12 19:50:01 2021
    On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:

    It uses state files in /var/spool/cron/lastrun/ to know when each interval was last run, so it only runs once per period. But: the age threshold for
    the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?

    My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at the next run, the state file is exactly one hour old. Since this is not old enough for the check, run-crons thinks that the last run is too recent and ignores this period. As a result, each period is only run on every other iteration.

    I don't use this, but I believe there should be an hourly crontab
    entry that deletes the cron.hourly file, which would mean it gets run
    on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
    are run in parallel or serial).

    The overall result is that things will get run at traditional crontab
    times no matter what, but if those are missed and enough time is
    accrued then run-crons will catch this and run the job on the next
    10min cycle, such as a missed daily overnight slot.

    So if you don't have your computer on a 3AM then the job will get run
    a bit more than 24h after the last time it was run. Then if you leave
    it on overnight that night it will run at 3AM despite it not being
    24h, but at that point if you leave it on it will run at 3AM daily
    since it will no longer be missed. The fixed times serve to anchor
    the actual execution times if you do leave the computer on.

    --
    Rich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Frank Steinmetzger@21:1/5 to All on Sun Dec 12 19:30:01 2021
    Hi, it’s me again, :)

    I’m currently improving the cron setup on my Gentoo-powered NAS. I installed zfs-auto-snapshot, which includes scripts for hourly, daily, weekly and
    monthly creation of snapshots.

    The NAS uses fcron for its cron daemon. In order for the scripts to work,
    I enabled the run-crons script from sys-process/cronbase, which runs all scripts in /etc/cron.{hourly,daily,...}. But I have the feeling that there
    is a bug in the script:

    It uses state files in /var/spool/cron/lastrun/ to know when each interval
    was last run, so it only runs once per period. But: the age threshold for
    the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?

    My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at
    the next run, the state file is exactly one hour old. Since this is not old enough for the check, run-crons thinks that the last run is too recent and ignores this period. As a result, each period is only run on every other iteration.

    Any thoughts on that?
    Cheerio.

    --
    Grüße | Greetings | Qapla’
    Please do not share anything from, with or about me on any social network.

    “Today I watched my first porn movie.” – “And?” – “I was so young back then.”

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmG2PaUACgkQizG+tUDU MMoJcQ/9FqCynBVH9PxZ4UqOYmVM0RGgvh3NJIvCfwqAqhj86ECw1MSE8alob3D4 VfBRkJ+hwC6urWUl8fVvl4Vqj4mWLneJpPLGiFT7ZydKsa+Ned/mgH7uI5Q6avrd Vjl7Umq2ONPO9bPdmQl9DTZxDQwvRFNqemqXJnQYy0JXC96ELvLXqnvkDhXkNK0e AmR2xRQKI1tqV7NHD9oJZfoUJV5crl9q+CHwaracgq4ReimXLqroTur1UE8ceH1a wkcWe+JiZkgU2daZt+XZWBQw7gdSTyMt2A8esjLGpK66LIYiHT2SWxx6zwwtRrrA YxT//CHm72O+M5991BeaCwxIYCPaR1udcKQODm5KQMqdA5dzeGI0HYF0Z2I1fH74 Pv/yBC9EAnSfRlSrVsgybhv4LKpSDjO3pKQcSnVZMEUpFFLXG0dci3QOyHcY5oUj M/pTYCc307J25eLNxhEnzO+g4RDprNKS8mqylBgYyNWEhYwL+N+XmJIRr5NNkBDa b/jFCdT8Yh8xY1b3W0pcLeCKnNLEp8tRG1GK7dMMG2eUeCm+ZP5oEedG5BzZM7ky 7gavoNlDn4vBvD2PidunEK3w3cJCGWH7+U1O5db6Gyket05h1R0sOg1xkQELg8Z+ aQSplBXX3T7Me0mdNUoEcJfM6nBVLqxJxxSwS7r3n2v5ZXLo5vU=
  • From Frank Steinmetzger@21:1/5 to All on Sun Dec 12 20:10:02 2021
    Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
    On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:

    It uses state files in /var/spool/cron/lastrun/ to know when each interval was last run, so it only runs once per period. But: the age threshold for the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?

    My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at the next run, the state file is exactly one hour old. Since this is not old enough for the check, run-crons thinks that the last run is too recent and ignores this period. As a result, each period is only run on every other iteration.

    I don't use this, but I believe there should be an hourly crontab
    entry that deletes the cron.hourly file, which would mean it gets run
    on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
    are run in parallel or serial).

    The check that I mentioned above is actually the deletion which you mention: run-crons looks for the state file for the given interval and - if it is old enough - deletes it.

    The part that executes the individual cron scripts is only executed if there
    is no state file. The first thing it then does is to create a new state file.


    In pseudo code:
    1. look for period state file that is older than interval + 5 mins
    found one?
    delete it

    2. Look for period state file
    none found?
    create state file
    execute cron scripts for that interval

    --
    Grüße | Greetings | Qapla’
    Please do not share anything from, with or about me on any social network.

    The boss is a human just like everyone else, he just doesn’t know.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmG2SIMACgkQizG+tUDU MMojqQ//aeTylLFPiaXYLQlwjA1pIQlNQllUf2scUTF9UW+kHL76d5bpVsHSc4rT q87f4DrYvzxuukqzBhSBxV5EQ7H5kVrexX4zlEQhbW9S6Wiz+YjyqM+aLiwqmRpZ KguC3fvL14/cPRl3bMGhGMxCCEBQm1miKMkVLBjVHfbq7zdJDCMCdIa3tknyP8CS aCmAQ/wEX1NEFa5eIxztUzGAhT4ROErvBY2LFysQiJ52hTTz4eyX6EiGr8u1WH// 4SfMedHmMNh6wTve9uaeRI1nKakUiYC/b+IsHYso1N5BAAD/RFSuyy+rQukHOKVF iBD5DZI2e3N5JFGzXCCA9Xsuiazoc/g8Etr+MaDeSE+wT4uzikOURHAl1MYdfksH P9RjdHutSFiRlzqLSQrDz3chmPidMBQPgmWOCtD8JqbfcqgInew9AQr4C4QnEs52 KxDkzr4Ah8ok6Jpv5MG3s7niEvAhC7PHOdNDqzed3tFmxNWLcY+/tPK2u5grojtf pFYioXuZwBodOTUtj8oZu1ljdXQ6VnsLsLqY77Y0gDcULgyWJ9cBBrHIWB4Bbrw+ /ktt3iKSGqTtzbDIEA2J53YQYl1mDV+yM4+YNb+ly4T76kvcamdJtzGdgkrQz5iU 03DpSla9xDUImq6iNwvOewIABAqyGA3g46AaDSbDLnUAR2FgkLA=
    =6tSu
    -----END PGP SIGN
  • From Rich Freeman@21:1/5 to Warp_7@gmx.de on Mon Dec 13 21:20:02 2021
    On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:

    Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
    On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:


    I don't use this, but I believe there should be an hourly crontab
    entry that deletes the cron.hourly file, which would mean it gets run
    on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
    are run in parallel or serial).

    The check that I mentioned above is actually the deletion which you mention: run-crons looks for the state file for the given interval and - if it is old enough - deletes it.

    The check I'm talking about isn't in run-crons at all. It is in
    /etc/crontab. It doesn't look at the age of the file and
    unconditionally deletes it every hour:
    59 * * * * rm -f /var/spool/cron/lastrun/cron.hourly

    --
    Rich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Orlitzky@21:1/5 to Frank Steinmetzger on Mon Dec 13 22:50:02 2021
    On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote:

    For the record: The checks in run-crons that I referred to earlier are actually more for those cases in which the machine was powered off for a while in order to restore cron completeness as early as possible after boot.


    The run-crons quackery has been causing problems since 2004:

    https://bugs.gentoo.org/69777

    One-liners with run-parts (NOT run-crons) are a lot more predictable.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Frank Steinmetzger@21:1/5 to All on Mon Dec 13 22:20:01 2021
    Am Mon, Dec 13, 2021 at 03:18:44PM -0500 schrieb Rich Freeman:
    On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:

    Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
    On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:


    I don't use this, but I believe there should be an hourly crontab
    entry that deletes the cron.hourly file, which would mean it gets run
    on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
    are run in parallel or serial).

    The check that I mentioned above is actually the deletion which you mention:
    run-crons looks for the state file for the given interval and - if it is old
    enough - deletes it.

    The check I'm talking about isn't in run-crons at all. It is in /etc/crontab. It doesn't look at the age of the file and
    unconditionally deletes it every hour:
    59 * * * * rm -f /var/spool/cron/lastrun/cron.hourly

    I had a look at files and docs on the net again. Thus I found exactly those
    rm entries in /etc/crontab, which by itself is not used by fcron. But after
    I understood all the logic behind it, I added them to fcron to be run
    serially before run-crons. Now everything is as I wanted it.

    For the record: The checks in run-crons that I referred to earlier are
    actually more for those cases in which the machine was powered off for a
    while in order to restore cron completeness as early as possible after boot.

    --
    Grüße | Greetings | Qapla’
    Please do not share anything from, with or about me on any social network.

    “Selfies are electronic masturbation.” — Karl Lagerfeld

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmG3uMcACgkQizG+tUDU MMp5ig/9GyOWE9NLVzBC3z1KkHZd7iP7qibEmn+kxo2OAnutLDoxttdb7J3QECSF InthD9PxsA11EzgL9zDaG0SSPTm+6aThacwy8vdmQtmb5v5t6ILD2jqX9nGfTH3l yEROYDVocvmiPtRquLKWXq2fo/P3BzLUhPCLdkDedyseCfasBJ98kM5nbETti7KB N1bhFshBeeqtkycV56pJaiu1Lpew/e1ll+PMK+safPgep8o+CZ5u1xK4BJ0PCc5l 47/AuO35J/xasaYm+50v5uXkwyrNs2mrLOe8Aj51rEOL5sBckm0u30x+T8KZMn9h iLNnPSYJZ0mZt6w/hb6liV4eBrOt521wfQn04E320O29vGmeQKAT+/yRV1lnWNNP AjDWKzIWB8OIim1POPb/qd5Xlz/0VrHsW7kxtcZDssRbm2ev5/nF3G8FECRKAcWo F40toEZXe0uorNWj58WiugGGlB2DGxoMwEWenVbv94I7EOM0BQcsnF36mHLwgXAB 9iZsK3lJnHEnqOeGjbK5fYt3nVkw7WkotHXFzHlGMygU6EYlN0cMx5OULouXfXN6 0DMs5I+LPoUTa15JXE2HSvtAGBXLM6ai7r8Iqu4i8vxCWt2baaF/ruF4ei0dNUBG 1YspiR1jIRVCCZ7KN2vYJaEchxHLWlQwqkQxEm380sjO39jEp8A=
    =igCH
    -----END PGP SIGNATURE--
  • From Frank Steinmetzger@21:1/5 to All on Mon Dec 13 22:50:03 2021
    Am Mon, Dec 13, 2021 at 04:33:32PM -0500 schrieb Michael Orlitzky:
    On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote:

    For the record: The checks in run-crons that I referred to earlier are actually more for those cases in which the machine was powered off for a while in order to restore cron completeness as early as possible after boot.


    The run-crons quackery has been causing problems since 2004:

    https://bugs.gentoo.org/69777

    One-liners with run-parts (NOT run-crons) are a lot more predictable.

    Well I *could* disable run-crons altogether and add entries to fcron’s own crontab which would run those scripts in /etc/cron.{hourly,daily,...}
    instead.

    However, I like predictable times at which those jobs will run. Especially
    if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes months. And when I power it up, it’s for a reason. And that reason usually
    is not a scrub, which—at the current zfs fill level—takes 10½ hours.

    --
    Grüße | Greetings | Qapla’
    Please do not share anything from, with or about me on any social network.

    A preposition is something you should never end a sentence with.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmG3vToACgkQizG+tUDU MMpbnRAAzQSReNV9hLH5g1S6EXA0mKM3wb831IjgMxkzIf62qgKEY51AWymYSxaG YBJPTeOUhux//4iRzvA7IKQlMsqBG6tVVtLKfR3I87nG23ZoYrAUE1TG/MDCJ/pd aWqhZJqwLH33PEAjYbA52tvKfvoHVzoQv04oRasCRRE0bBtoCfLkz4oHQ+ZmVDIW W0/HndFN08zrJ7brVabJr5LSa63d+NDMgJ0dDQuIdsJVs2ymAWgCEOaII+H+CuC/ p0lqGEUlXBo9g6n3U6boTXrTn8PtGlUCH7uxr74B/S2v20mx7lnHp4k58e4zWsKw dxyReTSH5ZY3R1qbzCCaL4hOONKWFmLn0oAH6Pbco5spZKXwHCJluTGWg5fO8c86 Q7Bndif3kZahwqU6BTiKa8ur/wF1SOmD6uGfWsswbIG+q2oIs8S+n/69xbr1PQrj 953ZOSN2HjGUhBwBDiz8ewog24t5EuaptEQ5BWyCtnmtO3nBdiAGYU2S0Z5lwfxD wFqeecv7ux0xDWvJGqCgMn/lOwDKA8BzrQrOoQBHtRz4NxtAXbZS2UJTbOuUGvxl 0oILCu5FK4g6YCRNOgWiIkH4iyzD5TD0kl/BzC/I5BwZm5SqEzdRdSizWQLyoaFv gsZQlmFZQzIC2FEzqMtPH9xGIMKCnP07Vp7qJyA+rLL9J37eqCw=
    =x8t2
    -----END PGP SIGNATUR
  • From Rich Freeman@21:1/5 to mjo@gentoo.org on Mon Dec 13 23:00:01 2021
    On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky <mjo@gentoo.org> wrote:

    On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:

    Well I *could* disable run-crons altogether and add entries to fcron’s own
    crontab which would run those scripts in /etc/cron.{hourly,daily,...} instead.

    However, I like predictable times at which those jobs will run. Especially if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes months. And when I power it up, it’s for a reason. And that reason usually
    is not a scrub, which—at the current zfs fill level—takes 10½ hours.


    Why choose fcron then? It sounds like you have the same rationale as I
    do: "no, I don't want to run the 4am backup job in the middle of the
    business day just because it wasn't run at 4am."

    If you pick a dumber cron, the crontab entries are run only at the
    specified times.

    There really are no one-size-fits all solutions obviously. You can
    either have predictable times and if the machine is down things get
    skipped, or you can run opportunistically in which case you lose
    predictable times at least until the machine goes back to running
    24x7. The machine has no way to know if you'll leave it on overnight
    the NEXT night - only if you did so the one before.

    fcron is perfectly capable of running jobs at either set times or at loosely-defined intervals that are compensated for if the machine is
    off. Systemd timers can also be set that way, and can also have
    limits put on scheduling (daily nominally at 3AM but allowed between
    11PM and 5AM or whatever). Obviously the more expressive the crontab equivalent is, the more you can tweak it to do what you want it to.

    The default configs are a one-size-fits all approach that tend to do
    the right thing if you leave the machine on 24x7, or if you run it at
    random times. If you want to specify exact behavior just edit the
    config files and make it do so.

    I'll also note that zfs scrubs checkpoint at shutdown and will
    auto-resume where you leave off. If they are involved multiple times
    with the default options I think any attempt to scrub something that
    is already being scrubbed is just a no-op. Obviously if you don't
    want all that IO during the day you'll have to do something more
    clever - you can instruct it to pause and resume so you could have a
    couple of crontab entries and scripts to do just that.

    --
    Rich

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Frank Steinmetzger@21:1/5 to All on Mon Dec 13 23:10:03 2021
    Am Mon, Dec 13, 2021 at 04:54:30PM -0500 schrieb Rich Freeman:
    On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky <mjo@gentoo.org> wrote:

    On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:

    Well I *could* disable run-crons altogether and add entries to fcron’s own
    crontab which would run those scripts in /etc/cron.{hourly,daily,...} instead.

    However, I like predictable times at which those jobs will run. Especially
    if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
    months. And when I power it up, it’s for a reason. And that reason usually
    is not a scrub, which—at the current zfs fill level—takes 10½ hours.


    Why choose fcron then? It sounds like you have the same rationale as I
    do: "no, I don't want to run the 4am backup job in the middle of the business day just because it wasn't run at 4am."

    Fair point.
    Fcron has a serialisation feature, so that for instance updatedb and mandb don’t run at the same time. Me kinda liky.

    fcron is perfectly capable of running jobs at either set times or at loosely-defined intervals that are compensated for if the machine is
    off.

    Indeed, I can say "@monthly * 0 *" to run a monthly job (like the scrub)
    only at night. *pondering*

    Systemd timers can also be set that way, and can also have
    limits put on scheduling (daily nominally at 3AM but allowed between
    11PM and 5AM or whatever). Obviously the more expressive the crontab equivalent is, the more you can tweak it to do what you want it to.

    I’ve gotten used to systemd on my Arch-based desktops. But the NAS is still on good-ol’ openrc. :)

    I'll also note that zfs scrubs checkpoint at shutdown and will
    auto-resume where you leave off.

    That’s a nice bit of knowledge.

    If they are involved multiple times
    with the default options I think any attempt to scrub something that
    is already being scrubbed is just a no-op. Obviously if you don't
    want all that IO during the day you'll have to do something more
    clever - you can instruct it to pause and resume so you could have a
    couple of crontab entries and scripts to do just that.

    Or, since I am the only user of that system, I could go back to my previous way: just run the scrub manually every other month or so before I go to bed. Because then I know that nothing will interfere with it and it won’t interfere with anything else.

    --
    Grüße | Greetings | Qapla’
    Please do not share anything from, with or about me on any social network.

    This message was written using only recycled electrons.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmG3wx4ACgkQizG+tUDU MMq1PA//aPfJ9mUZDRNwbNd2CMiXa+rQQy/PN6Gqr+TwahqIawC4OpIIykuaIRJH Xea9I9eVZz3u2ZXTKO+DU7Zw/574Nl1H+mky6Wi9q6SICmyhhjTWWJPfdgBtuFvV pDAUysTLFr/rHre7DmLFKNuKgPy0m7FqHq6l+QknLtePM8tiAXc5EmKNpJ6G4NWK EVBdq6m3bnAlZ5TZDRis583u/xeojLhsR7UAQR0XAfPLlownkiN9FhP3smhAnLov 9fPhYK7dHBj9VTmUXWJW6Y5eQP/id5JQfWzDgSwzzucK11RMH0pVVp6FgcrEOZIh je1xeHXhzcQsoHkzV7p/XmEUVelPFYC5Q6Qt1bDWmgJ6xJMm1LxUq8jinCOo32dd kBzh5Eav/WXE9B7UIiCSvxCtvMtyMOvUasO3Y33yKGJs+JU3HMtoIvRUgbfn1iM0 OPAVB0WKZ2L0LFmbrfJT8HUOmopQnpIyt0TPgOhWfNM3oZz25FZUkNQ5t3WV3r3Z ILBfH2G1H/kQ91te1t1I4nHAtUtSVlGjT4f+KOEIclNAtnpLXC8ddtUDyrf0brHy WlHPAhS6DMIWL6WMPeJuWTkqoW7Solxr3smMW6ZToj6b4e4viTHxWNOPddwtobNv f/pLu49bu8VP+hCM2gaFP5y8nhqIkS7X0aGNNsx26jgpPjaRTpI=
    =1HK3
    -----END PGP SIGNATURE-----
  • From Wols Lists@21:1/5 to Frank Steinmetzger on Mon Dec 13 23:30:01 2021
    On 13/12/2021 22:03, Frank Steinmetzger wrote:
    If they are involved multiple times
    with the default options I think any attempt to scrub something that
    is already being scrubbed is just a no-op. Obviously if you don't
    want all that IO during the day you'll have to do something more
    clever - you can instruct it to pause and resume so you could have a
    couple of crontab entries and scripts to do just that.

    Or, since I am the only user of that system, I could go back to my previous way: just run the scrub manually every other month or so before I go to bed. Because then I know that nothing will interfere with it and it won’t interfere with anything else.

    Yup. I was thinking that. The problem with scrubs et al auto-resuming or
    firing at boot is that disk i/o is then knackered for ages, and
    interferes with whatever it is you're actually trying to do :-)

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)