Hi all,

Need help with below query on python logging module.

I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know
why it is so but couldn't get how I can solve it.
Issue was because of serialization in logging when multiple processes are involved.

Below is log_config.py which is used by all other modules to get the logger and log.
import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s") LOG_FILE = "my_app.log"

def get_file_handler():
file_handler = TimedRotatingFileHandler(LOG_FILE, when='midnight')
file_handler.setFormatter(FORMATTER)
return file_handler

def get_logger(logger_name):
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG) # better to have too much log than not enough
logger.addHandler(get_file_handler())
#with this pattern, it's rarely necessary to propagate the error up to parent
logger.propagate = False
return logger

All other modules call, 'logging = log_config.get_logger(name)' and use it to log.
I came to know about QueueHandler and QueueListener but not sure how to use them in my code. How can I use these to serialize logs to single file.?

Any help is appreciated.
Thanks,
Chethan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

--
Lars Liedtke
Software Entwickler


Phone:
Fax: +49 721 98993-
E-mail: lal@solute.de


solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany


Marken der solute GmbH | brands of solute GmbH
billiger.de | Shopping.de


Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 110579
USt-ID | VAT ID: DE234663798


Informationen zum Datenschutz | Information about privacy policy http://solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php

Am 21.06.22 um 11:04 schrieb Chethan Kumar S:

I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to

know why it is so but couldn't get how I can solve it.

Issue

On 22 Jun 2022, at 11:06, Lars Liedtke <lal@solute.de> wrote:

Could be unrelated and only a part of a solution, but if you are on a unixoid system, you could use logrotate, instead of TimedRotatingFileHandler. logfrotate ensures that the logging service does not realize, its logs have been rotated. So it can log

as if nothing has happened.


The process that is writing the file must be told that rotation has happened for it to work.
Other wise all the logs keep being write to the original file via the FD that the process has.

logrotate's config include how to tell the process the log file needs reopening.

Barry

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

The process that is writing the file must be told that rotation has
happened for it to work.
Other wise all the logs keep being write to the original file via the
FD that the process has.

logrotate's config include how to tell the process the log file needs reopening.

Thanks for clearing.

--
Lars Liedtke
Software Entwickler


Phone:
Fax: +49 721 98993-
E-mail: lal@solute.de


solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany


Marken der solute GmbH | brands of solute GmbH
billiger.de | Shopping.de


Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 110579
USt-ID | VAT ID: DE234663798


Informationen zum Datenschutz | Information about privacy policy http://solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Chethan Kumar S wrote at 2022-6-21 02:04 -0700:

...
I have a main process which makes use of different other modules. And these modules also use other modules. I need to log all the logs into single log file. Due to use of TimedRotatingFileHandler, my log behaves differently after midnight. I got to know

why it is so but couldn't get how I can solve it.

Issue was because of serialization in logging when multiple processes are involved.

Below is log_config.py which is used by all other modules to get the logger and log.
import logging
import sys
from logging.handlers import TimedRotatingFileHandler

FORMATTER = logging.Formatter("%(asctime)s — %(name)s — %(message)s")

The usual logging usage pattern is:
the individual components decide what to log
but how the logging happens it decided centrally - common
for all components.
This implies that usually the individual components do not handle
handlers or formatters but use the configuration set up centrally.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-06-21 02:04:52 -0700, Chethan Kumar S wrote:

I have a main process which makes use of different other modules. And
these modules also use other modules. I need to log all the logs into
single log file. Due to use of TimedRotatingFileHandler, my log
behaves differently after midnight. I got to know why it is so but
couldn't get how I can solve it.
Issue was because of serialization in logging when multiple processes

^^^^^^^^^^^^^^^^^^

are involved.

I think this is crucial point. Not "multiple modules" (as you wrote in
the subject), but "multiple processes". If each of multiple processes
wants to rotate the logfile itself, it is very likely that they will
work at cross-purposes.

Somebody already suggested using logrotate (or similar external tools).
You can then use WatchedFileHandler (https://docs.python.org/3/library/logging.handlers.html#watchedfilehandler)
to automatically detect when a logile has been rotated.

Alternatively you can use a central logging service (like syslog) which
handles all that stuff.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmK5jNUACgkQ8g5IURL+ KF1KOhAApoeK0DJh1m5JBvM9SCB7J8Y8GCcfYR4cP0aUYMSDz0zUr5pO9TRQR+Am TK5wN/VWdl8W0QrMA4fB9ck2QHQ/T/9+EZJHB8gbQ/0z0J3KlmRD5Gk9AQlTfYF5 A9/bFdCstNjW+Ezj7zx3EJnU6IPer/ZK1sMQ1Y/9w4B98crw2cs4xU/6eB8O0K2i yzf0utSq5A4ColypzDC+X2qRyqrgg+dLwMbQMD2WGaUeJxz3dVIFEYoCHXtWxYBe OSAflXh84AINAc573wkDv3NOm7p9EMk4mz0Hwu6+2x0vdmh68kardJjvk5P61u2O ckvZgGqOYvc6bHWn6cX4lCDESJteE6YI99D2hNvL8GnBx4ONqDuWtgg+iBK46egh eQn3KqYnY8nOaddq3qxZXaTKnK0lAOI3UkrwRq1m8FOwvUt7/ZTg0BU6tiZxfAZZ EAcx7GG1Q+7Vo4GVghFVNisYniEnxmvzzbMDFdHZg/5hBtz0xyoC/45u9kYNeU0w Djl726gpXeL9C64WcsFUp3VTuOlcTcQzYXqo0KCftf9omm11MCE89iOSvcQFM89V pbyTG08A6Dfba9UHRPDHHGA7SXtapA1ecCxvDHBIpdikgK7EEc8XbLJO7CM6Vr5/ WCMIKYwPl2rSyAtuaNRGIzptbg2BDKwmVnMF5dN