Dynamically setting log handlers

1

We are using Graylog as our central logging application, set up in the zope.ini like so:

[handler_eventlog]
class = ims.graylog.handlers.GELFTCPHandler
args = ('graylogg.imsweb.com', 12201)
kwargs = {}
level = NOTSET
formatter = generic

What we are trying to do is have it write to a file if Graylog is down for some reason (this is a US gov NIST requirement I believe). As a proof of concept I was able to add a file handler on top of the Graylog connection like so:


[handlers]
keys = console, accesslog, eventlog, eventlog2

...

[logger_root]
level = INFO
handlers = console, eventlog, eventlog2

...

[handler_eventlog2]
class = FileHandler
args = (r'[file path]\var\log\event.log', 'a')
kwargs = {}
level = NOTSET
formatter = generic

And then in my handler class I can subclass createSocket to send the devs an email that the connection is down

    def createSocket(self) -> None:
        super().createSocket()
        if not self.sock and not self.emailed:
            self.emailed = 1
            # send email here

But what I'd like to do is only have this write to a file if it can't connect to Graylog for some reason. It's not a huge deal, but it would be nice to not have to worry about log rotation for a log I don't need. I thought maybe I could have take the file handler registration out of zope.ini and register it programmatically on Graylog connection failure. Perhaps by emulating how the handlers are registered from reading the zope.ini - but I don't know where that code is or how to do it?

2

I think it probably makes more sense to implement this fallback within the graylog handler rather than trying to configure multiple handlers. I'm not aware of support in the Python logging framework for doing a fallback like this.

3

I didn't want to put it into the existing Graylog handler I have because then that class would need to have full functionality for both file and socket. What I was looking for is the code that reads zope.ini and actually sets up the loggers. Then I could leave the graylog handler as is, effectively disabled, and just set up a new handler. Maybe that's just straight python, I don't know how it works. Is that not a good idea?

4

It doesn't make sense to me. I think you want the fallback to happen at the time when a log fails to be written, not when the process starts up and handlers are configured. Presumably Graylog is normally working fine when Zope starts but then there might be a connection problem later.

5

Ah, of course that's true. So I guess I will either need to replicate what FileHandler would do or just always log to a file after all.

I think I can do this all in emit. This is only partially tested

    def emit(self, record):
        super().emit(record)
        if not self.sock:
            # we couldn't connect to Graylog, write to backup log file
            backup_file = self.backup_file_location()
            with open(backup_file, 'a') as stream:
                msg = self.format(record)
                stream.write(msg + '\n')
            if not self.emailed:
                # send email here
                self.emailed = True