Multiple Processes Logging to ZeroMQ Appender

0
votes

I have multiple processes running on multiple machines using log4j (2.11). I need to consolidate the logging messages to be displayed on the front-end, and would like each process to use a ZeroMQ Appender to publish the log messages to a single connection. I will then have one subscriber receiving the messages, performing the consolidation, and then displaying the log messages.

I have a toy application working with one publisher (process logging); however, when multiple processes try to connect to the same endpoint, I receive the "Address already in use" error message. Which (most likely) means the log4j ZeroMQ (JeroMQ) appender is doing the "binding," since only one process can bind a zmq socket.

Is there a configuration option to have the log4j ZeroMQ Appender perform a "connect" instead of the "bind" or is there another option available to accomplish the same goal.

1
zeromqlog4j2publish-subscribeappenderjeromq
You may be well aware, there is no benefit in using PUB/SUB archetype, unless your infrastructure is willing to operate many-PUB-s + many-SUB-s, as if building an N+M-fault-resilience into your loging service. An ordinary PUSH/PULL would fit better, as there is Zero-need for subscription based TOPIC-filtering in doing plain logging, isn't it? ( BTW, is the 2.11 above referring to the ZeroMQ native API version? Many features now present in native API v4.2+ are not present there, so quite important to be aware of that ) – user3666197
The 2.11 is referring to the version of Log4J I am using (currently the latest) which offers a ZeroMQ (JeroMQ) Appender. Ideally, the subscriber will receive the logging messages without having to poll the other processes. I'll read on the PUSH/PULL architecture to see if that would be a better fit; however, with the PUB/SUB I can receive all of the messages without having to subscribe to each individual topic. So, unless I'm misunderstanding something, the TOPIC-filtering will not be a concern. – Walter

1 Answers

0
votes

There's two possible reasons for "hardware" rejected .bind()-s

One has been already in your post -- under an assumption, that the JeroMQ-services are indeed trying to rigidly .bind() in places, where you would like to .connect() instead. Review the code-base to either confirm that ( with a possible fork/mod circumventing any such discomfort in a refactored service fashion ) or reject this alternative, if the code is not the one to blame.

But let me mention another possible reason, which was quite often a problem during PoC / prototyping.

Early-stage mock-up tools are prone to crashes ( and yes, sometimes crash peacefully, but sometime do crash more times or harder than the PoC team is willing to live with ).

If the code-base is not ( yet ) well designed during this phase to clearly handle also the resources management ( proper use of explicit aSocket.setsockopt( LINGER, 0 ) ( a lethal must in native API versions pre-4.x ) {aSocket|aMessage}.close() and Context.term() methods ) that does a clean exit even after a crash, there are situations, that your code has left a still active ( not dismantled ) one or more Context()-instances, which ( still ) block the hardware resources, as it were not allowed to release them yet.

This has sometimes led to a need to reboot a platform, as a deadlocked Context()-instance has no other way to get "vacuum-cleaned" ( shame on those, who do not fuse/protect the code for surviving till a final stage in try: except: finally: triade, to enforce a graceful termination and clean release of all allocated resource ).