FileBeats vs elastic logger

[cyril94440]

Hi there,
Thanks for this great library.

I've read the section about filebeat/logstash VS this library, but there are certain points that I would love to be clarified.

I understand the point made about retrieving enough info, but how does it compare with using a son-handler with filebeat?
I mean what is the impact of this library at runtime? Will the python script wait for all calls to elastic search to be done before exiting?

To summarise, I am wondering how these two approaches impact the current python running script, and if there is a significant difference or it's basically the same. (Maybe Filebeat doesn't fire straight away the request and do it in batch, ...)

Sorry for all these questions, I hope this will help others make an educated decision too.

Thanks

[cmanaha]

hmmm, I'm not entirely sure what a "son-handler" is in filebeat any reference you can pass would help for comparison and I'd gladly comment.

Sorry for the long response;

TL;DR : IMHO, There are two completely different approaches. If unsure and you are not coding, use filebeats/logstaash, If you want your application/config to carry everything it needs to instrument itself think about using this or a similar module.

The long answer: both options are quite different in the approach; What do you think is more suited for your system: External Agent vs internal module ?

To me, they are complementary and both are suited for different scenarios; IMO the differences go from "how to get things deployed" to "what is the code path of my app" or to what is the information you'd like to get out from the system.

Aside from that, in the case of this library vs filebeats, there is another big difference in regards of the resources and people behind the projects ;) . elastic has a commercial support offering for filebeat worth considering.

So let's start with the good things about agents: They are not intrusive, don't have an impact on the application (orthogonal); the person setting up the agent doesn't need an understanding or knowledge of the app (I might be wrong, but the fact that you are asking yourself the questions above, probably means that you should pick up the agent approach).

Then some of the cons : Agents like filebeats/logstash require you to deploy a package and extra configuration. Then manage application deployment and version, the extra configuration making sure it's in sync with your application. Also, logging systems (on all languages) are more flexible than people give them credit for; they can take dictionary of parameters and log them according to pre-configured formats; This extra functionality is difficult to keep in sync when using agents. That's one of the strengths of handlers/APM's.

Architecture wise agents can end up in a infrastructure horror story. I've seen enterprise systems with more than 20 agents running on the basic OS image, fighting for system resources; Inventory discovery agents, OS Instrumentation agents, app specific instrumentation agents, and you could keep going to the point that when you pile agents up and put yourself on top, you are close to the stratosphere, with not much oxygen to breath.

Same with handlers, the good things first: They are part of your application, you move your application, and with it goes along the ability to get your logs still delivered to a central system. As stated logging systems provide functionality to pass arbitrary dictionary, this can be used to instrument applications:

log.debug("DB operation took %.3f seconds" % db_delta, extra={'db_execution_time': db_delta})

Calls like the one above in your code define a clear intent in your code and in our case implicitely create an entry directly on elastic so it's easy to create moving dashboards of how long it's taking to query a DB. There is no extra grok or processing. The developer doesn't need to be aware that if it change the log message all the dashboards will break and ops will have to re-code all grok expressions to capture the new value.

This approach is quite powerful combined with the ability to create python decorators (or aspect oriented programming, filters, pre-processors, etc in other languages). Extending an instrumentation system this way takes you to a technique traditionally known as "Application Performance Monitoring" systems (APM for abbreviation).

There are lots of cases following this approach (Appdynamics, Dynatrace, Elastic APM, yes! elastic has one of this too!) In fact look at the product here APM Django and if you are curious to spot the differences you can compare their log handler approach in the APM module with the one in this project

The cons: This approach is intrusive; You have to be careful with what does your program execute; What are the resources it consumes, the security that it gets, what is the impact to your code path and so on and so forth.

The other limitation to consider is the scope of the integration; This module does only fire logs to elastic, it does not push over kafka, redis and is not intended to be generic, however there are other modules for that, and redis too. All those just require to plug into whichever the system you need to integrate with and they mean maintaining more dependencies associated with your program (with additional complexity, dependency hell chances, etc)

I close with your question on "what's the impact of running this module" :

• You can define how much memory buffers up your messages as well as how often the messages are push on to elastic. buffer_size: An int, Once this size is reached on the internal buffer results are flushed into ES. flush_frequency_in_sec: A float representing how often and when the buffer will be flushed (regardless of if the buffer is full or not)

• We've tried to avoid blocking as much as we can and in the testing we run the performance impact is very low. As for your question of Will the python script wait for all calls to elastic search to be done before exiting? , if I'm not wrong line 208 here states self._timer.setDaemon(True), which means that the program will terminate where it was interrupted (more on daemon threads here). It will be easy to make this optional, so people can decide what they want at configuration time, I'll add that to the next iteration.

Regards,
Carlos.

[cyril94440]

Awesome Carlos, very detailed explanation! Thank you very much.
I'll definitely go with the library so that I don't have to bother with FileBeat install/config.
(Sorry for the typo, I meant json-handler which logs detailed json-formatted log to a file).

Yes I guess being able to set the thead as a non-daemon would be great. It would avoid making a script blindly sleep at the end to make sure the very last logs are properly sent to ES.

Cheers,
Cyril.