HACKER Q&A
📣 kirillzubovsky

Best Tools for Log Management?


What is your go to tool for viewing and understanding manually created application logs?

I know there are a few tools for high volume of automated logs (Papertrail, CloudWatch, Kinesis), but I am looking for something that can ingest small amount of debugging logs, and then make it searchable, ideally by time, or order of events.

For example, I would like to log from the server side:

1- Log("Event 1 happened") 2- Log("Event 2 happened") 3 -Log("Event 3 happened")

Then I want to get this information in a list/graph, that shows these flows in much the same way I would see user flow in modern analytics tools.

I guess I could use analytics tools, like Mixpanel, but that seems an overkill.

What would you use?

  👤 backslash_16 Accepted Answer ✓
Really easy and good homegrown logging. Use a structured logging library, this is key because it lets you easily filter on the properties attached to a log event.

Then log those to a database. If you don't have a ton of volume and want to use really reliable tech and be safe(ish) send your logs to a relational DB used only for logging. I have never tried sending to just a separate table within the production database but I know that can be done too.

Create read-only users on that logging DB or table.

Download a GUI client for that DB and you're up and running. You can use SQL to filter, order by timestamp, and create views.

If you still want to stay homegrown it's straightforward to put an API in front of it that powers some dashboards or easy investigation tools. For example, submit correlation/request ID and get back every log event in-order for that, color error logs red and boom - 1st level data visualization for investigations done too.

👤 aleixventa
If you plan to send logs from a mobile app or a javascript frontend app, we created a tool called Bugfender (https://bugfender.com) that will help you. You can use it for free up to 100K logs and then we have paid plans to fit almost any budget. For server logs, there are a lot of tools that will help you. I wouldn't recommend you to create your own service, might look easy at the beginning but it ends up being very complex (this is how we started Bugfender, creating a tool for ourselves but it get more and more complex).
👤 02020202
tech is irrelevant. all you need is to keep the correlation id, and possibly causation id, in the events and you generate the entire graph of user's movement(or whatever is the subject of the event stream).
👤 tstack
Try the Log File Navigator (http://lnav.org). It works directly with log files. Can understand many log formats by default and be customized. The log messages from separate files are collated into a single view where highlighting is also applied. You can even do analyses on the messages by executing SQLite queries (without having to insert the messages into a database).