It astounds me how hard it is to write good logging for software.
Like, as an example, I had the opportunity to work on software to integrate with a fairly complex system of microservices. This microservice system has no logging. Like, at all.
Here is, roughly, what it logs:
Information - Message queue listener is running.
Information - Message queue listener is running.
Information - Message queue listener is running.
Information - Message queue listener is running.
Information - Message queue listener is running.
Warning - Received message under topic A that contains payload A
Information - Message queue listener is running.
Information - Message queue listener is running.
Information - Message queue listener is running.
Information - Message queue listener is running.
It's unusable! It is, to be frank, godawful. I cannot in all honesty believe that whoever built it thought they were doing a good job. It upsets me every time I see it.
Because, how in the world am I supposed to track what was done with that message? Oh, it received it? What did it do with that message, hmm?
And yet, when I tried to upgrade the logging on the software I have full ownership over, it was really, surprisingly hard! It took like four days to:
- Identify the existing logging.
- Identify all areas that should be logged.
- Refactor a massive amount of the code, because as it stood it was virtually impossible to log effectively.
- Set up logging in this massive (1000+ line) background service's constructor.
- Verify the logging.
- Set up logging for all the code that handles incoming messages.
- Verify that logging.
It was nightmarish! And it's so much better now!
What did I learn?
This advice mostly applies to .NET.
Use an up-to-date, effective logging library!
I cannot stress this enough! Do not roll your own logging! Do not Console.WriteLine()! Use a tried and true, battle-tested logging system!
I personally recommend NLog as I work with .NET in my day-to-day, but there are a ton of good options! Please, for the love of god, pick a logging library and learn it!
Log EVERYTHING!
Seriously, this sounds dumb, but listen. There exists a logging level called Trace. This is meant for low-level, excessive logging that allows you to trace down an issue.
When you are setting up logging, log everything you can think of under this log level. Was a step in a process skipped due to a missing value? Log it! Was a database write completed? Log it! Want to verify a background service is running? Log it!
Under the Trace log level!
Then, bring stuff up to Debug, Information, Warning, Error, etcetera as needed. Now, you have fully formed logging. Congrats!
Have an analytics platform for your logging!
I personally like Loki and Grafana, but it's up to you. Just make sure that you do not have to parse through several layers of files to find your detailed logging. When you log a lot of stuff, it tends to bloat your log files up.
Log with Structured Logging!
Seriously! This is huge! There are human-readable logs, and machine-readable logs. Structured logging allows you to look up an id, and find all logs associated with that id. Want to check when a certain post was updated on a blog site? Search the logs for that post's slug! It's truly incredible!
I had no idea this was a thing until halfway through my big logging rewrite. God, could you imagine if I had just implemented it without that and found out later? I would've been so mad!
Postscript
If your software has bad logging, you are the worst. Please fix it. You're only screwing yourself.
Thanks for reading.