2 posts tagged “logging”

Every piece of code uses logging to tell the outside world what is happening inside - some verbose, some limited with their use of words. Nevertheless they talk all the time in a 1-way communication. It is as if they are making up for their creator's absence to tell about the glory and gory details of their existence.

Logging has to be done for 2 things:

   1. Provide feedback about its performance so it can be tuned till it is most efficient
   2. Call loud and clear when in need of help

This has to be done with minimal overhead to application performance or system resources.

As a side note log messages for tracing like "entered helloworld method" have no place in the final production executable code. They have to be ripped out of the code before it is checked in.

Feedback about performance

This data is primarily for analysis and the best place to store such data is in a structured way - something like a database. Data in a database is ready for analysis - why should there be log parsers to cull out such data from log files into a database or excel? Why not just log it in the right place? I can hear "a database write for a log record - that's so non-performant" and it contradicts the statement of "logging to be done with minimal overhead". But are database inserts that expensive? Yes if they are done one insert after another but when they are done in bulk - remember? databases are designed for set processing - it is not an overhead on the system. Obviously this should be done on a separate thread and in bulk. Our results on database performance and comparison of onelineJdbc with iBatis and Hibernate is here. I have done it in a previous application and it gives comfort to know that system is operating at 40% capacity.

Call For Help

An application should call for help when needed and never else. When it does it should be loud and clear with all details and how things can be fixed or corrected.  Assume  the code is decent enough not to swallow exceptions and making sure it has got all information before providing any service. In such a code any exception or error starts and ends with a single transaction. In that case all information regarding a transaction and its progress can be collected in memory. In case an error happens all this information is written to the log file otherwise it is discarded. At this time the application can also record any measures that can be taken for recovery if these are anticipated situations. Support team will thank you for these little considerations.

We are developing onelineLog to meet these 2 requirements, will announce when it is ready.

Btw have you seen a system designed from a support team's perspective? Let's talk about it in the next post. Until then good reading.

Below are links to some blogs from Hani Suleiman (of the bileblog fame) on logging. Not the best language for public consumption - you have been warned.

http://www.bileblog.org/?p=22
http://www.bileblog.org/?p=119
http://www.bileblog.org/?p=52

Software Architecture For Developing Countries

I was monitoring my EC2 linux instance. I suddenly found my free memory is depleting. I was cluless. With research I got an paper which explains about the linux cache memory. http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html 

    Memory Cached - Any modern operating system will cache files frequently accessed.

With detail research I found out one of the major I/O operations are log files. I use log4J for logging in the application. Though the log files are rotated, the system spits out huge amount of logging information. The monitoring scripts, scans through these log files in each 5mins to get if any exception has happend. The frequent access of log files and huge log file sizes makes the OS caching these files.

Major loggers of log4J are only in WARN level. However, each request entry is written for monitoring purpose. Then I asked myself why am I writting these logs. The answer was

1. For trouble shooting during a fault
2. For Monitoring
3. As a audit mechanism

Consider system has a fault / exception. This time I require to log the complete execution path as in case of debug to quickly identify the issue and solve it. With warning, it becomes difficult to get closure to actual circumstancecs.

Regarding this I had discussed with sunil before and he was talking about a in memory logging and on need flushing. This is the background of conceiving onelineLog -

A simple on need logging framework to optimize system resources and serving clients faster.

Software Architecture For Developing Countries