Why my LINUX cache memory is out (Experienced at EC2)
I was monitoring my EC2 linux instance. I suddenly found my free memory is depleting. I was cluless. With research I got an paper which explains about the linux cache memory. http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html
Memory Cached - Any modern operating system will cache files frequently accessed.
With detail research I found out one of the major I/O operations are log files. I use log4J for logging in the application. Though the log files are rotated, the system spits out huge amount of logging information. The monitoring scripts, scans through these log files in each 5mins to get if any exception has happend. The frequent access of log files and huge log file sizes makes the OS caching these files.
Major loggers of log4J are only in WARN level. However, each request entry is written for monitoring purpose. Then I asked myself why am I writting these logs. The answer was
- For trouble shooting during a fault
- For Monitoring
- As a audit mechanism
Consider system has a fault / exception. This time I require to log the complete execution path as in case of debug to quickly identify the issue and solve it. With warning, it becomes difficult to get closure to actual circumstancecs.
Regarding this I had discussed with sunil before and he was talking about a in memory logging and on need flushing. This is the background of conceiving onelineLog -
A simple on need logging framework to optimize system resources and serving clients faster.
