2 posts tagged “memory”

Software Architecture For Developing Countries

Most of the web applications work in a front controller pattern. I am going to base my discussion around this front controller.

What is the situation now

1. Do you know the "Path and Goals" of people in your application?
   The layout of paths will seem right and comfortable only when it is compatible with the process of walking. And the process of walking is far more subtle than one might imagine. - Christopher Alexander?
   
2. Do you know the request residence time at the server? Do your know the java application server memory state (Free, Used) for each request?
   
3. Do you know what people are using frequently, humans use 20% things 80% time. Are you aware of that 20%?
   
4. Do you know how many people you are serving now (Being served right now by the server), not the amount of sessions opened ?
   
5. Incase of http POST request, do you know the request meta data (Actions)
   

A little intimacy test!!  How much you scored?  Consider you know all these are available to you.

Now I am going to write how to boost your intimacy with your application. I increased my awareness of the application by implementing following access points in the application.

For each request, I recorded the start time when it enters the front controller. I also increases the no. of guests by one.

Long startTime = new Long(System.currentTimeMillis());
guest++;

When the request finishes the processing,

Recorded the endtime, got the diff, calculated max memory, free memory and total memory. Wrote down the detail about the request. Who accessed the system and what request, when....

StringBuffer strBuf =

new StringBuffer("

"

);
strBuf.append("<time>").append(new Datetime()).append("<time>");
strBuf.append("<residencetime>").append(diff).append("</residencetime>");
strBuf.append("<maxmem>").append(maxMem).append("</maxmem>");
strBuf.append("<totmem>").append(totalMem).append("</totmem>");
strBuf.append("<freemem>").append(freeMem).append("</freemem>");
strBuf.append("<guests>").append(guest).append("</guests>");
strBuf.append("</from>").append(req.getRemoteHost()).append("<from>");
.....
guest--;

Now you Introspect the data and provide what your application need..

 Drawing a path of walk with defined milestones for the remote host and the action ... It gives picture how people are moving?  In which milestone they flock around? When they come out of their houses?

99% communications are non-verbal in nature, Don't wait people to give you a feedback. They are telling you all time. Just be present to it. Be aware of it :)

 

Software Architecture For Developing Countries

Software Architecture For Developing Countries

I believed in one thing all these years.. It's better to take a vaccine rather than a Quinine. It's better to prevent than fix. A wellness check - A continous effort to get 0 complain from the application as well as server. A regular health checkup is what I am talking about.

I devide this checkup to five parts

• The machine - Memory, CPU, IO and Network
• The Application servers - Apache, Tomcat, My SQL, Mail ...
• The application - Log files to find slow performance, exceptions and other warnings
• The supporting agents - Are they doing their work regularly: Database backup, Email agents, SMS Agents and other SOA agents
• Heartbit checkup

In addition to this, some servicing is required like my car to make it running with no issues:

Monitoring - All servers are listening

email="support@XXX.com"

#Monitor the listening ports
 echo Monitoring the listening ports
 listeningports=`netstat -l -n -p -t -u -w | sed 's/ [ ]*/ / g' | cut -d' ' -f4`
 ports="8005 8009 3306 8080 80 22 25"
 
 for port in $ports;
 do
  isListening=`echo $listeningports | grep $port | wc -l`
  if [ $isListening -ne 1 ]; then
   echo "server is not listening in the port $port"
   netstat -l -n -p -t -u -w > emailbody
   mail -s "Application Servers are not lisening - $port " $email < emailbody
   cat emailbody;  rm -rf emailbody
  fi
 done

The sda1 primary disk is around 1.8GB only for the low end instances. So it is extermely necessary to check it's usage. I have here put a limit of 512MB. Once it goes down than this... It sends a mail to me.

#Monitor the primary disk usage

email=support@XXX.com 

echo Monitor the primary disk usage
 sda1=`df | grep "sda1" | sed 's/ [ ]*/ /g' | cut -d' ' -f4`
 if [ $sda1 -lt 512000 ]; then
  df -H > emailbody
  mail -s "Server diskspace is low in sda1 - $sda1 " $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

The sda2 Memory is huge. So this limit could be more. But we got to monitor this around less than 5GB free, let me know.

If my server is swapping, it's not a good news. It's important to investigate why is it swapping. So the first step to know that it swapped. Next is, an exploration to find out why it happend?

#SWAP monitoring
 echo Swap monitoring
 swap=`echo $stat | cut -d' ' -f3`
 if [ $swap -gt 0 ]; then
  vmstat -S M > emailbody
  mail -s "Server is swapping $swap" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

Memory monitoring is one more big aspect. Any time the server free memory is going down 256MB that means, thre is some gottcha going on. The box need my attention. Well, for the java applications, the memory might have been boxed at some upper limit. We will use other tricks to find out that too and flatten it.

#MEMORY monitoring
 echo memory monitoring
 mem=`echo $stat | cut -d' ' -f4`
 if [ $mem -lt 256 ]; then
  free > emailbody
  mail -s "Server is in low memory - $mem" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

How much weight lifing the CPU is doing. Are there spikes and cpu usage is surging up? If this is the case, those points need to be found out and removed or intentionally allowed to stay there.

#CPU monitoring
 echo CPU monitoring
 cpu=`echo $stat | cut -d' ' -f15`
 if [ $cpu -lt 30 ]; then
  top -n 1 -b > emailbody
  mail -s "Server cpu is busy  - $cpu" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

Tomcat write in catalina.out. Scanning the complete file is not an good idea. So a mechanism of incremental scanning needs to be there. The biggest question is what to look for in this file. If the stdout is not there, nothing will be written here. Then along with this, the output log file needs to be inspected. To answer, the question what am I looking for; that the developers can only say. I look for traces of Exception, Error or Warning. Let's do the incremental scanning:

# Check the /mnt/logs/catalina.out
 echo catalina.out monitoring
 logFile=/mnt/logs/catalina.out
 scanFile=/mnt/logs/catalina.out.scan
 lastline=0
 
 if [ -f $scanFile ]; then
         lastline=`cat $scanFile`
 fi
 newlastline=`wc -l $logFile | cut -d' ' -f1`
 
 if [ $lastline -gt $newlastline ]; then
         incremental=$newlastline
 else
         incremental=`expr $newlastline - $lastline`
 fi
 
 errors=`tail -n $incremental $logFile |  egrep "Exception|WARN|FATAL"`
 isErrors=${#errors}
 if [ $isErrors -gt 0 ]; then
  echo $errors > emailbody
  mail -s "Catalina Error,Fatal or warn." $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi
 echo $newlastline > $scanFile

Apache access file is the final moment of truth. Did the request went ahead propely. When I started putting a monitoring mechanism, I looked for 4** and 5** response codes. To my strange, there were many erroneous warnings. The erroneous warnings really dilute the importance of the system. So there should be continous effort to receive 0 warning from the system.

• I was receiving warning of favicon.ico
• When I released my new version, I found internet crawlers are looking for the old URL and are reporting page not found error.

# Check the Apache access file
 echo access.log monitoring
 logFile=/mnt/logs/apache.access.log
 scanFile=/mnt/logs/apache.access.log.scan
 lastline=0
 
 if [ -f $scanFile ]; then
         lastline=`cat $scanFile`
 fi
 newlastline=`wc -l $logFile | cut -d' ' -f1`
 
 if [ $lastline -gt $newlastline ]; then
         incremental=$newlastline
 else
         incremental=`expr $newlastline - $lastline`
 fi
 
 errors=`tail -n $incremental $logFile |  grep -v favicon.ico | sed 's/ [ ]*/ /g' | cut -d' ' -f9 | egrep "4..|5.."`
 isErrors=${#errors}
 if [ $isErrors -gt 0 ]; then
  tail -n $incremental $logFile | grep -v favicon.ico | egrep " 4.. | 5.. " > emailbody
  mail -s "Apache access Error." $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi
 echo $newlastline > $scanFile

Well, in my immediate next post; I will talk about the backup strategies. This is getting too big. So read on.

Software Architecture For Developing Countries