6 posts tagged “amazon”

I did read about this a while back when the challenges using EC2 was how not lose a database in case of a machine failure. Luckily it never happened and forgot all about EBS, the persistent disk for Amazon EC2. We went live with a new customer over the weekend and EBS came up all over again for data backup. Quite simple to use it.

Recently my EC2 instance was hung. The website was getting timed out. A clear signal that the web server is not responding and all socket communication is not happening. As the sockets were not reactive, so the ssh server. I was not able to login using the SSH terminal - PUTTY.

Now I just can't shut down that instance and start another one as I had some production data in mysql data files which had not backed up. I had told my customer 99% uptime, 256MBps connectivity but what about reliability.. Now they are going to lose production data. Literally, I was shocked. I desparetly tried to get my data back. Did googling and got depressed reading the comments of pool. We then put a urgent request in the forum and amazon replied that nothing is possible.

After I cooled down a bit, it was time to fight back. First thing I thought of trying was restarting the instance. However, I was not sure about my data. All my mysql data files were in /mnt. So the question was am I going to loose data if I restart my instance. Some links I referred confused me more; I left with a understanding that anything at MNT has the possibility of loss. But the mnt was my target. Now I took a decision to restart my server - The last resort.

I issued restart command from my desktop. Oops... there was no effect. Then I went to my another amazon instance. There I had the EC2 tools. I used those tools and issued a restart command. Did a ssh again. The server was still hung.  I sat tight for 5 mins. Again connected via my putty. Bingo, this time the server shown me the login prompt. My curiosity doubled up.. Am I going to have my data in /mnt. Then I changed directory to /mnt. Everything was there intact. I just relaxed. Thank God - I  survived a crash 

Software Architecture For Developing Countries

I was monitoring my EC2 linux instance. I suddenly found my free memory is depleting. I was cluless. With research I got an paper which explains about the linux cache memory. http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html 

    Memory Cached - Any modern operating system will cache files frequently accessed.

With detail research I found out one of the major I/O operations are log files. I use log4J for logging in the application. Though the log files are rotated, the system spits out huge amount of logging information. The monitoring scripts, scans through these log files in each 5mins to get if any exception has happend. The frequent access of log files and huge log file sizes makes the OS caching these files.

Major loggers of log4J are only in WARN level. However, each request entry is written for monitoring purpose. Then I asked myself why am I writting these logs. The answer was

1. For trouble shooting during a fault
2. For Monitoring
3. As a audit mechanism

Consider system has a fault / exception. This time I require to log the complete execution path as in case of debug to quickly identify the issue and solve it. With warning, it becomes difficult to get closure to actual circumstancecs.

Regarding this I had discussed with sunil before and he was talking about a in memory logging and on need flushing. This is the background of conceiving onelineLog -

A simple on need logging framework to optimize system resources and serving clients faster.

Software Architecture For Developing Countries

Software Architecture For Developing Countries

The definition of availability:

Present and ready for use; at hand; accessible:
Capable of being gotten; obtainable
Qualified and willing to be of service or assistance

Unavailability is not felt till the moment users reach out for the service and are not served. For me the counting of unavailability starts the moment absence of availability felt.

My application is a web based application. The potential of unavailability is bacause of:

• Absence of the desktop machines.
• Absence of network connection.
• Unable to serve due to fault on:
1. Website not available
2. Application servers are down or hung
3. Slow performance and high response time
4. Software bug and functionality is not available.

To make a high available application, the system requires:

1. Mobile presence
2. Smart client applications, provision to browse at client side during no network.
3. The website is up all the time.All machines and mobile devices are functioning properly.
4. All serves are functioning properly - Apache, tomcat, mysql, mail
5. All service agents are functioning properly - sms, email

During,

1. Single node failure
2. High load situation
3. New application version deployment
4. Regular machine patch management
5. Hacking attacks, intrusion attacks
6. Virus Attack
7. External applications unavailability

and immediately serving after a fault due to,

• Database failure
• System/node unavailability
• Application bugs

OK, the principles I am guided to provide high availability in my application are:

During Requirement

Availability for me is a functional requirement. All the requirements are analyzed from this perspective too. If system couldn't be available from world wide web, can he perform some operations from his mobile device via SMS, can he send a mail and do the same operation.

During Architecture & Design

• Stateless architecture on Amazon EC2 to quickly provide more servers as the number of users increase.
• Central point of information update and numerious points of information viewing to scale. In my application around 80% time information is viewed and 20% time it get's modified.
• Each component handles failure situation. Ex. The email agent if could not send mail using account X, it will try another email provider.

During execution - production environment

On production - During available period:

1. Preparing Stand by machine to take over during failure
2. Monitoring the health and providing necessary support.
3. Continuous revision of server deployment according to the measured data: 
      Staep -1 )   Moving database and other single point of failure to an dedicated machine.
      Step - 2)  Up scaling te machine by adding more memory, cpu, disk and other resources in the existing machine
      Step - 3) Hot backup and replication of all changed content.

On production - During Unavailable period

1. Browser always only refer to one url. This is the first single point of failure. To avoid this, we have a javascript at our yahoo hosting site. Application first loads the infromation from here. This failover is provided by yahoo. Now the web application stack maintains the list of one primary and another secondary server for processing at amazon instances. In case of access failure to primary, the browser switches to the secondary servers.
2. As the apache and tomcat servers are stateless. Once the server goes down, immediately a new instance is started by the clustered server.
3. Once the master database goes down, immediate replica takes over and chooses it's own back up node. It passes the next backup to all the clients when they connect to master for sync up. Master node pushes information to all the subscribed nodes. All updates, deletes, inserts happen in master node. Child nodes serve the lookup(select) queries.
4. Once a database replica goes down or a new instance is created, it fist takes yesterday backup and then replicates today's transaction querying from master node.

The Preparedness

1. Failover drill process. It forces the system to shut down and waits for the failover system to take care.

Software Architecture For Developing Countries

Software Architecture For Developing Countries

I believed in one thing all these years.. It's better to take a vaccine rather than a Quinine. It's better to prevent than fix. A wellness check - A continous effort to get 0 complain from the application as well as server. A regular health checkup is what I am talking about.

I devide this checkup to five parts

• The machine - Memory, CPU, IO and Network
• The Application servers - Apache, Tomcat, My SQL, Mail ...
• The application - Log files to find slow performance, exceptions and other warnings
• The supporting agents - Are they doing their work regularly: Database backup, Email agents, SMS Agents and other SOA agents
• Heartbit checkup

In addition to this, some servicing is required like my car to make it running with no issues:

Monitoring - All servers are listening

email="support@XXX.com"

#Monitor the listening ports
 echo Monitoring the listening ports
 listeningports=`netstat -l -n -p -t -u -w | sed 's/ [ ]*/ / g' | cut -d' ' -f4`
 ports="8005 8009 3306 8080 80 22 25"
 
 for port in $ports;
 do
  isListening=`echo $listeningports | grep $port | wc -l`
  if [ $isListening -ne 1 ]; then
   echo "server is not listening in the port $port"
   netstat -l -n -p -t -u -w > emailbody
   mail -s "Application Servers are not lisening - $port " $email < emailbody
   cat emailbody;  rm -rf emailbody
  fi
 done

The sda1 primary disk is around 1.8GB only for the low end instances. So it is extermely necessary to check it's usage. I have here put a limit of 512MB. Once it goes down than this... It sends a mail to me.

#Monitor the primary disk usage

email=support@XXX.com 

echo Monitor the primary disk usage
 sda1=`df | grep "sda1" | sed 's/ [ ]*/ /g' | cut -d' ' -f4`
 if [ $sda1 -lt 512000 ]; then
  df -H > emailbody
  mail -s "Server diskspace is low in sda1 - $sda1 " $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

The sda2 Memory is huge. So this limit could be more. But we got to monitor this around less than 5GB free, let me know.

If my server is swapping, it's not a good news. It's important to investigate why is it swapping. So the first step to know that it swapped. Next is, an exploration to find out why it happend?

#SWAP monitoring
 echo Swap monitoring
 swap=`echo $stat | cut -d' ' -f3`
 if [ $swap -gt 0 ]; then
  vmstat -S M > emailbody
  mail -s "Server is swapping $swap" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

Memory monitoring is one more big aspect. Any time the server free memory is going down 256MB that means, thre is some gottcha going on. The box need my attention. Well, for the java applications, the memory might have been boxed at some upper limit. We will use other tricks to find out that too and flatten it.

#MEMORY monitoring
 echo memory monitoring
 mem=`echo $stat | cut -d' ' -f4`
 if [ $mem -lt 256 ]; then
  free > emailbody
  mail -s "Server is in low memory - $mem" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

How much weight lifing the CPU is doing. Are there spikes and cpu usage is surging up? If this is the case, those points need to be found out and removed or intentionally allowed to stay there.

#CPU monitoring
 echo CPU monitoring
 cpu=`echo $stat | cut -d' ' -f15`
 if [ $cpu -lt 30 ]; then
  top -n 1 -b > emailbody
  mail -s "Server cpu is busy  - $cpu" $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi

Tomcat write in catalina.out. Scanning the complete file is not an good idea. So a mechanism of incremental scanning needs to be there. The biggest question is what to look for in this file. If the stdout is not there, nothing will be written here. Then along with this, the output log file needs to be inspected. To answer, the question what am I looking for; that the developers can only say. I look for traces of Exception, Error or Warning. Let's do the incremental scanning:

# Check the /mnt/logs/catalina.out
 echo catalina.out monitoring
 logFile=/mnt/logs/catalina.out
 scanFile=/mnt/logs/catalina.out.scan
 lastline=0
 
 if [ -f $scanFile ]; then
         lastline=`cat $scanFile`
 fi
 newlastline=`wc -l $logFile | cut -d' ' -f1`
 
 if [ $lastline -gt $newlastline ]; then
         incremental=$newlastline
 else
         incremental=`expr $newlastline - $lastline`
 fi
 
 errors=`tail -n $incremental $logFile |  egrep "Exception|WARN|FATAL"`
 isErrors=${#errors}
 if [ $isErrors -gt 0 ]; then
  echo $errors > emailbody
  mail -s "Catalina Error,Fatal or warn." $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi
 echo $newlastline > $scanFile

Apache access file is the final moment of truth. Did the request went ahead propely. When I started putting a monitoring mechanism, I looked for 4** and 5** response codes. To my strange, there were many erroneous warnings. The erroneous warnings really dilute the importance of the system. So there should be continous effort to receive 0 warning from the system.

• I was receiving warning of favicon.ico
• When I released my new version, I found internet crawlers are looking for the old URL and are reporting page not found error.

# Check the Apache access file
 echo access.log monitoring
 logFile=/mnt/logs/apache.access.log
 scanFile=/mnt/logs/apache.access.log.scan
 lastline=0
 
 if [ -f $scanFile ]; then
         lastline=`cat $scanFile`
 fi
 newlastline=`wc -l $logFile | cut -d' ' -f1`
 
 if [ $lastline -gt $newlastline ]; then
         incremental=$newlastline
 else
         incremental=`expr $newlastline - $lastline`
 fi
 
 errors=`tail -n $incremental $logFile |  grep -v favicon.ico | sed 's/ [ ]*/ /g' | cut -d' ' -f9 | egrep "4..|5.."`
 isErrors=${#errors}
 if [ $isErrors -gt 0 ]; then
  tail -n $incremental $logFile | grep -v favicon.ico | egrep " 4.. | 5.. " > emailbody
  mail -s "Apache access Error." $email < emailbody
  cat emailbody;  rm -rf emailbody
 fi
 echo $newlastline > $scanFile

Well, in my immediate next post; I will talk about the backup strategies. This is getting too big. So read on.

Software Architecture For Developing Countries