High availability at EC2

Software Architecture For Developing Countries

The definition of availability:

Present and ready for use; at hand; accessible:
Capable of being gotten; obtainable
Qualified and willing to be of service or assistance

Unavailability is not felt till the moment users reach out for the service and are not served. For me the counting of unavailability starts the moment absence of availability felt.

My application is a web based application. The potential of unavailability is bacause of:

• Absence of the desktop machines.
• Absence of network connection.
• Unable to serve due to fault on:
1. Website not available
2. Application servers are down or hung
3. Slow performance and high response time
4. Software bug and functionality is not available.

To make a high available application, the system requires:

1. Mobile presence
2. Smart client applications, provision to browse at client side during no network.
3. The website is up all the time.All machines and mobile devices are functioning properly.
4. All serves are functioning properly - Apache, tomcat, mysql, mail
5. All service agents are functioning properly - sms, email

During,

1. Single node failure
2. High load situation
3. New application version deployment
4. Regular machine patch management
5. Hacking attacks, intrusion attacks
6. Virus Attack
7. External applications unavailability

and immediately serving after a fault due to,

• Database failure
• System/node unavailability
• Application bugs

OK, the principles I am guided to provide high availability in my application are:

During Requirement

Availability for me is a functional requirement. All the requirements are analyzed from this perspective too. If system couldn't be available from world wide web, can he perform some operations from his mobile device via SMS, can he send a mail and do the same operation.

During Architecture & Design

• Stateless architecture on Amazon EC2 to quickly provide more servers as the number of users increase.
• Central point of information update and numerious points of information viewing to scale. In my application around 80% time information is viewed and 20% time it get's modified.
• Each component handles failure situation. Ex. The email agent if could not send mail using account X, it will try another email provider.

During execution - production environment

On production - During available period:

1. Preparing Stand by machine to take over during failure
2. Monitoring the health and providing necessary support.
3. Continuous revision of server deployment according to the measured data: 
      Staep -1 )   Moving database and other single point of failure to an dedicated machine.
      Step - 2)  Up scaling te machine by adding more memory, cpu, disk and other resources in the existing machine
      Step - 3) Hot backup and replication of all changed content.

On production - During Unavailable period

1. Browser always only refer to one url. This is the first single point of failure. To avoid this, we have a javascript at our yahoo hosting site. Application first loads the infromation from here. This failover is provided by yahoo. Now the web application stack maintains the list of one primary and another secondary server for processing at amazon instances. In case of access failure to primary, the browser switches to the secondary servers.
2. As the apache and tomcat servers are stateless. Once the server goes down, immediately a new instance is started by the clustered server.
3. Once the master database goes down, immediate replica takes over and chooses it's own back up node. It passes the next backup to all the clients when they connect to master for sync up. Master node pushes information to all the subscribed nodes. All updates, deletes, inserts happen in master node. Child nodes serve the lookup(select) queries.
4. Once a database replica goes down or a new instance is created, it fist takes yesterday backup and then replicates today's transaction querying from master node.

The Preparedness

1. Failover drill process. It forces the system to shut down and waits for the failover system to take care.

Software Architecture For Developing Countries