2 posts tagged “failover”

View other items tagged “failover:” Photos, Audio, Videos, Books or Links. View “failover” on Vox.

Scalability & Failover on Amazon EC2

Interesting aspect to tackle after putting applications on EC2 in a live production environment. Our first customer has been live for over 1 year on EC2 and never been down for a single time. We didn't have to reboot the linux server though we had to restart our java processes for upgrades. Touchwood. Database incremental backups happen every minute to S3 and a daily back up at night. This is mostly an enterprise app, so not very stringent requirements on real time failover. Shortly we are going live with a new retail customer facing application - a 24 x 7 uptime requirement. Scalability is not much of an issue as our server is still operating under 65MB with response times under 50ms most of the time. Failover is a key aspect for us - passive or active, active being the preferred option.


Scalr seems to be decent option (and the only one I could find) for EC2 environments. Server side clustering which also spawns new instances as load increases. Terracota has an offering on EC2 environment where the cache serves as the intermediate database for all practical purposes.

In our solution there is no http session or cache to hold data, every read and write operate on the database. There is only a small portion of read only configuration data that is cached in the server. So java server operates in a stateless mode - meaning a browser client has no affinity to a server. If there are 3 servers then client can utilize all 3 servers for its requests. So client itself can take of clustering and failover. When client first comes to server thru a url server loads the client and gives all server urls to it. Client discovers an available server out of this and sticks to it till server is either down or too loaded. Client switches to another server at that point from its list. No need for a gatekeeper on server.

Failover is largely needed at database layer. As long as database is replicated realtime (master-master replication) failover from a database perspective will be seamless. We are exploring solutions for this:

1) mysql clustering. There are some trials of this but none seem to be in a production EC2 setup.
2) C-JDBC or Sequoia: JDBC level clustering, seems the most transparent way. Have to check if JGroups etc can talk to EC2 servers only visible on http
3) HA-JDBC: Similar to C-JDBC.

Will check out in the next couple of weeks and post the results.

Post a comment Tags: mysql, database, ec2, clustering, failover, replication

Cricket site architecture

We advocate stateless architecture as an architecture for its numerous benefits. When combined with cloud computing from Amazon and failover/load balancing features built into the client it offers a very efficient way to handle sporadic heavy loads. Take cricket sites such as cricinfo, cricketnext or stock information sites such as moneycontrol which experience heavy loads during certain times on certain days. In all these cases a server has to push information constantly to the client.

Amazon gives the ability to start additional server instances on the fly. These can be started / shutdown as the load increases. Essentially an an infinite data center at a cost of Rs4 per hour per server for a 2Gig, 2CPU instance. As clients don't have affinity to the server any server can service any client. Who will handle the load balancing? Normally it is a reverse proxy sitting at the server end, sort of a gatekeeper. This can get overwhelmed if a million connections come in. We propose to do load balancing on the client. A client is given a bunch of server urls to connect to on its first request. Client tries out alternate servers till it finds an available server. A server can decide to service a request based on its load.

There is one more aspect in this scenario. A browser client has to make requests to the server to get up-to-date information. This creates a significant overhead in connection management on the server. Persistent connections can be used to solve this issue - details in a later post.


2 comments Tags: cricket, failover, load balancing, stateless architecture