3 posts tagged “replication”

View other items tagged “replication:” Photos, Audio, Videos, Books or Links. View “replication” on Vox.

Scalability & Failover on Amazon EC2

Interesting aspect to tackle after putting applications on EC2 in a live production environment. Our first customer has been live for over 1 year on EC2 and never been down for a single time. We didn't have to reboot the linux server though we had to restart our java processes for upgrades. Touchwood. Database incremental backups happen every minute to S3 and a daily back up at night. This is mostly an enterprise app, so not very stringent requirements on real time failover. Shortly we are going live with a new retail customer facing application - a 24 x 7 uptime requirement. Scalability is not much of an issue as our server is still operating under 65MB with response times under 50ms most of the time. Failover is a key aspect for us - passive or active, active being the preferred option.


Scalr seems to be decent option (and the only one I could find) for EC2 environments. Server side clustering which also spawns new instances as load increases. Terracota has an offering on EC2 environment where the cache serves as the intermediate database for all practical purposes.

In our solution there is no http session or cache to hold data, every read and write operate on the database. There is only a small portion of read only configuration data that is cached in the server. So java server operates in a stateless mode - meaning a browser client has no affinity to a server. If there are 3 servers then client can utilize all 3 servers for its requests. So client itself can take of clustering and failover. When client first comes to server thru a url server loads the client and gives all server urls to it. Client discovers an available server out of this and sticks to it till server is either down or too loaded. Client switches to another server at that point from its list. No need for a gatekeeper on server.

Failover is largely needed at database layer. As long as database is replicated realtime (master-master replication) failover from a database perspective will be seamless. We are exploring solutions for this:

1) mysql clustering. There are some trials of this but none seem to be in a production EC2 setup.
2) C-JDBC or Sequoia: JDBC level clustering, seems the most transparent way. Have to check if JGroups etc can talk to EC2 servers only visible on http
3) HA-JDBC: Similar to C-JDBC.

Will check out in the next couple of weeks and post the results.

Post a comment Tags: mysql, database, ec2, clustering, failover, replication

Immortal application in a mortal server..

I remember few years back when I was running the Websphere, it was very difficult to kill a instance. On killing of one instance, a deamon was spawning another. Killing that one, one more was coming up. I had tried "kill -9", "kill -KILL" - No help. When one was going down, other was coming up. It is not like, I have 2 process instances running and on killing of both, the application  is completely down. Rather, it is only one and taking birth each time it is murdered.

Why not this happens for a server. If I consider that is possible, virtually the application is immortal. 

[I have taken this section out from my regular writting as I want to highlight on one particular issue. Recently I had mapped a production instance to amazon EC2 instance in web DNS mapping. I issued a shutdown command on EC2 machine by mistake. It just took me 2 mins to bring up a snapshot of the last instance as I was taking the snapshot. Then I pointed to this new server address in web DNS mapping. This process took a good 30mins and the system remained down for so long :(]

Consider a machinsm is available, where web DNS mapping is instantaneous to the depployed server. In this spectrum, I have an application which transcends through server instances. The possibilities lies on the snapshot creation. Before death can I take an exact snapshot and all next instances starts from there. Amazon EC2 has infiniband networking to S3. So, network latency is not my concern. Hadoop is there for my rescue to transfer the big files. The big file is the tar ball of the complete mnt. If you are reading my previous posts, you know you can shove everyting to mnt. That means the complete harddisk portion of my interest to the S3. This helps on saving the database files, other contents, application state and everything else.

During shutdown,

  1. Create a snapshot tar ball of /mnt of the portion of harddisk of my interest. - 30sec
  2. Copy this to S3.
  3. Start another instance.
  4. When the next instance comes up, it self configures itself during start up taking the file from S3.

I have taken snapshot and created a new instance quickly but manually. It takes just a minute. How can I do it automatically?

First I looked for linux shutdown hooks like init. No luck. If you are aware of any, please let me know. Now I am going to explore by creating a blank java application and starting it during init and putting a shutdown hook there. This shutdown hook will grap the linux shutdown. I am concluding this reading the post at http://forum.java.sun.com/thread.jspa?threadID=504889 . Once I finish this experiement, I blog about my findings :) This will open up following possibilities in EC2:

  1. Immortal application in a mortal server
  2. No information loss when instance shutted down.
  3. Many app instance creation when load increases. [What about the database??? I have a big blog coming on this. I am half way through]

Don't forget to help me on implementing a linux hook of shutdown rather than going via the java way...  A work around not a clean solution :(

 

 

 

Post a comment Tags: database, ec2, replication

Database Replication

Databases replication is a really useful feature. It provides backup, load balancing, data aggregation to start with. A typical design of a database replication will be:

  • A data modification statement on the master database is written to the database log
  • These entries are read by a replication agent and put on a queue for child nodes to apply on their respective databases
  • A checkpoint thread scans the master node's database log and clears the entries which are read by the replication agent.


What happens when a child node goes down?

  • Entries off the queue (where the master rep agent puts in entries) are not removed by the child node so the queue starts building up
  • When the queue is full the rep agent stops reading entries off the master database log. If it read where will it put them?
  • The master database transaction log starts to get full and when it is full it stops accepting inbound transactions means master database becomes unavailable


In most of the cases the child comes up much before this happens and the child has all the information it has not consumed on the queue. So the queue is a important piece in the design. The master keeps it up and running, filling it with data while the child node is down. In essence the master node is responsible for the queue and hence takes on the risks associated with it. I don't think that is a very wise thing to do, you can think of any number of real world scenarios where this model does not work.

In the replication component Abinash is coding as I write this, we decided that each child node should take responsibility for its upkeep. it has to figure out how to recover from a failure and should not adversely affect the master in any way. We will publish the design details in a follow-up post.

Post a comment Tags: database, replication