5 posts tagged “database”

Interesting aspect to tackle after putting applications on EC2 in a live production environment. Our first customer has been live for over 1 year on EC2 and never been down for a single time. We didn't have to reboot the linux server though we had to restart our java processes for upgrades. Touchwood. Database incremental backups happen every minute to S3 and a daily back up at night. This is mostly an enterprise app, so not very stringent requirements on real time failover. Shortly we are going live with a new retail customer facing application - a 24 x 7 uptime requirement. Scalability is not much of an issue as our server is still operating under 65MB with response times under 50ms most of the time. Failover is a key aspect for us - passive or active, active being the preferred option.

The last post and this one I am writing the ideas that went in our products. If you need details on the product please visit drapsa site.

Any BI product needs a good data store to power the queries on a large dataset. Some of the work is done prior in way of building the cubes. Any adhoc queries or aggregation of data is done on the fly. Sybase and other database vendors including a couple of startups have proposed column oriented databases as a better model for the datastore as cubes aggregate data on a couple of columns at a time. Very compelling and results are there to see in live production environments.

In our design we made use of a search engine as the underlying datastore. Search engine index is optimized column wise like column oriented databases. Databases have largely grown from managing a small set of data very effectively with transaction semantics and ACID rules. As the dataset grew in BI scenarios more and more hardware and software designs have been employed to tackle it - example disk partitioning both physical and logical etc. Search engines on the other hand started with a paradigm of managing a large set of data primarily for read operations. No burden of supporting transaction semantics like a database. A BI product in that context is ideally suited to be served by a search engine.

Using flex on the front end alleviates load on the server to just serving search results. The heavy lifting of preparing visually rich charts is done very well by flex. So most of the server queries return in under 50ms and data transfer stays under 10K with compression enabled. Flex is optimized for the UI so the end response time is pretty good. If data is to be clustered (for redundancy or scalability) across search engine instances integration can happen at the UI so there is no single point of failure nor additional load on the server.

This is my viewpoint on database's role in software architecture during the last decade. I have a lot of respect for database vendors, they have made a very important data resource available with minimum fuss. Over the last decade or so databases have gone from a prominent player in software architecture to a much needed but far less desirable component. I refer to client server VB Unix Oracle applications where application development is faster and any significantly heavy processing is offloaded to database or to a unix batch process.

Mid to late 90s Java gained prominence and with it objects and middleware as well. Apps started to deal with objects instead of raw database records. A natural enhancement for a database was to provide an additional interface via objects. There have been some attempts in doing this but they never really picked up. At the same time ORM packages started to gain traction as the process of Object-Record mapping is a fairly straight forward procedure. Even though databases cached data in their layer to speed up access it proved to be little value for complex object models. The cost of OR mapping is so high that once done it made sense to cache the resulting object. So objects got cached at the middleware and caching products came into picture.

The fallouts:

• Majority of developers who started in last 5 years are familiar with ORM and caching products but never dealt with databases in a significant manner. So they are not able to leverage the database optimally. ORMs have abstracted the database but they did it sitting outside the database. So the tuning part is neither done by ORM nor developers.
• An organization dealing with huge databases has to provision for dual infrastructure - one for databases and another for caching the data. Typically caching is done by sucking the entire database table into memory.

I remember few years back when I was running the Websphere, it was very difficult to kill a instance. On killing of one instance, a deamon was spawning another. Killing that one, one more was coming up. I had tried "kill -9", "kill -KILL" - No help. When one was going down, other was coming up. It is not like, I have 2 process instances running and on killing of both, the application  is completely down. Rather, it is only one and taking birth each time it is murdered.

Why not this happens for a server. If I consider that is possible, virtually the application is immortal. 

[I have taken this section out from my regular writting as I want to highlight on one particular issue. Recently I had mapped a production instance to amazon EC2 instance in web DNS mapping. I issued a shutdown command on EC2 machine by mistake. It just took me 2 mins to bring up a snapshot of the last instance as I was taking the snapshot. Then I pointed to this new server address in web DNS mapping. This process took a good 30mins and the system remained down for so long :(]

Consider a machinsm is available, where web DNS mapping is instantaneous to the depployed server. In this spectrum, I have an application which transcends through server instances. The possibilities lies on the snapshot creation. Before death can I take an exact snapshot and all next instances starts from there. Amazon EC2 has infiniband networking to S3. So, network latency is not my concern. Hadoop is there for my rescue to transfer the big files. The big file is the tar ball of the complete mnt. If you are reading my previous posts, you know you can shove everyting to mnt. That means the complete harddisk portion of my interest to the S3. This helps on saving the database files, other contents, application state and everything else.

During shutdown,

1. Create a snapshot tar ball of /mnt of the portion of harddisk of my interest. - 30sec
2. Copy this to S3.
3. Start another instance.
4. When the next instance comes up, it self configures itself during start up taking the file from S3.

I have taken snapshot and created a new instance quickly but manually. It takes just a minute. How can I do it automatically?

First I looked for linux shutdown hooks like init. No luck. If you are aware of any, please let me know. Now I am going to explore by creating a blank java application and starting it during init and putting a shutdown hook there. This shutdown hook will grap the linux shutdown. I am concluding this reading the post at http://forum.java.sun.com/thread.jspa?threadID=504889 . Once I finish this experiement, I blog about my findings :) This will open up following possibilities in EC2:

1. Immortal application in a mortal server
2. No information loss when instance shutted down.
3. Many app instance creation when load increases. [What about the database??? I have a big blog coming on this. I am half way through]

Don't forget to help me on implementing a linux hook of shutdown rather than going via the java way...  A work around not a clean solution :(

Databases replication is a really useful feature. It provides backup, load balancing, data aggregation to start with. A typical design of a database replication will be:

• A data modification statement on the master database is written to the database log
  
• These entries are read by a replication agent and put on a queue for child nodes to apply on their respective databases
• A checkpoint thread scans the master node's database log and clears the entries which are read by the replication agent.

What happens when a child node goes down?

• Entries off the queue (where the master rep agent puts in entries) are not removed by the child node so the queue starts building up
• When the queue is full the rep agent stops reading entries off the master database log. If it read where will it put them?
• The master database transaction log starts to get full and when it is full it stops accepting inbound transactions means master database becomes unavailable

In most of the cases the child comes up much before this happens and the child has all the information it has not consumed on the queue. So the queue is a important piece in the design. The master keeps it up and running, filling it with data while the child node is down. In essence the master node is responsible for the queue and hence takes on the risks associated with it. I don't think that is a very wise thing to do, you can think of any number of real world scenarios where this model does not work.

In the replication component Abinash is coding as I write this, we decided that each child node should take responsibility for its upkeep. it has to figure out how to recover from a failure and should not adversely affect the master in any way. We will publish the design details in a follow-up post.