2 posts tagged “clustering”

Interesting aspect to tackle after putting applications on EC2 in a live production environment. Our first customer has been live for over 1 year on EC2 and never been down for a single time. We didn't have to reboot the linux server though we had to restart our java processes for upgrades. Touchwood. Database incremental backups happen every minute to S3 and a daily back up at night. This is mostly an enterprise app, so not very stringent requirements on real time failover. Shortly we are going live with a new retail customer facing application - a 24 x 7 uptime requirement. Scalability is not much of an issue as our server is still operating under 65MB with response times under 50ms most of the time. Failover is a key aspect for us - passive or active, active being the preferred option.

Software Architecture For Developing Countries

I once heard in some countries people cry when a new baby is born... Because the death is counted. Whatever has come to alive in this universe is going to die oneday. The information is no exception. For some time, it gets transacted, people access it more often and then it becomes dead. People open it from the coffin for investigation.

I am talking about this live and dead state of information. A transactional relational database is good for a live information storage. Why should we have the dead information there with just a flag off saying inactive or what ever you use in your application. I use states like "closed, settled, withdrawn, rejected, ...". But all stay together. Imaging in real life the dead and lives are staying in same house just a name board in dead rooms saying Late Mr. XXX

I can't. A horrible experience. Then why? This was my inspiration to architect onedata. Here information flows from the live storage to dead storage. From relational database to search engine. When we search, it searches both of these places, unifies the information and present in a one data fashion. The live storage enables frequent transactions.

The advantage

1. The database storage is super lite.
2. Search engine is good at finding information given a clue. Once a request goes to storage, people only remember a clue to find it again. So retrieval alligns with human nature.
3. Archieval becomes easy. The DNA of databases to handle loads of information.

The Solution

1. The state triggers the archieval process.
2. Records move to a staging area consists of a flattened record format.
3. From here it flows to search engine and gets indexed.
4. When a information retrieval happens, it goes to both if detail investigation is needed.
5. Inherently a search solution to all data with access based filteration.

I have implemented this for my product using solr and mysql. I will publish it soon in sourceforge. I have requested a project named oneline for this. If you are interested let me know. I will inform you once I release the alpha files. It spits an XML result for the clients. Now-a-days RIA flex, xslt technologies are very good in processing the XML results. In additon to this carrot2 kind of clustering on information, gives a buzz to the end user for quick information finding.

Don't forget to write a comment. If you liked it, just express yourself. Your view points, one comment will keep us writting.

 

A snippet of it:

/**

* Combines records from database as well as search engine.

* sensor=sensor.onedata&action=SEARCH&q=Coorg&fl=teaser,title&dq=o.title like 'Coorg~'&dfl=o.cat as cat, o.dn as dn

*

* ----- The stntax for the search -----

*

* q : This is executed in the search engine to catch the matching records.

* Ex. sensor:customererview AND created:2005-11-23

* fl: The fields which needs to be shown to user [sensor,teaser]

* dq: 1) Database queries. There are two tables represented as follows:

* onedata o, cnrole r

* So the search is like o.title like 'Coorg~'

* 2) All ~ character is replaed with a %. This can be used for a like query

* dfl: o.cat as cat, o.dn as dn

*

* ----- Basic requirement of data -----

*

* Both tables are joined to perform the search. So any missing oneDataId in

* cnrole table will make the search futile.

* dn - All records in that dn

* cn=any and role=guest - This is a public data. Everybody can access it.

* cn=xxx and role=guest - Any body belongs to the cn can access it.

* cn=xxx and role=yyy - The person belonging to the cn and role can only access it.

* cn=none and role=none - Nobody can view it except the dn.

* If nothing is specified, even dn can't view the record.

* No entry in cnrole - A jumbie record and no one can view this.

Software Architecture For Developing Countries