29 posts from 2008

We have been doing some work to host a mail server as part of our offering. Google search typically throws up Zombra but there are certain aspects of the license which we are not comfortable especially mobile support. We came up with 2 alternatives which fit our requirements to install on Amazon EC2

Security: We feel comfortable in providing our personal information to fulfill our own need.

Availability: The application should follow the user. Application should have presence in multiple gadgets in 24*7.

Scalability: The application should able to scale explonentially as WORD of mouth. 

Performance: Humans repeat 20% things 80% time.

Usability: The change need to happen at the vicinity of operation.

Adaptability: The environment surrounding us is changing all the time influenced by living bodies.

Communication: 95% communications are non-verbal.

Traceability: Actions by people at a place on time are not accident.

Collaboration: Virtual world and Real world complete each other.

Reporting: Every action has a equal and opposite reaction.

I am fascinated by the Google Map. This is the perfect exploration tool. The journey just gets interesting when I don't know what am I going to discover. 

Imagine a BI product which is built in such a way so that you can ZOOM, PAN, Rotate, Drive in different dimentions, ....  

Like looking at Bangalore from 200sq ft above ground on 1st Jan, 2008 trying to understand why my sales is dropping?  Then seeing the roofs of the profitable shops and loser shops. Finally descending down to the rack. Finally discovering a rat responsible for your losses ...

If this is a reality, what I need..

A raw implementation.. http://www.bizosys.com/tsr/TSR.html

TSR

What should be my backend to provide this infinite resource navigation to minute level ...

Putting the BI engine in steroid =>  Hadoop + Hive + Lucene Katta 

http://www.ewh.ieee.org/r6/scv/computer/nfic/2008/Facebook%20Hive%20by%20Ashish%20Thusoo.ppt
 
http://joa23.files.wordpress.com/2008/10/katta-in-production.pdf
 
What's your suggestion to make game interesting??

Anyone in India is prone to those automated daily marketing calls. I hated that someone could just call me anytime they wished to. Initially it was just one call a day at around 9AM but last 2 months it increased to 4-5 calls starting at 9 AM and as late as 8:30 PM. I created a contact for these numbers (there were 3 of them) and not take the call. One day I decided to just take the entire call, it lasted for about 4 mins, the automated message repeated for 5-6 times. I thought atleast the other side will incur some cost when they call me. Surprisingly the next day the calls came down to 3. I did the same 2 more times and calls stopped altogether. I do get an odd marketing sms but the calls completely stopped.

I wanted a multi tenancy database with no prior knowledge of data. It's like what database I should use to develop a SalesForce like infrastrcuture. Finally, one thing which came closure to my consideration was HBase. I liked the way they are going and the Amazon platform is best suitable for this. The HBase as mentioned is perfect for stortage and retrieving the information on a primary key. But what about the searching piece.  In the onelineweb we put the following mechanism:

Assumptions

1 - Data reading is 5 times than data creation.
2 - Active workflow records only change

Problem Faced

Each node maintains their local Solr instance. All solr indexes in the active nodes are alike. Updating the solr indexes need to be incremental in nature. The information can change in any node. 

The Solution

I created a sync table. When ever a record change, I put an entry to this sync table. Each machine maintans the last sync id. I get all the latest content from this last sync id and update the solr instnace.

The Solr index can go large. To distribute the solr index in many nodes, we need to have a platform to assemble it from all these places and serve it. The architecture is to have a common set of actively changing information and other nodef having frozen records which don't change often. We use the map reduce paradigm for managing the distributed solr index.

• The trend of "NO SOFTWARE" and only configurable system like SalesForce will be predominant in the market
• Softwares will be learning from expert humans. One of my friend shared how the robots paint a car. First time an expert painter paints the car. Robot records this and replays the operation from there on. 
• Most of the applications will communicate among themselves with no prior knowledge about their existance. JINI and browser will be their gossiping point and information exchange mandi.
• There will be huge need of Creative People ( Ex. Graphics Designers ). The human count of Indian Software Offshoring will be stagnant and will reduce.
• The new generation computers will aquire IO, CPU and Memory from the cloud
• Now humans follow the application in the sense come near to a desktop browser to access. In 2013, the application will follow the user with it's presence in all devices used by humans (Ex. Mobile, Watch, TV ).
• The applications will have multiple input medium (Not just mouse and keyboard).  Some of the new input sensors will be the things we look at , words we speek, the route we take, the friends we shake hand, our gestures, sitting  standing and sleeping positions  ....

The last post and this one I am writing the ideas that went in our products. If you need details on the product please visit drapsa site.

Any BI product needs a good data store to power the queries on a large dataset. Some of the work is done prior in way of building the cubes. Any adhoc queries or aggregation of data is done on the fly. Sybase and other database vendors including a couple of startups have proposed column oriented databases as a better model for the datastore as cubes aggregate data on a couple of columns at a time. Very compelling and results are there to see in live production environments.

In our design we made use of a search engine as the underlying datastore. Search engine index is optimized column wise like column oriented databases. Databases have largely grown from managing a small set of data very effectively with transaction semantics and ACID rules. As the dataset grew in BI scenarios more and more hardware and software designs have been employed to tackle it - example disk partitioning both physical and logical etc. Search engines on the other hand started with a paradigm of managing a large set of data primarily for read operations. No burden of supporting transaction semantics like a database. A BI product in that context is ideally suited to be served by a search engine.

Using flex on the front end alleviates load on the server to just serving search results. The heavy lifting of preparing visually rich charts is done very well by flex. So most of the server queries return in under 50ms and data transfer stays under 10K with compression enabled. Flex is optimized for the UI so the end response time is pretty good. If data is to be clustered (for redundancy or scalability) across search engine instances integration can happen at the UI so there is no single point of failure nor additional load on the server.

We have recently worked on a shopping cart optimized for grocery. Web is ready to a demo state and mobile is in the works. You can experience it here. It is done in flex so the first time load may take about 30 seconds.

But why build another shopping cart from scratch when the concept has been done to death in all conceivable forms - free, commercial, custom built, SaaS? There are certain characteristics to every shopping experience. Mobile shopping is largely centered on comparison (unless it is an iphone), nokia site illustrated this nicely. Similarly grocery shopping has to be done quickly as the list of items is known and it is typically a sizable one. Invite you to experience it and share your feedback. ---Thanks

Continuing on the previous post of comparing the onelineweb framework to Blazeds and Graniteds here is the summary of why an internal framework is developed.

1 - Scalability : It was impossible to introduce the plug-in architecture without creating/changing the class loading of graniteds and ibatis. The end result would have been a large monolithic package instead of smaller modules.

2 - Multi client support : Browser, Windows Mobile, Blackberry, Mashup API(XML API) and distribution (Ex. RSS feeds, Outlook).

4 - Performance : Fast response time moving out all intermediate data exchange layers and natively compressing the data from database layer (right out of the resultset object) to gain speed at network layer using apache native compressors and browser native decompressors.

5 - Lightweight : Less dependencies on 3rd party libraries.

Let the problem define the framework rather than the framework defining the solution. Simpler is Better.

Trying to put together a comparison of onelineWeb with BlazeDS and GraniteDS. There is one more decent implementation of remoting for flex from Hessian at http://hessian.caucho.com/#Flash/Flex

Hands on experience with the above: We started with GraniteDS (when it was beta) for our product and used it for 3-4 months before giving it up. I have checked out BlazeDS sample code and have a fairly good hang of how to go about if  I have to use it.

First things first: BlazeDS is a port out of LivecycleDS from Adobe, so it can be expected to have a decent start. GraniteDS seems to have evolved quite a bit in the last 6 months including server data push with Gravity. It seems to be more integrated with EJB, JBoss, Spring and other java software stack, so it may have an advantage in that area over Blaze. But if my need is for remoting I will go with Blazeds due to its history. Further on I will compare with Blazeds only. GraniteDS or Hessian information can be obtained from their respective sites.

Messaging / Server push: Blazeds has got good support and covered the fallback aspect when clients and servers behind firewalls. If messaging needs are primary it is the way to go. onelineWeb does not have anything in this space.

Remoting Options: Remoting can be done in an text(http, web services) or binary way using remote objects.

Text (xml) based data transfer:
Onelineweb supports only text (xml) via http. It does not support object graph but does support lists. The object graph is built on the client by the application code, not the framework. All class definitions (java and flex) of database tables are code generated including basic CRUD operations in java. These definitions are then used for data marshalling and unmarshalling from xml to java objects on server side and client side. Xml based data transfer support in Blazeds will be bare minimum and these have to be provisioned for by the application.

Binary (object) based data transfer:

Onelineweb does not support binary transfer. BlazeDS has got first class support for this including deep object graph serialization. The java object model has to be replicated in flex via Actionscript classes for this mode to work. It is best done using some sort of code generation otherwise the class definition difference can cause issues during transfer. Maintaining this manually is a chore which cannot be sustained. The major drawback of a binary protocol is clients should have the ability to deserialize binary data like Flex. Other clients will need a wrapper on the server side to communicate in text format, think regular http clients or mobile phones.

The bandwidth requirements will be the least in binary mode. One way to reduce bandwidth requirement in xml way is to zip the content on Apache.This typically reduces the payload by more than 90% as xml tags are very repetitive in nature.

Large data transfer:

Onelineweb has the provision to stream xml data right from the database result even without creating java objects. This greatly reduces the server cpu and memory requirements and client response times. We are able to send  100s of records this way. In binary transfer also the object graph has to be built first.

Time to code:

How long does it take to start coding your first business class? As all the client and server framework code is ready in onelineWeb it gets a developer up and running much faster. As an application is developed it will need a central piece of code to handle communication between server and client. This will aid in streamlining error handling, performance measurement etc. This central piece has to be built on top of BlazeDS.

Modular Applications - Plugin Architecture:

Flex recommends building modules as opposed to one monolithic application. It aids in multiple ways - smaller swf downloads at any time, code separation etc. onelineWeb has the development tools (ant scripts) and runtime support for modules both on the client and server side. Each module is packaged in its own directory structure and built as a separate jar and swf file. Only the required jars are loaded into memory and swfs loaded into the browser. BlazeDS  needs all the classes to be visible in the classpath and hence cannot support a full blown plugin architecture like Nutch or Eclipse.