Cloud Expo on Ulitzer

I was speaking with a company last week that would like to get into cloud event processing.  For the sake of discussion, I’m going to define ‘cloud’ as the ability to dynamically, and upon demand, use more or less compute, storage, or network capacity. This is often referred to as ‘elasticity.’  I’m also going to add a few other requirements; this cloud must be able to:

1. Host operations for a number of customers, all accessing the same data, but in different ways,
2. Provide access to a lot of big data (see #3 – it’s all stored too),
3. Able to handle messages rates approach 1MM per second (yes, that’s a million messages per second),
4. Do something with those messages before storing them (in fact, we’re not even worried about storing them),
5. And while we’re doing something with those messages, relate them to previously stored messages.

Cloud Event Processing Defined
And I’m going to define cloud event processing as, ‘using many varieties of event processing; including complex event processing (CEP), map/reduce, and event, or message, driven agents to process events.’  Using DarkStar, we’re able to deploy these various forms of event processing without regard for underlying storage, compute or network – the event processing services are simply deployed and begin processing; if a particular service needs more resources, it asks for them and when it receives them, it uses them.

Private Cloud Defined
It’s a private cloud when you deploy the above (Iaas) internally.  It’s a public cloud when it’s deployed publicly, and it’s a hybrid cloud when it’s deployed both privately and publicly.

Data Affinity
The large amount of inbound network bandwidth that would be required to get this particular cloud off the ground and into the sky already makes hosting with Amazon or Rackspace more than a ’sign up with a credit card’ process.  Having the elastic resources available to each customer of our customer makes things even more difficult.  See, our customer has 100’s of customers who want to do something with this data.  And each one of those customers will need access to both the old data, and the streaming 1MM messages per second.  Without creating a separate physical footprint in a cloud provider, it would be very difficult to solve this problem.

Buy vs Build
The more things change, the more they stay the same.  So, now we’re involved in the old, ‘how much to build vs buy?’ process.  The thing is, this company has the skills to run their own cloud so we’ll most likely start with a proof of concept and then grow it.  We’re cloud seeding at Cloud Event Processing!

The Test Platform
Our recent work with Twitter will be our first deployment in this cloud infant.  Using DarkStar, we’re able to analyze the Twitter feed to sense and respond to opportunities and threats.  This relatively simple project includes components of event driven agents, NoSQL storage, CEP sliding windows & aggregation, visualization, map/reduce and analytics.  (For those of you who’ve been reading all along, you’re asking, ‘analytics?’-we didn’t see any of those Colin! – and you’re right, you didn’t – we’ll share those when we’re ready.)

Kickin’ Tires & Lightin’ Fires
We’ll be starting off with some bog standard Intel hardware, Eucalyptus, and the Ubuntu Enterprise Cloud.  There’s a white paper available on Eucalyptus’s site that details our initial approach.

Thanks for reading!