Complex Event Processing: When Design Patterns Become Concrete

5 Jul 2007

Over the past few months at Lab49, we’ve thrown ourselves into complex event processing (CEP) — aka event stream processing (ESP) — and have been formulating exactly how and when it fits into the larger, more comprehensive technology stack found in global financial services institutions. We’ve formed a number of interesting vendor partnerships, attended product training, sampled, compared, and teased apart many of the popular products, and we’ve created several CEP-based demo applications that have been shown at recent events like SIFMA.

Along the way, we’ve all learned a lot about CEP, and the more I learn, the more I dig it. The more I put CEP into practice, the more I foresee its ultimate dominance as an architectural design pattern for everyday development.

What’s fascinating to me about CEP isn’t that it’s a new idea, despite how it may be touted by vendors. Regardless of the hype, CEP isn’t the most revolutionary technology you’ve never heard of. What’s fascinating is that out from a decades-old, primordial soup of ideas, research, and trial-and-error that, in and beyond academia, has been trying to create architectural models around complex data problems with real-time constraints, enough best practices and design patterns have emerged to evolve an ecosystem of market entrants, seemingly all at once.

It’s not the first time that a bundle of quality design patterns took concrete form as a technology. Object pooling, lifetime management, transaction enlistment, and crash domains begat COM+, Microsoft Component Services, and J2EE application servers. Logging levels, external configuration, adaptable logging sinks begat log4j, syslogd, and the Logging Application Block from the Microsoft Enterprise Library. Unit-testing and test-driven development begat JUnit and its children.

These transformations have been crucial. Once developers accepted these patterns and solutions as sufficiently solved and commoditized, they were saved considerable time and attention. Freed of coding logging libraries and unit testing frameworks for the umpteenth time, developers could focus more on the business problem being solved rather than the infrastructure details required to solve it.

But these transformations didn’t really upset the gross architecture of applications. They may have changed some of the design decisions and simplified the implementations, but they didn’t fundamentally change the abstraction you would use to model a problem and architect a solution.

CEP, on the other hand, does.

Instead of storing and indexing miles of cumulative data in a persistent store to service complex queries in batch/polling fashion, CEP inverts the whole shebang, storing and indexing the complex queries before streaming data across queries without storing a lick. The transformation of a business problem from tables, rows, and polling intervals into events, filters, triggers, and real-time reactions is not only quite enabling, it changes the very way you think about how business problems can be solved and which problems may have viable solutions.

Over the next few weeks, I’ll delve a bit more into CEP and how it relates to technologies you might be more familiar with. In the meantime, check out some of the in-depth blog entries other folks from Lab49 have been writing about CEP.

Advertisements

6 Responses to “Complex Event Processing: When Design Patterns Become Concrete”

  1. Max Palmer Says:

    Sounds really great. Looking forwards to hearing more about it. I’ll check out the other lab49 blogs too.

  2. Ben Says:

    I think everything CEP offers could be done on a SOA, where the DB also acts as a service on the ESB. If you want to be notified of events at the data-layer, then triggers could send messages to the application.

    This is something I’ve been thinking about doing: using Oracle Queues and an adapter to Tibco (our ESB). This would stop us from polling the database for changes, which is expensive when its idle. For example, we have a real-time webservice push with a 1-minute SLA that just kills the database. Other areas that poll, like retry logic, would be served better this way.

    If people started treating the DB as another service on the SOA, rather than just a data store, I wonder what would happen…

    That was about the only thing interesting I saw that CEP could do, but would be easy to adopt now without fundamental changes to the architecture.

  3. timbass Says:

    Glad to see you are diving into CEP. It will be interesting to see where your deep dives take you. Keep up the good work!

    http://www.thecepblog.com

  4. Ed Daniel Says:

    Glad to see you being enthused by this approach – you might find Bill’s post on “The message is the software” fits neatly into your view of CEP:

    http://billburnham.blogs.com/burnhamsbeat/2004/10/the_message_is_.html

  5. Marc Jacobs Says:

    Ben:

    In some ways, your proposal that “everything CEP offers could be done on a SOA” supports my point, that CEP is as much a set of design patterns (that can have multiple concrete implementations) as it is one concrete technology that implements those design patterns well. I agree with you that an enterprise service bus combined with an database exhibits several features of a CEP tools. [In fact, this is one of the topics I’ll be delving into in future posts in this series.]

    That said, though, I think you might be underestimating the value of several of the specific CEP tools on the market. First and foremost, CEP tools usually present these design patterns as a single unifying concept for event-driven software development. The solution you are looking into requires the piecemeal assemblage of several different technologies and platforms to get at the real goal. CEP tools, on the other hand, can provide a single, focused platform that can save considerable time and effort for the right set of business problems.

    The other major point is performance. Basing an event-driven system on a database, depending on the volume of data and the number of triggers/actions, can present knotty performance and scalability problems that are difficult to solve. One of the key advantages of CEP is its ability to index queries and perform its work in an almost stateless mode of operation. You can pile on triggers onto database tables to get similar functionality, but you will not get the same performance going down that path than you would from CEP.

  6. Marc Jacobs Says:

    Ed:

    Thanks for the link. It’s been a long time since I read that article.

    Message-oriented designs are not only providing simplifying architectural visions for time- and event-driven systems, but also for parallel, distributed systems. Erlang and CCR/DSS are both entirely message-driven, and an internal message bus is a key part of their story.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: