Monday, August 06, 2012

Real-time Pipeline

Here are a bunch of tools that might be useful for building a real-time pipeline:

  • Kafka: Persistent Message Broker with O(1) access time
  • Storm: Real time computation system
    • Trident: High-level abstraction on top of Storm
  • Esper: Complex Event Processing (CEP)

May talk more about these tools in future.

1 comment:

cresny said...

We did just this, though our messages were large enough, and our pipeline linear enough that we skipped Storm and rolled our own flow management using zeromq and redis.