Siddhi

Cloud Native Stream Processor

Fully open source, cloud-native, scalable, Streaming and Complex Event Processing System capable of building real-time analytics, data integration, notification and surveillance usecases.

Siddhi understands Streaming SQL queries in order to capture events from diverse data sources, process them, integrate with multiple services and data sources, and publish output to various endpoints in real-time.

Benefits

Faster Development

Agile development experience with SQL like query language and graphical drag-and-drop editor supporting event simulation.

Cloud Native

Lightweight runtime that natively runs in Kubernetes via Kubernetes CRD, and works with systems such as NATS, gRPC, and Prometheus.

Scalable Deployment

Embedded event processing within Java, Python applications to running on bare metal, Docker and massively scaling on Kubernetes.

System Integration

Integrates with messaging systems (NATS, Kafka, JMS), Databases (RDBMS, NoSQL), Services (HTTP, gRPC), File systems and others.

CI/CD Pipeline

Supports development lifecycle and seamless deployments, through simple file-based configurations, automated tests, and CI/CD pipelines.

Reliability

Battle-tested with billions of events at UBER, eBay, PayPal, HCA, TFL, WSO2 and in many Fortune 500 companies enabling exactly once event processing.

Overview

Siddhi can run as an embedded Java or Python library, run as a microservice on bare-metal, VM, or Docker, and run natively at scale in Kubernetes processing millions of events per second.

Use cases

Streaming Data Integration

  • Retrieve and publish data from various enterprise systems.
  • Perform data transformation on JSON, XML, Text, Avro, and CSV.
  • Integrate with databases, services, and realtime event streams.
  • Data preprocessing, fault tolerance, and error handling.

Streaming Data Analytics

  • Calculate aggregations over time, length, and session windows.
  • Long-running time-series aggregations from seconds to years.
  • Analyze event occurrence patterns and trends over time.
  • Realtime predictions with online and pre-trained ML models.

Alerts & Notifications

  • Generate alerts based on static and dynamic thresholds.
  • Correlate data to detect event anomalies and missing events.
  • Support scheduling, digest, and auto-retry of notifications.
  • Publish alerts via various event sinks such as email, and MQs.

Adaptive Decision Making

  • Static rule processing via predefined and database based rules.
  • Dynamic rule processing through stateful queries and system state.
  • Decision making through synchronous RPC (HTTP, gRPC).
  • Incremental learning and decision making online ML models.

Working with Siddhi