

Siddhi Extensions

Siddhi provides an extension architecture to enhance its functionality by incorporating other libraries and custom logic in a seamless manner. Each extension has a namespace that can be used to identify and specifically access the functionality of the relevant extension.

Following are the supported Siddhi extension types:

• Execution extension types

  • Function extension.
  • Aggregate Function extension.
  • Window extension.
  • Stream Function extension.
  • Stream Processor extension.

• IO extension types

  • Source extension.
  • Sink extension.

• Map extension types

  • Source Mapper extension.
  • Sink Mapper extension.

• Store extension types

  • Store extension.

• Script extension types

  • Script extension.

Available Extensions

All the Siddhi extensions are released under Apache 2.0 License.

Execution Extensions

Name	Description	Latest Tested Version
execution-string	Provides basic string handling capabilities such as concat, length, replace all, etc.	5.0.9
execution-regex	Provides basic RegEx execution capabilities.	5.0.5
execution-math	Provides useful mathematical functions.	5.0.4
execution-time	Provides time related functionality such as getting current time, current date, manipulating/formatting dates, etc.	5.0.4
execution-map	Provides the capability to generate and manipulate map data objects.	5.0.5
execution-json	Provides the capability to retrieve, insert, and modify JSON elements.	2.0.5
execution-unitconversion	Converts various units such as length, mass, time and volume.	2.0.2
execution-reorder	Orders out-of-order event arrivals using algorithms such as K-Slack and alpha K-Stack.	5.0.3
execution-unique	Retains and process unique events based on the given parameters.	5.0.5
execution-list	Provides the capability to send an array object inside Siddhi stream definitions and use it within queries.	1.0.0
execution-streamingml	Performs streaming machine learning (clustering, classification and regression) on event streams.	2.0.4
execution-tensorflow	Provides support for running pre-built TensorFlow models.	2.0.2
execution-pmml	Evaluates Predictive Model Markup Language (PMML). It is under GPL license.	5.0.1

Input/Output Extensions

Name	Description	Latest Tested Version
io-http	Receives and publishes events via http and https transports, calls external services, and serves incoming requests and provide synchronous responses.	2.2.0
io-nats	Receives and publishes events from/to NATS.	2.0.10
io-kafka	Receives and publishes events from/to Kafka.	5.0.7
io-email	Receives and publishes events via email using smtp, pop3 and imap protocols.	2.0.5
io-cdc	Captures change data from databases such as MySQL, MS SQL, Postgresql, H2 and Oracle.	2.0.5
io-tcp	Receives and publishes events through TCP transport.	3.0.4
io-googlepubsub	Receives and publishes events through Google Pub/Sub.	2.0.2
io-rabbitmq	Receives and publishes events from/to RabbitMQ.	3.0.4
io-file	Receives and publishes event data from/to files.	2.0.5
io-jms	Receives and publishes events via Java Message Service (JMS), supporting Message brokers such as ActiveMQ	2.0.3
io-prometheus	Consumes and expose Prometheus metrics from/to Prometheus server.	2.1.2
io-grpc	Receives and publishes events via gRpc.	1.0.8
io-mqtt	Allows to receive and publish events from/to mqtt broker.	3.0.0
io-sqs	Subscribes to a SQS queue and receive/publish SQS messages.	3.0.0
io-s3	Allows you to publish/retrieve events to/from Amazon AWS S3.	1.0.2
io-gcs	Receives/publishes events from/to Google Cloud Storage bucket.	1.0.0

Data Mapping Extensions

Name	Description	Latest Tested Version
map-json	Converts JSON messages to/from Siddhi events.	5.0.6
map-xml	Converts XML messages to/from Siddhi events.	5.0.3
map-text	Converts text messages to/from Siddhi events.	2.0.4
map-avro	Converts AVRO messages to/from Siddhi events.	2.0.6
map-keyvalue	Converts events having Key-Value maps to/from Siddhi events.	2.0.5
map-csv	Converts messages with CSV format to/from Siddhi events.	2.0.3
map-binary	Converts binary events that adheres to Siddhi format to/from Siddhi events.	2.0.4
map-protobuf	Converts protobuf messages to/from Siddhi events..	1.0.4

Store Extensions

Name	Description	Latest Tested Version
store-rdbms	Optimally stores, retrieves, and manipulates data on RDBMS databases such as MySQL, MS SQL, Postgresql, H2 and Oracle.	7.0.5
store-mongodb	Stores, retrieves, and manipulates data on MongoDB.	2.1.0
store-redis	Stores, retrieves, and manipulates data on Redis.	3.1.1
store-elasticsearch	Stores, retrieves, and manipulates data on Elasticsearch.	3.2.0

Script Extensions

Name	Description	Latest Tested Version
script-js	Allows writing user defined JavaScript functions within Siddhi Applications to process events.	5.0.3

Configure System Parameters for Extensions

Siddhi extensions cater use-case specific logic that is not available by default in Siddhi. Some of these extensions have system parameter configurations to define/modify their behavior. These extensions usually have default values for the parameters, but when needed, they can be overridden by configuring the parameters in Siddhi Configuration yaml and passing it at startup. Refer more details on this in here

Writing Custom Extensions

Custom extensions can be written in order to cater use case specific logics that are not by default available in Siddhi core, or in existing extensions.

The following Maven Archetypes can be used to generate the necessary Maven project to create the relevant extensions.

Writing Execution Extensions

Maven archetype for execution extension that is used to generate the extension types such as:

• Function extension, by extending io.siddhi.core.executor.function.FunctionExecutor.
• Aggregate Function extension, by extending io.siddhi.core.query.selector.attribute.aggregator.AttributeAggregatorExecutor.
• Window extension, by extending io.siddhi.core.query.processor.stream.window.WindowProcessor.
• Stream Function extension, by extending io.siddhi.core.query.processor.stream.function.StreamFunctionProcessor.
• Stream Processor extension, by extending io.siddhi.core.query.processor.stream.StreamProcessor.

The CLI command to generate the Maven project for the extension is as follows.

mvn archetype:generate
    -DarchetypeGroupId=io.siddhi.extension.archetype
    -DarchetypeArtifactId=siddhi-archetype-execution
    -DgroupId=io.siddhi.extension.execution
    -Dversion=1.0.0-SNAPSHOT

When the command is executed, the following properties will be prompted. At last, confirm that all the property values are correct by typing Y for yes, and N if they are not.

Property	Description	Mandatory	Default Value
_nameOfFunction	The name of the custom function (function, aggregate function, window, stream function, or stream processor) that is being created.	Yes	-
_nameSpaceOfFunction	The namespace of the function that groups similar custom functions.	Yes	-
groupIdPostfix	Postfix for the group ID. (As a convention, the function namespace is used).	No	${_nameSpaceOfFunction}
artifactId	Artifact ID for the project.	No	siddhi-execution-${_nameSpaceOfFunction}
classNameOfFunction	Class name of the Function.	No	${_nameOfFunction}Function
classNameOfAggregateFunction	Class name of the Aggregate Function.	No	${_nameOfFunction}AggregateFunction
classNameOfWindow	Class name of the Window.	No	${_nameOfFunction}Window
classNameOfStreamFunction	Class name of the Stream Function.	No	${_nameOfFunction}StreamFunction
classNameOfStreamProcessor	Class name of the Stream Processor.	No	${_nameOfFunction}StreamProcessor

Writing Input/Output Executions

Maven archetype for input/output execution that is used to generate the extension types such as:

• Source extension, by extending io.siddhi.core.stream.input.source.Source.
• Sink extension, by extending io.siddhi.core.stream.output.sink.Sink.

The CLI command to generate the Maven project for the extension is as follows.

mvn archetype:generate
    -DarchetypeGroupId=io.siddhi.extension.archetype
    -DarchetypeArtifactId=siddhi-archetype-io
    -DgroupId=io.siddhi.extension.io
    -Dversion=1.0.0-SNAPSHOT

When the command is executed, the following properties will be prompted. At last, confirm that all the property values are correct by typing Y for yes, and N if they are not.

Property	Description	Mandatory	Default Value
_IOType	The name of the custom IO type that is being created.	Yes	-
groupIdPostfix	Postfix for the group ID. (As a convention, the name of the IO type is used).	No	${_IOType}
artifactId	Artifact ID for the project.	No	siddhi-io-${_IOType}
classNameOfSink	Class name of the Sink.	No	${_IOType}Sink
classNameOfSource	Class name of the Source.	No	${_IOType}Source

Writing Data Mapping Executions

Maven archetype for data mapping execution that is used to generate the extension types such as:

• Source Mapper extension, by extending io.siddhi.core.stream.output.sink.SourceMapper.
• Sink Mapper extension, by extending io.siddhi.core.stream.output.sink.SinkMapper.

The CLI command to generate the Maven project for the extension is as follows.

mvn archetype:generate
    -DarchetypeGroupId=io.siddhi.extension.archetype
    -DarchetypeArtifactId=siddhi-archetype-map
    -DgroupId=io.siddhi.extension.map
    -Dversion=1.0.0-SNAPSHOT

When the command is executed, the following properties will be prompted. At last, confirm that all the property values are correct by typing Y for yes, and N if they are not.

Property	Description	Mandatory	Default Value
_mapType	The name of the custom mapper type that is being created.	Yes	-
groupIdPostfix	Postfix for the group ID. (As a convention, the name of the mapper type is used).	No	${_mapType}
artifactId	Artifact ID of the project.	No	siddhi-map-${_mapType}
classNameOfSinkMapper	Class name of the Sink Mapper.	No	${_mapType}SinkMapper
classNameOfSourceMapper	Class name of the Source Mapper.	No	${_mapType}SourceMapper

Writing Store Executions

Maven archetype to generate store extension, by extending either io.siddhi.core.table.record.AbstractRecordTable or io.siddhi.core.table.record.AbstractQueryableRecordTable. Here, the former allows Siddhi queries to perform conditional filters on the store while the latter allows performing both the conditional filters and select aggregations directly on the store.

The CLI command to generate the Maven project for the extension is as follows.

mvn archetype:generate
   -DarchetypeGroupId=io.siddhi.extension.archetype
   -DarchetypeArtifactId=siddhi-archetype-store
   -DgroupId=io.siddhi.extension.store
   -Dversion=1.0.0-SNAPSHOT

When the command is executed, the following properties will be prompted. At last, confirm that all the property values are correct by typing Y for yes, and N if they are not.

Property	Description	Mandatory	Default Value
_storeType	The name of the custom store that is being created.	Yes	-
groupIdPostfix	Postfix for the group ID. (As a convention, the name of the store type is used).	No	${_storeType}
artifactId	Artifact ID for the project.	No	siddhi-store-${_storeType}
className	Class name of the store.	No	${_storeType}EventTable

Writing Script Executions

Maven archetype to generate script extension, by extending either io.siddhi.core.function.Script.

The CLI command to generate the Maven project for the extension is as follows.

mvn archetype:generate
    -DarchetypeGroupId=io.siddhi.extension.archetype
    -DarchetypeArtifactId=siddhi-archetype-script
    -DgroupId=io.siddhi.extension.script
    -Dversion=1.0.0-SNAPSHOT

When the command is executed, the following properties will be prompted. At last, confirm that all the property values are correct by typing Y for yes, and N if they are not.

Property	Description	Mandatory	Default Value
_nameOfScript	The name of the custom script that is being created.	Yes	-
groupIdPostfix	Postfix for the group ID. (As a convention, the name of the script type is used).	No	${_nameOfScript}
artifactId	Artifact ID for the project.	No	siddhi-script-${_nameOfScript}
classNameOfScript	Class name of the Script.	No	Eval${_nameOfScript}

Top