Cloud EPN

Cloud-based Event Processing Network (EPN).

Pattern Definition

ProblemHow do I use Cloud resources to process high volume incoming events?
ContextThis pattern is applicable when there is a desire to leverage Cloud computing to process high volume events where the volume or types of events would overwhelm a single Stream Processing Platform.Frequently this pattern is used to provide edge filtering of events where events are first filtered to eliminate duplicates, eliminate unimportant events, and reduce the overall volume as close to the event generator as possible.
Forces
  • Devices and event generators may be on premises or Cloud based.
  • Very high volume of data is being generated.
  • It is possible to reduce the volume of events by filtering.
  • Event types require different processing.
SolutionThis pattern employs multiple Stream Processing Platforms working in parallel and in sequence to handle a large volume of events. The conceptual model shows three Stream Processing Platforms, but the number employed could be much larger both in parallel processing and in the number of sequential processing steps. 
ConsequencesPros:
  • Provides the quickest way to add high volume event processing capability.
  • Leverages Cloud-based capacity to handle high volumes of events.
Cons:
  • Complexity in managing multiple Stream Processing Platforms.
  • Overkill for low volumes of events.

Interaction Views

All Cloud

EPN composed of streaming processing platforms

Interactions

  1. A high volume of events is received by the first tier of the EPN. 
  2. After filtering, correlation, aggregation, and enhancement, a much smaller number of events is sent to the next tier of the EPN.
  3. After filtering, correlation, aggregation, and enhancement, an appropriate message is output by the second tier.

Notes

  1. The number of instances of the streaming processing platform in the initial tier is shown as two, but could be any number.
  2. The number of instances of streaming processing platform in the initial tier is determined by the incoming number of events and/or the geographic location of the event generators. For example, if events are being generated worldwide, there could be an instance of streaming processing platform hosted in each geographic region.
  3. Using an application integration platform as the second tier of event processing allows the other capabilities (e.g. orchestration, messaging, adapters) to be used post event processing.
  4. This interaction shows two tiers of event processing, but that could be more than two.

IoT

EPN composed of an IoT platform and a stream processing platform

Interactions

  1. The stream processing platform activates and registers a callback URL with each instance of the IoT platform
  2. A high volume of device generated events is received by the first tier of the EPN consisting of the IoT Platform.
  3. After filtering, correlation, aggregation, and enhancement, a much smaller number of events is sent to the next tier of the EPN composed of the stream processing platform.
  4. After filtering, correlation, aggregation, and enhancement, an appropriate message is output by the second tier.

Notes

  1. The events are received in the Cloud, but the devices generating the events could be on premises as well.
  2. The number of instances of the IoT platform in the initial tier is shown as two, but could be any number.
  3. The number of instances of IoT platform in the initial tier is determined by the incoming number of events and/or the geographic location of the devices generating events. For example, if devices are located worldwide, there could be an instance of the IoT platform hosted in each geographic region.
  4. Using the IoT platform in the first tier provides the device management capability that the stream processing platform does not have.
  5. Using an application integration platform as the second tier of event processing allows the other capabilities (e.g. orchestration, messaging, adapters) to be used post event processing.
  6. This interaction shows two tiers of event processing, but that could be more than two with IoT platform as the first tier and
    stream processing platform in all subsequent tiers.
  7. The stream processing platform must be registered as a device in the IoT platform before it can activate and register a callback.

Cloud and On-Premise

EPN composed of stream processing platform as well as on-premises stream processing platform. 

Interactions

  1. A high volume of events is sent to the Stream Processing Platform
  2. A high volume of events is sent from the on-premises Stream Processing Platform.
  3. After filtering, correlation, aggregation, and enhancement, a much smaller number of events is sent the second tier of event processing.
  4. After filtering, correlation, aggregation, and enhancement, an appropriate message is output by the second tier.

Notes

  1. Using both Cloud and on-premises stream processing allows events to be processed closest to the source of the events.
  2. Using an application integration platform as the second tier of event processing allows the other capabilities (e.g. orchestration, messaging, adapters) to be used post event processing.
  3. The number of instances of Stream Processing Platform in the first tier is shown as two, but could be any number.
  4. This interaction shows two tiers of event processing, but that could be more than two, with each tier Cloud based, on premise, or a mixture.