The modern IT infrastructure requires to react immediately to many kinds of critical events that must be collected for critical proceedings of heterogeneous devices on any layer of the IT infrastructure. Collecting and processing more and more monitoring events (e.g., events, logs, performances) to react in near real-time becomes more challenging with exponentially growing environments.
Complex Event Processing (CEP) overcomes these problems by passively receiving and reacting (instead of actively collecting) to events (e.g., monitoring checks, emails, telegram messages) and, based on their typology and content, possibly trigger various operations (e.g, notifications, logging to files, annotating events in a time series database).
In this talk, I will present Tornado, our Open Source CEP solution developed in the Rust programming language that provides:
- Safety: compilation time guarantees protecting the developer from various kinds of common errors without losing performance
- Optimized resource usage: In cloud context where users pay per usage, resource optimization is extremely important
- Scalability: micro-service architecture
- Modularity: developers can easily add new event processor or output actions executors
I will explain how we met these requirements leveraging the three main components of the Tornado architecture:
- A Tornado Collector receives and converts heterogeneous external events into a homogeneous Tornado Event structure, and forwards them to the Tornado Engine. Examples of collectors are Emails, DNS Cloud monitoring (AWS, Azure, Cisco/Meraki, etc.)
- The Tornado Engine receives and processes Tornado events produced by Collectors and generates an outcome defined by Rules arranged in a processing tree which match conditions, extract fields and produce actions.
- A Tornado Executor receives the actions produced from the Engine and triggers the associated executable instructions. Rest API calls and File archive, custom scripts are examples of executors.
