Pipelines

Question and objective

How do you apply processing continuously to new events? Pipelines move recurring steps such as filtering and calculation upstream of the actual analysis.

Implementation in Bytefabrik

The pipeline editor is based on Apache StreamPipes. Data streams provide the input, processors handle events, and sinks determine the output. Available elements depend on the installed extensions. Configuration maps input schema fields to an element's requirements.

Best practices for a data flow

1. Select a validated input stream and name the operational objective.
2. Add the required processing steps and check their field mappings.
3. Configure the output destination. Historical analysis requires suitable storage.
4. Test the pipeline with known data, then check the actual result.
5. Document time logic, units, and the meaning of newly calculated fields.

Example: A temperature stream is restricted to one machine and prepared for analysis. Store the processed result separately if you still need raw values for later troubleshooting.

What to consider

A filter discards events for downstream steps. Aggregation changes the resolution and meaning of values. Both should be deliberate operational decisions.

A saved pipeline draft does not mean processing is running. Check its operating status and output. When making changes, clarify whether a restart is required and how interruptions are handled.

A pipeline works on its input stream; it does not automatically recalculate previously stored history. Additional operating guidance: StreamPipes Pipelines.