Pipeline Patterns

Today we cannot imagine our CI\CD processes without pipelines. They’re everywhere: building, linting, testing, verifying compliance, deploying, and even handling maintenance tasks.

Have you ever seen the internals of those pipelines? I've seen: it's often a full mess.
So it's not a big surprise that someone started to think about how to write pipelines in resource efficient and easy to support way. That's exactly one of the topics from recent NDC Oslo conference: Pipeline Patterns and Antipatterns by Daniel Raniz Raneland.

It may not be any rocket science, but there is a good set of useful recipes:
🔸 Right pipeline for the job: Select only required steps for the task. For example, in build pipeline we can execute unit tests on PR and on the main, but we should not execute them on nightly CI with integration tests.
🔸 Conditional steps: Define a logic to skip not needed steps. For example, if you change only docs, you don't need to run build and tests.
🔸 Steps results reuse: Use artifacts from one step as the input to another steps.
🔸 Fail fast: Put steps that are failed more frequently to the beginning of the pipe.
🔸 Parallel run: Execute steps in parallel where it's possible.
🔸 Isolation: Result of one pipeline should not affect the results of the another.
🔸 Artifacts Housekeeping: Define cleanup policies for the artifacts.
🔸 Reasonable HWE: Carefully define required HWE to execute pipeline steps.

The key idea from the talk is that we should treat pipelines as any other software and we should apply the same architecture principles and best practices as for any other application.

#engineering