How is 𝗖𝗜/𝗖𝗗 𝗽𝗿𝗼𝗰𝗲𝘀𝘀 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗳𝗼𝗿 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀 compared to 𝗥𝗲𝗴𝘂𝗹𝗮𝗿 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲?

The important difference that the Machine Learning aspect of the projects brings to the CI/CD process is the treatment of the Machine Learning Training pipeline as a first class citizen of the software world.

➡️ CI/CD pipeline is a separate entity from Machine Learning Training pipeline. There are frameworks and tools that provide capabilities specific to Machine Learning pipelining needs (e.g. KubeFlow Pipelines, Sagemaker Pipelines etc.).
➡️ ML Training pipeline is an artifact produced by Machine Learning project and should be treated in the CI/CD pipelines as such.

What does it mean? Let’s take a closer look:

Regular CI/CD pipelines will usually be composed of at-least three main steps. These are:

𝗦𝘁𝗲𝗽 𝟭: Unit Tests - you test your code so that the functions and methods produce desired results for a set of predefined inputs.

𝗦𝘁𝗲𝗽 𝟮: Integration Tests - you test specific pieces of the code for ability to integrate with systems outside the boundaries of your code (e.g. databases) and between the pieces of the code itself.

𝗦𝘁𝗲𝗽 𝟯: Delivery - you deliver the produced artifact to a pre-prod or prod environment depending on which stage of GitFlow you are in.

What does it look like when ML Training pipelines are involved?

𝗦𝘁𝗲𝗽 𝟭: Unit Tests - in mature MLOps setup the steps in ML Training pipeline should be contained in their own environments and Unit Testable separately as these are just pieces of code composed of methods and functions.

𝗦𝘁𝗲𝗽 𝟮: Integration Tests - you test if ML Training pipeline can successfully integrate with outside systems, this includes connecting to a Feature Store and extracting data from it, ability to hand over the ML Model artifact to the Model Registry, ability to log metadata to ML Metadata Store etc. This CI/CD step also includes testing the integration between each of the Machine Learning Training pipeline steps, e.g. does it succeed in passing validation data from training step to evaluation step.

𝗦𝘁𝗲𝗽 𝟯: Delivery - the pipeline is delivered to a pre-prod or prod environment depending on which stage of GitFlow you are in. If it is a production environment, the pipeline is ready to be used for Continuous Training. You can trigger the training or retraining of your ML Model ad-hoc, periodically or if the deployed model starts showing signs of Feature/Concept Drift.