Source: towardsdatascience.com
Why ML should be written as pipelines from the get-goCategory: Data, automation, yaml
Today, Machine Learning powers the top 1% of the most valuable organizations in the world (FB, ALPH, AMZ, N etc).
The graphic below shows an admittedly simplified representation of a typical setup for machine learning: There are three stages to the above process: Each of these stages requires different skills, tooling, and organization.
If training happens in isolation from the deployment strategy, that is never going to translate well in production scenarios — leading to inconsistencies, silent failures, and eventually failed model deployments.
However, the devil is really in the details — how do you give data scientists the flexibility they need for experimentation in a framework that is robust enough to be taken all the way to production?
Additionally, the interfaces exposed for individual steps are mostly set up in a way to be easy to extend in an idempotent, and therefore a distributed, manner.
The graphic below shows an admittedly simplified representation of a typical setup for machine learning: There are three stages to the above process: Each of these stages requires different skills, tooling, and organization.
If training happens in isolation from the deployment strategy, that is never going to translate well in production scenarios — leading to inconsistencies, silent failures, and eventually failed model deployments.
However, the devil is really in the details — how do you give data scientists the flexibility they need for experimentation in a framework that is robust enough to be taken all the way to production?
Additionally, the interfaces exposed for individual steps are mostly set up in a way to be easy to extend in an idempotent, and therefore a distributed, manner.
Related Articles
Community Partners
DevOps Careers