Dear friends,
Last week, I talked about how best practices for machine learning projects are not one-size-fits-all, and how they vary depending on whether a project uses structured or unstructured data, and whether the dataset is small or big. Another dimension that affects best practices is which phase of development a project is in: proof of concept or production.
During the proof of concept (POC) phase, the primary goal is to determine if a system is worth building and deploying. During this phase, you might ask:
- For a visual inspection system, can we build a model that matches the performance of human inspectors?
- For face detection, can we build an edge (on-device) implementation that’s nearly as accurate as the cloud version while avoiding an unacceptable level of bias?
- For a sales-lead scoring application, how much will estimated revenue increase by using machine learning to prioritize leads?
When building a POC, my goal is to move fast. We’ve all been told we should build replicable, robust, and scalable systems — but when I haven’t even determined if a project is technically feasible, I often trade replicability for speed. I hope I don’t get too much hate mail for this, but if it buys you speed, it is okay to hard-code parameters, compute key variables in a Jupyter notebook, use local copies of data, and operate with lightweight code review or versioning processes.
If you already have a platform for experimentation, you may be able to build POCs in a systematic and robust way without sacrificing speed. But if you don’t, avoid over-investing in infrastructure at this stage. Instead, focus on getting the key information you need: whether this project is worth taking to production.
(Those of you who are familiar with the lean startup philosophy will see the parallel to building a minimum viable product, which is often a clunky piece of software that helps validate or falsify a hypothesis.)
In contrast, during the production phase, the goal is to build and deploy a system that generates practical value. I might go back to the messy POC and make sure that every step is replicable and documented. I put a lot of thought into scalable data pipelines, monitoring systems, and reliability.
For example, if a researcher wrote preprocessing routines (say, a sequence of scripts and regexps to remove data associated with spam accounts), these now need to be documented, tested, and incorporated into the system. You’ll likely want to document everything to make sure models can be replicated and maintained: hyperparameters, model choices, data provenance (where the data came from), data lineage (how it was processed). During this phase, tools like TensorFlow Transform and Apache Beam can be lifesavers.
If you’re building a project, don’t confuse the POC and production phases! Both are important, but the best practices depend on whether you’re deciding as quickly as possible if a project is worth putting into production or building a system that delivers real results to real users.
Keep learning!
Andrew