Besides prediction results, there are several other important factors to consider when comparing machine learning prototypes. These factors provide valuable insights into the overall suitability and effectiveness of the models in real-life scenarios. By focusing not only on predictive power, your chances of getting your machine learning solution to production increase.
The factors are grouped in four categories: maintenance, implementation complexity, costs, and business requirements. Up front, the project team should decide which factors are important for the project. During creation of the prototype solutions, developers can already take notes about the different factors.
How hard is it to collect data or to perform feature engineering? Do you use many different libraries and is the model sensitive to parameter tuning? Is the project using standard API’s you can place in a pipeline? These aspects make a solution easier or harder to maintain.
If your data is coming from many different internal and external sources, it presents a disadvantage compared to a solution that solely relies on internal company data. The reason is that you cannot completely rely on the external sources to remain unchanged, and any alterations or updates in those sources would require refactoring or adjustments in your solution. This is an example of a maintenance issue that can arise.
Another part of maintainability is monitoring. This involves tracking metrics, detecting anomalies or degradation in performance, and debugging issues that may arise. Some models provide robust monitoring and debugging capabilities. This can be an advantage over other models.
Implementation complexity measures the difficulty and effort involved in deploying a model into a production system. It takes into account factors such as the availability of necessary libraries, the complexity of the model architecture, and the compatibility with existing infrastructure. A model that is straightforward to implement and integrate into existing systems can save valuable time and resources during the deployment phase.
Another factor that can influence implementation complexity is familiarity with the approach. Choosing a model that aligns with the team’s skill set can significantly impact the development timeline.
It’s easy to develop a model that costs a lot of money. Costs are an important factor for almost any company. If you need an expensive license for a certain solution, you should be able to justify why that license is worth the costs.
You can spend money on data acquisition, data storage, (re)training, inference or licenses and subscriptions. Also the resources for developing the solution have a certain cost. By making an educated guess about these costs upfront for every solution, it becomes another factor to compare solutions on.
If the costs exceed the budget (or the value the model will bring), you should reconsider an approach. It can also be the case that two solutions score the same on all factors except the costs. In that case the choice is easy, the cheaper solution is the better one.
Last but not least are the requirements of the business. They can come in many forms; here are some common ones:
- Interpretability
Being able to understand and explain specific predictions is a vital part of some business processes. In that case, a model that is easy to explain can be of higher importance than predictive power. If interpretability is important, you should try to keep the model simple. You can experiment with different interpretation techniques and score how easy it is to use the technique together with the model. - Time-to-market
In competitive industries or when addressing time-sensitive opportunities, the speed at which the model can be developed and deployed may be a critical business requirement. Minimizing the time-to-market can be essential to gain a competitive advantage. Models that can be developed and deployed quickly, with minimal iterations or complex preprocessing steps, can be advantageous in such scenarios. - Regulatory compliance
Certain industries, such as finance, healthcare, and insurance, have strict regulations and compliance standards. Business requirements may include ensuring that the selected models adhere to these regulations, such as data privacy laws (e.g., GDPR), industry-specific guidelines, or ethical considerations. Models must be compliant with relevant regulations to avoid legal and reputational risks. - Real-time inference
Some applications require (near) real-time predictions, where decisions need to be made within strict time constraints. Business requirements may specify the need for low-latency models that can quickly process incoming data and generate predictions in real-time. Models that offer efficient real-time inference capabilities are crucial for time-sensitive applications like fraud detection or recommendation systems.
This post originally appeared on TechToday.