ML Project Life Cycle Part 3 – Deployment and Monitoring

Once the system is deployed, the job is only 50% done. The other 50% ahead is now the substantial effort required to monitor and maintain the system, which can be far more taxing than monitoring and maintaining traditional software.

To start, ML systems are typically not deployed broadly. Rather, start small and progressively roll-out to more users as you have success. Options include:

• Canary deployment: Canary deployment is a strategy for releasing a new version of software or a machine learning model to a small group of users first (like 5% of the total users) to test its performance. If everything works well with this smaller group, the new version is then gradually rolled out to more users. The goal is to catch any potential issues early with a smaller group before fully deploying it to everyone, ensuring a safer and smoother update process.

• Blue/green deployment: Blue/green deployment is a method used to safely switch from an old system to a new one. In this approach, two environments are created: “blue” (the old system) and “green” (the new system). All user traffic initially goes to the blue environment. When the green environment is ready, traffic is gradually switched over to it. The blue environment stays live in case anything goes wrong, allowing for a quick rollback if needed. Once everything works perfectly on the green environment, the blue environment can be shut down.

• Shadow mode, decision support, partial automation: In programs where the goal is to automate what was previously manual work, consider first running the algorithm alongside the human worker (shadow mode, to see if the ML’s actions closely match what the human does), providing advice to the human (decision support), and perhaps aiming for only partial automation of specific subsets of tasks rather than trying to take on complete automation right away.

Models eventually lose their predictive power, called “model drift” or “model decay”. This can be due to data drift and concept drift. Data drift is when the live data coming in to your model no longer has the expected distribution. For example, in an e-commerce store selling shoes, data drift may occur as the seasons change and customers’ preferences for different types of shoes change. Concept drift is when the basic concepts your model has been trained to recognize start to be labelled differently altogether. For example, fashion changes and the shoes of today no longer look like shoes the model was trained to recognize a few years ago.

Data drift and concept drift are two common challenges faced in machine learning models that can significantly impact their performance. While they might seem similar, they are distinct concepts.

• Definition: Changes in the distribution of the input data over time.
• Example: Imagine a model trained on historical customer data to predict future purchases. If the demographics of the customer base shift significantly (e.g., younger customers becoming more prevalent), the distribution of input features (age, income, etc.) will change. This is data drift.

Scenario: A model is trained to predict house prices based on square footage and number of bedrooms. Initially, the data is primarily for suburban houses.

Original Data:

New Data (Data Drift): The model starts receiving data for urban houses.

Explanation: The distribution of the input features (square footage and number of bedrooms) has shifted. The model was trained on suburban data, but now it’s encountering urban data with smaller houses and higher prices per square foot. This is data drift.

• Definition: Changes in the underlying relationship between the input data and the target variable.
• Example: Consider a model predicting loan default risk. If a major economic event (like a recession) occurs, the relationship between loan characteristics (e.g., income, credit score) and default risk might change. This is concept drift.

Scenario: A model is trained to predict house prices based on square footage, number of bedrooms, and near to schools.

Original Data:

New Data (Concept Drift): The local government introduces a new zoning law that limits the construction of new schools.

Explanation: The relationship between the features (square footage, number of bedrooms, and proximity to schools) and the target variable (house price) has changed. The zoning law has made schools more scarce, increasing their value as a feature. As a result, houses closer to schools are now worth more than they were before, even if they have the same square footage and number of bedrooms. This is concept drift.

In essence, data drift is about changes in the what of the data, while concept drift is about changes in the how the data relates to the target.

Since both drifts involve a statistical change in the data, the best approach to detect them is by monitoring its statistical properties, the model’s predictions, and their correlation with other factors. Drifts can be slow or very sudden. Usually consumer changes are slow trends over time, whereas in a B2B context an entire enterprise can shift quite suddenly (If for example a company tells all of its workers to change their behaviour one day, or installs a new company-wide software).

Regular monitoring is essential for maintaining the health and performance of a system. By implementing a robust monitoring strategy that encompasses software metrics, input metrics, and output metrics, we can detect issues proactively and ensure optimal operation.

Software metrics are crucial for assessing the performance and reliability of the system. Key metrics to monitor include:

• Memory Usage: Track the amount of memory consumed by the application to identify potential memory leaks or resource exhaustion.
• Server Load: Monitor the server load to understand how well the system is handling incoming requests. High server load may indicate the need for scaling or optimization.
• Throughput: Measure the number of transactions or requests processed over a given time frame to evaluate the system’s efficiency.
• Latency: Monitor the response time of the system to ensure that it meets user expectations. High latency can lead to user dissatisfaction and increased abandonment rates.

Input metrics provide insights into the quality and nature of the data being processed. Important input metrics to monitor include:

• Input Length: Track the average length of input queries to identify trends or anomalies.
• Input Values: Monitor the diversity and frequency of input values to ensure relevance and coverage.
• Volume of Input: Measure the total number of inputs received to assess system load and capacity.
• Number of Missing Values: Keep track of missing or incomplete data points to maintain data quality.
• Average Image Brightness: For image-based inputs, monitor average brightness to detect potential issues with sensor performance or data quality.

Output metrics help evaluate the effectiveness of the system in delivering user value. Key output metrics to monitor include:

• Clickthrough Rates (CTR): Measure the percentage of users who click on recommended results to assess relevance and engagement.
• Null Value Returns: Track the frequency of null or empty responses to gauge system performance and accuracy.
• User Query Repeats: Monitor how often users repeat queries or switch to manual typing, indicating potential dissatisfaction or search failures.
• User Engagement: Measure metrics such as session duration and user retention to assess overall user experience.

In conclusion, deploying machine learning models into production environments represents a significant milestone in transforming data-driven insights into actionable, real-world applications. Successful production deployment requires careful consideration of various factors including model performance, scalability, integration with existing systems, and ongoing maintenance. Ensuring robust monitoring, regular updates, and a clear strategy for handling model drift and evolving data patterns are essential for maintaining the efficacy and relevance of machine learning solutions over time. By prioritizing these elements, organizations can effectively harness the power of machine learning to drive innovation, improve decision-making, and deliver tangible business value. In this way ,Machine learning project lifecycle completes from scoping to deployment thank you for the valuable time to all the readers.