When it comes to data science projects, rigorously defining success criteria and establishing a way to measure them is key to overall user adoption.
Consider a project that aims to optimize prices on products. Is your goal to maximize total company revenue, which is appropriate for market share gain, or to maximize profits? And if you want to maximize profits, do you want to also measure long-term customer retention? How will retention be measured? And how will you define project success if short-term profits increase at the expense of long-term loyalty?
When you consider multiple performance indicators, success becomes even harder to nail down. Imagine a grocer of short-life merchandise that wants to optimize its in-store inventory. The grocer determines that it wants to evaluate success based on a) increasing on-shelf availability, b) reducing inventory, and c) reducing spoilage. All three are worthy goals.
- On-shelf availability directly impacts customer satisfaction and long-term customer loyalty
- Reduced inventory lowers inventory carrying costs and frees up capital
- Reduced spoilage increases margins and is socially conscientious
Before this inventory optimization project starts, the grocer needs to identify clear guidelines for what constitutes success so that it can adjust when the metrics do not all move in the desired direction.
Measuring effectiveness means observing the outcomes of two conditions - one where the insight/recommendations from the data science work are used (known as the test) and one where they are not (the control), all else equal. Having all else equal is a critical consideration in any test and control. It is imperative that we neutralize factors that influence the outcomes in test/control apart from the fact that one uses the recommendations and the other does not.
You can create a control group by looking at time, location, or product dimension in situations where the recommendations are not being applied. The randomized the selection, the better, but business realities typically dictate how test and control groups are created.
One common long-term issue crops up when a project is deemed successful and recommendations are rolled out to the entire business: the present-day control is lost. To account for this, often businesses will compare the present (test) to the past (control), but this is only effective as long as we believe the past is still comparable. Other solutions include attempting to simulate what the control would have been based on rules for how decisions were previously made. This is difficult when those previous decisions involved human evaluation/interpretation, but it is very doable when we have one machine replacing another.
When implementing a new measurement process, one of the most common pain points I see is a difference in opinions between the business decision maker and the end user. Gaining buy-in from the end user is critical for user adoption, and I've seen more than one test/control measure suffer as a result of low adoption numbers that are unrelated to the quality of the science implemented.
One way to engage the end user is to create measures of project success that impact their day-to-day activities. For example, how much faster can they now accomplish existing tasks? If the project has an interface, what is the user experience like? Do they enjoy doing their job more when making use of the insights from the project? While this may take more time upfront, ultimately, this will mean better measurement for how well (or poorly) the project is going, allowing issues to be dealt with more readily.
From there, setting up a test and control group that can be monitored for the KPIs of interest, where the test group is impacted by the project and the control is not, is a good way to ensure you're staying on track. As an extra layer of effectiveness, tasking one person with reporting on test vs. control via an executive dashboard/summary that circulates to key individuals on a regular interval can cut down on confusion and disagreement on the team.
See the resources below to learn how data science and machine learning projects are helping to transform analytics and processes in industries and beyond.