The Data Gauge R&R
“Ay, there’s the rub”
It's your first day on the job at Acme Medical Catheters, a manufacturing plant. As you walk through the shop, you notice how complicated the manufacturing lines are.
"I mean, they're just vascular access catheters, how complex could a tube be?" you chuckle to yourself.
After rounding the bend at the end of your tour, your manager gives you your first task; reduce the variability in the outer diameter of the catheter.
"Alright" you say to yourself, "let's get to it".
After several days of meticulously observing the lines, the workers, and taking copious of measurements yourself, you come to an odd conclusion. You check your spreadsheet for the 5th time.
"Huh, they're already within spec." Why, then, were you put on this task? As expected, it's not that simple. A quick perusal over the quality logs shows dramatic historical variation in the outer diameter measurements.
Then the most important question of your month surfaces:
"Where, then, does the variability come from?"
Enter the gauge R&R.
In any field where predictability comes into play, variability is key. Quantifying and attributing the randomness in our results is the first step in gaining control over the outcome. If we want to have more confidence in our results, we need to have more control. To have more control, the attribution of the sources of variability is essential, otherwise our efforts may be totally misinformed and ineffective. This attribution creates the clarity necessary to identify the most effective action we can take.
Clarity.
In manufacturing, and in particular the observation of manufacturing results (i.e. measuring), the Gauge R&R (repeatability and reproducibility) is an indispensable staple. This statistical tool is used to asses how reliable a measurements system is. Importantly, it's purpose is to determine the source of the variation we see in the measurements; whether it is due to the system itself (an inaccurate or imprecise "gauge"), or due to the actual variation in the part being measured.
Determining the source of the variability is essential in gaining the clarity needed to take effective action. If our goal is to increase our confidence that a part is within specifications, we need to know what to influence in order to achieve that goal. The Gauge R&R gives us that clarity.
Though a little more abstract, within ML development the mental model of a Gauge R&R is equally indispensable. A key component to this indispensability is explainability. Where does the variability in our output come from? Why does a given sample produce a certain result? Why does another, similar sample, product a different result? Explainability plays an important part for many a successful model deployments in the wild. It's crucial, however, for medical inference development.
How do we have confidence in our assessment of the risks associated with our device? By having greater control over their behavior. How can we increase our control on the models outputs? By having greater clarity on what causes unpredictability. Once we have clarity, we can take action to mitigate the unpredictability.
Volumes of journal articles, books, and blog posts have detailed nearly inumerable attempts to increase model explainability, as such going through these in detail is out of scope for this essay. That said, whether through ablation studies, latent space visualizations, manual outlier inspection, SHAP plots, cross validation, or other means, our goal should be to increase our clarity into the source of variability. Is it from samples that are at the edge of our sensor's range? From class imbalance? From noisy labels? Only after confidently indentifying the sources of variance are we then able to increase our control over the model behavoir, thereby increasing our confidence in the risks involved with using our device.
- Increasing confidence requires control.
- increasing control requires clarity.
- increasing clarity creates agency.