- August 14, 2014
- Posted by: Cheryl Rybka
- Category: Blog
By: Russ Aikman
Part 3: Confirming the Validity of the Measurement System
- In order to respond appropriately to changes in key measures, managers must develop the ability to distinguish between everyday variation (noise) and unusual variation (signal)
- Various graphical and statistical tools can be used to distinguish between signal and noise. The best tools include Box Plots and Process Behavior Charts (Control Charts)
This is the 3rd in a 3-part series on distinguishing between signal and noise. Part 1 covered the basics on what is meant by signal and noise. Part 2 addressed the risks of confusing these two phenomena, and common mistakes associated with that confusion. Part 3 covers some tools to help distinguish between the two types of variation.
Before examining data to look for signal amidst noise it is important to stress that without a good measurement system in place there is still risk of making bad business decisions. That is, if a data value is not accurate due to problems with the measurement system then a manager could conclude that something unusual has occurred (signal) when in reality it has not. The basic question a manager should ask: Can I trust the data? The old adage of ‘Garbage In, Garbage Out’ still holds.
Managers should periodically check to make sure they are working with a valid measurement system. Validation typically involves some combination of one or more of the following activities:
- Confirming operational definitions (i.e., make sure everyone agrees on a consistent way of measuring the business process)
- Checking the data for errors due to data collection or data entry
- Calibrating the measuring device (only required if a mechanical device is used such as a caliper or micrometer)
- Performing Measurement System Analysis, such as a Gage R&R Study
- Checking for bias amongst those collecting data
- Ensuring the data is representative of the process
Tools to Help Identify Signal Amidst Noise
There are a number of different tools available to help separate signal from noise. The specific tool chosen would depend on the question to be answered, how the data is collected, and how comfortable the manager is with these tools. Any good statistical software can be used to generate these charts and graphs. The most commonly used tools include:
- Run Charts, also called Time Series Plots
- Process Behavior Charts, also called Control Charts
The first two types of graphs are used to show data in aggregate. The last two types of charts allow you to display data over time, which also helps to identify trends and patterns. While both histograms and run charts make it easy to identify potential signals, boxplots and control charts offer a more analytical approach to determine true signals.
Here are examples of each chart using the same data set of 100 values:
Histogram: Signal may be shown as bars well outside the main part of the histogram
Boxplot: Outliers (Signal) displayed as asterisks
Time Series Chart: Signal may be shown by the highest (or lowest) data points
Control Chart: Signal shown by data points outside the red lines
Of these four graphs, the Control Chart is considered the most reliable tool to identify signal from noise. In fact, the tool was designed for that specific purpose.
A key skill for all managers is the ability to distinguish between signal and noise amongst process data. Before examining any process data for signals it is important to confirm that the measurement system can be trusted. Once that has been done a manager can make use of several graphical tools to perform this task. The best tool is the Control Chart, followed by the Boxplot because they combine both a graphical output and an analytical method to separate signal from noise. All statistical software packages can be used to generate these charts. Finally: The appropriate management response to process variation depends on whether a specific data value is due to signal or noise, and also whether that value is ‘good’ or ‘bad’ relative to the requirements for the process.