Validation Approaches - Prediction Accuracy Measures
In the domain of Lean Six Sigma, the use of statistical tools and analyses is paramount for improving processes, reducing variability, and eliminating defects. Multiple Regression Analysis (MRA) is one such powerful tool, enabling practitioners to understand and model the relationship between a dependent variable and several independent variables. This article delves into a critical aspect of MRA within the context of Lean Six Sigma: Model Validation in Multiple Regression, with a specific focus on Validation Approaches and Prediction Accuracy Measures.
Introduction to Model Validation
Model validation in multiple regression is an essential step to ensure that the model accurately represents the data and can reliably predict future outcomes. Validation approaches assess the model's performance and its ability to generalize to new, unseen data. This step is crucial because a model that performs well on the training dataset but poorly on new data is of limited use in practical applications.
Validation Approaches
The primary goal of validation approaches in multiple regression analysis is to test the model's predictive power and reliability. These approaches can be broadly classified into two categories: internal validation and external validation.
Internal Validation involves techniques that use the same dataset on which the model was developed. The most common internal validation method is the split-sample method, which divides the dataset into two parts: a training set and a validation set. The model is built on the training set and tested on the validation set. Another popular internal validation technique is cross-validation, especially k-fold cross-validation, which divides the dataset into k subsets, trains the model on k-1 subsets, and validates it on the remaining subset. This process is repeated k times, with each subset used exactly once as the validation data.
External Validation refers to the evaluation of the model on a completely new dataset that was not used during the model development process. This approach provides the most rigorous test of a model's predictive power and generalizability but requires access to additional data, which may not always be available.
Prediction Accuracy Measures
Prediction accuracy measures are quantitative metrics used to assess how well a multiple regression model predicts the outcome variable. These measures help in comparing different models and in selecting the best model for practical use. Key prediction accuracy measures include:
Mean Absolute Error (MAE): It measures the average magnitude of errors in a set of predictions, without considering their direction. It's calculated as the average of the absolute differences between the predicted values and the actual values.
The chart above illustrates the concept of Mean Absolute Error (MAE). Here, the blue dots represent the true values of a dataset, while the red dots depict the predicted values. The grey dashed lines between the true and predicted values highlight the errors for individual predictions.
MAE is calculated as the average of these absolute differences (errors) between the predicted and actual values, without considering the direction of the errors.
This example yields an MAE of approximately 7.88, indicating the average magnitude of errors in the predictions. The smaller the MAE, the closer the predictions are to the actual values, reflecting a more accurate model.
Mean Squared Error (MSE): It measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value. MSE gives more weight to larger errors.
The charts above illustrate the concept of Mean Squared Error (MSE) in the context of actual vs. predicted values and the distribution of squared errors.
The first chart on the left displays a scatter plot of the actual values against the predicted values. The red dashed line represents perfect predictions where actual values equal predicted values. Points deviating from this line indicate errors in prediction, with more substantial deviations representing larger errors.
The second chart, a histogram on the right, shows the distribution of squared errors. Squared errors are the square of the differences between actual and predicted values, highlighting that MSE emphasizes larger errors due to squaring the differences. The distribution of these squared errors gives us an idea of how frequently different magnitudes of errors occur.
The calculated MSE for our sample data is approximately 0.269. This value quantifies the average of the squares of the errors, giving us a single metric to gauge the prediction accuracy of the model, with a focus on penalizing larger errors more severely than smaller ones.
Root Mean Squared Error (RMSE): It is the square root of the MSE and provides a measure of the magnitude of the error. The RMSE is beneficial because it is in the same units as the response variable, making interpretation easier.
The chart above illustrates a simple linear regression model applied to a set of observed data (in blue), the true underlying relationship between the independent and dependent variables (in green), and the predictions made by the model (in red). The Root Mean Squared Error (RMSE) is calculated based on the differences between the observed values and the model's predictions, providing a measure of the model's prediction error magnitude. The RMSE value is displayed in the chart title, offering insight into the average magnitude of the prediction errors, and is in the same units as the dependent variable, facilitating easier interpretation of the model's accuracy.
R-squared (R²) and Adjusted R-squared: R² measures the proportion of the variance in the dependent variable that is predictable from the independent variables. Adjusted R² also considers the number of predictors in the model, adjusting for the number of terms in the model.
The plots above illustrate three hypothetical scenarios regarding the impact of adding predictors to a model on R-squared (R²) and Adjusted R-squared values:
Adding Relevant Predictors: Both R² and Adjusted R² increase, demonstrating the value of including relevant predictors that contribute significantly to explaining the variance in the dependent variable. However, R² increases more quickly because it doesn't penalize the model for the number of predictors, unlike Adjusted R².
Adding Irrelevant Predictors: Here, R² shows a slight increase or remains almost constant because it does not penalize for the number of predictors. In contrast, Adjusted R² decreases, reflecting the penalty for adding predictors that do not improve the model's explanatory power. This scenario highlights the importance of Adjusted R² in identifying models burdened with unnecessary predictors.
Adding a Mix of Relevant and Irrelevant Predictors: This scenario, which is quite common in practice, shows a moderate increase in both R² and Adjusted R². However, the increase in Adjusted R² is less pronounced, illustrating its role in balancing the model's complexity against its predictive power. This balance is crucial for developing models that are both accurate and generalizable.
AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion): Both are used for model selection, with a lower value indicating a better model. They take into account the number of parameters in the model and the likelihood of the model, penalizing models with more parameters to prevent overfitting.
Conclusion
In Lean Six Sigma projects, validating multiple regression models through appropriate approaches and accurately measuring prediction accuracy are vital steps. These processes ensure that the models are reliable, generalizable, and useful for making data-driven decisions. By carefully applying validation approaches and rigorously evaluating prediction accuracy measures, practitioners can enhance process improvements, reduce variability, and contribute significantly to organizational excellence.