Minimum maximum accuracy, mean absolute percent error, median absolute error, root mean square error, coefficient of variation, and Efron's pseudo r-squared

Produces a table of fit statistics for multiple models.

Usage

accuracy(fits, plotit = FALSE, digits = 3, ...)

Arguments

fits: A series of model object names. Must be a list of model objects or a single model object.
plotit: If TRUE, produces plots of the predicted values vs. the actual values for each model.
digits: The number of significant digits in the output.
...: Other arguments passed to plot.

Value

A list of two objects: The series of model calls, and a data frame of statistics for each model.

Details

Produces a table of fit statistics for multiple models: minimum maximum accuracy, mean absolute percentage error, median absolute error, root mean square error, normalized root mean square error, Efron's pseudo r-squared, and coefficient of variation.

For minimum maximum accuracy, larger indicates a better fit, and a perfect fit is equal to 1.

For mean absolute error (MAE), smaller indicates a better fit, and a perfect fit is equal to 0. It has the same units as the dependent variable. Note that here, MAE is simply the mean of the absolute values of the differences of predicted values and the observed values (MAE = mean(abs(predy - actual))). There are other definitions of MAE and similar-sounding terms.

Median absolute error (MedAE) is similar, except employing the median rather than the mean.

For mean absolute percent error (MAPE), smaller indicates a better fit, and a perfect fit is equal to 0. The result is reported as a fraction. That is, a result of 0.1 is equal to 10 percent.

Root mean square error (RMSE) has the same units as the predicted values.

Normalized root mean square error (NRMSE) is RMSE divided by the mean or the median of the values of the dependent variable.

Efron's pseudo r-squared is calculated as 1 minus the residual sum of squares divided by the total sum of squares. For linear models (lm model objects), Efron's pseudo r-squared will be equal to r-squared. For other models, it should not be interpreted as r-squared, but can still be useful as a relative measure.

CV.prcnt is the coefficient of variation for the model. Here it is expressed as a percent. That is, a result of 10 = 10 percent.

Model objects currently supported: lm, glm, nls, betareg, gls, lme, lmer, lmerTest, glmmTMB, rq, loess, gam, glm.nb, glmRob, mblm, and rlm.

References

https://rcompanion.org/handbook/G_14.html

Author

Salvatore Mangiafico, mangiafico@njaes.rutgers.edu

Examples

data(BrendonSmall)
BrendonSmall$Calories = as.numeric(BrendonSmall$Calories)
BrendonSmall$Calories2 = BrendonSmall$Calories ^ 2
model.1 = lm(Sodium ~ Calories, data = BrendonSmall)

accuracy(model.1, plotit=FALSE)
#> $Models
#>   Call                                                  
#> 1 "lm(formula = Sodium ~ Calories, data = BrendonSmall)"
#> 
#> $Fit.criteria
#>   Min.max.accuracy  MAE MedAE   MAPE  MSE RMSE NRMSE.mean NRMSE.median
#> 1            0.976 32.7    30 0.0244 1440   38     0.0282       0.0275
#>   Efron.r.squared CV.prcnt
#> 1           0.721     2.82
#> 

model.2 = lm(Sodium ~ Calories + Calories2, data = BrendonSmall)
model.3 = glm(Sodium ~ Calories, data = BrendonSmall, family="Gamma")
quadplat = function(x, a, b, clx) {
          ifelse(x  < clx, a + b * x   + (-0.5*b/clx) * x   * x,
                           a + b * clx + (-0.5*b/clx) * clx * clx)}
model.4 = nls(Sodium ~ quadplat(Calories, a, b, clx),
              data = BrendonSmall,
              start = list(a=519, b=0.359, clx = 2300))
              
accuracy(list(model.1, model.2, model.3, model.4), plotit=FALSE)
#> $Models
#>   Call                                                                         
#> 1 "lm(formula = Sodium ~ Calories, data = BrendonSmall)"                       
#> 2 "lm(formula = Sodium ~ Calories + Calories2, data = BrendonSmall)"           
#> 3 "glm(formula = Sodium ~ Calories, family = \"Gamma\", data = BrendonSmall)"  
#> 4 "nls(formula = Sodium ~ quadplat(Calories, a, b, clx), data = BrendonSmall, "
#> 
#> $Fit.criteria
#>   Min.max.accuracy  MAE MedAE   MAPE  MSE RMSE NRMSE.mean NRMSE.median
#> 1            0.976 32.7  30.0 0.0244 1440 38.0     0.0282       0.0275
#> 2            0.983 22.6  20.9 0.0170  713 26.7     0.0198       0.0194
#> 3            0.975 34.4  31.6 0.0257 1630 40.4     0.0300       0.0293
#> 4            0.984 22.0  20.0 0.0166  700 26.5     0.0196       0.0192
#>   Efron.r.squared CV.prcnt
#> 1           0.721     2.82
#> 2           0.862     1.98
#> 3           0.684     3.00
#> 4           0.865     1.96
#> 

### Perfect and poor model fits
X = c(1, 2,  3,  4,  5,  6, 7, 8, 9, 10, 11, 12)
Y = c(1, 2,  3,  4,  5,  6, 7, 8, 9, 10, 11, 12)
Z = c(1, 12, 13, 6, 10, 13, 4, 3, 5,  6, 10, 14)
perfect = lm(Y ~ X)
poor    = lm(Z ~ X)
accuracy(list(perfect, poor), plotit=FALSE)
#> $Models
#>   Call                 
#> 1 "lm(formula = Y ~ X)"
#> 2 "lm(formula = Z ~ X)"
#> 
#> $Fit.criteria
#>   Min.max.accuracy      MAE MedAE     MAPE      MSE     RMSE NRMSE.mean
#> 1            1.000 1.39e-16  0.00 7.71e-17 9.14e-32 3.02e-16   4.65e-17
#> 2            0.597 3.92e+00  4.36 1.03e+00 1.78e+01 4.22e+00   5.22e-01
#>   NRMSE.median Efron.r.squared CV.prcnt
#> 1     4.65e-17          1.0000 4.65e-15
#> 2     5.28e-01          0.0135 5.22e+01
#>