Understanding overfitting: an inaccurate meme in supervised learning

Feed: R-bloggers.
Author: msuzen.

1. Generalized: A model should not depend on the dataset. This step is called model validation.

Figure 1: A workflow for model validation and  selection in supervised learning.

Generalization of a model can be measured by goodness-of-fit. It essentially tells us how good our model (chosen function) explains the dataset. To find a minimally complex model requires comparison against another model. 

Overfitting of models is widely recognized as a concern. It is less recognized however that overfitting is not an absolute but involves a comparison. A model overfits if it is more complex than another model that fits equally well.

The important point here what do we mean by complex model, or how can we quantify model complexity? Unfortunately, again there is no unique way of doing this. One of the most used approaches is that a model having more parameters is getting more complex. But this is again a bit of a meme and not generally true. One could actually resort to different measures of complexity. For example, by this definition $f_{1}(a,x)=ax$ and $f_{2}(a,x)=ax^2$ have the same complexity by having the same number of free parameters, but intuitively $f_{2}$ is more complex, while it is nonlinear. There are a lot of information theory based measures of complexity but discussion of those are beyond the scope of our post. For demonstration purposes, we will consider more parameters and degree of nonlinearity as more complex a model.

Figure 2: Simulated data and the non-stochastic part of the data.

Hand on example We have intuitively covered the reasons behind how we can’t resolve model validation and judge overfitting simultaneously. Now try to demonstrate this with a simple dataset and models, yet essentially capturing the above premise.
A usual procedure is to generate a synthetic dataset, or simulated dataset, from a model, as a gold standard and use this dataset to build other models. Let’s use the following functional form, from classic text of Bishop, but with an added Gaussian noise $$ f(x) = sin(2pi x) + mathcal{N}(0,0.1).$$ We generate large enough set, 100 points to avoid sample size issue discussed in Bishop’s book, see Figure 2. Let’s decide on two models we would like to apply to this dataset in supervised learning task. Note that, we won’t be discussing Bayesian interpretation here, so equivalency of these model under a strong prior assumption is not an issue as we are using this example for ease of demonstrating the concept. A polynomial model of degree $3$ and degree $5$, we call them $g(x)$ and $h(x)$ respectively are used to learn from the simulated data. $$g(x) = a_{0} + a_{1} x + a_{2} x^{2} + a_{3} x^{3}$$ and $$h(x) = b_{0} + b_{1} x + b_{2} x^{2} + b_{3} x^{3} + b_{4} x^{4} + b_{5} x^{5} + b_{6} x^{6}.$$

Figure 3: Overtraining occurs at around after 40 percent of the data usage for g(x).

Overtraining is not overfitting Overtraining means a model performance degrades in learning model parameters against an objective variable that effects how model is build, for example, an objective variable can be a training data size or iteration cycle in neural network. This is more prevalent in neural networks (see Dayhoff 2011). In our practical example, this will manifest in hold out method to measure RMSD in modelling with g(x). In other words finding an optimal number of data points to use to train the model to give a better performance on unseen data, See Figure 3 and 4.

Overfitting with low validation error We can also estimate 10-fold cross-validation error, CV-RMSD. For this sampling, g and h have 0.13 and 0.12 CV-RMSD respectively. So as we can see, we have a situation that more complex model reaches similar predictive power with cross validation and we can not distinguish this overfitting by just looking at CV-RMSD value or detecting ‘overtraining’ curve from Figures 4. We need two models to compare, hence both Figure 3 and 4, with both CV-RMSD values. We might argue that in small data sets we might be able tell the difference by looking at test and training error differences, this is exactly how Bishop explains overfitting; where he points out overtraining in small datasets.
Which trained model to deploy? Now the question is, we found out best performing model with minimal complexity empirically. All well, but which trained model should we use in production?
Actualy we have already build the model in model selection. In above case, since we got similar
predictive power from g and h, we obviously will use g, trained on the splitting sweet spot from Figure 3.

Figure 4: Overtraining occurs at around after 30 percent of the data usage for h(x)

Conclusion The essential message here is poor validation performance would not guarantee the detection of an overfitted model. As we have seen from examples using synthetic data in one dimension. Overtraining is actually what most practitioners mean when they use the term overfitting.

Outlook As more and more people are using techniques from machine learning or inverse problems, both in academia and industry, some key technical concepts are deviated a bit and take different definitions and meaning for different people, due to the fact that people learn some concepts not from reading the literature carefully but from their line managers or senior colleagues verbally. This creates memes which are actually wrong or at least creating lots of confusion in jargon. It is very important for all of us as practitioners that we must question all technical concepts and try to seek origins from the published scientific literature and not rely entirely on verbal explanations from our experienced colleagues. Also, we should strongly avoid ridiculing question from colleagues even they sound too simple, at the end of the day we don’t stop learning and naive looking questions might have very important consequences in fundamentals of the field.

Figure 5: A deployed models h and g  on the testing set with the original data.

Appendix: Reproducing the example using R

The code used in producing the synthetic data, modelling step and visualising the results can be found in github [repo]. In this appendix, we present this R code with detailed comments, but visualisation codes are omitted, they are available in the github repository.

R (GNU S) provides very powerful formula interface. It is probably the most advanced and expressive formula interface in statistical computing, of course along with S.







Above two polynomials can be expressed as formula and as well as a function where we can evaluate.

1
2
3
4
5
6
7
8
9

#'
#' Two polynomial models: g and h, 3rd and 5th degree respectively.
#' 
g_fun  function(x,params) as.numeric(params[1]+x*params[2]+x^2*params[3]+x^3*params[4])
h_fun  function(x,params) as.numeric(params[1]+x*params[2]+x^2*params[3]+params[4]*x^3+
params[5]*x^4+params[6]*x^{5}+
params[7]*x^{6})
g_formula  ysim ~ I(x) + I(x^2) + I(x^3) 
h_formula  ysim ~ I(x) + I(x^2) + I(x^3) + I(x^4) + I(x^5) + I(x^6)  

A learning from data will be achieved with lm function from R,

1
2
3
4
5
6
7
8

#'
#' Given data.frame with x and ysim, and an R formula with ysim=f(x), 
#' fit a linear model 
#'
get_coefficients  function(data_portion, model_formula) {
model  lm(model_formula, data=data_portion)
return(model$coefficients)
}

and the resulting approximated function can be applied to new data set with the following helper functions with measuring RMSD as a performance metric.

1
2
3
4
5
6
7
8
9

#'
#' Find the prediction error for a given model_function and model_formula
#'
lm_rmsd  function(x_train, y_train, x_test, y_test, model_function, model_formula) {
params  get_coefficients(data.frame(x=x_train,ysim=y_train), model_formula)
params[as.numeric(which(is.na(params)))]  0 # if there is any co-linearity
f_hat   sapply(x_test, model_function, params=params)
return(sqrt(sum((f_hat-y_test)^2)/length(f_hat)))
}

We can generate a simulated data as we discussed above by using runif.

 1
2
3
4
5
6
7
8
9
10
11
12
13

#' 
#' Generate a synthetic dataset
#' A similar model from Bishop : 
#' 
#' f(x) = sin(2pi*x) + N(0, 0.1)
#'  
set.seed(424242)
f        function(x) return(sin(2*pi*x))
fsim     function(x) return(sin(2*pi*x)+rnorm(1,0,0.1))
x        seq(0,1,1e-2)
y        sapply(x,f)
ysim     sapply(x,fsim)
simdata  data.frame(x=x, y=y, ysim=ysim)

To detect overtraining we can split the data in different places in increasing training size, and measure the the performance on the training data itself and unseen test data.

 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

#'
#' Demonstration of overtraining with g
#' 
#'
set.seed(424242)
model_function  g_fun
model_formula   g_formula
split_percent   seq(0.05,0.95,0.03)
split_len       length(split_percent)
data_len        length(simdata$ysim) 
splits          as.integer(data_len*split_percent)
test_rmsd       vector("integer", split_len-1)
train_rmsd      vector("integer", split_len-1)
for(i in 2:split_len) {
train_ix  sample(1:data_len,splits[i-1]) 
test_ix   (1:data_len)[-train_ix]
train_rmsd[i-1]   lm_rmsd(simdata$x[train_ix], 
simdata$ysim[train_ix], 
simdata$x[train_ix], 
simdata$ysim[train_ix],
model_function, 
model_formula)
test_rmsd[i-1]   lm_rmsd(simdata$x[train_ix], 
simdata$ysim[train_ix], 
simdata$x[test_ix], 
simdata$ysim[test_ix],
model_function, 
model_formula)
}
rmsd_df             data.frame(test_rmsd=test_rmsd, train_rmsd=train_rmsd, percent=split_percent[-1]) 
rmsd_df2            melt(rmsd_df, id=c("percent"))
colnames(rmsd_df2)  c("percent", "Error_on", "rmsd")
rmsd_df2$test_train  as.factor(rmsd_df2$Error_on)

And the last portion of the code does 10-fold cross-validation.

 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#' CV for g(x) and h(x)
split_percent   seq(0,1,0.1)
split_len       length(split_percent)
data_len        length(simdata$ysim) 
splits          as.integer(data_len*split_percent)
cv_rmsd_g       0
cv_rmsd_h       0
for(i in 2:split_len) { # 10-fold cross validation
test_ix   (splits[i-1]+1):splits[i]
train_ix    (1:data_len)[-test_ix]
x_train    simdata$x[train_ix]
y_train    simdata$ysim[train_ix]
x_test     simdata$x[test_ix]
y_test     simdata$ysim[test_ix]
cv_rmsd_g   lm_rmsd(x_train,
y_train,
x_test,
y_test,
g_fun, 
g_formula)+cv_rmsd_g
cv_rmsd_h    lm_rmsd(x_train,
y_train,
x_test,
y_test,
h_fun, 
h_formula)+cv_rmsd_h
}
cat("10-fold CV error G = ", cv_rmsd_g/split_len,"n") # 0.1304164
cat("10-fold CV error H = ", cv_rmsd_h/split_len,"n") # 0.1206458

Related