Applying k-fold Cross Validation model using caret package

Question

Let me start by saying that I have read many posts on Cross Validation and it seems there is much confusion out there. My understanding of that it is simply this:

Answer 1

An important thing to be noted here is not confuse model selection and model error estimation.

You can use cross-validation to estimate the model hyper-parameters (regularization parameter for example).

Usually that is done with 10-fold cross validation, because it is good choice for the bias-variance trade-off (2-fold could cause models with high bias, leave one out cv can cause models with high variance/over-fitting).

After that, if you don't have an independent test set you could estimate an empirical distribution of some performance metric using cross validation: once you found out the best hyper-parameters you could use them in order to estimate de cv error.

Note that in this step the hyperparameters are fixed but maybe the model parameters are different accross the cross validation models.