Reply To: Regularization in R

Home Forums Analytics / Predictive Analytics Regularization in R Reply To: Regularization in R

Michael Barr

PPS – the glmnet package allows you to fit a variety of distributions, similar to the glm() r function. both methods are likelihood-based and do not perform OLS (the two are only equivalent in the gaussian + identity case you identified). In any case, you will not receive back the MLE estimates from glmnet except for the special case of \lambda==0 (no weight on penalty). In all cases, the loss function being used in the estimation is negative log-likelihood + lambda * penalty (either ||L1||, ||L2||, or a*||L1|| + (1-a)*||L2||).


A vanilla GLM would simply maximize negative log-likelihood. But because the penalties are measures of the magnitudes of coefficients, including that term in the loss function means that you reduce the magnitudes of the estimates, moving away from the MLE estimate.