Regularization: how complicated the model is? Regularization, measures complexity of model 使预测准确、平稳 predictive stable

http://www.kdd.org/kdd2016/papers/files/rfp0697-chenAemb.pdf

https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf

【Training Loss measures how well model fit on training data】

【Regularization, measures complexity of model】

【simple】

【predictive】

• Why do we want to contain two component in the objective?

• Optimizing training loss encourages predictive models

 Fitting well in training data at least get you close to training data which is hopefully close to the underlying distribution

• Optimizing regularization encourages simple models

 Simpler models tends to have smaller variance in future predictions, making prediction stable

【训练损失：量化对点的拟合度】

Training Loss: How will the function fit on the points?

【泛化、正则化：量化复杂度、通用性】

Regularization: How do we define complexity of the function?