scikit-learn：3.2. Grid Search: Searching for estimator parameters

參考：http://scikit-learn.org/stable/modules/grid_search.html

GridSearchCV通过（蛮力）搜索參数空间（參数的全部可能组合）。寻找最好的 Cross-validation:
evaluating estimator performance score相应的超參数（翻译文章參考：http://blog.csdn.net/mmc2015/article/details/47099275）。比如Support
Vector Classifier的 C, kernel and gamma ，Lasso的alpha。etc。

A search consists of:

• an estimator (regressor or classifier such as sklearn.svm.SVC());
• a parameter space;
• a method for searching or sampling candidates;
• a cross-validation scheme
• a score
  function.

RandomizedSearchCV 通过一定的分布sample候选參数。而不是搜索全部參数组合。

本节我们介绍 GridSearchCV、RandomizedSearchCV 、以及parameter
search的小Tips，最后介绍蛮力搜索的alternatives。

1、Exhaustive
Grid Search

GridSearchCV的參数param_grid定义搜索网格。

两个样例说明一切：

• See Parameter
  estimation using grid search with cross-validation for an example of Grid Search computation on the digits dataset.
• See Sample
  pipeline for text feature extraction and evaluation for an example of Grid Search coupling parameters from a text documents feature extractor (n-gram count vectorizer and TF-IDF transformer) with a classifier (here a linear SVM trained with SGD with
  either elastic net or L2 penalty) using a pipeline.Pipeline instance.

2、Randomized
Parameter Optimization

RandomizedSearchCV 通过在參数可能的取值的某个分布中sample一组參数。优点是：能够设定独立于參数（及全部取值）详细数量的一个搜索次数；加入无效的參数也不会减少效率。

搜索的次数通过 n_iter 设定，对于每个參数，假设是连续的取值。则通过一定的分布sample，假设是离散的取值，则通过uniform分布sample，比如：

[{'C': scipy.stats.expon(scale=100), 'gamma': scipy.stats.expon(scale=.1),
  'kernel': ['rbf'], 'class_weight':['auto', None]}]

scipy.stats module提供了非常多用来sample參数的distributions，如expon, gamma, uniform or randint.

对于连续的參数，如 C ，一定要选择连续的分布来sample，而且适当增大 n_iter 通常会搜索到更好的參数组合。

给个样例：

• Comparing
  randomized search and grid search for hyperparameter estimation compares the usage and efficiency of randomized search and grid search.

3、Tips
for parameter search（这几个建议很靠谱。。。

）

1）详细化目标函数

參数搜索默认使用score function（
即，分类用sklearn.metrics.accuracy_score 回归用sklearn.metrics.r2_score ）来衡量參数的好坏对于有些应用（比方分类unbalance，score不是非常好的标准），通过详细化GridSearchCV和RandomizedSearchCV 的scoring parameter。See The
scoring parameter: defining model evaluation rules for more details.

2）综合estimators和parameter sapces（同一时候考虑预測器和參数空间）

Pipeline:
chaining estimators describes building composite estimators whose parameter space can be searched with these tools.

3）模型选择：先训练、再评估

用训练集选择模型。用測试集验证模型（using
the cross_validation.train_test_split utility
function.）（it is recommended to split the data into a development set (to be
fed to the GridSearchCV instance)
and an evaluation set to compute performance metrics.）

4）并行搜索

n_jobs=-1.
自己主动使用全部核。

5）robustness to failure（增强搜索错误的鲁棒性）

有些參数组合对于某些folds
of the data会failure，进而导致整个search failure，虽然其它的參数组合没有问题。

设定 error_score=0 (or =np.NaN)
能够使search过程忽略这种failure，只抛出一个warning，并将这种search结果设为0 (or =np.NaN)
，可以提高搜索遇到错误时的鲁棒性！

4、Alternatives
to brute force parameter search（没太看懂，还是不翻译了）

3.2.4.1. Model specific cross-validation

Some models can fit data for a range of value of some parameter almost as efficiently as fitting the estimator for a single value of the parameter. This feature can be leveraged to perform
a more efficient cross-validation used for model selection of this parameter.

The most common parameter amenable to this strategy is the parameter encoding the strength of the regularizer. In this case we say that we compute theregularization path of
the estimator.

Here is the list of such models:

linear_model.ElasticNetCV([l1_ratio, eps, ...])	Elastic Net model with iterative fitting along a regularization path
linear_model.LarsCV([fit_intercept, ...])	Cross-validated Least Angle Regression model
linear_model.LassoCV([eps, n_alphas, ...])	Lasso linear model with iterative fitting along a regularization path
linear_model.LassoLarsCV([fit_intercept, ...])	Cross-validated Lasso, using the LARS algorithm
linear_model.LogisticRegressionCV([Cs, ...])	Logistic Regression CV (aka logit, MaxEnt) classifier.
linear_model.MultiTaskElasticNetCV([...])	Multi-task L1/L2 ElasticNet with built-in cross-validation.
linear_model.MultiTaskLassoCV([eps, ...])	Multi-task L1/L2 Lasso with built-in cross-validation.
linear_model.OrthogonalMatchingPursuitCV([...])	Cross-validated Orthogonal Matching Pursuit model (OMP)
linear_model.RidgeCV([alphas, ...])	Ridge regression with built-in cross-validation.
linear_model.RidgeClassifierCV([alphas, ...])	Ridge classifier with built-in cross-validation.

3.2.4.2. Information Criterion

Some models can offer an information-theoretic closed-form formula of the optimal estimate of the regularization parameter by computing a single regularization path (instead of several when
using cross-validation).

Here is the list of models benefitting from the Aikike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) for automated model selection:

linear_model.LassoLarsIC([criterion, ...])

Lasso model fit with Lars using BIC or AIC for model selection

3.2.4.3. Out of Bag Estimates

When using ensemble methods base upon bagging, i.e. generating new training sets using sampling with replacement, part of the training set remains unused. For each classifier in the ensemble,
a different part of the training set is left out.

This left out portion can be used to estimate the generalization error without having to rely on a separate validation set. This estimate comes “for free” as no additional data is needed and
can be used for model selection.

This is currently implemented in the following classes:

ensemble.RandomForestClassifier([...])	A random forest classifier.
ensemble.RandomForestRegressor([...])	A random forest regressor.
ensemble.ExtraTreesClassifier([...])	An extra-trees classifier.
ensemble.ExtraTreesRegressor([n_estimators, ...])	An extra-trees regressor.
ensemble.GradientBoostingClassifier([loss, ...])	Gradient Boosting for classification.
ensemble.GradientBoostingRegressor([loss, ...])	Gradient Boosting for regression.