XGBOOST应用及调参示例

该示例所用的数据可从该链接下载，提取码为3y90，数据说明可参考该网页。该示例的“模型调参”这一部分引用了这篇博客的步骤。

数据前处理

导入数据

import pandas as pd

import numpy as np

from sklearn.cross_validation import train_test_split

### Load data

### Split the data to train and test sets

data = pd.read_csv('data/loan/Train.csv', encoding = "ISO-8859-1")

train, test = train_test_split(data,train_size=0.7,random_state=123,stratify=data['Disbursed'])

### Check number of nulls in each feature column

nulls_per_column = train.isnull().sum()

print(nulls_per_column)

将特征拆分成数值型和种类型

### Drop the useless columns

train_1 = train.drop(['ID','Lead_Creation_Date','LoggedIn'],axis=1)

### Split the columns to numerical and categorical

category_cols = train_1.columns[train_1.dtypes==object].tolist()

category_cols.remove('DOB')

category_cols.append('Var4')

numeric_cols = list(set(train_1.columns)-set(category_cols))

分析并处理种类型特征

### explore the categorical columns

for v in category_cols:

    print('Ratio of missing value for variable {0}: {1}'.format(v,nulls_per_column[v]/train_1.shape[0]))

print('-----------------------------------------------------------')

counts = dict()

for v in category_cols:

    print('\nFrequency count for variable %s'%v)

    counts[v] = train_1[v].value_counts()

    print(counts[v])

### merge the cities that counts<200

merge_city = [c for c in counts['City'].index if counts['City'][c]<200]

train_1['City'] = train_1['City'].apply(lambda x: 'others' if x in merge_city else x)

### merge the salary accounts that counts<100

merge_sa = [c for c in counts['Salary_Account'].index if counts['Salary_Account'][c]<100]

train_1['Salary_Account'] = train_1['Salary_Account'].apply(lambda x: 'others' if x in merge_sa else x)

### merge the sources that counts<100

merge_sr = [c for c in counts['Source'].index if counts['Source'][c]<100]

train_1['Source'] = train_1['Source'].apply(lambda x: 'others' if x in merge_sr else x)

### impute the missing value

train_1['City'].fillna('Missing',inplace=True)

train_1['Salary_Account'].fillna('Missing',inplace=True)

### delete the column Employer_Name since too many categories

train_2 = train_1.drop('Employer_Name',axis=1)

分析并处理数值型特征

### Explore the numerical columns

for v in numeric_cols:

    print('Ratio of missing value for variable {0}: {1}'.format(v,nulls_per_column[v]/train_2.shape[0]))

print('-----------------------------------------------------------')

for v in numeric_cols:

    print('\nStatistical summary for variable %s'%v)

    print(train_2[v].describe())

### Create Age column:

train_2['Age'] = train_2['DOB'].apply(lambda x: 118 - int(x[-2:]))

### High proportion missing so create a new variable stating whether this is missing or not:

train_2['Loan_Amount_Submitted_Missing'] = train_2['Loan_Amount_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

train_2['Loan_Tenure_Submitted_Missing'] = train_2['Loan_Tenure_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

train_2['EMI_Loan_Submitted_Missing'] = train_2['EMI_Loan_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

train_2['Interest_Rate_Missing'] = train_2['Interest_Rate'].apply(lambda x: 1 if pd.isnull(x) else 0)

train_2['Processing_Fee_Missing'] = train_2['Processing_Fee'].apply(lambda x: 1 if pd.isnull(x) else 0)

### Impute the missing value

train_2['Existing_EMI'].fillna(train_2['Existing_EMI'].median(), inplace=True)

train_2['Loan_Amount_Applied'].fillna(train_2['Loan_Amount_Applied'].median(),inplace=True)

train_2['Loan_Tenure_Applied'].fillna(train_2['Loan_Tenure_Applied'].median(),inplace=True)

### Drop original columns

train_3 = train_2.drop(['DOB','Loan_Amount_Submitted','Loan_Tenure_Submitted','EMI_Loan_Submitted', \

                        'Interest_Rate','Processing_Fee'],axis=1)

One-Hot encoding

from sklearn.preprocessing import LabelEncoder

dropped_columns = ['ID','Lead_Creation_Date','LoggedIn','Employer_Name','DOB','Loan_Amount_Submitted', \

                'Loan_Tenure_Submitted','EMI_Loan_Submitted','Interest_Rate','Processing_Fee']

le = LabelEncoder()

var_to_encode = list(set(category_cols)-set(dropped_columns))

for col in var_to_encode:

    train_3[col] = le.fit_transform(train_3[col])

### pd.get_dummies can also be used directly without LabelEncoder

train_3 = pd.get_dummies(train_3, columns=var_to_encode)

模型调参

建立基础模型并使用early_stop调整迭代次数

import xgboost as xgb

import matplotlib.pyplot as plt

from sklearn import metrics

### base model

target = 'Disbursed'

predictors = [x for x in train_3.columns if x!=target]

xgb1 = xgb.XGBClassifier(learning_rate=0.1, n_estimators=1000, max_depth=5, min_child_weight=1, gamma=0, \

                         subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

### use early_stop in xgb.cv

def get_n_estimators(alg, dtrain, predictors, target, cv_folds=5, early_stopping_rounds=50):

    xgb_param = alg.get_xgb_params()

    xgtrain = xgb.DMatrix(dtrain[predictors], label=dtrain[target])

    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds, \

                      metrics='auc', early_stopping_rounds=early_stopping_rounds, stratified=True)

    alg.set_params(n_estimators=cvresult.shape[0])

    #Print model report:

    print("\nModel Report")

    print("Set n_estimators to {0}".format(cvresult.shape[0]))

    print(cvresult.tail(1)['test-auc-mean'])

    #Fit the algorithm on the data

    alg.fit(dtrain[predictors], dtrain[target], eval_metric='auc')

    #Feature importance

    feat_imp = pd.Series(alg.get_booster().get_fscore()).sort_values(ascending=False)

    feat_imp.plot(kind='bar', title='Feature Importances', figsize=(20,6))

    plt.ylabel('Feature Importance Score')

    return

### get n_estimators

get_n_estimators(xgb1, train_3, predictors, target)

Tune max_depth and min_child_weight

from sklearn.model_selection import GridSearchCV

### optimal: {'max_depth':5,'min_child_weight':5}

param_test1 = {'max_depth':range(3,10,2),'min_child_weight':range(1,6,2)}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=141, max_depth=5, min_child_weight=1, gamma=0, \

                        subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

gsearch1 = GridSearchCV(estimator = alg, param_grid = param_test1, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch1.fit(train_3[predictors],train_3[target])

print(gsearch1.best_params_)

print(gsearch1.best_score_)

### optimal: {'max_depth':4,'min_child_weight':6}

param_test2 = {'max_depth':[4,5,6],'min_child_weight':[4,5,6]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=141, max_depth=5, min_child_weight=5, gamma=0, \

                        subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

gsearch2 = GridSearchCV(estimator = alg, param_grid = param_test2, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch2.fit(train_3[predictors],train_3[target])

print(gsearch2.best_params_)

print(gsearch2.best_score_)

### optimal: {'min_child_weight':6}

param_test2b = {'min_child_weight':[6,8,10,12]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=141, max_depth=4, min_child_weight=6, gamma=0, \

                        subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

gsearch2b = GridSearchCV(estimator = alg, param_grid = param_test2b, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch2b.fit(train_3[predictors],train_3[target])

print(gsearch2b.best_params_)

print(gsearch2b.best_score_)

Tune gamma

### optimal: {'gamma':0.2}

param_test3 = {'gamma':[i/10.0 for i in range(0,5)]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=141, max_depth=4, min_child_weight=6, gamma=0, \

                        subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

gsearch3 = GridSearchCV(estimator = alg, param_grid = param_test3, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch3.fit(train_3[predictors],train_3[target])

print(gsearch3.best_params_)

print(gsearch3.best_score_)

### get n_estimators

xgb2 = xgb.XGBClassifier(learning_rate=0.1, n_estimators=1000, max_depth=4, min_child_weight=6, gamma=0.2, \

                         subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

get_n_estimators(xgb2, train_3, predictors, target)

Tune subsample and colsample_bytree

### optimal: {'colsample_bytree': 0.7, 'subsample': 0.7}

param_test4 = {'subsample':[i/10.0 for i in range(6,11)], 'colsample_bytree':[i/10.0 for i in range(6,11)]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, \

                        subsample=0.8, colsample_bytree=0.8, objective= 'binary:logistic', nthread=4, seed=27)

gsearch4 = GridSearchCV(estimator = alg, param_grid = param_test4, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch4.fit(train_3[predictors],train_3[target])

print(gsearch4.best_params_)

print(gsearch4.best_score_)

### optimal: {'colsample_bytree': 0.75, 'subsample': 0.7}

param_test5 = {'subsample':[i/100.0 for i in range(65,80,5)], 'colsample_bytree':[i/100.0 for i in range(65,80,5)]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, \

                        subsample=0.7, colsample_bytree=0.7, objective= 'binary:logistic', nthread=4, seed=27)

gsearch5 = GridSearchCV(estimator = alg, param_grid = param_test5, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch5.fit(train_3[predictors],train_3[target])

print(gsearch5.best_params_)

print(gsearch5.best_score_)

Tune reg_alpha

### optimal: {'reg_alpha': 0.01}

param_test6 = {'reg_alpha':[0, 1e-5, 1e-2, 0.1, 1, 100]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, \

                        subsample=0.7, colsample_bytree=0.75, objective= 'binary:logistic', nthread=4, seed=27)

gsearch6 = GridSearchCV(estimator = alg, param_grid = param_test6, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch6.fit(train_3[predictors],train_3[target])

print(gsearch6.best_params_)

print(gsearch6.best_score_)

### optimal: {'reg_alpha': 0.01}

param_test7 = {'reg_alpha':[0.001, 0.005, 0.01, 0.05]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, reg_alpha=0.01, \

                        subsample=0.7, colsample_bytree=0.75, objective= 'binary:logistic', nthread=4, seed=27)

gsearch7 = GridSearchCV(estimator = alg, param_grid = param_test7, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch7.fit(train_3[predictors],train_3[target])

print(gsearch7.best_params_)

print(gsearch7.best_score_)

Tune reg_lambda

### optimal: {'reg_lambda': 1}

param_test8 = {'reg_lambda':[0, 0.01, 0.1, 1, 10, 100]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, reg_alpha=0.01, \

                        subsample=0.7, colsample_bytree=0.75, objective= 'binary:logistic', nthread=4, seed=27)

gsearch8 = GridSearchCV(estimator = alg, param_grid = param_test8, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch8.fit(train_3[predictors],train_3[target])

print(gsearch8.best_params_)

print(gsearch8.best_score_)

### optimal: {'reg_lambda': 1}

param_test9 = {'reg_lambda':[0.5, 0.7, 1, 3, 5]}

alg = xgb.XGBClassifier(learning_rate=0.1, n_estimators=142, max_depth=4, min_child_weight=6, gamma=0.2, reg_alpha=0.01, \

                        subsample=0.7, colsample_bytree=0.75, objective= 'binary:logistic', nthread=4, seed=27)

gsearch9 = GridSearchCV(estimator = alg, param_grid = param_test9, scoring='roc_auc', n_jobs=4, iid=False, cv=5)

gsearch9.fit(train_3[predictors],train_3[target])

print(gsearch9.best_params_)

print(gsearch9.best_score_)

### get n_estimators

xgb3 = xgb.XGBClassifier(learning_rate=0.1, n_estimators=1000, max_depth=4, min_child_weight=6, gamma=0.2, \

                         reg_alpha=0.01, reg_lambda=1, subsample=0.7, colsample_bytree=0.75, \

                         objective= 'binary:logistic', nthread=4, seed=27)

get_n_estimators(xgb3, train_3, predictors, target)

Reduce learning rate

xgb4 = xgb.XGBClassifier(learning_rate=0.01, n_estimators=5000, max_depth=4, min_child_weight=6, gamma=0.2, \

                         reg_alpha=0.01, reg_lambda=1, subsample=0.7, colsample_bytree=0.75, \

                         objective= 'binary:logistic', nthread=4, seed=27)

get_n_estimators(xgb4, train_3, predictors, target)

根据上述过程构建完整的Pipeline

import pandas as pd

import numpy as np

import xgboost as xgb

import matplotlib.pyplot as plt

from sklearn import metrics

from sklearn.cross_validation import train_test_split

from sklearn.preprocessing import Imputer, FunctionTransformer, LabelBinarizer

from sklearn_pandas import DataFrameMapper, CategoricalImputer

from sklearn.pipeline import Pipeline

data = pd.read_csv('Train.csv', encoding = "ISO-8859-1")

train, test = train_test_split(data,train_size=0.7,random_state=123,stratify=data['Disbursed'])

target_raw = 'Disbursed'

predictors_raw = [col for col in train.columns if col!=target_raw]

train_X, train_y = train[predictors_raw], train[target_raw]

category_cols = train_X.columns[train_X.dtypes==object].tolist()

category_cols.remove('DOB')

category_cols.append('Var4')

numeric_cols = list(set(train_X.columns)-set(category_cols))

numeric_cols = numeric_cols+['Age', 'Loan_Amount_Submitted_Missing', 'Loan_Tenure_Submitted_Missing', \

                             'EMI_Loan_Submitted_Missing', 'Interest_Rate_Missing', 'Processing_Fee_Missing']

counts = dict()

for v in category_cols:

    counts[v] = train_X[v].value_counts()

non_merge_city = [c for c in counts['City'].index if counts['City'][c]>=200]

non_merge_sa = [c for c in counts['Salary_Account'].index if counts['Salary_Account'][c]>=100]

non_merge_sr = [c for c in counts['Source'].index if counts['Source'][c]>=100]

dropped_columns = ['ID','Lead_Creation_Date','LoggedIn','Employer_Name','DOB','Loan_Amount_Submitted', \

                   'Loan_Tenure_Submitted','EMI_Loan_Submitted','Interest_Rate','Processing_Fee']

# Function Transform

def preprocess(X):

    X['City'] = X['City'].apply(lambda x: 'others' if x not in non_merge_city and not pd.isnull(x) else x)

    X['Salary_Account'] = X['Salary_Account'].apply(lambda x: 'others' if x not in non_merge_sa and not pd.isnull(x) else x)

    X['Source'] = X['Source'].apply(lambda x: 'others' if x not in non_merge_sr and not pd.isnull(x) else x)

    X['Age'] = X['DOB'].apply(lambda x: 118 - int(x[-2:]))

    X['Loan_Amount_Submitted_Missing'] = X['Loan_Amount_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

    X['Loan_Tenure_Submitted_Missing'] = X['Loan_Tenure_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

    X['EMI_Loan_Submitted_Missing'] = X['EMI_Loan_Submitted'].apply(lambda x: 1 if pd.isnull(x) else 0)

    X['Interest_Rate_Missing'] = X['Interest_Rate'].apply(lambda x: 1 if pd.isnull(x) else 0)

    X['Processing_Fee_Missing'] = X['Processing_Fee'].apply(lambda x: 1 if pd.isnull(x) else 0)

    return X.drop(dropped_columns, axis=1)

# Apply numeric imputer

numeric_imputer = [([feature], Imputer(strategy="median")) for feature in numeric_cols if feature not in dropped_columns]

# Apply categorical imputer and one-hot encode

category_imputer = [(feature, [CategoricalImputer(strategy='constant', fill_value='Missing'),LabelBinarizer()]) \

                    for feature in category_cols if feature not in dropped_columns]

# Combine the numeric and categorical transformations

numeric_categorical_union = DataFrameMapper(numeric_imputer+category_imputer,input_df=True,df_out=True)

# Tuned Classifier

tuned_xgb = xgb.XGBClassifier(learning_rate=0.01, n_estimators=1480, max_depth=4, min_child_weight=6, gamma=0.2, \

                              reg_alpha=0.01, reg_lambda=1, subsample=0.7, colsample_bytree=0.75, \

                              objective= 'binary:logistic', nthread=4, seed=27)

# Create full pipeline

pipeline = Pipeline([("preprocessor", FunctionTransformer(preprocess, validate=False)), \

                     ("featureunion", numeric_categorical_union), ("classifier", tuned_xgb)])

pipeline.fit(train_X, train_y)

#Feature importance

feat_imp = pd.Series(pipeline.named_steps['classifier'].get_booster().get_fscore()).sort_values(ascending=False)

feat_imp.plot(kind='bar', title='Feature Importances', figsize=(20,6))

plt.ylabel('Feature Importance Score')

# individual prediction

print(pipeline.predict_proba(test.iloc[[1]][predictors_raw]))

# test data predictions

# AUC Score (Test): 0.8568

predprob=pipeline.predict_proba(test[predictors_raw])[:,1]

print("AUC Score (Test): %f" % metrics.roc_auc_score(test[target_raw], predprob))

XGBOOST应用及调参示例的更多相关文章

XGBoost 重要参数(调参使用)
XGBoost 重要参数(调参使用) 数据比赛Kaggle,天池中最常见的就是XGBoost和LightGBM. 模型是在数据比赛中尤为重要的,但是实际上,在比赛的过程中,大部分朋友在模型上花的时间却 ...
xgboost/gbdt在调参时为什么树的深度很少就能达到很高的精度？
问题: 用xgboost/gbdt在在调参的时候把树的最大深度调成6就有很高的精度了.但是用DecisionTree/RandomForest的时候需要把树的深度调到15或更高.用RandomFore ...
【Python机器学习实战】决策树与集成学习（七）——集成学习（5）XGBoost实例及调参
上一节对XGBoost算法的原理和过程进行了描述,XGBoost在算法优化方面主要在原损失函数中加入了正则项,同时将损失函数的二阶泰勒展开近似展开代替残差(事实上在GBDT中叶子结点的最优值求解也是使 ...
xgboost参数及调参
常规参数General Parameters booster[default=gbtree]:选择基分类器,可以是:gbtree,gblinear或者dart.gbtree和draf基于树模型,而gb ...
xgboost的遗传算法调参
遗传算法适应度的选择: 机器学习的适应度可以是任何性能指标 —准确度,精确度,召回率,F1分数等等.根据适应度值,我们选择表现最佳的父母(“适者生存”),作为幸存的种群. 交配: 存活下来的群体中的父 ...
Xgboost调参总结
一.参数速查参数分为三类: 通用参数:宏观函数控制. Booster参数:控制每一步的booster(tree/regression). 学习目标参数:控制训练目标的表现. 二.回归 from xg ...
xgboost使用调参
欢迎关注博主主页,学习python视频资源 https://blog.csdn.net/q383700092/article/details/53763328 调参后结果非常理想 from sklea ...
xgboost的sklearn接口和原生接口参数详细说明及调参指点
from xgboost import XGBClassifier XGBClassifier(max_depth=3,learning_rate=0.1,n_estimators=100,silen ...
xgboost入门与实战（实战调参篇）
https://blog.csdn.net/sb19931201/article/details/52577592 xgboost入门与实战(实战调参篇) 前言前面几篇博文都在学习原理知识,是时候上 ...

随机推荐

JavaWeb学习 (二十八)————文件上传和下载
在Web应用系统开发中,文件上传和下载功能是非常常用的功能,今天来讲一下JavaWeb中的文件上传和下载功能的实现. 对于文件上传,浏览器在上传的过程中是将文件以流的形式提交到服务器端的,如果直接使用 ...
libtorch初体验
环境 Ubuntu -18.04.1, opencv3.4.0 , python 3.6, cmake 3.5.0, pytorch 1.0. pytorch官网下载对应版本:https://py ...
C#基础知识回顾-- 反射（2）
使用反射调用方法: 一旦知道一个类型所支持的方法,就可以对方法进行调用.调用时,需使用包含在 MethodInfo中的Invoke()方法.调用形式: object Invoke(object ...
使用Nexus2.x为Maven3.x搭建私服构件仓库
前言在笔者的前一篇博文<Use Maven3.x>中,笔者详细的讲解了如何通过使用Maven3.x来构建及管理你的项目.其中笔者提到过些关于私服的概念,但却没有明确的对私服做出详细的阐述 ...
【18】观察者模式（Observer Pattern）
一.引言在现实生活中,处处可见观察者模式.例如,微信中的订阅号,订阅博客和QQ微博中关注好友,这些都属于观察者模式的应用.在这一章将分享我对观察者模式的理解,废话不多说了,直接进入今天的主题. 二. ...
tomcat启动时卡住
tomcat启动时卡住进入jdk/jre/lib/security/java.security文件找到securerandom.source将这一行隐藏并在下面一行加入securerandom. ...
功率因数cosφ仪表盘
一.截图二.说明本篇博客主要是有三个亮点: ① 刻度标注在仪表盘标线外 ② 仪表盘存在两个刻度值,分别是(正)0.5~1 和(负)-1~-0.5 ③ 仪表盘内标注,分别是“超前”和“滞后” 三.代 ...
inheritPrototypeChain.js
// 原型链 // 其基本思路是利用原型让一个引用类型继承另一个引用类型的属性和方法 function Person(){ this.name = "Person"; } Pers ...
React 入门学习笔记整理（九）——路由
(1)安装路由 React-router React-router提供了一些router的核心api,包括Router, Route, Switch等,但是它没有提供dom操作进行跳转的api. Re ...
简单实用的jQuery分页插件
在做商城和订单管理的时候,常常会用到分页功能,所以我封装了一个jQuery的分页插件,该插件主要实现上下翻页,输入数字跳转等功能. 具体实现如下: 输入参数需要当前页码pageNo,总页码totalP ...