【第四课】kaggle案例分析四

Evernote Export

比赛题目介绍

facebook想要准确的知道用户登录的地点，从而可以为用户提供更准确的服务
为了比赛，facebook创建了一个虚拟世界地图，地图面积为100km2，其中包含了超过1000000个地点
通过给定的坐标，以及坐标准确性，判断用户登录地点
训练集和测试集是根据时间划分的，而在公共排行榜和私人排行榜上的测试集数据是随机划分的
row_id 登录事件的id，作为标识符使用
x，y：坐标数值
accuracy：坐标的准确性
time：时间戳
place_id：地点id，需要预测的变量
其中，accuracy和time的具体含义并没有给出，关于这两个变量的探索也是比赛的一部分内容

XGboost

XGboost就是梯度提升树的改进(速度快)
kaggle神器 XGboost
模型： 如何在已知xi而预测y^i
线性模型：y^i=∑jwjxij包含线性模型和逻辑回归模型
预测分数y^i可以有基于任务的不同解读
- 线性回归 y^i是预测分数
- 逻辑回归 1+exp(−y^i)1是对积极的实例的可能性预测
- 其他，比如排名预测
参数：我们需要从数据中学习到的参数
线性模型：wj∣j=1,...,d
损失函数的使用
Obj(Θ)=L(Θ)+Ω(Θ)
训练数据中的损失：L=∑i=1nl(yi,y^i)
- 方差损失 l(yi,y^i)=(yi−y^i)2
- 逻辑损失 l(yi,y^i)=yiln(1+e−y^i)+(1−yi)ln(1+eey^i)
模型的复杂度
- L2规范 Ω(w)=λ∣∣w∣∣2
- L1规范 Ω(w)=λ∣∣w∣∣1
正则项(惩罚模型的复杂度) ∑i=1n(yi−wTxi)2+λ∣∣w∣∣2
Lasso ∑i=1n(yi−wTxi)2+λ∣∣w∣∣1
逻辑回归 ∑i=1n[yiln(1+e−wTxi)+(1−yi)ln(1+ewTxi)]+λ∣∣w∣∣2

回归树

线性回归问题就是用折线或者折平面(高维度)去拟合训练集
用小的决策树，不剪枝，用投票的方式将决策树组合起来
折线回归树预测：
y^i=k=1∑Kfk(xi),fk∈F

数据探索

特征工程

与坐标相关的特征
与时间相关的特征
与准确性相关的特征
Z-值

%23%23%23%20%E6%AF%94%E8%B5%9B%E9%A2%98%E7%9B%AE%E4%BB%8B%E7%BB%8D%0A*%20facebook%E6%83%B3%E8%A6%81%E5%87%86%E7%A1%AE%E7%9A%84%E7%9F%A5%E9%81%93%E7%94%A8%E6%88%B7%E7%99%BB%E5%BD%95%E7%9A%84%E5%9C%B0%E7%82%B9%EF%BC%8C%E4%BB%8E%E8%80%8C%E5%8F%AF%E4%BB%A5%E4%B8%BA%E7%94%A8%E6%88%B7%E6%8F%90%E4%BE%9B%E6%9B%B4%E5%87%86%E7%A1%AE%E7%9A%84%E6%9C%8D%E5%8A%A1%0A*%20%E4%B8%BA%E4%BA%86%E6%AF%94%E8%B5%9B%EF%BC%8Cfacebook%E5%88%9B%E5%BB%BA%E4%BA%86%E4%B8%80%E4%B8%AA%E8%99%9A%E6%8B%9F%E4%B8%96%E7%95%8C%E5%9C%B0%E5%9B%BE%EF%BC%8C%E5%9C%B0%E5%9B%BE%E9%9D%A2%E7%A7%AF%E4%B8%BA%24100km%5E2%24%EF%BC%8C%E5%85%B6%E4%B8%AD%E5%8C%85%E5%90%AB%E4%BA%86%E8%B6%85%E8%BF%871000000%E4%B8%AA%E5%9C%B0%E7%82%B9%0A*%20%E9%80%9A%E8%BF%87%E7%BB%99%E5%AE%9A%E7%9A%84%E5%9D%90%E6%A0%87%EF%BC%8C%E4%BB%A5%E5%8F%8A%E5%9D%90%E6%A0%87%E5%87%86%E7%A1%AE%E6%80%A7%EF%BC%8C%E5%88%A4%E6%96%AD%E7%94%A8%E6%88%B7%E7%99%BB%E5%BD%95%E5%9C%B0%E7%82%B9%0A*%20%20%E8%AE%AD%E7%BB%83%E9%9B%86%E5%92%8C%E6%B5%8B%E8%AF%95%E9%9B%86%E6%98%AF%E6%A0%B9%E6%8D%AE%E6%97%B6%E9%97%B4%E5%88%92%E5%88%86%E7%9A%84%EF%BC%8C%E8%80%8C%E5%9C%A8%E5%85%AC%E5%85%B1%E6%8E%92%E8%A1%8C%E6%A6%9C%E5%92%8C%E7%A7%81%E4%BA%BA%E6%8E%92%E8%A1%8C%E6%A6%9C%E4%B8%8A%E7%9A%84%E6%B5%8B%E8%AF%95%E9%9B%86%E6%95%B0%E6%8D%AE%E6%98%AF%E9%9A%8F%E6%9C%BA%E5%88%92%E5%88%86%E7%9A%84%0A*%20row_id%20%E7%99%BB%E5%BD%95%E4%BA%8B%E4%BB%B6%E7%9A%84id%EF%BC%8C%E4%BD%9C%E4%B8%BA%E6%A0%87%E8%AF%86%E7%AC%A6%E4%BD%BF%E7%94%A8%0A*%20x%EF%BC%8Cy%EF%BC%9A%E5%9D%90%E6%A0%87%E6%95%B0%E5%80%BC%0A*%20accuracy%EF%BC%9A%E5%9D%90%E6%A0%87%E7%9A%84%E5%87%86%E7%A1%AE%E6%80%A7%0A*%20time%EF%BC%9A%E6%97%B6%E9%97%B4%E6%88%B3%0A*%20place_id%EF%BC%9A%E5%9C%B0%E7%82%B9id%EF%BC%8C%E9%9C%80%E8%A6%81%E9%A2%84%E6%B5%8B%E7%9A%84%E5%8F%98%E9%87%8F%0A*%20%E5%85%B6%E4%B8%AD%EF%BC%8Caccuracy%E5%92%8Ctime%E7%9A%84%E5%85%B7%E4%BD%93%E5%90%AB%E4%B9%89%E5%B9%B6%E6%B2%A1%E6%9C%89%E7%BB%99%E5%87%BA%EF%BC%8C%E5%85%B3%E4%BA%8E%E8%BF%99%E4%B8%A4%E4%B8%AA%E5%8F%98%E9%87%8F%E7%9A%84%E6%8E%A2%E7%B4%A2%E4%B9%9F%E6%98%AF%E6%AF%94%E8%B5%9B%E7%9A%84%E4%B8%80%E9%83%A8%E5%88%86%E5%86%85%E5%AE%B9%0A%23%23%23%20XGboost%0A*%20XGboost%E5%B0%B1%E6%98%AF%E6%A2%AF%E5%BA%A6%E6%8F%90%E5%8D%87%E6%A0%91%E7%9A%84%E6%94%B9%E8%BF%9B(%E9%80%9F%E5%BA%A6%E5%BF%AB)%0A*%20kaggle%E7%A5%9E%E5%99%A8%20XGboost%0A*%20**%E6%A8%A1%E5%9E%8B%EF%BC%9A**%20%E5%A6%82%E4%BD%95%E5%9C%A8%E5%B7%B2%E7%9F%A5%24x_i%24%E8%80%8C%E9%A2%84%E6%B5%8B%24%5Chat%20y_i%24%0A*%20%E7%BA%BF%E6%80%A7%E6%A8%A1%E5%9E%8B%EF%BC%9A%24%5Chat%20y_i%20%3D%20%5Csum_j%20w_jx_%7Bij%7D%24%E5%8C%85%E5%90%AB%E7%BA%BF%E6%80%A7%E6%A8%A1%E5%9E%8B%E5%92%8C%E9%80%BB%E8%BE%91%E5%9B%9E%E5%BD%92%E6%A8%A1%E5%9E%8B%0A*%20%E9%A2%84%E6%B5%8B%E5%88%86%E6%95%B0%24%5Chat%20y_i%24%E5%8F%AF%E4%BB%A5%E6%9C%89%E5%9F%BA%E4%BA%8E%E4%BB%BB%E5%8A%A1%E7%9A%84%E4%B8%8D%E5%90%8C%E8%A7%A3%E8%AF%BB%0A%20%20%20%20*%20%E7%BA%BF%E6%80%A7%E5%9B%9E%E5%BD%92%20%24%5Chat%20y_i%24%E6%98%AF%E9%A2%84%E6%B5%8B%E5%88%86%E6%95%B0%0A%20%20%20%20*%20%E9%80%BB%E8%BE%91%E5%9B%9E%E5%BD%92%20%24%5Cfrac%7B1%7D%7B1%2Bexp(-%20%5Chat%20y_i)%7D%24%E6%98%AF%E5%AF%B9%E7%A7%AF%E6%9E%81%E7%9A%84%E5%AE%9E%E4%BE%8B%E7%9A%84%E5%8F%AF%E8%83%BD%E6%80%A7%E9%A2%84%E6%B5%8B%0A%20%20%20%20*%20%E5%85%B6%E4%BB%96%EF%BC%8C%E6%AF%94%E5%A6%82%E6%8E%92%E5%90%8D%E9%A2%84%E6%B5%8B%0A%20*%20%E5%8F%82%E6%95%B0%EF%BC%9A%E6%88%91%E4%BB%AC%E9%9C%80%E8%A6%81%E4%BB%8E%E6%95%B0%E6%8D%AE%E4%B8%AD%E5%AD%A6%E4%B9%A0%E5%88%B0%E7%9A%84%E5%8F%82%E6%95%B0%0A%20*%20%E7%BA%BF%E6%80%A7%E6%A8%A1%E5%9E%8B%EF%BC%9A%24%7Bw_j%7Cj%3D1%2C...%2Cd%7D%24%0A%20*%20%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0%E7%9A%84%E4%BD%BF%E7%94%A8%0A%20*%20%24%24Obj(%5CTheta)%20%3D%20L(%5CTheta)%20%2B%20%5COmega(%5CTheta)%20%24%24%0A%20*%20%E8%AE%AD%E7%BB%83%E6%95%B0%E6%8D%AE%E4%B8%AD%E7%9A%84%E6%8D%9F%E5%A4%B1%EF%BC%9A%24L%20%3D%20%5Csum%5En_%7Bi%3D1%7Dl(y_i%2C%5Chat%20y_i)%24%0A%20%20%20%20*%20%E6%96%B9%E5%B7%AE%E6%8D%9F%E5%A4%B1%20%24l(y_i%2C%5Chat%20y_i)%20%3D%20(y_i%20-%20%5Chat%20y_i)%5E2%24%0A%20%20%20%20*%20%E9%80%BB%E8%BE%91%E6%8D%9F%E5%A4%B1%20%24l(y_i%2C%5Chat%20y_i)%20%3D%20y_iln(1%2Be%5E%7B-%20%5Chat%20y_i%7D)%2B(1-y_i)ln(1%2Be%5E%7Be%20%5Chat%20y_i%7D)%24%0A%20*%20%E6%A8%A1%E5%9E%8B%E7%9A%84%E5%A4%8D%E6%9D%82%E5%BA%A6%0A%20%20%20%20*%20L2%E8%A7%84%E8%8C%83%20%24%5COmega%20(w)%20%3D%20%5Clambda%20%7C%7Cw%7C%7C%5E2%24%0A%20%20%20%20*%20L1%E8%A7%84%E8%8C%83%20%24%5COmega(w)%20%3D%20%5Clambda%20%7C%7Cw%7C%7C_1%24%0A%20%0A*%20%E6%AD%A3%E5%88%99%E9%A1%B9(%E6%83%A9%E7%BD%9A%E6%A8%A1%E5%9E%8B%E7%9A%84%E5%A4%8D%E6%9D%82%E5%BA%A6)%20%24%5Csum%5En_%7Bi%3D1%7D(y_i-w%5ETx_i)%5E2%2B%5Clambda%7C%7Cw%7C%7C%5E2%24%0A*%20Lasso%20%24%5Csum%5En_%7Bi%3D1%7D(y_i-w%5ETx_i)%5E2%2B%5Clambda%7C%7Cw%7C%7C_1%24%0A%0A*%20%E9%80%BB%E8%BE%91%E5%9B%9E%E5%BD%92%20%24%5Csum%5En_%7Bi%3D1%7D%5By_iln(1%2Be%5E%7B-w%5ETx_i%7D)%2B(1-y_i)ln(1%2Be%5E%7Bw%5ETx_i%7D)%5D%2B%5Clambda%7C%7Cw%7C%7C%5E2%24%0A%0A%23%23%23%23%20%E5%9B%9E%E5%BD%92%E6%A0%91%0A*%20%E7%BA%BF%E6%80%A7%E5%9B%9E%E5%BD%92%E9%97%AE%E9%A2%98%E5%B0%B1%E6%98%AF%E7%94%A8%E6%8A%98%E7%BA%BF%E6%88%96%E8%80%85%E6%8A%98%E5%B9%B3%E9%9D%A2(%E9%AB%98%E7%BB%B4%E5%BA%A6)%E5%8E%BB%E6%8B%9F%E5%90%88%E8%AE%AD%E7%BB%83%E9%9B%86%0A*%20%E7%94%A8%E5%B0%8F%E7%9A%84%E5%86%B3%E7%AD%96%E6%A0%91%EF%BC%8C%E4%B8%8D%E5%89%AA%E6%9E%9D%EF%BC%8C%E7%94%A8%E6%8A%95%E7%A5%A8%E7%9A%84%E6%96%B9%E5%BC%8F%E5%B0%86%E5%86%B3%E7%AD%96%E6%A0%91%E7%BB%84%E5%90%88%E8%B5%B7%E6%9D%A5%0A*%20%E6%8A%98%E7%BA%BF%E5%9B%9E%E5%BD%92%E6%A0%91%E9%A2%84%E6%B5%8B%EF%BC%9A%0A*%20%24%24%5Chat%20y_i%20%3D%20%5Csum%5EK_%7Bk%3D1%7Df_k(x_i)%2Cf_k%20%5Cin%20F%24%24%0A%0A%0A%0A!%5B01cc019ccf72cd1a39c053867d03f1fe.png%5D(en-resource%3A%2F%2Fdatabase%2F1362%3A1)%0A%0A%0A%23%23%23%20%E6%95%B0%E6%8D%AE%E6%8E%A2%E7%B4%A2%0A%0A%23%23%23%20%E7%89%B9%E5%BE%81%E5%B7%A5%E7%A8%8B%0A*%20%E4%B8%8E%E5%9D%90%E6%A0%87%E7%9B%B8%E5%85%B3%E7%9A%84%E7%89%B9%E5%BE%81%0A*%20%E4%B8%8E%E6%97%B6%E9%97%B4%E7%9B%B8%E5%85%B3%E7%9A%84%E7%89%B9%E5%BE%81%0A*%20%E4%B8%8E%E5%87%86%E7%A1%AE%E6%80%A7%E7%9B%B8%E5%85%B3%E7%9A%84%E7%89%B9%E5%BE%81%0A*%20Z-%E5%80%BC%0A%0A%23%23%23%20XGBoost%0A*%20%E4%B8%89%E7%B1%BB%E5%8F%82%E6%95%B0%0A%20%20%20%20*%20General%20Parameters%0A%20%20%20%20*%20Booster%20Parameters%0A%20%20%20%20*%20Learning%20Task%20Parameters%0A*%20eta%0A*%20gamma%0A*%20max_depth%0A*%20min_child_weight%0A*%20max_delta_step%0A*%20subsample%0A*%20colsample%0A*%20colsample_bylevel%0A*%20lambda%0A*%20alpha%0A*%20tree_method%0A*%20sketch_eps%0A*%20scale_pos_weight%0A*%20updater%0A*%20refresh_leaf%0A*%20process_type%0A*%20grow_plilcy%0A*%20max_leaves%0A*%20max_bins%0A*%20**%E9%80%89%E6%8B%A9%E8%BE%83%E9%AB%98%E7%9A%84eta**%0A*%20**%E7%A1%AE%E5%AE%9A%E5%90%88%E9%80%82%E7%9A%84%E5%BE%AA%E7%8E%AF%E6%AC%A1%E6%95%B0**%0A*%20**%E7%A1%AE%E5%AE%9A%E8%B0%83%E6%95%B4%E6%A0%91%E7%BB%93%E6%9E%84%E7%9A%84%E7%89%B9%E5%AE%9A%E5%8F%82%E6%95%B0**%0A*%20**%E8%B0%83%E6%95%B4XGBoost%E7%9A%84%E6%AD%A3%E5%88%99%E5%8C%96%E5%8F%82%E6%95%B0**%0A*%20**%E9%80%90%E6%AD%A5%E9%99%8D%E4%BD%8Eeta%EF%BC%8C%E6%8F%90%E9%AB%98%E5%BE%AA%E7%8E%AF%E6%AC%A1%E6%95%B0%E9%87%8D%E6%96%B0%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B%EF%BC%8C%E5%AF%BB%E6%89%BE%E5%90%88%E9%80%82%E7%9A%84eta**