Handling Missing Values
1) A Simple Option: Drop Columns with Missing Values
如果这些列具有有用信息(在未丢失的位置),则在删除列时,模型将失去对此信息的访问权限。 此外,如果您的测试数据在您的训练数据没有的地方缺少值,则会导致错误。
data_without_missing_values = original_data.dropna(axis=1) #同时操作tran和test部分
cols_with_missing = [col for col in original_data.columns
if original_data[col].isnull().any()]
redued_original_data = original_data.drop(cols_with_missing, axis=1)
reduced_test_data = test_data.drop(cols_with_missing, axis=1)
2) A Better Option: Imputation
默认行为填写了插补的平均值。 统计学家已经研究了更复杂的策略,但是一旦将结果插入复杂的机器学习模型,那些复杂的策略通常没有任何好处。
关于Imputation的一个(很多)好处是它可以包含在scikit-learn Pipeline中。 管道简化了模型构建,模型验证和模型部署。
from sklearn.impute import SimpleImputer
my_imputer = SimpleImputer()
data_with_imputed_values = my_imputer.fit_transform(original_data)
3) An Extension To Imputation
估算是标准方法,通常效果很好。 但是,估算值可能系统地高于或低于其实际值(未在数据集中收集)。 或者具有缺失值的行可能以某种其他方式看来是唯一的。 在这种情况下,您的模型会通过考虑最初缺少哪些值来做出更好的预测。
# make copy to avoid changing original data (when Imputing)
new_data = original_data.copy() # make new columns indicating what will be imputed
cols_with_missing = (col for col in new_data.columns
if new_data[col].isnull().any())
for col in cols_with_missing:
new_data[col + '_was_missing'] = new_data[col].isnull() # Imputation
my_imputer = SimpleImputer()
new_data = pd.DataFrame(my_imputer.fit_transform(new_data))
new_data.columns = original_data.columns
Example (Comparing All Solutions)
import pandas as pd # Load data
melb_data = pd.read_csv('../input/melbourne-housing-snapshot/melb_data.csv') from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split melb_target = melb_data.Price
melb_predictors = melb_data.drop(['Price'], axis=1) # For the sake of keeping the example simple, we'll use only numeric predictors.
melb_numeric_predictors = melb_predictors.select_dtypes(exclude=['object']) from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(melb_numeric_predictors,
melb_target,
train_size=0.7,
test_size=0.3,
random_state=0) def score_dataset(X_train, X_test, y_train, y_test):
model = RandomForestRegressor()
model.fit(X_train, y_train)
preds = model.predict(X_test)
return mean_absolute_error(y_test, preds) # Get Model Score from Dropping Columns with Missing Values
# 直接丢弃含有缺失值的列
cols_with_missing = [col for col in X_train.columns
if X_train[col].isnull().any()]
reduced_X_train = X_train.drop(cols_with_missing, axis=1)
reduced_X_test = X_test.drop(cols_with_missing, axis=1)
print("Mean Absolute Error from dropping columns with Missing Values:")
print(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test)) # Get Model Score from Imputation
# 插入值
from sklearn.impute import SimpleImputer my_imputer = SimpleImputer()
imputed_X_train = my_imputer.fit_transform(X_train)
imputed_X_test = my_imputer.transform(X_test)
print("Mean Absolute Error from Imputation:")
print(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test)) # Get Score from Imputation with Extra Columns Showing What Was Imputed
# 添加额外列显示缺失值
imputed_X_train_plus = X_train.copy()
imputed_X_test_plus = X_test.copy() cols_with_missing = (col for col in X_train.columns
if X_train[col].isnull().any())
for col in cols_with_missing:
imputed_X_train_plus[col + '_was_missing'] = imputed_X_train_plus[col].isnull()
imputed_X_test_plus[col + '_was_missing'] = imputed_X_test_plus[col].isnull() # Imputation
my_imputer = SimpleImputer()
imputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)
imputed_X_test_plus = my_imputer.transform(imputed_X_test_plus) print("Mean Absolute Error from Imputation while Track What Was Imputed:")
print(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))
Handling Missing Values的更多相关文章
- [sklearn]官方例程-Imputing missing values before building an estimator 随机填充缺失值
官方链接:http://scikit-learn.org/dev/auto_examples/plot_missing_values.html#sphx-glr-auto-examples-plot- ...
- [sklearn] 官方例程-Imputing missing values before building an estimator 随机填充缺失值
官方链接:http://scikit-learn.org/dev/auto_examples/plot_missing_values.html#sphx-glr-auto-examples-plot- ...
- Multi-batch TMT reveals false positives, batch effects and missing values(解读人:胡丹丹)
文献名:Multi-batch TMT reveals false positives, batch effects and missing values (多批次TMT定量方法中对假阳性率,批次效应 ...
- 缺失值处理(Missing Values)
什么是缺失值?缺失值指数据集中某些变量的值有缺少的情况,缺失值也被称为NA(not available)值.在pandas里使用浮点值NaN(Not a Number)表示浮点数和非浮点数组中的缺失值 ...
- Web Scraping with R: How to Fill Missing Value (爬虫:如何处理缺失值)
网络上有大量的信息与数据.我们可以利用爬虫技术来获取这些巨大的数据资源. 这次用 IMDb 网站的2018年100部最欢迎的电影 来练练手,顺便总结一下 R 爬虫的方法. >> Prepa ...
- A Complete Tutorial on Tree Based Modeling from Scratch (in R & Python)
A Complete Tutorial on Tree Based Modeling from Scratch (in R & Python) MACHINE LEARNING PYTHON ...
- Kaggle:Home Credit Default Risk 特征工程构建及可视化(2)
博主在之前的博客 Kaggle:Home Credit Default Risk 数据探索及可视化(1) 中介绍了 Home Credit Default Risk 竞赛中一个优秀 kernel 关于 ...
- 【转】The most comprehensive Data Science learning plan for 2017
I joined Analytics Vidhya as an intern last summer. I had no clue what was in store for me. I had be ...
- data cleaning
Cleaning data in Python Table of Contents Set up environments Data analysis packages in Python Cle ...
随机推荐
- [自学]数据库ER图基础概念整理(转)
ER图分为实体.属性.关系三个核心部分.实体是长方形体现,而属性则是椭圆形,关系为菱形. ER图的实体(entity)即数据模型中的数据对象,例如人.学生.音乐都可以作为一个数据对象,用长方体来表示, ...
- HTML编码的用户输入------阻止向Controller的方法传入参数时用链接注入javascript代码或者HTML标记
- vue 监听返回
mounted: function() { //使用keep-alive时可以放在activated内 if (window.history && window.history.pus ...
- COCI2014/2015 Contest#1 D MAFIJA【基环树最大独立点集】
T1725 天黑请闭眼 Online Judge:COCI2014/2015 Contest#1 D MAFIJA(原题) Label:基环树,断环+树形Dp,贪心+拓扑 题目描述 最近天黑请闭眼在 ...
- NOIP2018提高组初赛选讲
说实话,这次的初赛比上一次的要简单. 不过还有些变态的题目. 在一条长度为1 的线段上随机取两个点,则以这两个点为端点的线段的期望 长度是( ). A. 1 / 2 B. 1 / 3 C. 2 / 3 ...
- 廖雪峰Java11多线程编程-1线程的概念-3线程的状态
1线程的状态 线程终止的的原因: run()或call()方法执行完成,线程正常结束 线程抛出一个未捕获的Exception或Error 直接调用该线程的stop()方法来结束该线程--该方法容易导致 ...
- 廖雪峰Java10加密与安全-3摘要算法-1MD5
1.摘要算法 1.1 摘要算法(哈希算法/Hash/数字指纹): 计算任意长度数据的摘要(固定长度) 相同的输入数据始终得到相同的输出 不同的输入尽量得到不同的输出 1.2 摘要算法目的: 验证数据和 ...
- day48作业
使用Bootstrap框架编写一个简单的web静态页面 效果图: <!DOCTYPE html> <html lang="en"> <head> ...
- select有条件in要按照in中的数据排序
mybatis中的写法 <select id="selectByIds" resultType="com.hoohui.electronic.util.ExHash ...
- 出现大量rcuob进程
方法一: vim /etc/default/grub,在GRUB_CMDLINE_LINUX这一行末尾添加nr_cpus=2,然后执行命令grub2-mkconfig -o /boot/grub2/g ...