The Dataset was acquired from https://www.kaggle.com/c/titanic

For data preprocessing, I firstly defined three transformers:

  • DataFrameSelector: Select features to handle.
  • CombinedAttributesAdder: Add a categorical feature Age_cat which divided all passengers into three catagories according to their ages.
  • ImputeMostFrequent: Since the SimpleImputer( ) method was only suitable for numerical variables, I wrote an transformer to impute string missing values with the mode value. Here I was inspired by https://stackoverflow.com/questions/25239958/impute-categorical-missing-values-in-scikit-learn.

Then I wrote pipelines separately for different features

  • For numerical features, I applied DataFrameSelector, SimpleImputer and StandardScaler
  • For categorical features, I applied DataFrameSelector, ImputeMostFrequent and OneHotEncoder
  • For the new created feature Age_cat, since itself was a category but was derived from a numerical feature, I wrote an individual pipeline to impute the missing values and encode the categories.

Finally, we can build a full pipeline through FeatureUnion. Here is the code:

 # Read data
import pandas as pd
import numpy as np
import os
titanic_train = pd.read_csv('Dataset/Titanic/train.csv')
titanic_test = pd.read_csv('Dataset/Titanic/test.csv')
submission = pd.read_csv('Dataset/Titanic/gender_submission.csv') # Divide attributes and labels
titanic_labels = titanic_train['Survived'].copy()
titanic = titanic_train.drop(['Survived'],axis=1) # Feature Selection
from sklearn.base import BaseEstimator, TransformerMixin class DataFrameSelector(BaseEstimator, TransformerMixin):
def __init__(self,attribute_name):
self.attribute_name = attribute_name
def fit(self, X):
return self
def transform (self, X, y=None):
if 'Pclass' in self.attribute_name:
X['Pclass'] = X['Pclass'].astype(str)
return X[self.attribute_name] # Feature Creation
class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self # nothing else to do
def transform(self, X, y=None):
Age_cat = pd.cut(X['Age'],[0,18,60,100],labels=['child', 'adult', 'old'])
Age_cat=np.array(Age_cat)
return pd.DataFrame(Age_cat,columns=['Age_Cat']) # Impute Categorical variables
class ImputeMostFrequent(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
self.fill = pd.Series([X[c].value_counts().index[0] for c in X],index=X.columns)
return self
def transform(self, X, y=None):
return X.fillna(self.fill) #Pipeline
from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import FeatureUnion num_pipeline = Pipeline([
('selector',DataFrameSelector(['Age','SibSp','Parch','Fare'])),
('imputer', SimpleImputer(strategy="median")),
('std_scaler', StandardScaler()),
]) cat_pipeline = Pipeline([
('selector',DataFrameSelector(['Pclass','Sex','Embarked'])),
('imputer',ImputeMostFrequent()),
('encoder', OneHotEncoder()),
]) new_pipeline = Pipeline([
('selector',DataFrameSelector(['Age'])),
#('imputer', SimpleImputer(strategy="median")),
('attr_adder',CombinedAttributesAdder()),
('imputer',ImputeMostFrequent()),
('encoder', OneHotEncoder()),
]) full_pipeline = FeatureUnion([
("num", num_pipeline),
("cat", cat_pipeline),
("new", new_pipeline),
]) titanic_prepared = full_pipeline.fit_transform(titanic)

Another thing I want to mention is that the output of a pipeline should be a 2D array rather a 1D array. So if you wanna choose only one feature, don't forget to transform the 1D array by reshape() method. Otherwise, you will receive an error like

ValueError: Expected 2D array, got 1D array instead

Specifically, apply reshape(-1,1) for column and reshape(1,-1). More about the issue can be found at https://stackoverflow.com/questions/51150153/valueerror-expected-2d-array-got-1d-array-instead.


												

[Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset的更多相关文章

  1. Getting started with machine learning in Python

    Getting started with machine learning in Python Machine learning is a field that uses algorithms to ...

  2. 《Learning scikit-learn Machine Learning in Python》chapter1

    前言 由于实验原因,准备入坑 python 机器学习,而 python 机器学习常用的包就是 scikit-learn ,准备先了解一下这个工具.在这里搜了有 scikit-learn 关键字的书,找 ...

  3. Python (1) - 7 Steps to Mastering Machine Learning With Python

    Step 1: Basic Python Skills install Anacondaincluding numpy, scikit-learn, and matplotlib Step 2: Fo ...

  4. 【Machine Learning】Python开发工具:Anaconda+Sublime

    Python开发工具:Anaconda+Sublime 作者:白宁超 2016年12月23日21:24:51 摘要:随着机器学习和深度学习的热潮,各种图书层出不穷.然而多数是基础理论知识介绍,缺乏实现 ...

  5. Machine Learning的Python环境设置

    Machine Learning目前经常使用的语言有Python.R和MATLAB.如果采用Python,需要安装大量的数学相关和Machine Learning的包.一般安装Anaconda,可以把 ...

  6. [Machine Learning with Python] Data Preparation through Transformation Pipeline

    In the former article "Data Preparation by Pandas and Scikit-Learn", we discussed about a ...

  7. [Machine Learning with Python] Data Preparation by Pandas and Scikit-Learn

    In this article, we dicuss some main steps in data preparation. Drop Labels Firstly, we drop labels ...

  8. [Machine Learning with Python] Familiar with Your Data

    Here I list some useful functions in Python to get familiar with your data. As an example, we load a ...

  9. [Machine Learning with Python] How to get your data?

    Using Pandas Library The simplest way is to read data from .csv files and store it as a data frame o ...

随机推荐

  1. JVM——Java类加载机制总结

    )解析:解析阶段是把虚拟机中常量池的符号引用替换为直接引用的过程. 2.3 初始化 类初始化时类加载的最后一步,前面除了加载阶段用户可以通过自定义类加载器参与以外,其余都是虚拟机主导和控制.到了初始化 ...

  2. C#操作XML配置文件

    代码为C#操作xml配置文件的范例类,函数SetValue用于向配置文件写入一个值,GetValue用于根据Key获取相应值。这种方法的配置文件不需要手动创建,程序在运行后会自动处理创建。 注意:1. ...

  3. 【Best Time to Buy and Sell Stock II】cpp

    题目: Say you have an array for which the ith element is the price of a given stock on day i. Design a ...

  4. 【Remove Elements】cpp

    题目: Given an array and a value, remove all instances of that value in place and return the new lengt ...

  5. Mybatis使用-Error attempting to get column 'type' from result set. / '255' in column '4' is outside valid range for the datatype TINYINT.

    一.遇到的问题是这样的: [RemoteTestNG] detected TestNG version 6.9.10log4j: Parsing for [root] with value=[DEBU ...

  6. csapp读书笔记-并发编程

    这是基础,理解不能有偏差 如果线程/进程的逻辑控制流在时间上重叠,那么就是并发的.我们可以将并发看成是一种os内核用来运行多个应用程序的实例,但是并发不仅在内核,在应用程序中的角色也很重要. 在应用级 ...

  7. linux压缩文件——解压方法

    linux下 tar解压 gz解压 bz2等各种解压文件使用方法 .tar 解包:tar xvf FileName.tar 打包:tar cvf FileName.tar DirName (注:tar ...

  8. docker (centOS 7) 使用笔记6 - skydns

    skydns被用于kubenets作为DNS服务.本次测试是单独使用skydns作为DNS服务器,且作为loadbalance使用. 前提:需要先安装配置etcd服务 (在前面的文章里,已经安装部署了 ...

  9. filesystem

    1 tmpfs 以下来源于维基百科: tmpfs是类Unix系统上暂存档存储空间的常见名称,通常以挂载文件系统方式实现,并将数据存储在易失性存储器而非永久存储设备中.和RAM disk的概念近似,但后 ...

  10. nginx进行项目域名配置时提示Job for nginx.service failed

    ps aux | grep nginx /bin/systemctl stop nginx.service /bin/systemctl start nginx.service /bin/system ...