[Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset

The Dataset was acquired from https://www.kaggle.com/c/titanic

For data preprocessing, I firstly defined three transformers:

• DataFrameSelector: Select features to handle.
• CombinedAttributesAdder: Add a categorical feature Age_cat which divided all passengers into three catagories according to their ages.
• ImputeMostFrequent: Since the SimpleImputer( ) method was only suitable for numerical variables, I wrote an transformer to impute string missing values with the mode value. Here I was inspired by https://stackoverflow.com/questions/25239958/impute-categorical-missing-values-in-scikit-learn.

Then I wrote pipelines separately for different features

• For numerical features, I applied DataFrameSelector, SimpleImputer and StandardScaler
• For categorical features, I applied DataFrameSelector, ImputeMostFrequent and OneHotEncoder
• For the new created feature Age_cat, since itself was a category but was derived from a numerical feature, I wrote an individual pipeline to impute the missing values and encode the categories.

Finally, we can build a full pipeline through FeatureUnion. Here is the code:

 # Read data
 import pandas as pd
 import numpy as np
 import os
 titanic_train = pd.read_csv('Dataset/Titanic/train.csv')
 titanic_test = pd.read_csv('Dataset/Titanic/test.csv')
 submission = pd.read_csv('Dataset/Titanic/gender_submission.csv')

 # Divide attributes and labels
 titanic_labels = titanic_train['Survived'].copy()
 titanic = titanic_train.drop(['Survived'],axis=1)

 # Feature Selection
 from sklearn.base import BaseEstimator, TransformerMixin

 class DataFrameSelector(BaseEstimator, TransformerMixin):
     def __init__(self,attribute_name):
         self.attribute_name = attribute_name
     def fit(self, X):
         return self
     def transform (self, X, y=None):
         if 'Pclass' in self.attribute_name:
             X['Pclass'] = X['Pclass'].astype(str)
         return X[self.attribute_name]

 # Feature Creation
 class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
     def fit(self, X, y=None):
         return self  # nothing else to do
     def transform(self, X, y=None):
         Age_cat = pd.cut(X['Age'],[0,18,60,100],labels=['child', 'adult', 'old'])
         Age_cat=np.array(Age_cat)
         return pd.DataFrame(Age_cat,columns=['Age_Cat'])

 # Impute Categorical variables
 class ImputeMostFrequent(BaseEstimator, TransformerMixin):
     def fit(self, X, y=None):
         self.fill = pd.Series([X[c].value_counts().index[0] for c in X],index=X.columns)
         return self
     def transform(self, X, y=None):
         return X.fillna(self.fill)

 #Pipeline
 from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+
 from sklearn.pipeline import Pipeline
 from sklearn.preprocessing import StandardScaler
 from sklearn.preprocessing import OneHotEncoder
 from sklearn.pipeline import FeatureUnion

 num_pipeline = Pipeline([
     ('selector',DataFrameSelector(['Age','SibSp','Parch','Fare'])),
     ('imputer', SimpleImputer(strategy="median")),
     ('std_scaler', StandardScaler()),
 ])

 cat_pipeline = Pipeline([
     ('selector',DataFrameSelector(['Pclass','Sex','Embarked'])),
     ('imputer',ImputeMostFrequent()),
     ('encoder', OneHotEncoder()),
 ])

 new_pipeline = Pipeline([
     ('selector',DataFrameSelector(['Age'])),
     #('imputer', SimpleImputer(strategy="median")),
     ('attr_adder',CombinedAttributesAdder()),
     ('imputer',ImputeMostFrequent()),
     ('encoder', OneHotEncoder()),
 ])

 full_pipeline = FeatureUnion([
     ("num", num_pipeline),
     ("cat", cat_pipeline),
     ("new", new_pipeline),
 ])

 titanic_prepared = full_pipeline.fit_transform(titanic)

Another thing I want to mention is that the output of a pipeline should be a 2D array rather a 1D array. So if you wanna choose only one feature, don't forget to transform the 1D array by reshape() method. Otherwise, you will receive an error like

ValueError: Expected 2D array, got 1D array instead

Specifically, apply reshape(-1,1) for column and reshape(1,-1). More about the issue can be found at https://stackoverflow.com/questions/51150153/valueerror-expected-2d-array-got-1d-array-instead.