Kaggle-tiantic数据建模与分析

1.数据可视化

kaggle中数据解释：https://www.kaggle.com/c/titanic/data

数据形式：

读取数据，并显示数据信息

data_train = pd.read_csv("./data/train.csv")

print(data_train.info())

数据结果如下：

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 891 entries, 0 to 890

Data columns (total 12 columns):

PassengerId    891 non-null int64

Survived       891 non-null int64

Pclass         891 non-null int64

Name           891 non-null object

Sex            891 non-null object

Age            714 non-null float64

SibSp          891 non-null int64

Parch          891 non-null int64

Ticket         891 non-null object

Fare           891 non-null float64

Cabin          204 non-null object

Embarked       889 non-null object

数据解释：

PassengerId => 乘客ID

Survive => 乘客是否生还（仅在训练集中有，测试集中没有）

Pclass => 乘客等级(1/2/3等舱位)

Name => 乘客姓名

Sex => 性别

Age => 年龄

SibSp => 堂兄弟/妹个数

Parch => 父母与小孩个数

Ticket => 船票信息

Fare => 票价

Cabin => 客舱

Embarked => 登船港口

1.1 生存/死亡人数统计

# # 统计 存活/死亡 人数

def sur_die_analysis(data_train):

    fig = plt.figure()

    fig.set(alpha=0.2)  # 设定图表颜色alpha参数

    data_train.Survived.value_counts().plot(kind='bar')# 柱状图

    plt.title(u"获救情况 (1为获救)") # 标题

    plt.ylabel(u"人数")

    plt.show()

1.2 PClass

# PClass

def pclass_analysis(data_train):

    fig = plt.figure()

    fig.set(alpha=0.2)  # 设定图表颜色alpha参数

    sur_data = data_train.Pclass[data_train.Survived == 1].value_counts()

    die_data = data_train.Pclass[data_train.Survived == 0].value_counts()

    pd.DataFrame({'Survived':sur_data,'Died':die_data}).plot(kind='bar')

    plt.ylabel(u"人数")

    plt.title(u"乘客等级分布")

    plt.show()

通过数据分布可以很明显的看出 Pclass 为 1/2 的乘客存活率比 3 的高很多

1.3 Sex

#Sex

def sex_analysis(data_train):

    no_survived_g = data_train.Sex[data_train.Survived == 0].value_counts()

    no_survived_g.to_csv("no_survived_g.csv")

    survived_g = data_train.Sex[data_train.Survived == 1].value_counts()

    df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})

    df_g.plot(kind='bar', stacked=True)

    plt.title('性别存活率分析')

    plt.xlabel('People')

    plt.ylabel('Survive')

    plt.show()

女性的存活率比男性高

1.4 Age

# age : 将年龄分成十段，分别统计 存活人数和死亡人数

def age_analysis(data_train):

    data_series = pd.DataFrame(columns=['Survived', 'dies'])

    cloms = []

    for num in range(0, 10):

        clo  = "" + str(num * 10) + "-" + str((num + 1) * 10)

        cloms.append(clo)

        sur_df = data_train.Age[(10 * (num + 1) > data_train.Age) & (10 * num < data_train.Age) & (data_train.Survived == 1)].shape[0]

        die_df = data_train.Age[(10 * (num + 1) > data_train.Age) & (10 * num < data_train.Age) & (data_train.Survived == 0)].shape[0]

        data_series.loc[num] = [sur_df,die_df]

    data_series.index = cloms

    data_series.plot(kind='bar', stacked=True)

    plt.ylabel(u"存活率")  # 设定纵坐标名称

    plt.grid(b=True, which='major', axis='y')

    plt.title(u"按年龄看获救分布")

    plt.show()

低年龄段的获救的百分比明显占的比例较多

1.5 Family ： SibSp + Parch

定义Family项，代表家庭成员数量，并离散分类为三个等级：

0：代表没有任何成员

1： 1-4

2： > 4

# Family: Sibsp + Parch 家庭成员人数

def family_analysis(data_train):

    data_train['Family'] = data_train['SibSp'] + data_train['Parch']

    data_train.loc[(data_train.Family == 0), 'Family'] = 0

    data_train.loc[((data_train.Family > 0) & (data_train.Family < 4)), 'Family'] = 1

    data_train.loc[((data_train.Family >= 4)), 'Family'] = 2

    no_survived_g = data_train.Family[data_train.Survived == 0].value_counts()

    survived_g = data_train.Family[data_train.Survived == 1].value_counts()

    df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})

    df_g.plot(kind='bar', stacked=True)

    plt.title('家庭成员分析')

    plt.xlabel('等级：0-无 1-（1~4） 2-（>4）')

    plt.ylabel('存活情况')

    plt.show()

由于数据分布很不均衡，sibsp 是否和存活率的关系，可以将所有列都除以该列总人数。这里不再赘述。

1.6 Fare

费用统计：

当费用升高到一定时，存活人数已经超过了死亡人数

# Fare

def fare_analysis(data_train):

    # data_train.Fare[data_train.Survived == 1].plot(kind='kde')

    # data_train.Fare[data_train.Survived == 0].plot(kind='kde')

    # data_train["Fare"].plot(kind='kde')

    # plt.legend(('survived', 'died','all'), loc='best')

    # plt.show()

    data_train['NewFare'] = data_train['Fare']

    data_train.loc[(data_train.Fare < 50), 'NewFare'] = 0

    data_train.loc[((data_train.Fare>=50) & (data_train.Fare<100)), 'NewFare'] = 1

    data_train.loc[((data_train.Fare >= 100) & (data_train.Fare < 150)), 'NewFare'] = 2

    data_train.loc[((data_train.Fare >= 150) & (data_train.Fare < 200)), 'NewFare'] = 3

    data_train.loc[(data_train.Fare >= 200), 'NewFare'] = 4

    no_survived_g = data_train.NewFare[data_train.Survived == 0].value_counts()

    survived_g = data_train.NewFare[data_train.Survived == 1].value_counts()

    df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})

    df_g.plot(kind='bar', stacked=True)

    plt.title('费用-生存分析')

    plt.xlabel('费用等级')

    plt.ylabel('存活情况')

    plt.show()

很明显可以看出费用等级较高的人存活率会高很多。

优化：

上述只是任意的选取了五个费用段，作为五类，但是具体是多少类才能最好的拟合数据？

这里可以通过聚类的方法查找最佳的分类个数，再将每个费用数据映射为其中一类：

def fare_kmeans(data_train):

    for i in range(2,10):

        clusters = KMeans(n_clusters=i)

        clusters.fit(data_train['Fare'].values.reshape(-1,1))

        # intertia_ 参数是衡量聚类的效果，越大则表明效果越差

        print("" + str(i) + "" + str(clusters.inertia_))

打印结果：

 2 846932.9762272763

 3 399906.26606199215

 4 195618.50643749788

 5 104945.73652631264

 6 52749.474696547695

 7 35141.316334118805

 8 26030.553497795216

 9 19501.242236941747

由此可以看出看出当类别数为 5 时分类的效果最好。所以这里将所有的费用映射到为这五类。

#将费用进行聚类，发现 类别数为 5 时聚合的效果最好

def fare_kmeans(data_train):

    clusters = KMeans(n_clusters=5)

    clusters.fit(data_train['Fare'].values.reshape(-1, 1))

    predict = clusters.predict(data_train['Fare'].values.reshape(-1, 1))

    print(predict)

    data_train['NewFare'] = predict

    print(data_train[['NewFare','Survived']].groupby(['NewFare'],as_index=False).mean())

    print(""  + str(clusters.inertia_))

等级映射后每个等级的存活率如下：(效果明显比上面随便分类的好)

  NewFare  Survived

0        0  0.319832

1        1  0.647059

2        2  0.606557

3        3  1.000000

4        4  0.757576

1.7 Embarked

#Embarked 上船港口情况

def embarked_analysis(data_train):

    no_survived_g = data_train.Embarked[data_train.Survived == 0].value_counts()

    survived_g = data_train.Embarked[data_train.Survived == 1].value_counts()

    df_g = pd.DataFrame({'Survived': survived_g, 'Died': no_survived_g})

    df_g.plot(kind='bar', stacked=True)

    plt.title('登陆港口-存活情况分析')

    plt.xlabel('Embarked')

    plt.ylabel('Survive')

    plt.show()

至于就登陆港口而言，三个港口并看不出明显的差距，C港生还率略高于S港与Q港。

2. 数据预处理

由开头部分数据信息可以看出，有几栏的数据是部分缺失的： Age / Cabin / Embarked

对于缺失数据这里选择简单填充的方式进行处理：（可以以中值/均值/众数等方式填充）

同时对费用进行分类

def dataPreprocess(df):

    df.loc[df['Sex'] == 'male', 'Sex'] = 0

    df.loc[df['Sex'] == 'female', 'Sex'] = 1

    # 由于 Embarked中有两个数据未填充，需要先将数据填满

    df['Embarked'] = df['Embarked'].fillna('S')

    # 部分年龄数据未空， 填充为 均值

    df['Age'] = df['Age'].fillna(df['Age'].median())

    df.loc[df['Embarked']=='S', 'Embarked'] = 0

    df.loc[df['Embarked'] == 'C', 'Embarked'] = 1

    df.loc[df['Embarked'] == 'Q', 'Embarked'] = 2

    df['FamilySize'] = df['SibSp'] + df['Parch']

    df['IsAlone'] = 0

    df.loc[df['FamilySize']==0,'IsAlone'] = 1

    df.drop('FamilySize',axis = 1)

    df.drop('Parch',axis=1)

    df.drop('SibSp',axis=1)

    return  fare_kmeans(df)

def fare_kmeans(data_train):

    clusters = KMeans(n_clusters=5)

    clusters.fit(data_train['Fare'].values.reshape(-1, 1))

    predict = clusters.predict(data_train['Fare'].values.reshape(-1, 1))

    data_train['NewFare'] = predict

    data_train.drop('Fare')

    # print(data_train[['NewFare','Survived']].groupby(['NewFare'],as_index=False).mean())

    # print(" "  + str(clusters.inertia_))

    return data_train

这里对与分类特征通过了普通的编码方式进行实现，也可以通过onehot编码使每种分类之间的间隔相等。

3. 特征选择

上述感性的认识了各个特征与存活率之间的关系，其实sklearn库中提供了对每个特征打分的函数，可以很方便的看出各个特征的重要性

predictors = ["Pclass", "Sex", "Age", "NewFare", "Embarked",'IsAlone']

    # Perform feature selection

    selector = SelectKBest(f_classif, k=5)

    selector.fit(data_train[predictors], data_train["Survived"])

    # Plot the raw p-values for each feature,and transform from p-values into scores

    scores = -np.log10(selector.pvalues_)

    # Plot the scores.   See how "Pclass","Sex","Title",and "Fare" are the best?

    plt.bar(range(len(predictors)),scores)

    plt.xticks(range(len(predictors)),predictors, rotation='vertical')

    plt.show()

上图可以看到输入的6个特征中那些特征比较重要

4. 线性回归建模

def linearRegression(df):

    predictors = ['Pclass', 'Sex', 'Age', 'IsAlone', 'NewFare', 'Embarked']

    #predictors = ['Pclass', 'Sex', 'Age', 'IsAlone', 'NewFare', 'EmbarkedS','EmbarkedC','EmbarkedQ']

    alg = LinearRegression()

    X = df[predictors]

    Y = df['Survived']

    X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.2)

    # 打印 训练集 测试集 样本数量

    print (X_train.shape)

    print (Y_train.shape)

    print (X_test.shape)

    print (Y_test.shape)

    # 进行拟合

    alg.fit(X_train, Y_train)

    print (alg.intercept_)

    print (alg.coef_)

    Y_predict = alg.predict(X_test)

    Y_predict[Y_predict >= 0.5 ] = 1

    Y_predict[Y_predict < 0.5] = 0

    acc = sum(Y_predict==Y_test) / len(Y_predict)

    return acc

测试模型预测准确率： 0.79

5. 随机森林建模

选取最有价值的5个特征进行模型训练，并验证模型的效果：

def randomForest(data_train):

    # Pick only the four best features.

    predictors = ["Pclass", "Sex", "NewFare", "Embarked", 'IsAlone']

    X_train, X_test, Y_train, Y_test = train_test_split(data_train[predictors], data_train['Survived'], test_size=0.2)

    alg = RandomForestClassifier(random_state=1, n_estimators=50, min_samples_split=8, min_samples_leaf=4)

    alg.fit(X_train, Y_train)

    Y_predict = alg.predict(X_test)

    acc = sum(Y_predict == Y_test) / len(Y_predict)

    return acc

经过测试该模型的准确率为 0.811

初步原因分析：选取的5个特征中没有Age，Age可能因为缺失很大部分数据对预测的准确率有一定的影响。

代码已经提交git： https://github.com/lsfzlj/kaggle

欢迎指正交流

参考：

https://blog.csdn.net/han_xiaoyang/article/details/49797143

https://blog.csdn.net/CSDN_Black/article/details/80309542

https://www.kaggle.com/sinakhorami/titanic-best-working-classifier