pandas（进阶操作）-- 政治献金项目数据分析

博客地址：https://www.cnblogs.com/zylyehuo/

开发环境

• anaconda
  • 集成环境：集成好了数据分析和机器学习中所需要的全部环境
  • 安装目录不可以有中文和特殊符号
• jupyter
  • anaconda提供的一个基于浏览器的可视化开发工具

import numpy as np
import pandas as pd

为了方便操作，将月份和参选人以及所在政党进行定义

months = {'JAN' : 1, 'FEB' : 2, 'MAR' : 3, 'APR' : 4, 'MAY' : 5, 'JUN' : 6,
          'JUL' : 7, 'AUG' : 8, 'SEP' : 9, 'OCT': 10, 'NOV': 11, 'DEC' : 12}
of_interest = ['Obama, Barack', 'Romney, Mitt', 'Santorum, Rick',
               'Paul, Ron', 'Gingrich, Newt']
parties = {
  'Bachmann, Michelle': 'Republican',
  'Romney, Mitt': 'Republican',
  'Obama, Barack': 'Democrat',
  "Roemer, Charles E. 'Buddy' III": 'Reform',
  'Pawlenty, Timothy': 'Republican',
  'Johnson, Gary Earl': 'Libertarian',
  'Paul, Ron': 'Republican',
  'Santorum, Rick': 'Republican',
  'Cain, Herman': 'Republican',
  'Gingrich, Newt': 'Republican',
  'McCotter, Thaddeus G': 'Republican',
  'Huntsman, Jon': 'Republican',
  'Perry, Rick': 'Republican'
 }

需求

• 加载数据
• 查看数据的基本信息
• 指定数据截取，将如下字段的数据进行提取，其他数据舍弃
  • cand_nm ：候选人姓名
  • contbr_nm ： 捐赠人姓名
  • contbr_st ：捐赠人所在州
  • contbr_employer ： 捐赠人所在公司
  • contbr_occupation ： 捐赠人职业
  • contb_receipt_amt ：捐赠数额（美元）
  • contb_receipt_dt ： 捐款的日期
• 对新数据进行总览,查看是否存在缺失数据
• 用统计学指标快速描述数值型属性的概要。
• 空值处理。可能因为忘记填写或者保密等等原因，相关字段出现了空值，将其填充为NOT PROVIDE
• 异常值处理。将捐款金额<=0的数据删除
• 新建一列为各个候选人所在党派party
• 查看party这一列中有哪些不同的元素
• 统计party列中各个元素出现次数
• 查看各个党派收到的政治献金总数contb_receipt_amt
• 查看具体每天各个党派收到的政治献金总数contb_receipt_amt
• 将表中日期格式转换为'yyyy-mm-dd'
• 查看老兵(捐献者职业)DISABLED VETERAN主要支持谁

加载数据并查看数据的基本信息

df = pd.read_csv('./data/usa_election.txt')
df

对数据进行总览,查看是否存在缺失数据

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 536041 entries, 0 to 536040
Data columns (total 16 columns):
 #   Column             Non-Null Count   Dtype
---  ------             --------------   -----
 0   cmte_id            536041 non-null  object
 1   cand_id            536041 non-null  object
 2   cand_nm            536041 non-null  object
 3   contbr_nm          536041 non-null  object
 4   contbr_city        536026 non-null  object
 5   contbr_st          536040 non-null  object
 6   contbr_zip         535973 non-null  object
 7   contbr_employer    525088 non-null  object
 8   contbr_occupation  530520 non-null  object
 9   contb_receipt_amt  536041 non-null  float64
 10  contb_receipt_dt   536041 non-null  object
 11  receipt_desc       8479 non-null    object
 12  memo_cd            49718 non-null   object
 13  memo_text          52740 non-null   object
 14  form_tp            536041 non-null  object
 15  file_num           536041 non-null  int64
dtypes: float64(1), int64(1), object(14)
memory usage: 65.4+ MB

用统计学指标快速描述数值型属性的概要

df.describe()

空值处理。可能因为忘记填写或者保密等等原因，相关字段出现了空值，将其填充为NOT PROVIDE

df.fillna(value='NOT PROVIDE',inplace=True)
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 536041 entries, 0 to 536040
Data columns (total 16 columns):
 #   Column             Non-Null Count   Dtype
---  ------             --------------   -----
 0   cmte_id            536041 non-null  object
 1   cand_id            536041 non-null  object
 2   cand_nm            536041 non-null  object
 3   contbr_nm          536041 non-null  object
 4   contbr_city        536041 non-null  object
 5   contbr_st          536041 non-null  object
 6   contbr_zip         536041 non-null  object
 7   contbr_employer    536041 non-null  object
 8   contbr_occupation  536041 non-null  object
 9   contb_receipt_amt  536041 non-null  float64
 10  contb_receipt_dt   536041 non-null  object
 11  receipt_desc       536041 non-null  object
 12  memo_cd            536041 non-null  object
 13  memo_text          536041 non-null  object
 14  form_tp            536041 non-null  object
 15  file_num           536041 non-null  int64
dtypes: float64(1), int64(1), object(14)
memory usage: 65.4+ MB

异常值处理。将捐款金额<=0的数据删除

df['contb_receipt_amt'] <= 0  # 判断哪些值为小于等于0
df.loc[df['contb_receipt_amt'] <= 0]  # 捐赠金额小于等于0的行数据
drop_indexs = df.loc[df['contb_receipt_amt'] <= 0].index
df.drop(labels=drop_indexs,axis=0,inplace=True)
df

新建一列为各个候选人所在党派party

df['party'] = df['cand_nm'].map(parties)
df.head()

查看party这一列中有哪些不同的元素

df['party'].unique()

array(['Republican', 'Democrat', 'Reform', 'Libertarian'], dtype=object)

统计party列中各个元素出现次数

df['party'].value_counts()

Democrat       289999
Republican     234300
Reform           5313
Libertarian       702
Name: party, dtype: int64

查看各个党派收到的政治献金总数contb_receipt_amt

df.groupby(by='party')['contb_receipt_amt'].sum()

party
Democrat       8.259441e+07
Libertarian    4.132769e+05
Reform         3.429658e+05
Republican     1.251181e+08
Name: contb_receipt_amt, dtype: float64

查看具体每天各个党派收到的政治献金总数contb_receipt_amt

df.groupby(by=['contb_receipt_dt','party'])['contb_receipt_amt'].sum()

contb_receipt_dt  party
01-APR-11         Reform             50.00
                  Republican      12635.00
01-AUG-11         Democrat       182198.00
                  Libertarian      1000.00
                  Reform           1847.00
                                   ...
31-MAY-11         Republican     313839.80
31-OCT-11         Democrat       216971.87
                  Libertarian      4250.00
                  Reform           3205.00
                  Republican     751542.36
Name: contb_receipt_amt, Length: 1183, dtype: float64

将表中日期格式转换为'yyyy-mm-dd'

def trandformDate(d):
    day,month,year = d.split('-')
    month = months[month]#将英文形式的月份转换成了数字形式的月份
    return '20'+year+'-'+str(month)+'-'+day

df['contb_receipt_dt'] = df['contb_receipt_dt'].map(trandformDate)
df.head()

查看老兵(捐献者职业)DISABLED VETERAN主要支持谁

• 给谁捐赠的钱越多表示越支持谁

# 可以先将源数据中的老兵这个职业对应的行数据取出
df['contbr_occupation'] == 'DISABLED VETERAN'

0         False
1         False
2         False
3         False
4         False
          ...
536036    False
536037    False
536038    False
536039    False
536040    False
Name: contbr_occupation, Length: 530314, dtype: bool

df_old = df.loc[df['contbr_occupation'] == 'DISABLED VETERAN']
df_old

# 分组：根据候选人分组，对捐赠金额求和
df_old.groupby(by='cand_nm')['contb_receipt_amt'].sum()

cand_nm
Cain, Herman       300.00
Obama, Barack     4205.00
Paul, Ron         2425.49
Santorum, Rick     250.00
Name: contb_receipt_amt, dtype: float64