kaggle Data Leakage

What is Data Leakage¶

Data leakage is one of the most important issues for a data scientist to understand. If you don't know how to prevent it, leakage will come up frequently, and it will ruin your models in the most subtle and dangerous ways. Specifically, leakage causes a model to look accurate until you start making decisions with the model, and then the model becomes very inaccurate. This tutorial will show you what leakage is and how to avoid it.

There are two main types of leakage: Leaky Predictors and a Leaky Validation Strategies.

Leaky Predictors

This occurs when your predictors include data that will not be available at the time you make predictions.

For example, imagine you want to predict who will get sick with pneumonia. The top few rows of your raw data might look like this:

got_pneumonia	age	weight	male	took_antibiotic_medicine	...
False	65	100	False	False	...
False	72	130	True	False	...
True	58	100	False	True	...

-

People take antibiotic medicines after getting pneumonia in order to recover. So the raw data shows a strong relationship between those columns. But took_antibiotic_medicine is frequently changed after the value for got_pneumonia is determined. This is target leakage.

The model would see that anyone who has a value of False for took_antibiotic_medicine didn't have pneumonia. Validation data comes from the same source, so the pattern will repeat itself in validation, and the model will have great validation (or cross-validation) scores. But the model will be very inaccurate when subsequently deployed in the real world.

To prevent this type of data leakage, any variable updated (or created) after the target value is realized should be excluded. Because when we use this model to make new predictions, that data won't be available to the model.

Leaky Validation Strategy

A much different type of leak occurs when you aren't careful distinguishing training data from validation data. For example, this happens if you run preprocessing (like fitting the Imputer for missing values) before calling train_test_split. Validation is meant to be a measure of how the model does on data it hasn't considered before. You can corrupt this process in subtle ways if the validation data affects the preprocessing behavoir.. The end result? Your model will get very good validation scores, giving you great confidence in it, but perform poorly when you deploy it to make decisions.

Preventing Leaky Predictors

There is no single solution that universally prevents leaky predictors. It requires knowledge about your data, case-specific inspection and common sense.

However, leaky predictors frequently have high statistical correlations to the target. So two tactics to keep in mind:

• To screen for possible leaky predictors, look for columns that are statistically correlated to your target.
• If you build a model and find it extremely accurate, you likely have a leakage problem.

Preventing Leaky Validation Strategies

If your validation is based on a simple train-test split, exclude the validation data from any type of fitting, including the fitting of preprocessing steps. This is easier if you use scikit-learn Pipelines. When using cross-validation, it's even more critical that you use pipelines and do your preprocessing inside the pipeline.

Example

We will use a small dataset about credit card applications, and we will build a model predicting which applications were accepted (stored in a variable called card). Here is a look at the data:

 

import pandas as pd

data = pd.read_csv('../input/AER_credit_card_data.csv',
                   true_values = ['yes'],
                   false_values = ['no'])
print(data.head())

 

   card  reports       age  income     share  expenditure  owner  selfemp  \
0  True        0  37.66667  4.5200  0.033270   124.983300   True    False
1  True        0  33.25000  2.4200  0.005217     9.854167  False    False
2  True        0  33.66667  4.5000  0.004156    15.000000   True    False
3  True        0  30.50000  2.5400  0.065214   137.869200  False    False
4  True        0  32.16667  9.7867  0.067051   546.503300   True    False   

   dependents  months  majorcards  active
0           3      54           1      12
1           3      34           1      13
2           4      58           1       5
3           0      25           1       7
4           2      64           1       5

 

We can see with data.shape that this is a small dataset (1312 rows), so we should use cross-validation to ensure accurate measures of model quality

 

data.shape

 

(1319, 12)

 

from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

y = data.card
X = data.drop(['card'], axis=1)

# Since there was no preprocessing, we didn't need a pipeline here. Used anyway as best practice
modeling_pipeline = make_pipeline(RandomForestClassifier())
cv_scores = cross_val_score(modeling_pipeline, X, y, scoring='accuracy')
print("Cross-val accuracy: %f" %cv_scores.mean())

 

Cross-val accuracy: 0.979528

 

With experience, you'll find that it's very rare to find models that are accurate 98% of the time. It happens, but it's rare enough that we should inspect the data more closely to see if it is target leakage.

Here is a summary of the data, which you can also find under the data tab:

• card: Dummy variable, 1 if application for credit card accepted, 0 if not
• reports: Number of major derogatory reports
• age: Age n years plus twelfths of a year
• income: Yearly income (divided by 10,000)
• share: Ratio of monthly credit card expenditure to yearly income
• expenditure: Average monthly credit card expenditure
• owner: 1 if owns their home, 0 if rent
• selfempl: 1 if self employed, 0 if not.
• dependents: 1 + number of dependents
• months: Months living at current address
• majorcards: Number of major credit cards held
• active: Number of active credit accounts

A few variables look suspicious. For example, does expenditure mean expenditure on this card or on cards used before appying?

At this point, basic data comparisons can be very helpful:

 

expenditures_cardholders = data.expenditure[data.card]
expenditures_noncardholders = data.expenditure[~data.card]

print('Fraction of those who received a card with no expenditures: %.2f' \
      %(( expenditures_cardholders == 0).mean()))
print('Fraction of those who received a card with no expenditures: %.2f' \
      %((expenditures_noncardholders == 0).mean()))

 

Fraction of those who received a card with no expenditures: 0.02
Fraction of those who received a card with no expenditures: 1.00

 

Everyone with card == False had no expenditures, while only 2% of those with card == True had no expenditures. It's not surprising that our model appeared to have a high accuracy. But this seems a data leak, where expenditures probably means *expenditures on the card they applied for.**.

Since share is partially determined by expenditure, it should be excluded too. The variables active, majorcards are a little less clear, but from the description, they sound concerning. In most situations, it's better to be safe than sorry if you can't track down the people who created the data to find out more.

We would run a model without leakage as follows:

 

potential_leaks = ['expenditure', 'share', 'active', 'majorcards']
X2 = X.drop(potential_leaks, axis=1)
cv_scores = cross_val_score(modeling_pipeline, X2, y, scoring='accuracy')
print("Cross-val accuracy: %f" %cv_scores.mean())

 

Cross-val accuracy: 0.806677

 

This accuracy is quite a bit lower, which on the one hand is disappointing. However, we can expect it to be right about 80% of the time when used on new applications, whereas the leaky model would likely do much worse then that (even in spite of it's higher apparent score in cross-validation.).

Conclusion

Data leakage can be multi-million dollar mistake in many data science applications. Careful separation of training and validation data is a first step, and pipelines can help implement this separation. Leaking predictors are a more frequent issue, and leaking predictors are harder to track down. A combination of caution, common sense and data exploration can help identify leaking predictors so you remove them from your model.

有极高的精准度，代表可能发生数据泄露。

Data Leakage是因果关系的纰漏，是由于数据准备过程中出现的失误，使模型沿着有纰漏的，甚至是颠倒的因果关系进行预测，但得到极好的预测结果。

按我的理解就是，用于训练X中的某个或某几个特征与最终要预测的y有强烈的因果关系，预测后这些特征的权重很大(对应的theta值)，导致模型最终由这几个特诊所主导。

Data Leakage 基本都是在准备数据的时候，或者数据采样的时候出了问题，误将与结果直接相关的feature纳入了数据集。这样的纰漏，比较难以发现。

处理方法是丢掉这些特征列。