Three ways to detect outliers
Z-score
import numpy as np def outliers_z_score(ys):
threshold = 3 mean_y = np.mean(ys)
stdev_y = np.std(ys)
z_scores = [(y - mean_y) / stdev_y for y in ys]
return np.where(np.abs(z_scores) > threshold)
Modified Z-score
import numpy as np def outliers_modified_z_score(ys):
threshold = 3.5 median_y = np.median(ys)
median_absolute_deviation_y = np.median([np.abs(y - median_y) for y in ys])
modified_z_scores = [0.6745 * (y - median_y) / median_absolute_deviation_y
for y in ys]
return np.where(np.abs(modified_z_scores) > threshold)
IQR(interquartile range)
import numpy as np def outliers_iqr(ys):
quartile_1, quartile_3 = np.percentile(ys, [25, 75])
iqr = quartile_3 - quartile_1
lower_bound = quartile_1 - (iqr * 1.5)
upper_bound = quartile_3 + (iqr * 1.5)
return np.where((ys > upper_bound) | (ys < lower_bound))
Conclusion
It is important to reiterate that these methods should not be used mechanically.
They should be used to explore the data – they let you know which points might be worth a closer look.
What to do with this information depends heavily on the situation.
Sometimes it is appropriate to exclude outliers from a dataset to make a model trained on that dataset more predictive.
Sometimes, however,
the presence of outliers is a warning sign that the real-world process generating the data is more complicated than expected. As an astute commenter on CrossValidated put it:
“Sometimes outliers are bad data, and should be excluded, such as typos.
Sometimes they are Wayne Gretzky or Michael Jordan, and should be kept.” Domain knowledge and practical wisdom are the only ways to tell the difference.
摘自:http://colingorrie.github.io/outlier-detection.html
Three ways to detect outliers的更多相关文章
- MATLAB remove outliers.
Answer by Richard Willey on 9 Jan 2012 Hi Michael MATLAB doesn't provide a specific function to remo ...
- 异常值处理outlier
python信用评分卡(附代码,博主录制) https://study.163.com/course/introduction.htm?courseId=1005214003&utm_camp ...
- 深度学习图像配准 Image Registration: From SIFT to Deep Learning
Image Registration is a fundamental step in Computer Vision. In this article, we present OpenCV feat ...
- (转)利用libcurl和国内著名的两个物联网云端通讯的例程, ubuntu和openwrt下调试成功(四)
1. libcurl 的参考文档如下 CURLOPT_HEADERFUNCTION Pass a pointer to a function that matches the following pr ...
- 如何查找物理cpu,cpu核心和逻辑cpu的数量
环境 Red Hat Enterprise Linux 4 Red Hat Enterprise Linux 5 Red Hat Enterprise Linux 6 Red Hat Enterpri ...
- Gartner 2018 年WAF魔力象限报告:云WAF持续增长,Bot管理与API安全拥有未来
Gartner 2018 年WAF魔力象限报告:云WAF持续增长,Bot管理与API安全拥有未来 来源 https://www.freebuf.com/articles/paper/184903.ht ...
- A Gentle Guide to Machine Learning
A Gentle Guide to Machine Learning Machine Learning is a subfield within Artificial Intelligence tha ...
- [linux]linuxI/O测试的方法之dd
参考http://www.thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_dd Measuring Write Performan ...
- WGCNA构建基因共表达网络详细教程
这篇文章更多的是对于混乱的中文资源的梳理,并补充了一些没有提到的重要参数,希望大家不会踩坑. 1. 简介 1.1 背景 WGCNA(weighted gene co-expression networ ...
随机推荐
- 【English】20190415
approximately大约 [əˈprɑ:ksɪmətli] This install will take + minutes and requires the download of appro ...
- solr 7.7 搭建和搜索
最近做了个solr搜索的demo, 用的是solr7.7,之前网上好多文章搭建solr都是5.5+tomcat.. 自己摆弄了下,发现solr7.7和5.5 的部分文件夹有些改动,没有深究原理也就没有 ...
- Django-CRM项目学习(二)-模仿admin实现stark
开始今日份整理 1.stark模块基本操作 1.1 stark模块的启动 保证django自动的加载每一个app下的stark.py文件 创建django项目,创建stark项目,start app ...
- android 获取通话记录
在manifest添加以下权限<uses-permission android:name="android.permission.READ_CALL_LOG" />&l ...
- 使用springMVC时的web.xml配置文件
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" " ...
- 前端——JavaScript
何谓JavaScript?它与Java有什么关系? JavaScript与HTML.CSS组合使用应用于前端开发,JavaScript是一门独立的语言,浏览器内置了JS的解释器.它除了和Java名字长 ...
- P5239 回忆京都(洛谷3月月赛T2)
题目描述 射命丸文在取材中发现了一个好玩的东西,叫做组合数. 组合数的定义如下:从n个不同元素中,任取m(m≤n)个元素并成一组,叫做从n个不同元素中取出m个元素的一个组合.所有组合的数量,就是组合数 ...
- 六招轻松搞定你的CentOS系统安全加固
Redhat是目前企业中用的最多的一类Linux,而目前针对Redhat攻击的黑客也越来越多了.我们要如何为这类服务器做好安全加固工作呢? 一. 账户安全 1.1 锁定系统中多余的自建帐号 检查方 ...
- C# 递归构造树状数据结构(泛型),如何构造?如何查询?
十年河东,十年河西,莫欺少年穷. 学无止境,精益求精 难得有清闲的一上午,索性写篇博客. 首先,我们需要准备一张表,如下范例: create table TreeTable ( TreeId ) no ...
- 定时任务 cron命令
一名linux新手,因为项目需要现在在学习linux操作系统. 因项目需要做一个定时任务,每个小时访问某一个url一次,通过参考很多大神的博客学会了crontab命令的使用方法 crontab -u ...