Your Prediction Gets As Good As Your Data

May 5, 2015 by Kazem

In the past, we have seen software engineers and data scientists assume that they can keep increasing their prediction accuracy by improving their machine learning algorithm. Here, we want to approach the classification problem from a different angle where we recommend data scientists should analyze the distribution of their data first to measure information level in data. This approach can givesus an upper bound for how far one can improve the accuracy of a predictive algorithm and make sure our optimization efforts are not wasted!

Entropy and Information

In information theory, mathematician have developed a few useful techniques such as entropy to measure information level in data in process. Let's think of a random coin with a head probability of 1%.

If one filps such a coin, we will get more information when we see the head event since it's a rare event compared to tail which is more likely to happen. We can formualte the amount of information in a random variable with the negative logarithm of the event probability. This captures the described intuition. Mathmatician also formulated another measure called entropy by which they capture the average information in a random process in bits. Below we have shown the entropy formula for a discrete random variable:



For the first example, let's assume we have a coin with P(H)=0% and P(T)=100%. We can compute the entropy of the coin as follows:



For the second example, let's consider a coin where P(H)=1% and P(T)=1-P(H)=99%. Plugging numbers one can find that the entropy of such a coin is:



Finally, if the coin has P(H) = P(T) = 0.5 (i.e. a fair coin), its entropy is calculated as follows:



Entropy and Predictability

So, what these examples tell us? If we have a coin with head probability of zero, the coin's entropy is zero meaning that the average information in the coin is zero. This makes sense because flipping such a coin always comes as tail. Thus, the prediction accuracy is 100%. In other words, when the entropy is zero, we have the maximum predictibility.

In the second example, head probability is not zero but still very close to zero which again makes the coin to be very predictable with a low entropy.

Finally, in the last example we have 50/50 chance of seeing head/tail events which maximizes the entropy and consequently minimizes the predictability. In words, one can show that a fair coin has the meaximum entropy of 1 bit making the prediction as good as a random guess.

Kullback–Leibler divergence

As last example, it's important to give another example of how we can borrow ideas from information theory to measure the distance between two probability distributions. Let's assume we are modeling two random processes by their pmf's: P(.) and Q(.). One can use entropy measure to compute the distance between two pmf's as follows:



Above distance function is known as KL divergence which measures the distance of Q's pmf from P's pmf. The KL divergence can come handy in various problems such as NLP problems where we'd like to measure the distance between two sets of data (e.g. bag of words).

Wrap-up

In this post, we showed that the entropy from information theory provides a way to measure how much information exists in our data. We also highlighted the inverse relationship between the entropy and the predictability. This shows that we can use the entropy measure to calculate an upper bound for the accuracy of the prediction problem in hand.

Feel free to share with us if you have any comments or questions in the comment section below.

You can also reach us at info@AIOptify.com

Your Prediction Gets As Good As Your Data的更多相关文章

  1. Lessons Learned from Developing a Data Product

    Lessons Learned from Developing a Data Product For an assignment I was asked to develop a visual ‘da ...

  2. A Brief Review of Supervised Learning

    There are a number of algorithms that are typically used for system identification, adaptive control ...

  3. 微软职位内部推荐-Software Engineer II

    微软近期Open的职位: Job Description Group: Search Technology Center Asia (STCA)/Search Ads Title: SDEII-Sen ...

  4. 在opencv3中实现机器学习算法之:利用最近邻算法(knn)实现手写数字分类

    手写数字digits分类,这可是深度学习算法的入门练习.而且还有专门的手写数字MINIST库.opencv提供了一张手写数字图片给我们,先来看看 这是一张密密麻麻的手写数字图:图片大小为1000*20 ...

  5. 在opencv3中的机器学习算法练习:对OCR进行分类

    OCR (Optical Character Recognition,光学字符识别),我们这个练习就是对OCR英文字母进行识别.得到一张OCR图片后,提取出字符相关的ROI图像,并且大小归一化,整个图 ...

  6. Libsvm:脚本(subset.py、grid.py、checkdata.py) | MATLAB/OCTAVE interface | Python interface

    1.脚本 This directory includes some useful codes: 1. subset selection tools. (子集抽取工具) subset.py 2. par ...

  7. Nagios工作原理

    图解Nagios的工作原理 Nagios的主动模式和被动模式 被动模式:就如同上图所显示的那样,客户端起nrpe进程,服务端通过check_nrpe插件向客户端发送命令,客户端根据服务端的指示来调用相 ...

  8. 学习笔记TF020:序列标注、手写小写字母OCR数据集、双向RNN

    序列标注(sequence labelling),输入序列每一帧预测一个类别.OCR(Optical Character Recognition 光学字符识别). MIT口语系统研究组Rob Kass ...

  9. OpenCV OpenGL手写字符识别

    另外一篇文章地址:这个比较详细,但是程序略显简单,现在这个程序是比较复杂的 http://blog.csdn.net/wangyaninglm/article/details/17091901 整个项 ...

随机推荐

  1. Yaml学习文档

    pdf文档地址 http://yaml.org/spec/ JS-Yaml demo地址 http://nodeca.github.io/js-yaml/

  2. DRF框架QQ登录功能

    用户模块---QQ登录 流程图 QQ登录文档:http://wiki.connect.qq.com/%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C_oauth2-0 流程简述 ...

  3. [T-ARA][놀아볼래?][要玩吗]

    歌词来源:http://music.163.com/#/song?id=22704479 作曲 : 赵英秀/김태현 [作曲 : 赵英秀/k/gim-Tae-hyeon] 作词 : 安英民 [作词 : ...

  4. mysql数据出现Unknown column 'user_uid' in 'field list' sql错误

    来源:https://blog.csdn.net/gnail_oug/article/details/53606608 在操作mysql数据库时提示com.mysql.jdbc.exceptions. ...

  5. Spring学习总结之装配bean

    1.  XML中显式配置 规范,文件头: <?xml version=”1.0” encoding=”UTF-8”?>            <beans xmlns=http:// ...

  6. sql not in 优化问题

    问题情境: not in 耗时过长.想用join或exits代替.结果并不明显,这里先记录3种写法,以后探讨速度问题. sql语句: // not exists sql = @"select ...

  7. Daily Scrum - 12/04

    Meeting Minutes 与Qiufeng, Bojie, Zhongqiu, Ran, Travis一起讨论. 确定了最后的UI及Feature. 开始Code Review, 以及有计划的M ...

  8. 4种PHP回调函数风格

    4种PHP回调函数风格 匿名函数 $server->on('Request', function ($req, $resp) use ($a, $b, $c) { echo "hell ...

  9. 在PHPStorm中快速插入当前日期

    在EditPlus中使用快捷键Ctrl+D即可插入当前日期,但在PHPStorm中似乎没有这样的快捷键,那如何实现快速插入当前日期呢?其实很简单,跟我做一遍你就会了: 目标 为PHPStorm定义一个 ...

  10. 使用navicat 链接数据库时乱码

    在建立数据库链接时设置  高级->编码->uft-8 其他版本使用下面方法