在机器学习中,导致overfitting的原因之一是noise,这个noise可以分为两种,即stochastic noise,随机噪声来自数据产生过程,比如测量误差等,和deterministic noise,确定性噪声来自added complexity,即model too complex。这两种类型的造成来源不同,但是对于学习的影响是相似的,large noise总会导致overfitting。


This is a very subtle question!

The most important thing to realize is that in learning, H is fixed and D is given, and so can be assumed fixed. Now we can ask, what is going on in this learning scenario. Here is what we can say:

i) If there is stochastic noise with ‘magnitude’ σ2, then you are in trouble.

ii) If there deterministic noise then you are in trouble.

The stochastic noise can be viewed as one part of the data generation process (eg. measurement errors). The deterministic noise can similarly be viewed as another part of the data generation process, namely f. The deterministic and stochastic noise are fixed. In your analogy, you can increase the stochastic noise by increasing the noise variance and you get into deeper trouble. Similarly, you can increase the deterministic noise by making f more complex and you will get into deeper trouble.

I just need to tell you what ‘trouble’ means. Well, we actually use another word instead of ‘trouble’ - overfitting.

This means you may be likely to make an inferior choice over the superior choice because the inferior choice has lower in-sample error. Doing stuff that looks good in-sample that leads to disasters out-of-sample is the essence of overfitting. An example of this is trying to choose the regularization parameter. If you pick a lower regularization parameter, then you have lower in-sample error, but it leads to higher out-of-sample error - you picked the λ with lowerEinbut it gave higher Eout. We call that overfitting. Underfitting is just the name we give to the opposite process in the context of picking the regularization parameter. Once the regularization parameter gets too high, as you pick a higher λ you get both higher Einand higher Eout. It also turns out that this means you over regularized and obtained an over-simplistic g - i.e. you ‘underfitted’, you didn’t fit the data enough. The underfitting and overfitting are just terms. The substance of what is going on under the hood is how the deterministic and stochastic noise are affecting what you should and should not do in-sample.

Now let’s get back to the subtle part of your question. There is actually another way to decrease the deterministic noise - increase the complexity of H (the other way is to decrease the complexity of f which we discussed above). Now is where the difference with stochastic noise pops up. With stochastic noise, it either goes up or down; if down, then things get better. With deterministic noise, if you just tell me that it went down, I need to ask you how. Did your target function get simpler - if yes, then great, it is just as if the stochastic noise went down. If it is that your H got more complicated, then things get interesting. To understand what is going on, the Bias Variance decomposition helps (bottom of page 125 in the textbook).

Eout=σ2+bias+var

σ2is the direct impact of the stochastic noise. bias is the direct impact of the deterministic noise. The var term is interesting and is the indirect impact of the noise, through H. The var term is mostly controlled by the size of H in relation to the number of data points. So getting back to the point, if you make H more complex, you will decrease the det. noise (bias) but you will increase the var (its indirect impact). Usually the latter dominates (overfitting, not because of the direct impact of the noise, but because of its indirect impact) … unless you are in the underfitting regime when the former dominates.

上面一段主要摘自《learning from data》一书,主要说明的内容是overfitting的含义以及noise对于overfitting的效用。

下面是对overfitting的很好的总结:

VC维大=>模型复杂度高=>error in sample 小=>模型不够平滑=>generalization能力弱=>error out of sample大=>overfitting=>模型并没有卵用。

总的来说,deterministic noise是由于你选择的H中的最好的hypothesis h∗对于不在H中的function f进行估计时的差。在给定x后,这个deterministic noise就确定了。

deterministic function可用来生成伪随机数(pseudo-random generator)。

详细的论述可以参看《learning from data》


2015-8-27

艺少

stochastic noise and deterministic noise的更多相关文章

  1. Perlin Noise 及其应用

    Perlin Noise 可以用来表现自然界中无法用简单形状来表达的物体的形态,比如火焰.烟雾.表面纹路等.要生成 Perlin Noise 可以使用工具离线生成,也可以使用代码运行时生成.最简单常用 ...

  2. python perlin noise

    python 利用 noise 生成纹理. # -*- coding: utf-8 -*- """ Created on Mon Apr 23 20:04:41 2018 ...

  3. GraphicsLab Project 之 Curl Noise

    作者:i_dovelemon 日期:2020-04-25 主题:Perlin Noise, Curl Noise, Finite Difference Method 引言 最近在研究流体效果相关的模拟 ...

  4. 台大《机器学习基石》课程感受和总结---Part 1(转)

    期末终于过去了,看看别人的总结:http://blog.sina.com.cn/s/blog_641289eb0101dynu.html 接触机器学习也有几年了,不过仍然只是个菜鸟,当初接触的时候英文 ...

  5. 过度拟合(overfitting)

    我们之前解决过一个理论问题:机器学习能不能起作用?现在来解决另一个理论问题:过度拟合. 正如之前我们看到的,很多时候我们必须进行nonlinear transform.但是我们又无法确定Q的值.Q过小 ...

  6. 【Regularization】林轩田机器学习基石

    正则化的提出,是因为要解决overfitting的问题. 以Linear Regression为例:低次多项式拟合的效果可能会好于高次多项式拟合的效果. 这里回顾上上节nonlinear transf ...

  7. 【Hazard of Overfitting】林轩田机器学习基石

    首先明确了什么是Overfitting 随后,用开车的例子给出了Overfitting的出现原因 出现原因有三个: (1)dvc太高,模型过于复杂(开车开太快) (2)data中噪声太大(路面太颠簸) ...

  8. 过拟合产生的原因(Root of Overfitting)

    之前在<过拟合和欠拟合(Over fitting & Under fitting)>一文中简要地介绍了过拟合现象,现在来详细地分析一下过拟合产生的原因以及相应的解决办法. 过拟合产 ...

  9. {ICIP2014}{收录论文列表}

    This article come from HEREARS-L1: Learning Tuesday 10:30–12:30; Oral Session; Room: Leonard de Vinc ...

随机推荐

  1. 在运维中的shell经验总结

    来自良许Linux公众号 编写 脚本开头部分应有脚本功能说明.参数使用说明.作者姓名.创建/修改日期.版本信息,格式为: 脚本编写时,注意格式对齐,如所有的循环或者判断语句前后的语句进行对齐,以及ca ...

  2. 第12组 Alpha冲刺(2/6)

    Header 队名:To Be Done 组长博客 作业博客 团队项目进行情况 燃尽图(组内共享) 展示Git当日代码/文档签入记录(组内共享) 注: 由于GitHub的免费范围内对多人开发存在较多限 ...

  3. 理解 ES6 语法中 yield* 关键字的作用

    在 ES6 中新增了生成器函数的语法,本文解释了与生成器函数有关的 yield* 关键字,及其使用场景. 描述 根据语法规范,yield* 的作用是代理 yield 表达式,将需要函数本身产生(yie ...

  4. 「SCOI2011」棘手的操作

    传送门 Description 有\(N\)个节点,标号从\(1\)到\(N\),这\(N\)个节点一开始相互不连通.第$ i\(个节点的初始权值为\)a_i$ ,接下来有如下一些操作: U x y ...

  5. Java 中HashTable、HashMap、TreeMap三者区别,以及自定义对象是否相同比较,自定义排序等

    /* Map集合:该集合存储键值对.一对一对往里存.而且要保证键的唯一性. Map |--Hashtable:底层是哈希表数据结构,不可以存入null键null值.该集合是线程同步的.效率低.基本已废 ...

  6. adb-andorid记录当前手机的日志当前显示的app进程及activity,

    adb logcat -v time> /home/sumsang.log adb shell dumpsys window | grep mCurrentFocus

  7. HTTPWebrequest上传文件--Upload files with HTTPWebrequest (multipart/form-data)

    使用HTTPWebrequest上传文件遇到问题,可以参考Upload files with HTTPWebrequest (multipart/form-data)来解决 https://stack ...

  8. 000 基于Spring boot发送邮件

    发送邮件的程序,使用QQ的服务器,经过测试,完全可行.可复现 一:准备工作 1.找到账号的授权码 这个是程序需要使用的. 在设置中查找. 2.新建项目的目录 二:完整的程序代码 1.pom.xml & ...

  9. 【转载】 tensorflow中 tf.train.slice_input_producer 和 tf.train.batch 函数

    原文地址: https://blog.csdn.net/dcrmg/article/details/79776876 ----------------------------------------- ...

  10. ISO/IEC 9899:2011 条款6.7——声明

    6.7 声明 语法 1.declaration: declaration-specifiers    init-declarator-listopt    ; static_assert-declar ...