Logistic Regression – Geometric Intuition

Everybody who has taken a machine learning course probably knows the geometric intuition behind a support vector machine (SVM, great book): A SVM is a large margin classifier. In other words, it maximizes the geometric distance between the decision boundary and the classes of samples. Often you’ll find plots similar to this one:

But what about logistic regression? What is the geometric intuition behind it and how does it compare to linear SVMs? Let’s find out.

Geometric intuition behind logistic regression

First, a quick reminder about the definition of the logistic function, given features:

With that out of the way, let’s dive into the geometric aspects of logistic regression, starting with a toy example of only one feature:

The blue samples of our feature data belong to class , while the red dots are supposed to get assigned to . As we can see in the figure above, our data is fully separable and the logistic regression is able to demonstrate this property. From a geometric perspective, the algorithm essentially adds a new dimension for our dependent variable and fits a logistic function  to the arising two-dimensional data in such a way that it separates the samples as good as possible within the range of . Once we have such a fitted curve, the usual method of prediction is to assign to everything and vice versa.

When thinking about logistic regression, I usually have its tight connection to linear regression somewhere in the back of my head (no surprise, both have “regression” as part of their names). Therefore, I was wondering what the line would look like and added it to the plot. As we can easily see, differs significantly from the potential result of a linear regression for the given input data. While that was to be expected, I think it’s still nice to confirm it visually.

Note two details of the plot: For one, the logistic curve is not as steep as you might expect at first. This is due to the use of regularization, which logistic regression basically can’t live without. Second, for and not, as my intuition initially suggested, for . Not sure if this is particularly helpful, but I find it interesting nonetheless.

So far, so good. But what happens when we introduce an additional dimension? Let’s have a look at such a two feature scenario (feel free to pinch and zoom the plot):

What we see is the surface of the fitted logistic function  for a classification task with two features. This time, instead of transforming a line to a curve, the equivalent happened for a plane. To make it easier to understand how to carry out predictions given such a model, the following plot visualizes the data points projected onto the surface of the logistic regression function:

Now we can again make decisions for our class based on a threshold , which usually is :

So much about the geometric intuition behind logistic regression. Let’s see how that compares to linear SVMs.

Comparison of logistic regression to linear SVM

First of all, notice that with logistic regression we can use the value of a sample as a probability estimate for . The responsive 3D-plots above created with the help of plotly’s fantastic library illustrate this property of logistic regression very well. I have yet to come across a real-world classification task where such probability estimates are not at least somewhat useful. Alas, SVMs (in their original version) aren’t able to provide them. Due to the way the model works, all it can tell you is whether a sample belongs to class  or not.

I think at this point the most effective way of comparing logistic regression to linear SVM geometrically, is to add the decision boundary of logistic regression to the initial figure of the post. To make this happen, we need to project the decision boundary of the three-dimensional plot above onto the two-dimensional feature space. Doing so, we end up with the following figure:

As we can see, the logistic regression does not follow the same large margin characteristic as a linear SVM. I’ll leave it to you to decide which is the better fit for the given data.

If you’d like to know further differences between logistic regression and linear SVMs aside from the geometrically motivated ones mentioned here, have a look at the excellent answers on quora.

The code for generating the plots of this post can be found on github.

Logistic Regression – Geometric Intuition的更多相关文章

  1. 课程一(Neural Networks and Deep Learning),第二周(Basics of Neural Network programming)—— 4、Logistic Regression with a Neural Network mindset

    Logistic Regression with a Neural Network mindset Welcome to the first (required) programming exerci ...

  2. 机器学习 (三) 逻辑回归 Logistic Regression

    文章内容均来自斯坦福大学的Andrew Ng教授讲解的Machine Learning课程,本文是针对该课程的个人学习笔记,如有疏漏,请以原课程所讲述内容为准.感谢博主Rachel Zhang 的个人 ...

  3. 逻辑回归 Logistic Regression

    逻辑回归(Logistic Regression)是广义线性回归的一种.逻辑回归是用来做分类任务的常用算法.分类任务的目标是找一个函数,把观测值匹配到相关的类和标签上.比如一个人有没有病,又因为噪声的 ...

  4. logistic regression与SVM

    Logistic模型和SVM都是用于二分类,现在大概说一下两者的区别 ① 寻找最优超平面的方法不同 形象点说,Logistic模型找的那个超平面,是尽量让所有点都远离它,而SVM寻找的那个超平面,是只 ...

  5. Logistic Regression - Formula Deduction

    Sigmoid Function \[ \sigma(z)=\frac{1}{1+e^{(-z)}} \] feature: axial symmetry: \[ \sigma(z)+ \sigma( ...

  6. SparkMLlib之 logistic regression源码分析

    最近在研究机器学习,使用的工具是spark,本文是针对spar最新的源码Spark1.6.0的MLlib中的logistic regression, linear regression进行源码分析,其 ...

  7. [OpenCV] Samples 06: [ML] logistic regression

    logistic regression,这个算法只能解决简单的线性二分类,在众多的机器学习分类算法中并不出众,但它能被改进为多分类,并换了另外一个名字softmax, 这可是深度学习中响当当的分类算法 ...

  8. Stanford机器学习笔记-2.Logistic Regression

    Content: 2 Logistic Regression. 2.1 Classification. 2.2 Hypothesis representation. 2.2.1 Interpretin ...

  9. Logistic Regression vs Decision Trees vs SVM: Part II

    This is the 2nd part of the series. Read the first part here: Logistic Regression Vs Decision Trees ...

随机推荐

  1. B1030 完美数列 (25 分)

    这是一道二分法的题目,许久不使用二分法,感觉有点生疏. #include<bits/stdc++.h> using namespace std; const int MAXN=100000 ...

  2. 区块链学习:Windows下搭建以太坊私有链环境

    一:安装geth客户端 Windows要求必须是64位系统,从官方网站下载编译好的win64客户端,下载解压后只有一个Geth.exe问价,运行安装即可,下载地址如下: https://github. ...

  3. javascript 数组对象及其方法

    数组声明:通过let arr = new Array(); 或者 let arr = []; 数组对象可调用的方法: 1)find方法,使用情况是对数组进行筛选遍历,find方法要求某个函数(A)作为 ...

  4. webpack 学习笔记 (一)

    webpack 作为当下前端前沿最受欢迎的打包工具,作为一个前端开发人员是很有必要去了解下它的. 题外话: npm i -D 是 npm install  --save-dev的简写,是安装模块并保存 ...

  5. 利用mask-image蒙层编写异形头像

    需求:后台给了一个规规矩矩的头像,或圆或方,UI要求展示成水滴的形状.正在想到底如何实现的时候,不由自主去翻了鑫神的博客,正好找到了答案,窃喜(·_·) UI给的形状: 后台给的头像(忽略橙色背景色, ...

  6. [转帖]Programmer’s guide to the big tech companies 💻

    Programmer’s guide to the big tech companies

  7. [知乎]BAT占线

    黑色自有,蓝色全资收够,红色入股. https://www.zhihu.com/question/304396738/answer/547766603

  8. react & monaco editor & vs code

    react & monaco editor & vs code monaco-editor https://microsoft.github.io/monaco-editor/inde ...

  9. suoi14 子树查找 (dfs)

    把询问记下来,然后开个桶差分 #include<bits/stdc++.h> #define pa pair<int,int> #define lowb(x) ((x)& ...

  10. 【bzoj2844】 albus就是要第一个出场

    http://www.lydsy.com/JudgeOnline/problem.php?id=2844 (题目链接) 题意 给出${n}$个数,它们可以异或出${n^2}$个数,将这些数从小到大排列 ...