Logistic Regression – Geometric Intuition
Logistic Regression – Geometric Intuition
Everybody who has taken a machine learning course probably knows the geometric intuition behind a support vector machine (SVM, great book): A SVM is a large margin classifier. In other words, it maximizes the geometric distance between the decision boundary and the classes of samples. Often you’ll find plots similar to this one:

But what about logistic regression? What is the geometric intuition behind it and how does it compare to linear SVMs? Let’s find out.
Geometric intuition behind logistic regression
First, a quick reminder about the definition of the logistic function, given features:
With that out of the way, let’s dive into the geometric aspects of logistic regression, starting with a toy example of only one feature:

The blue samples of our feature data belong to class , while the red dots are supposed to get assigned to
. As we can see in the figure above, our data is fully separable and the logistic regression is able to demonstrate this property. From a geometric perspective, the algorithm essentially adds a new dimension for our dependent variable
and fits a logistic function
to the arising two-dimensional data in such a way that it separates the samples as good as possible within the range of
. Once we have such a fitted curve, the usual method of prediction is to assign
to everything
and vice versa.
When thinking about logistic regression, I usually have its tight connection to linear regression somewhere in the back of my head (no surprise, both have “regression” as part of their names). Therefore, I was wondering what the line would look like and added it to the plot. As we can easily see,
differs significantly from the potential result of a linear regression for the given input data. While that was to be expected, I think it’s still nice to confirm it visually.
Note two details of the plot: For one, the logistic curve is not as steep as you might expect at first. This is due to the use of regularization, which logistic regression basically can’t live without. Second, for
and not, as my intuition initially suggested, for
. Not sure if this is particularly helpful, but I find it interesting nonetheless.
So far, so good. But what happens when we introduce an additional dimension? Let’s have a look at such a two feature scenario (feel free to pinch and zoom the plot):

What we see is the surface of the fitted logistic function for a classification task with two features. This time, instead of transforming a line to a curve, the equivalent happened for a plane. To make it easier to understand how to carry out predictions given such a model, the following plot visualizes the data points projected onto the surface of the logistic regression function:

Now we can again make decisions for our class based on a threshold
, which usually is
:
So much about the geometric intuition behind logistic regression. Let’s see how that compares to linear SVMs.
Comparison of logistic regression to linear SVM
First of all, notice that with logistic regression we can use the value of a sample as a probability estimate for
. The responsive 3D-plots above created with the help of plotly’s fantastic library illustrate this property of logistic regression very well. I have yet to come across a real-world classification task where such probability estimates are not at least somewhat useful. Alas, SVMs (in their original version) aren’t able to provide them. Due to the way the model works, all it can tell you is whether a sample belongs to class
or not.
I think at this point the most effective way of comparing logistic regression to linear SVM geometrically, is to add the decision boundary of logistic regression to the initial figure of the post. To make this happen, we need to project the decision boundary of the three-dimensional plot above onto the two-dimensional feature space. Doing so, we end up with the following figure:

As we can see, the logistic regression does not follow the same large margin characteristic as a linear SVM. I’ll leave it to you to decide which is the better fit for the given data.
If you’d like to know further differences between logistic regression and linear SVMs aside from the geometrically motivated ones mentioned here, have a look at the excellent answers on quora.
The code for generating the plots of this post can be found on github.
Logistic Regression – Geometric Intuition的更多相关文章
- 课程一(Neural Networks and Deep Learning),第二周(Basics of Neural Network programming)—— 4、Logistic Regression with a Neural Network mindset
Logistic Regression with a Neural Network mindset Welcome to the first (required) programming exerci ...
- 机器学习 (三) 逻辑回归 Logistic Regression
文章内容均来自斯坦福大学的Andrew Ng教授讲解的Machine Learning课程,本文是针对该课程的个人学习笔记,如有疏漏,请以原课程所讲述内容为准.感谢博主Rachel Zhang 的个人 ...
- 逻辑回归 Logistic Regression
逻辑回归(Logistic Regression)是广义线性回归的一种.逻辑回归是用来做分类任务的常用算法.分类任务的目标是找一个函数,把观测值匹配到相关的类和标签上.比如一个人有没有病,又因为噪声的 ...
- logistic regression与SVM
Logistic模型和SVM都是用于二分类,现在大概说一下两者的区别 ① 寻找最优超平面的方法不同 形象点说,Logistic模型找的那个超平面,是尽量让所有点都远离它,而SVM寻找的那个超平面,是只 ...
- Logistic Regression - Formula Deduction
Sigmoid Function \[ \sigma(z)=\frac{1}{1+e^{(-z)}} \] feature: axial symmetry: \[ \sigma(z)+ \sigma( ...
- SparkMLlib之 logistic regression源码分析
最近在研究机器学习,使用的工具是spark,本文是针对spar最新的源码Spark1.6.0的MLlib中的logistic regression, linear regression进行源码分析,其 ...
- [OpenCV] Samples 06: [ML] logistic regression
logistic regression,这个算法只能解决简单的线性二分类,在众多的机器学习分类算法中并不出众,但它能被改进为多分类,并换了另外一个名字softmax, 这可是深度学习中响当当的分类算法 ...
- Stanford机器学习笔记-2.Logistic Regression
Content: 2 Logistic Regression. 2.1 Classification. 2.2 Hypothesis representation. 2.2.1 Interpretin ...
- Logistic Regression vs Decision Trees vs SVM: Part II
This is the 2nd part of the series. Read the first part here: Logistic Regression Vs Decision Trees ...
随机推荐
- 最简单的iOS网络请求
做iOS开发,说到网络请求,大家可能都不约而同的提到AFN,可以说大家的网络请求都是用AFN封装而成,AFN的强大易用的确很好. 但是版本升级就会出现一些问题,所以就自己基于iOS原生封装了一个网络请 ...
- grunt源码解析:整体运行机制&grunt-cli源码解析
前端的童鞋对grunt应该不陌生,前面也陆陆续续的写了几篇grunt入门的文章.本篇文章会更进一步,对grunt的源码进行分析.文章大体内容内容如下: grunt整体设计概览 grunt-cli源码分 ...
- Win环境 Android Studio使用Git 教程 ( 生成SSH key )
Github和码云都提供SSH协议,即用户可以用公钥认证方式连接到码云的SSH服务器.这就需要生成并部署SSH Key.下面就是我生成SSH Key的步骤,希望有所帮助: Git生成SSH key 在 ...
- Alpha版本项目展示得分
团队名称 得分 newbe 80 C705 100 ourteam 60 sevens 50 sixsix 190 dxteam 75 hots 200 Echo 90
- M1阶段事后总结
设想和目标 1. 我们的软件要解决什么问题?是否定义得很清楚?是否对典型用户和典型场景有清晰的描述?我们组要爬取网上的内容供下一组使用,定义的不太清楚,因为用户只有下一个团队所以没有进行详细的需求分析 ...
- We're Chronos! ----- Meet the team 团队作业 #1
Meet Us —————————————————La ligne de séparation————————————————— Kane Lim [林珣玙] < PM, Programmer ...
- dispatch_block_t
通常我写一个不带参数的块回调函数是这样写的 在 . h 头文件中 定义类型 typedef void (^leftBlockAction)(); 在定义一个回调函数 -(void)leftButton ...
- PAT 甲级 1145 Hashing - Average Search Time
https://pintia.cn/problem-sets/994805342720868352/problems/994805343236767744 The task of this probl ...
- Docker 将一堆镜像 导成一个文件
docker save istio/galley istio/citadel istio/mixer istio/sidecar_injector istio/proxy_init istio/pro ...
- ES6学习笔记(五):Class和Module
Class Class 只是一个语法糖,其功能用es5也能实现,但是比es5更符合类的期待 定义: constructor代表构造方法,而this指向new 生成的实例 定义类方法时,可以不使用fun ...