Pegasos: Primal Estimated sub-GrAdient Solver for SVM
Abstract
We describe and analyze a simple and effective iterative algorithm for solving the optimization problem cast by Support Vector Machines (SVM). Our method alternates between stochastic gradient descent steps and projection steps. We prove that the number of iterations required to obtain a solution of accuracy is . In contrast, previous analyses of stochastic gradient descent methods require iterations. As in previous devised SVM solvers, the number of iterations also scales linearly with , where is the regularization parameter of SVM. For a linear kernel, the total run-time of our method is , where is a bound on the number of non-zero features in each example. Since the run-time does not depend directly on the size of the training set, the resulting algorithm is especially suited for learning from large datasets. Our approach can seamlessly be adapted to employ non-linear kernels while working solely on the primal objective function. We demonstrate the efficiency and applicability of our approach by conducting experiments on large text classification problems, comparing our solver to existing state-of-the art SVM solvers. For example, it takes less than 5 seconds for our solver to converge when solving a text classification problem from Reuters Corpus Volume 1 (RCV 1) with training examples.
1. Introduction
Support Vector Machines (SVMs) are effective and popular classification learning tool. The task of learning a support vector machine is cast as a constrained quadratic programming. However, in its native form, it is in fact an unconstrained empirical loss minimization with a penalty term for the norm of the classifier that is being learned. Formally, given a training set , where and , we would like to find the minimizer of the problem
(1)
where
(1)
We denote the objective function of Eq. (1) by . An optimization method finds an -accurate solution if . The original SVM problem also includes a bias term, . We omit the bias throughout the first sections and defer the description of an extension which employs a bias term to Sec. 4.
We describe and analyze in this paper a simple iterative algorithm, called Pegasos, for solving Eq. (1). The algorithm performs iterations and also requires an additional parameter , whose role is explained in the sequel. Pegasos alternates between stochastic subgradient descent steps and projection steps. The parameter determines the number of examples from the algorithm uses on each iteration for estimating the subgradient. When , Pegasos reduces to a variant of the subgradient projection method. We show that in this case the number of iterations that is required in order to achieve an -accurate solution is . At the other extreme, when , we recover a variant of the stochastic (sub) gradient method. In the stochastic case, we analyze the probability of obtaining a good approximate solution. Specifically, we show that with probability of at least our algorithm finds an -accurate solution using only iterations, while each iteration involves a single inner product between and . This rate of convergence does not depend on the size of the training set and thus our algorithm is especially suited for large datasets.
2. The Pegasos Algorithm
In this section we describe the Pegasos algorithm for solving the optimization problem given in Eq. (1). The algorithm receives as input two parameters: - the number of iterations to perform; - the number of examples to use for calculating sub-gradients. Initially, we set to any vector whose norm is at most . On iteration of the algorithm, we first choose a set of size . Then, we replace the objective in Eq. (1) with an approximate objective function,
.
Note that we overloaded our original definition of as the original objective can be denoted either as or as . We interchangeably use both notations depending on the context. Next, we set the learning rate and define to be the set of examples for which suffers a non-zero loss. We now perform a two-step update as follows. We scale by and for all examples we add to the vector . We denote the resulting vector by . This step can be also written as , where
(1)
The definition of the hinge-loss implies that is a sub-gradient of at . Last, we set to be the projection of onto the set
(1)
That is, is obtained by scaling by . As we show in our analysis below, the optimal solution of SVM is in the set . Informally speaking, we can always project back onto the set as we only get closer to the optimum. The output of Pegasos is the last vector .
Note that if we choose on each round then we obtain the sub-gradient projection method. On the other extreme, if we choose to contain a single randomly selected example, then we recover a variant of the stochastic gradient method. In general, we allow to be a set of examples sampled i.i.d. from .
We conclude this section with a short discussion of implementation details when the instances are sparse, namely, when each instance has very few non-zero elements. In this case, we can represent as a triplet where is a dense vector and are scalars. The vector is defined through the triplet as follows: and stores the squared norm of , . Using this representation, it is easily verified that the total number of operations required for performing one iteration of Pegasos with is , where is the number of non-zero elements in .
3. Analysis
In this section we analyze the convergence properties of Pegasos. Throughout this section we denote
(1)
Recall that on each iteration of the algorithm, we focus on an instantaneous objective function
Pegasos: Primal Estimated sub-GrAdient Solver for SVM的更多相关文章
- [转] 从零推导支持向量机 (SVM)
原文连接 - https://zhuanlan.zhihu.com/p/31652569 摘要 支持向量机 (SVM) 是一个非常经典且高效的分类模型.但是,支持向量机中涉及许多复杂的数学推导,并需要 ...
- 损失函数(Loss Function) -1
http://www.ics.uci.edu/~dramanan/teaching/ics273a_winter08/lectures/lecture14.pdf Loss Function 损失函数 ...
- 常见数据挖掘算法的Map-Reduce策略(2)
接着上一篇文章常见算法的mapreduce案例(1)继续挖坑,本文涉及到算法的基本原理,文中会大概讲讲,但具体有关公式的推导还请大家去查阅相关的文献文章.下面涉及到的数据挖掘算法会有:L ...
- 逻辑回归原理_挑战者飞船事故和乳腺癌案例_Python和R_信用评分卡(AAA推荐)
sklearn实战-乳腺癌细胞数据挖掘(博客主亲自录制视频教程) https://study.163.com/course/introduction.htm?courseId=1005269003&a ...
- 基于MNIST数据的softmax regression
跟着tensorflow上mnist基本机器学习教程联系 首先了解sklearn接口: sklearn.linear_model.LogisticRegression In the multiclas ...
- [Scikit-learn] 1.1 Generalized Linear Models - Logistic regression & Softmax
二分类:Logistic regression 多分类:Softmax分类函数 对于损失函数,我们求其最小值, 对于似然函数,我们求其最大值. Logistic是loss function,即: 在逻 ...
- Factorization Machine算法
参考: http://stackbox.cn/2018-12-factorization-machine/ https://baijiahao.baidu.com/s?id=1641085157432 ...
- LibLinear(SVM包)使用说明之(一)README
转自:http://blog.csdn.net/zouxy09/article/details/10947323/ LibLinear(SVM包)使用说明之(一)README zouxy09@qq.c ...
- SVM应用
我在项目中应用的SVM库是国立台湾大学林智仁教授开发的一套开源软件,主要有LIBSVM与LIBLINEAR两个,LIBSVM是对非线性数据进行分类,大家也比较熟悉,LIBLINEAR是对线性数据进行分 ...
随机推荐
- Spring基于注解ehCache缓存整合
注解的使用参照:http://blog.csdn.net/wjacketcn/article/details/50945887 (侵删) ehCache是纯java实现的缓存组件,spring从3.1 ...
- 非标准JSON解析
http://blog.csdn.net/superit401/article/details/51734591 String category = "{'v-soft-list':[{ty ...
- spark快速大数据分析学习笔记*初始化sparkcontext(一)
初始化SparkContext 1// 在java中初始化spark import org.apache.spark.SparkConf; import org.apache.spark.api.ja ...
- SQL server 常用语句
SQL Server中常用的SQL语句 1.概述 2.查询概述 3.单表查询 4.连接查询 5.带有exists的相关子查询 6.SQL的集合操作 7.插入操作 8.删除操作 9.修改操作 10. ...
- linux配置3-安装tomcat
下载文件:apache-tomcat-7.0.73.tar.gz 通过共享传到Ubuntu, 复制到/tmp 解压 移动解压后的文件到到/opt/tomcat7,完成可见:/opt/tomcat7/a ...
- jquery 基础-Ajax应用
使用load()方案异步请求数据 调用格式:load(url,[data],[callback]) 参数url为加载服务器地址,可选项data参数为请求时发送的数据,callback参数为数据请求成功 ...
- 很重要的vue的生命周期
- angularjs 嵌套控制器,子控制器访问父控制器
<pre> http://www.lovelucy.info/understanding-scopes-in-angularjs.html http://blog.csdn.net/jfk ...
- Windows 10系统更换Windows 7系统磁盘分区注意事项一
新买的电脑预装系统是WIN10,考虑到兼容性问题,打算更换为WIN7,但在新机上不能直接装WIN7系统,需要在BIOS启动中做一点小改动. 原因分析:由于Windows 8采用的是UEFI引导和GPT ...
- SharpDevelop的亮点—优化提示
用惯了Visual Studio,像被惯坏了孩子,很难适应别的IDE. 上个月Win7系统崩溃了,重装后,自然VS也没了.这次下定决心,绝对不在自己电脑上用VS了,于是重新捡起SharpDevel ...