2019-ICLR-DARTS: Differentiable Architecture Search-论文阅读
DARTS
2019-ICLR-DARTS Differentiable Architecture Search
- Hanxiao Liu、Karen Simonyan、Yiming Yang
- GitHub:2.8k stars
- Citation:557
Motivation
Current NAS method:
- Computationally expensive: 2000/3000 GPU days
- Discrete search space, leads to a large number of architecture evaluations required.
Contribution
- Differentiable NAS method based on gradient decent.
- Both CNN(CV) and RNN(NLP).
- SOTA results on CIFAR-10 and PTB.
- Efficiency: (2000 GPU days VS 4 GPU days)
- Transferable: cifar10 to ImageNet, (PTB to WikiText-2).
Method
Search Space
Search for a cell as the building block of the final architecture.
The learned cell could either be stacked to form a CNN or recursively connected to form a RNN.
A cell is a DAG consisting of an ordered sequence of N nodes.
\(\bar{o}^{(i, j)}(x)=\sum_{o \in \mathcal{O}} \frac{\exp \left(\alpha_{o}^{(i, j)}\right)}{\sum_{o^{\prime} \in \mathcal{O}} \exp \left(\alpha_{o^{\prime}}^{(i, j)}\right)} o(x)\)
\(x^{(j)}=\sum_{i<j} o^{(i, j)}\left(x^{(i)}\right)\)
Optimization Target
Our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters).
\(\min _{\alpha} \mathcal{L}_{v a l}\left(w^{*}(\alpha), \alpha\right)\) ......(3)
s.t. \(\quad w^{*}(\alpha)=\operatorname{argmin}_{w} \mathcal{L}_{\text {train}}(w, \alpha)\) .......(4)
The idea is to approximate w∗(α) by adapting w using only a single training step, without solving the inner optimization (equation 4) completely by training until convergence.
\(\nabla_{\alpha} \mathcal{L}_{v a l}\left(w^{*}(\alpha), \alpha\right)\) ......(5)
\(\approx \nabla_{\alpha} \mathcal{L}_{v a l}\left(w-\xi \nabla_{w} \mathcal{L}_{t r a i n}(w, \alpha), \alpha\right)\) ......(6)
- When ξ = 0, the second-order derivative in equation 7 will disappear.
- ξ = 0 as the first-order approximation,
- ξ > 0 as the second-order approximation.
Discrete Arch
To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes.
we use k = 2 for convolutional cells and k = 1 for recurrent cellsThe strength of an operation is defined as \(\frac{\exp \left(\alpha_{o}^{(i, j)}\right)}{\sum_{o^{\prime} \in \mathcal{O}} \exp \left(\alpha_{o^{\prime}}^{(i, j)}\right)}\)
Experiments
We include the following operations in O:
- 3 × 3 and 5 × 5 separable convolutions,
- 3 × 3 and 5 × 5 dilated separable convolutions,
- 3 × 3 max pooling,
- 3 × 3 average pooling,
- identity (skip connection?)
- zero.
All operations are of
- stride one (if applicable)
- the feature maps are padded to preserve their spatial resolution.
We use the
- ReLU-Conv-BN order for convolutional operations,
- Each separable convolution is always applied twice
- Our convolutional cell consists of N = 7 nodes, the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded).
The first and second nodes of cell k are set equal to the outputs of cell k−2 and cell k−1
Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two.
The architecture encoding therefore is (αnormal, αreduce),
where αnormal is shared by all the normal cells
and αreduce is shared by all the reduction cells.
To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB).
This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive (Fig. 3)
Arch Evaluation
- To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set.
- To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set.
Result Analysis
- DARTS achieved comparable results with the state of the art while using three orders of magnitude less computation resources.
- (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet)
- The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization (Fig. 3).
- It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design.
Results in Table 3 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet.
The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search.
The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest.
Conclusion
- We presented DARTS, a simple yet efficient NAS algorithm for both CNN and RNN.
- SOTA
- efficiency improvement by several orders of magnitude.
Improve
- discrepancies between the continuous architecture encoding and the derived discrete architecture. (softmax…)
- It would also be interesting to investigate performance-aware architecture derivation schemes based on the shared parameters learned during the search process.
Appendix
2019-ICLR-DARTS: Differentiable Architecture Search-论文阅读的更多相关文章
- 论文笔记:DARTS: Differentiable Architecture Search
DARTS: Differentiable Architecture Search 2019-03-19 10:04:26accepted by ICLR 2019 Paper:https://arx ...
- 论文笔记系列-DARTS: Differentiable Architecture Search
Summary 我的理解就是原本节点和节点之间操作是离散的,因为就是从若干个操作中选择某一个,而作者试图使用softmax和relaxation(松弛化)将操作连续化,所以模型结构搜索的任务就转变成了 ...
- 论文笔记:Progressive Differentiable Architecture Search:Bridging the Depth Gap between Search and Evaluation
Progressive Differentiable Architecture Search:Bridging the Depth Gap between Search and Evaluation ...
- 2019-ICCV-PDARTS-Progressive Differentiable Architecture Search Bridging the Depth Gap Between Search and Evaluation-论文阅读
P-DARTS 2019-ICCV-Progressive Differentiable Architecture Search Bridging the Depth Gap Between Sear ...
- 论文笔记系列-Auto-DeepLab:Hierarchical Neural Architecture Search for Semantic Image Segmentation
Pytorch实现代码:https://github.com/MenghaoGuo/AutoDeeplab 创新点 cell-level and network-level search 以往的NAS ...
- Research Guide for Neural Architecture Search
Research Guide for Neural Architecture Search 2019-09-19 09:29:04 This blog is from: https://heartbe ...
- 小米造最强超分辨率算法 | Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
本篇是基于 NAS 的图像超分辨率的文章,知名学术性自媒体 Paperweekly 在该文公布后迅速跟进,发表分析称「属于目前很火的 AutoML / Neural Architecture Sear ...
- 论文笔记系列-Neural Architecture Search With Reinforcement Learning
摘要 神经网络在多个领域都取得了不错的成绩,但是神经网络的合理设计却是比较困难的.在本篇论文中,作者使用 递归网络去省城神经网络的模型描述,并且使用 增强学习训练RNN,以使得生成得到的模型在验证集上 ...
- 论文笔记:Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation2019-03-18 14:4 ...
随机推荐
- 《C程序设计语言》 练习2-1
问题描述 编写一个程序以确定分别由signed及unsigned限定的char.short.int及long类型变量的取值范围.采用打印标准头文件中的相应值以及直接计算两种方式实现 Write a p ...
- Educational Codeforces Round 77 (Rated for Div. 2) C. Infinite Fence
C. Infinite Fence 题目大意:给板子涂色,首先板子是顺序的,然后可以涂两种颜色,如果是r的倍数涂成红色,是b的倍数涂成蓝色, 连续的k个相同的颜色则不能完成任务,能完成任务则输出OBE ...
- Java 常用API(一)
目录 Java 常用API(一) 1. Scanner类 引用类型的一般使用步骤 Scanner的使用步骤 例题 2. 匿名对象 概述 匿名对象作为方法的参数 匿名对象作为方法的返回值 3. Rand ...
- java使用window builder图形界面开发简易计算器
界面效果: /** * */ package calculator; import java.awt.BorderLayout; import java.awt.EventQueue; import ...
- qgis控制滚轮转动地图比例尺的变化幅度
需求:在比例尺1万-10万之间,鼠标滚轮转动比例尺的变化幅度为1万重写滚轮事件 void Xx::wheelEvent(QWheelEvent*event){ double curScale = sc ...
- R语言:计算现金工资的币数
新入职的员工,有的没有相应银行卡,需要计算现金工资的币数.实发工资,一般取整数. 简化计算,纸币面值只有100.10.1.4278除以100等于42余78,78除以10等于7余8,8除以1等于8. 复 ...
- ASP.NET 开源导入导出库Magicodes.IE 完成Csv导入导出
Magicodes.IE Csv导入导出 说明 本章主要说明如何使用Magicodes.IE.Csv进行Csv导入导出. 主要步骤 1.安装包Magicodes.IE.Csv Install-Pack ...
- 在没有RedirectAttributes的环境中如何在重定向环境中报错错误提示信息供页面使用
Spring在无RedirectAttributes的情况下(如Interceptor.filter中)使用Flash scope 今天遇到一个应用场景: 在需要在自定义的Interceptor中判断 ...
- Web_php_unserialize
0x01 <?php class Demo { private $file = 'index.php'; public function __construct($file) { $this-& ...
- shell脚本常用命令汇总
一.shell脚本概述和入门 (1)shell脚本是一个命令行解释器,它接收应用程序/用户命令,然后调用操作系统内核 (2)shell脚本的常用执行方式: 第一种:采用bash或sh+脚本的相对路径或 ...