DARTS

2019-ICLR-DARTS Differentiable Architecture Search

  • Hanxiao Liu、Karen Simonyan、Yiming Yang
  • GitHub:2.8k stars
  • Citation:557

Motivation

Current NAS method:

  • Computationally expensive: 2000/3000 GPU days
  • Discrete search space, leads to a large number of architecture evaluations required.

Contribution

  • Differentiable NAS method based on gradient decent.
  • Both CNN(CV) and RNN(NLP).
  • SOTA results on CIFAR-10 and PTB.
  • Efficiency: (2000 GPU days VS 4 GPU days)
  • Transferable: cifar10 to ImageNet, (PTB to WikiText-2).

Method

Search Space

Search for a cell as the building block of the final architecture.

The learned cell could either be stacked to form a CNN or recursively connected to form a RNN.

A cell is a DAG consisting of an ordered sequence of N nodes.

\(\bar{o}^{(i, j)}(x)=\sum_{o \in \mathcal{O}} \frac{\exp \left(\alpha_{o}^{(i, j)}\right)}{\sum_{o^{\prime} \in \mathcal{O}} \exp \left(\alpha_{o^{\prime}}^{(i, j)}\right)} o(x)\)

\(x^{(j)}=\sum_{i<j} o^{(i, j)}\left(x^{(i)}\right)\)

Optimization Target

Our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters).

\(\min _{\alpha} \mathcal{L}_{v a l}\left(w^{*}(\alpha), \alpha\right)\) ......(3)

s.t. \(\quad w^{*}(\alpha)=\operatorname{argmin}_{w} \mathcal{L}_{\text {train}}(w, \alpha)\) .......(4)

The idea is to approximate w∗(α) by adapting w using only a single training step, without solving the inner optimization (equation 4) completely by training until convergence.

\(\nabla_{\alpha} \mathcal{L}_{v a l}\left(w^{*}(\alpha), \alpha\right)\) ......(5)

\(\approx \nabla_{\alpha} \mathcal{L}_{v a l}\left(w-\xi \nabla_{w} \mathcal{L}_{t r a i n}(w, \alpha), \alpha\right)\) ......(6)

  • When ξ = 0, the second-order derivative in equation 7 will disappear.
  • ξ = 0 as the first-order approximation,
  • ξ > 0 as the second-order approximation.

Discrete Arch

To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes.

we use k = 2 for convolutional cells and k = 1 for recurrent cells

The strength of an operation is defined as \(\frac{\exp \left(\alpha_{o}^{(i, j)}\right)}{\sum_{o^{\prime} \in \mathcal{O}} \exp \left(\alpha_{o^{\prime}}^{(i, j)}\right)}\)

Experiments

We include the following operations in O:

  • 3 × 3 and 5 × 5 separable convolutions,
  • 3 × 3 and 5 × 5 dilated separable convolutions,
  • 3 × 3 max pooling,
  • 3 × 3 average pooling,
  • identity (skip connection?)
  • zero.

All operations are of

  • stride one (if applicable)
  • the feature maps are padded to preserve their spatial resolution.

We use the

  • ReLU-Conv-BN order for convolutional operations,
  • Each separable convolution is always applied twice
  • Our convolutional cell consists of N = 7 nodes, the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded).

  • The first and second nodes of cell k are set equal to the outputs of cell k−2 and cell k−1

  • Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two.

  • The architecture encoding therefore is (αnormal, αreduce),

  • where αnormal is shared by all the normal cells

  • and αreduce is shared by all the reduction cells.

  • To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB).

  • This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive (Fig. 3)

Arch Evaluation

  • To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set.

  • To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set.

Result Analysis

  • DARTS achieved comparable results with the state of the art while using three orders of magnitude less computation resources.
  • (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet)
  • The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization (Fig. 3).

  • It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design.

Results in Table 3 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet.

  • The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search.

  • The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest.

Conclusion

  • We presented DARTS, a simple yet efficient NAS algorithm for both CNN and RNN.
  • SOTA
  • efficiency improvement by several orders of magnitude.

Improve

  • discrepancies between the continuous architecture encoding and the derived discrete architecture. (softmax…)
  • It would also be interesting to investigate performance-aware architecture derivation schemes based on the shared parameters learned during the search process.

Appendix

2019-ICLR-DARTS: Differentiable Architecture Search-论文阅读的更多相关文章

  1. 论文笔记:DARTS: Differentiable Architecture Search

    DARTS: Differentiable Architecture Search 2019-03-19 10:04:26accepted by ICLR 2019 Paper:https://arx ...

  2. 论文笔记系列-DARTS: Differentiable Architecture Search

    Summary 我的理解就是原本节点和节点之间操作是离散的,因为就是从若干个操作中选择某一个,而作者试图使用softmax和relaxation(松弛化)将操作连续化,所以模型结构搜索的任务就转变成了 ...

  3. 论文笔记:Progressive Differentiable Architecture Search:Bridging the Depth Gap between Search and Evaluation

    Progressive Differentiable Architecture Search:Bridging the Depth Gap between Search and Evaluation ...

  4. 2019-ICCV-PDARTS-Progressive Differentiable Architecture Search Bridging the Depth Gap Between Search and Evaluation-论文阅读

    P-DARTS 2019-ICCV-Progressive Differentiable Architecture Search Bridging the Depth Gap Between Sear ...

  5. 论文笔记系列-Auto-DeepLab:Hierarchical Neural Architecture Search for Semantic Image Segmentation

    Pytorch实现代码:https://github.com/MenghaoGuo/AutoDeeplab 创新点 cell-level and network-level search 以往的NAS ...

  6. Research Guide for Neural Architecture Search

    Research Guide for Neural Architecture Search 2019-09-19 09:29:04 This blog is from: https://heartbe ...

  7. 小米造最强超分辨率算法 | Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search

    本篇是基于 NAS 的图像超分辨率的文章,知名学术性自媒体 Paperweekly 在该文公布后迅速跟进,发表分析称「属于目前很火的 AutoML / Neural Architecture Sear ...

  8. 论文笔记系列-Neural Architecture Search With Reinforcement Learning

    摘要 神经网络在多个领域都取得了不错的成绩,但是神经网络的合理设计却是比较困难的.在本篇论文中,作者使用 递归网络去省城神经网络的模型描述,并且使用 增强学习训练RNN,以使得生成得到的模型在验证集上 ...

  9. 论文笔记:Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

    Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation2019-03-18 14:4 ...

随机推荐

  1. 自定义比较器(IComparer接口的实现)

    class FileNameSort : IComparer { [System.Runtime.InteropServices.DllImport("Shlwapi.dll", ...

  2. 你真的会用Flutter日期类组件吗

    Flutter系统提供了一些日期选择类组件,比如DayPicker.MonthPicker.YearPicker.showDatePicker.CupertinoDatePicker等,其中前4个为M ...

  3. 完了!CPU一味求快出事儿了!

    自我介绍 我叫阿Q,是CPU一号车间里的员工,我所在的这个CPU足足有8个核,就有8个车间,干起活来杠杠滴. 我所在的一号车间里,除了负责执行指令的我,还有负责取指令的小A,负责分析指令的小胖和负责结 ...

  4. 06_CSS入门和高级技巧(4)

    复习 CSS : 负责样式层,层叠式样式表cascading style sheet.CSS2.1,最新版本CSS3. CSS选择器: 选择哪些元素加样式.基本选择3种:标签p.id选择器#id.cl ...

  5. 谈谈Spring中的BeanPostProcessor接口

    一.前言   这几天正在复习Spring的相关内容,在了解bean的生命周期的时候,发现其中涉及到一个特殊的接口--BeanPostProcessor接口.由于网上没有找到比较好的博客,所有最后花了好 ...

  6. 安装stanfordcorenlp成功,import stanfordcorenlp失败,出现错误:importerror-no-module-named-psutil

    1.问题描述 安装stanfordcorenlp成功,import stanfordcorenlp失败,pycharm中输入import stanfordcorenlp,然后运行,出现错误:impor ...

  7. 【Linux系列汇总】小白博主的嵌入式Linux实战快速进阶之路(持续更新)

    我把之前在学习嵌入式Linux和工作中遇到问题和相关经验等,一起整理到这里,方便自己查阅和学习,温故而知新,如果能帮助到您,请帮忙点个赞: 本文的宗旨 嵌入式Linux 的项目通常需要一个团队来开发, ...

  8. OpenCV Error: Unspecified Error(The Function is not implemented)

    Ubuntu 或者 Debian 系统显示窗口的时候遇到了这个问题 error: (-2:Unspecified error) The function is not implemented. Reb ...

  9. input在IOS中的聚焦问题

    关于input输入框在iPhone手机中的聚焦问题,开发中是会经常遇到的,在一般的浏览器中,我们一般是通过 document.getElementById('opop').focus(); 来获取焦点 ...

  10. [hdu3572]最大流(dinic)

    题意:有m台机器,n个任务,每个任务需要在第si~ei天之间,且需要pi天才能完成,每台机器每天只能做一个任务,不同机器每天不能做相同任务,判断所有任务是否可以做完. 思路: 把影响答案的对象提取出来 ...