Feature Extractor[content]

0. AlexNet

1. VGG

VGG网络相对来说，结构简单，通俗易懂，作者通过分析2013年imagenet的比赛的最好模型，并发现感受野还是小的好，然后再加上《network in network》中的\(1*1\)卷积核，使得全文只在卷积网络的深度上做文章，从而得出了网络还是越深越好的结论
VGG

2. Inception

与VGG同期出来的有googlenet，该网络通过关注减少模型参数，而不降低模型性能的角度出发，设计出了inception结构，提出了googlenet；

然后google人员发现因为网络在训练过程中，下层参数的变化会导致下层输出的数据分布不断变化，从而当前层乃至后面层都需要不断地去拟合新的分布，这个想法也是来自于迁移学习中的covariate shift问题(即，训练样本和测试样本其实不一致)。
而借鉴迁移学习的这个概念，如果细化到网络内部的每一层的话，该现象又叫做internal covariate shift。
那么通过白化角度，的确可以解决这个问题，可是如果白化独立于网络的优化算法而存在，会导致网络没什么改变，而且会因为两者不相关而让其中的网络参数不断变大，即使loss稳定了，也还是会变大。所以就需要设计出一种即放入优化算法，而又能够对每层网络的输入进行类白化的效果。而如果基于mini-batch进行0均值和1方差，本身又会引入新的问题：
从概率论角度出发：后验概率=先验概率*联合概率。
如果只是做类白化，那么就相当于引入了联合概率，从而数据的分布其实还是会变化的，而且拿sigmoid举例的话，也会导致网络根本达不到其非线性部分，而一直处在线性部分。
那么解决该问题的角度就是，引入修复变量，进行线性变换，使得变换后的数据分布能够修复之前0均值1方差带来的损失，从而提出了BN(被大家认为是inception v2)

inception v2
inception V3
inception V4

3. HighwayNet

4. ResNet

ResNet v1
ResNet v2

5. DenseNet

DenseNet

6. MobileNet

7. ShuffleNet

8. SENet

SENet

9. NASNet

参考文献：

[原理] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[C]//European conference on computer vision. Springer, Cham, 2014: 818-833.
[alexnet] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[C]//Advances in neural information processing systems. 2012: 1097-1105.
[vgg] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.
[googlenet] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1-9.
.. [bn&inception v2] Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]//Proceedings of the 32nd International Conference on Machine Learning (ICML-15). 2015: 448-456.
.. [BRN] Ioffe S. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models[J]. arXiv preprint arXiv:1702.03275, 2017
.. [inception v3] Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2818-2826.
.. [inception v4] Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[C]//AAAI. 2017: 4278-4284.
[highway net] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv:1505.00387, 2015.
.. [highway net v2] Srivastava R K, Greff K, Schmidhuber J. Training very deep networks[C]//Advances in neural information processing systems. 2015: 2377-2385.
[resnet v1] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
.. [resnet v2] He K, Zhang X, Ren S, et al. Identity mappings in deep residual networks[C]//European Conference on Computer Vision. Springer International Publishing, 2016: 630-645.
.. [wider or deeper] Wu Z, Shen C, Hengel A. Wider or deeper: Revisiting the resnet model for visual recognition[J]. arXiv preprint arXiv:1611.10080, 2016.
.. [WRN] Zagoruyko S, Komodakis N. Wide residual networks[J]. arXiv preprint arXiv:1605.07146, 2016.
..[ResNext] Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 5987-5995.
[Dense Net] Huang G, Liu Z, Weinberger K Q, et al. Densely connected convolutional networks[J]. arXiv preprint arXiv:1608.06993, 2016.
.. [Dense Net] Pleiss G, Chen D, Huang G, et al. Memory-Efficient Implementation of DenseNets[J]. arXiv preprint arXiv:1707.06990, 2017.
.. [why&how] .DenseNet 的“what”、“why”和“how”
.. [多尺度DenseNet] Huang G, Chen D, Li T, et al. Multi-Scale Dense Convolutional Networks for Efficient Prediction[J]. arXiv preprint arXiv:1703.09844, 2017.
[MobileNet] Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, 2017.
..[MobileNetV2] Sandler M, Howard A, Zhu M, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4510-4520.
[SENet] Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks[J]. arXiv preprint arXiv:1709.01507, 2017.
[xception] Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions[J]. arXiv preprint arXiv:1610.02357, 2016.
[NASNet] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
.. [NASNet] Zoph B, Vasudevan V, Shlens J, et al. Learning transferable architectures for scalable image recognition[J]. arXiv preprint arXiv:1707.07012, 2017. .