Fine-Grained（细粒度） Image – Papers, Codes and Datasets

Table of contents

1. Introduction

2. Survey papers

3. Benchmark datasets

4. Fine-grained image recognition

   1. Fine-grained recognition by localization-classification subnetworks

   2. Fine-grained recognition by end-to-end feature encoding

   3. Fine-grained recognition with external information

      1. Fine-grained recognition with web data / auxiliary data

      2. Fine-grained recognition with multi-modality data

      3. Fine-grained recognition with humans in the loop

5. Fine-grained image retrieval

   1. Unsupervised with pre-trained models

   2. Supervised with metric learning

6. Fine-grained image generation

   1. Generating from fine-grained image distributions

   2. Generating from text descriptions

7. Future directions of FGIA

   1. Automatic fine-grained models

   2. Fine-grained few shot learning

   3. Fine-grained hashing

   4. FGIA within more realistic settings

8. Leaderboard

1. Introduction

---

This homepage lists some representative papers/codes/datasets all about deep learning based fine-grained image, including fine-grained image recognition, fine-grained image retrieval, fine-grained image generation, etc. If you have any questions, please feel free to leave message.

2. Survey papers

---

• A Survey on Deep Learning-based Fine-Grained Object Classification and Semantic Segmentation. 
  Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. International Journal of Automation and Computing, 2017.

3. Benchmark datasets

---

Summary of popular fine-grained image datasets. Note that ‘‘BBox’’ indicates whether this dataset provides object bounding box supervisions. ‘‘Part anno.’’ means providing the key part localizations. ‘‘HRCHY’’ corresponds to hierarchical labels. ‘‘ATR’’ represents the attribute labels (e.g., wing color, male, female, etc). ‘‘Texts’’ indicates whether fine-grained text descriptions of images are supplied.

Dataset name	Year	Meta-class	images	categories	BBox	Part anno.	HRCHY	ATR	Texts
Oxford flower	2008	Flowers	8,189	102
CUB200	2011	Birds	11,788	200
Stanford Dog	2011	Dogs	20,580	120
Stanford Car	2013	Cars	16,185	196
FGVC Aircraft	2013	Aircrafts	10,000	100
Birdsnap	2014	Birds	49,829	500
NABirds	2015	Birds	48,562	555
DeepFashion	2016	Clothes	800,000	1,050
Fru92	2017	Fruits	69,614	92
Veg200	2017	Vegetable	91,117	200
iNat2017	2017	Plants & Animals	859,000	5,089
RPC	2019	Retail products	83,739	200

4. Fine-grained image recognition

---

Fine-grained recognition by localization-classification subnetworks

• Part-based R-CNNs for Fine-Grained Category Detection. 
  Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. ECCV, 2014. [code]

• Interaction Part Mining: A Mid-Level Approach for Fine-Grained Action Recognition. 
  Yang Zhou, Bingbing Ni, Richang Hong, Meng Wang, and Qi Tian. CVPR, 2015.

• Fine-Grained Recognition without Part Annotations. 
  Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. CVPR, 2015. [code]

• The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification. 
  Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. CVPR, 2015.

• Deep LAC: Deep Localization, Alignment and Classification for Fine-grained Recognition. 
  Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia. CVPR, 2015.

• Spatial Transformer Networks. 
  Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. NIPS, 2015. [code]

• Part-Stacked CNN for Fine-Grained Visual Categorization. 
  Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. CVPR, 2016.

• Mining Discriminative Triplets of Patches for Fine-Grained Classification. 
  Yaming Wang, Jonghyun Choi, Vlad I. Morariu, and Larry S. Davis. CVPR, 2016.

• SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-grained Recognition. 
  Han Zhang, Tao Xu, Mohamed Elhoseiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. CVPR, 2016.

• Picking Deep Filter Responses for Fine-grained Image Recognition. 
  Xiaopeng Zhang, Hongkai, Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. CVPR, 2016.

• Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition. 
  Jianlong Fu, Heliang Zheng, and Tao Mei. CVPR, 2017.

• Fine-Grained Recognition as HSnet Search for Informative Image Parts. 
  Michael Lam, Behrooz Mahasseni, and Sinisa Todorovic. CVPR, 2017.

• Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition. 
  Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. ICCV, 2017. [code]

• Weakly Supervised Learning of Part Selection Model with Spatial Constraints for Fine-Grained Image Classification. 
  Xiangteng He, and Yuxin Peng. AAAI, 2017.

• Localizing by Describing: Attribute-Guided Attention Localization for Fine-Grained Recognition. 
  Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin. AAAI, 2017.

• Learning to Navigate for Fine-grained Classification. 
  Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. ECCV, 2018. [code]

• Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition. 
  Ming Sun, Yuchen Yuan, Feng Zhou, and Errui Ding. ECCV, 2018. [code]

• Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up. 
  Weifeng Ge, Xiangru Lin, and Yizhou Yu. CVPR, 2019.

Fine-grained recognition by end-to-end feature encoding

• Hyper-Class Augmented and Regularized Deep Learning for Fine-Grained Image Classification. 
  Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. CVPR, 2015.

• Subset Feature Learning for Fine-Grained Category Classification. 
  ZongYuan Ge, Christopher McCool, Conrad Sanderson, and Peter Corke. CVPR, 2015.

• Bilinear CNN Models for Fine-grained Visual Recognition. 
  Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. ICCV, 2015. [code]

• Multiple Granularity Descriptors for Fine-Grained Categorization. 
  Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. ICCV, 2015.

• Compact Bilinear Pooling. 
  Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. CVPR, 2016. [code]

• Fine-Grained Image Classification by Exploring Bipartite-Graph Labels. 
  Feng Zhou, and Yuanqing Lin. CVPR, 2016. [project page]

• Kernel Pooling for Convolutional Neural Networks. 
  Yin Cui, Feng Zhou, Jiang Wang, Xiao Liu, Yuanqing Lin, and Serge Belongie. CVPR, 2017.

• Low-rank Bilinear Pooling for Fine-Grained Classification. 
  Shu Kong, and Charless Fowlkes. CVPR, 2017. [code]

• Higher-order Integration of Hierarchical Convolutional Activations for Fine-Grained Visual Categorization. 
  Sijia Cai, Wangmeng Zuo, and Lei Zhang. ICCV, 2017. [code]

• Learning a Discriminative Filter Bank within a CNN for Fine-grained Recognition. 
  Yaming Wang, Vlad I. Morariu, and Larry S. Davis. CVPR, 2018. [code]

• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization. 
  Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. CVPR, 2018. [code]

• Maximum-Entropy Fine Grained Classification. 
  Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. NIPS, 2018.

• Pairwise Confusion for Fine-Grained Visual Classification. 
  Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik. ECCV, 2018. [code]

• DeepKSPD: Learning Kernel-matrix-based SPD Representation for Fine-Grained Image Recognition. 
  Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu. ECCV, 2018.

• Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition. 
  Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. ECCV, 2018. [code]

• Grassmann Pooling as Compact Homogeneous Bilinear Pooling for Fine-Grained Visual Classification. 
  Xing Wei, Yue Zhang, Yihong Gong, Jiawei Zhang, and Nanning Zheng. ECCV, 2018.

• Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition. 
  Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. CVPR, 2019. [code]

• Destruction and Construction Learning for Fine-grained Image Recognition. 
  Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. CVPR, 2019. [code]

5. Fine-grained recognition with external information

Fine-grained recognition with web data / auxiliary data

• Augmenting Strong Supervision Using Web Data for Fine-Grained Categorization. 
  Zhe Xu, Shaoli Huang, Ya Zhang, and Dacheng Tao. ICCV, 2015.

• Webly Supervised Learning Meets Zero-shot Learning: A Hybrid Approach for Fine-grained Classification. 
  Li Niu, Ashok Veeraraghavan, and Vshu Sabbarwal. CVPR, 2018.

• Fine-Grained Visual Categorization using Meta-Learning Optimization with Sample Selection of Auxiliary Data. 
  Yabin Zhang, Hui Tang, and Kai Jia. ECCV, 2018. [code]

• Learning from Web Data using Adversarial Discriminative Neural Networks for Fine-Grained Classification. 
  Xiaoxiao Sun, Liyi Chen, and Jufeng Yang. AAAI, 2019.

Fine-grained recognition with multi-modality data

• Fine-Grained Image Classification via Combining Vision and Language. 
  Xiangteng He, and Yuxin Peng. CVPR, 2017.

• Audio Visual Attribute Discovery for Fine-Grained Object Recognition. 
  Hua Zhang, Xiaochun Cao, and Rui Wang. AAAI, 2018.

• Fine-grained Image Classification by Visual-Semantic Embedding. 
  Huapeng Xu, Guilin Qi, Jingjing Li, Meng Wang, Kang Xu, and Huan Gao. IJCAI, 2018.

• Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition. 
  Tianshui Chen, Liang Lin, Riquan Chen, Yang Wu, and Xiannan Luo. IJCAI, 2018.

Fine-grained recognition with humans in the loop

• Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop. 
  Yin Cui, Feng Zhou, Yuanqing Lin, and Serge Belongie. CVPR, 2016.

5. Fine-grained image retrieval

---

Unsupervised with pre-trained models

• Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval. 
  Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. TIP, 2017. [project page]

Supervised with metric learning

• Centralized Ranking Loss with Weakly Supervised Localization for Fine-Grained Object Retrieval. 
  Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. IJCAI, 2018.

• Towards Optimal Fine Grained Retrieval via Decorrelated Centralized Loss with Normalize-Scale layer. 
  Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. AAAI, 2019.

6. Fine-grained image generation

---

Generating from fine-grained image distributions

• CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training. 
  Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. ICCV, 2017. [code]

• FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery. 
  Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. CVPR, 2019. [code]

Generating from text descriptions

• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks. 
  Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. CVPR, 2018. [code]

7. Future directions of FGIA

---

Fine-grained few shot learning

• Piecewise classifier mappings: Learning fine-grained learners for novel categories with few examples. 
  Xiu-Shen Wei, Peng Wang, Lingqiao Liu, Chunhua Shen, and Jianxin Wu. TIP, 2019.

FGIA within more realistic settings

• Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach. 
  Timnit Geru, Judy Hoffman, and Li Fei-Fei. ICCV, 2017.

• The iNaturalist Species Classification and Detection Dataset. 
  Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. CVPR 2018.

• RPC: A Large-Scale Retail Product Checkout Dataset. 
  Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. arXiv: 1901.07249, 2019. [project page]

8. Leaderboard

---

The section is being continually updated. Since CUB200-2011 is the most popularly used fine-grained dataset, we list the fine-grained recognition leaderboard by treating it as the test bed.

Method	Publication	BBox?	Part?	External information?	Base model	Image resolution	Accuracy
PB R-CNN	ECCV 2014				Alex-Net	224x224	73.9%
MaxEnt	NIPS 2018				GoogLeNet	TBD	74.4%
PB R-CNN	ECCV 2014				Alex-Net	224x224	76.4%
PS-CNN	CVPR 2016				CaffeNet	454x454	76.6%
MaxEnt	NIPS 2018				VGG-16	TBD	77.0%
Mask-CNN	PR 2018				Alex-Net	448x448	78.6%
PC	ECCV 2018				ResNet-50	TBD	80.2%
DeepLAC	CVPR 2015				Alex-Net	227x227	80.3%
MaxEnt	NIPS 2018				ResNet-50	TBD	80.4%
Triplet-A	CVPR 2016			Manual labour	GoogLeNet	TBD	80.7%
Multi-grained	ICCV 2015			WordNet etc.	VGG-19	224x224	81.7%
Krause et al.	CVPR 2015				CaffeNet	TBD	82.0%
Multi-grained	ICCV 2015			WordNet etc.	VGG-19	224x224	83.0%
TS	CVPR 2016				VGGD+VGGM	448x448	84.0%
Bilinear CNN	ICCV 2015				VGGD+VGGM	448x448	84.1%
STN	NIPS 2015				GoogLeNet+BN	448x448	84.1%
LRBP	CVPR 2017				VGG-16	224x224	84.2%
PDFS	CVPR 2016				VGG-16	TBD	84.5%
Xu et al.	ICCV 2015			Web data	CaffeNet	224x224	84.6%
Cai et al.	ICCV 2017				VGG-16	448x448	85.3%
RA-CNN	CVPR 2017				VGG-19	448x448	85.3%
MaxEnt	NIPS 2018				Bilinear CNN	TBD	85.3%
PC	ECCV 2018				Bilinear CNN	TBD	85.6%
CVL	CVPR 2017			Texts	VGG	TBD	85.6%
Mask-CNN	PR 2018				VGG-16	448x448	85.7%
GP-256	ECCV 2018				VGG-16	448x448	85.8%
KP	CVPR 2017				VGG-16	224x224	86.2%
T-CNN	IJCAI 2018				ResNet	224x224	86.2%
MA-CNN	ICCV 2017				VGG-19	448x448	86.5%
MaxEnt	NIPS 2018				DenseNet-161	TBD	86.5%
DeepKSPD	ECCV 2018				VGG-19	448x448	86.5%
OSME+MAMC	ECCV 2018				ResNet-101	448x448	86.5%
StackDRL	IJCAI 2018				VGG-19	224x224	86.6%
DFL-CNN	CVPR 2018				VGG-16	448x448	86.7%
PC	ECCV 2018				DenseNet-161	TBD	86.9%
KERL	IJCAI 2018			Attributes	VGG-16	224x224	87.0%
HBP	ECCV 2018				VGG-16	448x448	87.1%
Mask-CNN	PR 2018				ResNet-50	448x448	87.3%
DFL-CNN	CVPR 2018				ResNet-50	448x448	87.4%
NTS-Net	ECCV 2018				ResNet-50	448x448	87.5%
HSnet	CVPR 2017				GoogLeNet+BN	TBD	87.5%
MetaFGNet	ECCV 2018			Auxiliary data	ResNet-34	TBD	87.6%
DCL	CVPR 2019				ResNet-50	448x448	87.8%
TASN	CVPR 2019				ResNet-50	448x448	87.9%
Ge et al.	CVPR 2019				GoogLeNet+BN	Shorter side is 800 px	90.4%