The R-CNN method is a way to localize objects in an image. It is restricted to finding one of each object in an image. 

1. Regions are generated based on any method including brute force sliding window.
2. Each region is classified using AlexNet.
3. The classifications for each label are searched to find the location which expresses that label the most.

The [R-CNN](http://arxiv.org/abs/1311.2524) paper presents a method based on convolutional neural networks (CNNs) for object detection. It does so by region proposals (hence the "R"). The key insight was to train CNNs on classification tasks and use the learned features for the region proposals. The do *not* use a sliding window approach such as Overfeat. They create around 2000 category-independent region proposals. For each proposal, they crop the part of that image. Then they resize the cropped part to fit into the CNN and classify it.


Notable follow-ups are:

* [Fast R-CNN](http://www.shortscience.org/paper?bibtexKey=conf/iccv/Girshick15)
* [Faster R-CNNs](http://www.shortscience.org/paper?bibtexKey=conf/nips/RenHGS15)

This paper presents R-CNN, an approach to do object detection using CNNs pre-trained for image classification. Object proposals are extracted from the image using Selective Search, dilated by few pixels, warped to CNN input size and fed into the CNN to extract features (they experiment with pool5, fc6, fc7). These extracted feature vectors are scored using SVMs, one per class. Bounding box regression, where they predict parameters to move the proposal closer to ground-truth, further boosts localization.

The authors use AlexNet, pre-trained on ImageNet and finetuned for detection. Object proposals with IOU overlap greater than 0.5 are treated as positive examples, and others as negative, and a 21-way classification (20 object categories + background) is set up to finetune the CNN. After finetuning, SVMs are trained per class, taking only the ground-truth boxes as positives, and IOU <= 0.3 as negatives.

R-CNN achieves major performance improvements on PASCAL VOC 2007/2010 and ILSVRC2013 detection datasets. Finally, this method is extended to do semantic segmentation and achieves competitive results.

## Strengths

- The method is simple and effective.
- Extensive ablation studies show why R-CNN works.
    - FC7 is the best feature to use (against pool5, fc6).
    - Fine-tuning provides a large boost in performance.
    - VGG performs better than AlexNet.
    - Bounding box regression further improves localization.

## Weaknesses / Notes

- Each region proposal is treated independently, which adds up to compute time.
- There are lots of different parts; the network can't be trained end-to-end.

  * Previously, methods to detect bounding boxes in images were often based on the combination of manual feature extraction with SVMs.
  * They replace the manual feature extraction with a CNN, leading to significantly higher accuracy.
  * They use supervised pre-training on auxiliary datasets to deal with the small amount of labeled data (instead of the sometimes used unsupervised pre-training).
  * They call their method R-CNN ("Regions with CNN features").

### How
  * Their system has three modules: 1) Region proposal generation, 2) CNN-based feature extraction per region proposal, 3) classification.
    * ![Architecture](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Rich_feature_hierarchies_for_accurate_object_detection_and_semantic_segmentation__architecture.jpg?raw=true "Architecture")
  * Region proposals generation
    * A region proposal is a bounding box candidate that *might* contain an object.
    * By default they generate 2000 region proposals per image.
    * They suggest "simple" (i.e. not learned) algorithms for this step (e.g. objectneess, selective search, CPMC).
    * They use selective search (makes it comparable to previous systems).
  * CNN features
    * Uses a CNN to extract features, applied to each region proposal (replaces the previously used manual feature extraction).
    * So each region proposal ist turned into a fixed length vector.
    * They use AlexNet by Krizhevsky et al. as their base CNN (takes 227x227 RGB images, converts them into 4096-dimensional vectors).
    * They add `p=16` pixels to each side of every region proposal, extract the pixels and then simply resize them to 227x227 (ignoring aspect ratio, so images might end up distorted).
    * They generate one 4096d vector per image, which is less than what some previous manual feature extraction methods used. That enables faster classification, less memory usage and thus more possible classes.
  * Classification
    * A classifier that receives the extracted feature vectors (one per region proposal) and classifies them into a predefined set of available classes (e.g. "person", "car", "bike", "background / no object").
    * They use one SVM per available class.
    * The regions that were not classified as background might overlap (multiple bounding boxes on the same object).
      * They use greedy non-maximum suppresion to fix that problem (for each class individually).
      * That method simply rejects regions if they overlap strongly with another region that has higher score.
      * Overlap is determined via Intersection of Union (IoU).
  * Training method
    * Pre-Training of CNN
      * They use AlexNet pretrained on Imagenet (1000 classes).
      * They replace the last fully connected layer with a randomly initialized one that leads to `C+1` classes (`C` object classes, `+1` for background).
    * Fine-Tuning of CNN
      * The use SGD with learning rate `0.001`.
      * Batch size is 128 (32 positive windows, 96 background windows).
      * A region proposal is considered positive, if its IoU with any ground-truth bounding box is `>=0.5`.
    * SVM
      * They train one SVM per class via hard negative mining.
      * For positive examples they use here an IoU threshold of `>=0.3`, which performed better than 0.5.

### Results
  * Pascal VOC 2010
    * They: 53.7% mAP
    * Closest competitor (SegDPM): 40.4% mAP
    * Closest competitor that uses the same region proposal method (UVA): 35.1% mAP
    * ![Scores on Pascal VOC 2010](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Rich_feature_hierarchies_for_accurate_object_detection_and_semantic_segmentation__scores.jpg?raw=true "Scores on Pascal VOC 2010")
  * ILSVRC2013 detection
    * They: 31.4% mAP
    * Closest competitor (OverFeat): 24.3% mAP
  * The feed a large number of region proposals through the network and log for each filter in the last conv-layer which images activated it the most:
    * ![Activations](https://raw.githubusercontent.com/aleju/papers/master/neural-nets/images/Rich_feature_hierarchies_for_accurate_object_detection_and_semantic_segmentation__activations.jpg?raw=true "Activations")
  * Usefulness of layers:
    * They remove later layers of the network and retrain in order to find out which layers are the most useful ones.
    * Their result is that both fully connected layers of AlexNet seemed to be very domain-specific and profit most from fine-tuning.
  * Using VGG16:
    * Using VGG16 instead of AlexNet increased mAP from 58.5% to 66.0% on Pascal VOC 2007.
    * Computation time was 7 times higher.
  * They train a linear regression model that improves the bounding box dimensions based on the extracted features of the last pooling layer. That improved their mAP by 3-4 percentage points.
  * The region proposals generated by selective search have a recall of 98% on Pascal VOC and 91.6% on ILSVRC2013 (measured by IoU of `>=0.5`).

This paper presents a object detection algorithm that improves mAP on PASCAL VOC dataset by over 20% to previous state-of-the-art. Unlike image classification which take an image or the center part of an image as input, object detection task requires an algorithm to detect bounding boxes of objects in an image. To use the high capacity CNN features in object detection, the proposed algorithm first generates region proposals. CNN features are extracted from those region proposals and are feed to a set of class-specific linear SVMs which tell whether objects are detected in those regions.
Technical details

The figure below show the object detection system in this paper.
![](http://3.bp.blogspot.com/-O6e43qcpcYA/VWapFWyXt5I/AAAAAAAAA8c/rcjlQJAQ35s/s320/system.png)

Because the PASCAL VOC dataset is not large enough for training high capacity CNN features, this paper use supervised pre-training on a large auxiliary dataset (ILSVRC 2012). The CNN is then fine-tuned with a portion of the PASCAL VOC dataset.

Results

The following table shows the detection mAP on VOC 2007 test.

![](http://1.bp.blogspot.com/-AmEd1cI6iWs/VWaqnD1YYwI/AAAAAAAAA8o/07pfZpdvwck/s400/table%2B2.png)