#### Introduction

* Task of translating natural language queries into regular expressions without using domain specific knowledge.
* Proposes a methodology for collecting a large corpus of regular expressions to natural language pairs.
* Reports performance gain of 19.6% over state-of-the-art models.
* [Link to the paper](http://arxiv.org/abs/1608.03000v1)

#### Architecture

* LSTM based sequence to sequence neural network (with attention)
* Six layers
    * One-word embedding layer
    * Two encoder layers
    * Two decoder layers
    * One dense output layer.
* Attention over encoder layer.
* Dropout with the probability of 0.25.
* 20 epochs, minibatch size of 32 and learning rate of 1 (with decay rate of 0.5)

#### Dataset Generation

* Created a public dataset - **NL-RX** - with 10K pair of (regular expression, natural language) 
* Two step generate-and-paraphrase approach
    * Generate step
        * Use handcrafted grammar to translate regular expressions to natural language.
    * Paraphrase step
        * Crowdsourcing the task of translating the rigid descriptions into more natural expressions.


#### Results

* Evaluation Metric
    * Functional equality check (called DFA-Equal) as same regular expression could be written in many ways.
* Proposed architecture outperforms both the baselines - Nearest Neighbor classifier using Bag of Words (BoWNN) and Semantic-Unify

Deep convolutional neural networks (DCNN) has been a popular model for image classification over the last few years. This paper proposes a DCNN structure, also known as AlexNet, for the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC). To train AlexNet, which has 60 million parameters, this paper uses Rectified Linear Units (ReLU) and multiple GPU to accelerate training. This paper also report that using local response normalization and overlapping pooling can reduce error rate. To prevent over fitting, they suggest data augmentation and apply dropout in the fully connected layer. 
Technical details

The following figure shows the architecture of AlexNet. It contains five convolutional and three fully connected layers. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers follow the first and second response-normalization layers and the fifth convolutional layer.

![](http://i.imgur.com/2iqwCq1.png)

It's not clear to me where the difference between DenseNets and DelungeNets are.

## Evaluation

* Cifar-10: 3.76% error (DenseNet: )
* Cifar-100: 19.02% error

## See also

* [reddit](https://www.reddit.com/r/MachineLearning/comments/5l0k6w/r_delugenets_deep_networks_with_massive_and/)
* [DenseNet](http://www.shortscience.org/paper?bibtexKey=journals/corr/1608.06993)

**TL;DR:** There are 'place cells' in the hippopotamus that are fired when passing through a location. You can take a rat and measure how its cells are activated in a maze, then monitor neurons during planning, rest or sleep. You'll see patterns that show it's thinking of locations in order and focusing on interesting locations. This paper looks at how RL agents do 'prioritized experience replay' and compare it to place cells in animals. The authors do a RL simulation and *qualitatively* compare the results to the activity observed in place cells.

**Key paragraphs:**

> Neural activity recorded from hippocampal place cells during spatial navigation typically represents the animal’s spatial position, though it can sometimes represent locations ahead of the animal. For instance, during “sharp wave ripple” events, activity might progress sequentially from the animal’s current location towards a goal location. These “forward replay” ´sequences predict subsequent behavior and have been suggested to support a planning mechanism that links actions to their deferred consequences along a spatial trajectory. However, analogously to the human evidence, remote activity in the hippocampus can also represent locations behind the animal, and even altogether disjoint, ´remote locations (especially during rest or sleep) (Fig. 1a).

...

> we develop a normative theory to predict not just whether but which memories should be accessed at each time
to enable the most rewarding future decisions.

...

> To test the implications of our theory, we simulate a spatial navigation task where an agent generates and stores experiences which can be later retrieved. We show that an agent that accesses memories sequentially and in order of utility
produces patterns of sequential state consideration that resemble place cell replay, and reproduces qualitatively and with
no parameter fitting a wealth of empirical findings including (i) the existence and balance between forward and reverse replay; (ii) the content of replay; and (iii) effects of experience. 

...

> we propose the unifying view that all patterns of replay during behavior, rest, and sleep reflect different instances of a more general state retrieval operation that integrates experiences across space and time to propagate value and guide decisions. 

**My 2 cents**: I like this paper because prioritized experience replay reminds me of how we often dream or daydream of novel good or bad events that happened or that we anticipate. This paper drills much deeper into this connection.

## Structured segmented network

### **key word**: action detection in video; computing complexity reduction; structurize proposal 

**Abstract**: using a temporal action grouping scheme (TAG) to generate accurate proposals, using a structured pyramid to model the temporal structure of each action instance to tackle the issue that detected actions are not complete, using two classifiers to determine class and completeness and using a regressor for each category to further modify the temporal bound. In this paper, Yue Zhao et al mainly tackle the problem of high computing complexity by sampling video frame and remove redundant proposals in video detection and the lack of action stage modeling.  

**Model**: 
1. generate proposals: find continuous temporal regions with mostly high actioness. $P = \{ p_i = [s_i,e_i]\}_{i = 1}^N$
2. splitting proposals into 3 stages: start, course, and end: first augment the proposal by 2 times symmetrical to center, and course part is the original proposal, while start and end is the left part and right part of the difference between the transformed proposal and original one.
3. build temporal pyramid representation for each stage: first  L samples are sampled from the augmented proposal, then two-stream feature extractor is used on each one of them and pooling features for each stage 
4. build global representation for each proposal by concatenating stage-level representations
5. a global representation for each proposal is used as input for classifiers 

* input  =  ${S_t}_{t = 1} ^{T}$a sequence of T snippet representing the video. each snippet  = the frames + an optical  flow stack
* network: two linear classifiers; L two-steam feature extractor and several pooling layer  
* output: category and completeness and modification for each proposals.
https://i.imgur.com/thM9oWz.png

**Training**: 
* joint loss for classifiers: $L_{cls} = -log(P(c_i|p_i)* P(b_i,c_i,p_i)) $
* loss for location regression:  $\lambda * 1(c_i>=1, b_i = 1)  L(u_i,\varphi _i;p_i)$

**Summary**:
This paper has three highlights:
1. Parallel: it uses a paralleled network structure where proposals can be processed in paralleled which will shorten the processing time based on GPU
2. temporal structure modeling and regression: give each proposal certain structure so that completeness of proposals can be achieved
3. reduce computing complexity: use two tricks: remove video redundancy by sampling frame; remove proposal redundance