This paper is about a new model for language which uses a convolutional approach instead of LSTMs.


## General Language modeling

Statistical language models estimate the probability distribution of a sequence of words. They are important for ASR (automatic speech recognition) and translation. The usual approach is to embedd words into $\mathbb{R}^n$ and then apply RNNs to the vector sequences.


## Evaluation

* [WikiText-103](http://metamind.io/research/the-wikitext-long-term-dependency-language-modeling-dataset/): [Perplexity](https://en.wikipedia.org/wiki/Perplexity) of 44.9 (lower is better)
* new best single-GPU result on the Google Billion Word benchmark: Perplexity of 43.9


## Idea

* uses Gated Linear Units (GLU)
* uses pre-activation residual blocks
* adaptive softmax
* no tanh in the gating mechanism
* use gradient clipping

## See also

* [Reddit](https://www.reddit.com/r/MachineLearning/comments/5kbsjb/r_161208083_language_modeling_with_gated/)
* [Improving Neural Language Models with a Continuous Cache](https://arxiv.org/abs/1612.04426): Test perplexity of **40.8 on WikiText-103** 

## Temporal unit regression network

keyword: temporal action proposal; computing efficiency 

**Summary**: In this paper, Jiyang et al designed a proposal generation and refinement network with high computation efficiency by reusing unit feature on coordinated regression and classification network.  Especially,  a new metric against temporal proposal called AR-F is raised to meet 2 metric criteria: 1. evaluate different method on the same dataset efficiently. 2. capable to evaluate same method's performance across several datasets(generalization capability)


**Model**:
* decompose video and extract feature to form clip pyramid: 
1. A video is first decomposed into short units where each unit has 16/32 frames.
2. extract each unit's feature using C3D/Two-stream CNN model.
3. several units' features are average pooled to compose clip level feature. In order to provide context and adaptive for different length action, clip level feature also concatenate surround feature and scaled to different length by concatenating more or fewer clips.
Feature for a slip is $f_c = P(\{u_j\}_{s_u-n_{ctx}}^{s_u})||P(\{u_j\}_{s_u}^{e_u})||P(\{u_j\}_{e_u}^{e_u+n_{ctx}})  $
4. for each proposal pyramid, a classifier is used to judge if the proposal contains an action and a regressor is used to provide an offset for each proposal to refine proposal's temporal boundary. 
5. finally, during prediction, NMS is used to remove redundant proposal thus provide high accuracy without changing the recall rate.

https://i.imgur.com/zqvHOxj.png

**Training**: There are two output need to be optimized, the classification result and the regression offset. Intuitively, the distance between the proposal and corresponding ground truth should be measured. In this paper, the authors used the L-1 metric for regressor targeted the positive proposals. total loss is measured as follow:
$L = \lambda L_{reg}+L_{cls}$


$L_{reg} = \frac{1}{N_{pos}}\sum_{i = 1}^N*l_s^*|o_{s,i} - o_{s,i}^*+o_{e,i} - o_{e,i}^*|$


During training, the ratio between positive samples and negative samples is set to 1:10. And for each positive proposal, its ground truth is the one with which it has the highest IOU or which it has IOU more than 0.5.

 
**result**: 
1. Computation complexity: 880 fps using the C3D feature on TITAN X GPU, while 260 FPS using flow CNN feature on the same machine.
2. Accuracy: mAP@0.5  = 25.6% on THUMOS14


**Conclusion**: Within this paper, it generates proposals by generate candidate at each unit with different scale and then using regression to refine the boundary. 


*However, there are a lot of redundant proposals for each unit which is an unnecessary waste of computing source; Also, proposals are generated with the pre-defined length which restricted its adaptivity to different length action; Finally the proposals are generated on the unit level which will suffer granularity problem*

Reports of rapid growth in nature-based tourism and recreation add significant weight to the economic case for biodiversity conservation but seem to contradict widely voiced concerns that people are becoming increasingly isolated from nature. This apparent paradox has been highlighted by a recent study showing that on a per capita basis, visits to natural areas in the United States and Japan have declined over the last two decades. These results have been cited as evidence of "a fundamental and pervasive shift away from nature-based recreation" - but how widespread is this phenomenon? We address this question by looking at temporal trends in visitor numbers at 280 protected areas (PAs) from 20 countries. This more geographically representative dataset shows that while PA visitation (whether measured as total or per capita visit numbers) is indeed declining in the United States and Japan, it is generally increasing elsewhere. Total visit numbers are growing in 15 of the 20 countries for which we could get data, with the median national rate of change unrelated to the national rate of population growth but negatively associated with wealth. Reasons for this reversal of growth in the richest countries are difficult to pin down with existing data, but the pattern is mirrored by trends in international tourist arrivals as a whole and so may not necessarily be caused by disaffection with nature. Irrespective of the explanation, it is clear that despite important downturns in some countries, nature-related tourism is far from declining everywhere, and may still have considerable potential both to generate funds for conservation and to shape people's attitudes to the environment.

Nature-based tourism is frequently described as one of the fastest growing sectors of the world's largest industry, and a very important justification for conservation. However, a recent, high profile report has interpreted declining visit rates to US and Japanese national parks as evidence of a pervasive shift away from nature tourism. Here we use the largest database so far compiled on trends in visits to Protected Areas around the world to resolve this apparent paradox. We find that, while visit rates—measured in two different ways—are indeed declining in some wealthy countries, in roughly three-quarters of the nations where data are available, visits to Protected Areas are increasing. Internationally, rates of growth in the number of visits to such areas show a clear negative association with per capita income, which interestingly is matched by trends in foreign arrivals as a whole. Our results therefore suggest that, despite worrying local downturns, nature-related tourism is far from declining everywhere, and may still have considerable potential to generate funds for conservation and engage people with the environment.

Proposing character-based extensions to a neural MT system for grammatical error correction. OOV words are represented in the encoder and decoder using character-based RNNs. They evaluate on the CoNLL-14 dataset, integrate probabilities from a large language model, and achieve good results.

https://i.imgur.com/r0Bsxp5.png

TLDR; The authors propose a novel encoder-decoder neural network architecture. The encoder RNN encodes a sequence into a fixed length vector representation and the decoder generates a new variable-length sequence based on this representation. The authors also introduce a new cell type (now called GRU) to be used with this network architecture. The model is evaluated on a statistical machine translation task where it is fed as an additional feature to a log-linear model. It leads to improved BLEU scores. The authors also find that the model learns syntactically and semantically meaningful representations of both words and phrases.

#### Key Points:

- New encoder-decoder architecture, seq2seq. Decoder conditioned on thought vector.
- Architecture can be used for both scoring or generation
- New hidden unit type, now called GRU. Simplified LSTM.
- Could replace whole pipeline with this architecture, but this paper doesn't
- 15k vocabulary (93% of dataset cover). 100d embeddings, 500 maxout units in final affine layer, batch size of 64, adagrad, 384M words, 3 days training time.
- Architecture is trained without frequency information so we expect it to capture linguistic information rather than statistical information.
- Visualizations of both words embeddings and thought vectors.


#### Questions/Notes

- Why not just use LSTM units?