If you’ve ever read a paper on Generative Adversarial Networks (from now on: GANs), you’ve almost certainly heard the author refer to the scourge upon the land of GANs that is mode collapse. When a generator succumbs to mode collapse, that means that, instead of modeling the full distribution, of input data, it will choose one region where there is a high density of data, and put all of its generated probability weight there. Then, on the next round, the discriminator pushes strongly away from that region (since it now is majority-occupied by fake data), and the generator finds a new mode. 

In the view of the authors of the Unrolled GANs paper,  one reason why this happens is that, in the typical GAN, at each round the generator implicitly assumes that it’s optimizing itself against the final and optimal discriminator. And, so, it makes its best move given that assumption, which is to put all its mass on a region the discriminator assigns high probability. Unfortunately for our short-sighted robot friend, this isn’t a one-round game, and this mass-concentrating strategy gives the discriminator a really good way to find fake data during the next round: just dramatically downweight how likely you think data is in the generator’s prior-round sweet spot, which it’s heavy concentration allows you to do without impacting your assessment of other data. Unrolled GANs operate on this key question: what if we could give the generator an ability to be less short-sighted, and make moves that aren’t just optimizing for the present, but are also defensive against the future, in ways that will hopefully tamp down on this running-around-in-circles dynamic illustrated above. If the generator was incentivized not only to make moves that fool the current discriminator, but also make moves that make the next-step discriminator less likely to tell it apart, the hope is that it will spread out its mass more, and be less likely to fall into the hole of a mode collapse. 

This intuition was realized in UnrolledGANs, through a mathematical approach that is admittedly a little complex for this discussion format. Essentially, in addition to the typical GAN loss (which is based on the current values of the generator and discriminator), this model also takes one “step forward” of the discriminator (calculates what the new parameters of the discriminator would be, if it took one update step), and backpropogates backward through that step. The loss under the next-step discriminator parameters is a function of both the current generator, and the next-step parameters, which come from the way the discriminator reacts to the current generator. When you take the gradient with respect to the generator of both of these things, you get something very like the ideal we described earlier: a generator that is trying to put its mass into areas the current discriminator sees as high-probability, but also change its parameters such that it gives the discriminator a less effective response strategy. 

https://i.imgur.com/0eEjm0g.png

Empirically: UnrolledGANs do a quite good job at their stated aim of reducing mode collapse, and the unrolled training procedure is now a common building-block technique used in other papers. 

Joint summary from https://mitpress.mit.edu/books/developmental-robotics

Developmental robotics is the interdisciplinary approach to the autonomous design of behavioural and cognitive capabilities in artificial agents (robots) that takes direct inspiration from the developmental principles and mechanisms observed in the natural cognitive systems.  It relies on a highly interdisciplinary effort of empirical developmental sciences such as developmental psychology, neuroscience, and comparative psychology, and computational and engineering disciplines such as robotics and artificial intelligence.  The implementation of these principles and mechanisms into a robot-s control architecture and the testing through experiments where the robot interacts with its physical and social environment simultaneously permits the validation of such principles and the actual design of complex behavioural and mental capabilities in robots. Developmental psychology and developmental robotics mutually benefit from such a combined effort.

Synonym terms of Developmental Robotics include cognitive developmental robotics, autonomous mental development as well as epigenetic robotics \cite{Cangelosi18}. The  later term borrows the term ‘epigenetic’ from Piaget’s Epigenetic Theory of human development, where the child’s cognitive system develops as a result of the interaction between genetic predispositions and the organism’s interaction with the environment.  Therefore, ‘Epigenetic robotics’ was justified by Piaget’s stress on the importance of the role of interaction with the environment (later complemented with the emphasis on social interaction of Zlatev and Balkenius in 2001), and in particular, on the sensorimotor bases of higher-order cognitive capabilities. 
Due to the challenges of robotics domain not only on acquisition of skills in an incremental manner but also in open-ended environments, it is perhaps worth noticing that the state of the art is not as advanced as in other more constrained well defined unique task settings. Because of the importance of embodiment in robotics for open ended learning \cite{Cangelosi18} and because of the scarcity of continual learning strategies on robotics and merely assessed  on perception tasks, we believe both fields methodologies should borrow from each other. The blend could result not only in more effective or robust lifelong learning but also in less forgetful agents (i.e., the main rationale behind CL), which has not yet empirically shown is impossible to reach. We hope this survey helps reach this goal.

Dynamical systems development: In math it is characterized by complex changes over time in the phase state. The complex interaction of nonlinear phenomena results in the production of unpredictable states of the system, often referred to as emergent states. This concept was borrowed by developmental psychologists \cite{Thelen94} to explain child development as the emergent product of the intricate and dynamic interaction of many decentralized and local interactions related to the child growing body and brain and her environment.


Some definitions: 

Novelty, curiosity and surprise: focuses on the autonomy of learning, i.e. on the agents freedom to choose what, when and how it will learn. Intrinsic motivation (IM) is a mechanism to drive autonomous learning, not only in developmental robotics but also more broadly within the field of machine learning \cite{Mirolli13}\cite{Oudeyer07}. IM is task-independent, the agent could be placed in a completely new environment with no prior knowledge  or experience, and through self-directed exploration the robot will potentially learn not only important features of the environment but also the behavioural skills necessary for dealing with the environment. Second, IM promotes the hierarchical learning rather than solving a specific, predefined task. 

The Theory of MInd (ToM): A robot can use its own theory of mind to improve the interaction with human users, for example. It aims at reading the intention of the others and reacting appropriately to the emotional, attentional and cognitive states of the other agents, to anticipate their reactions, and to modify its own behaviour to satisfy these expectation and needs \cite{Scasellati02}.

Cascades: developmental theories refer to the far reach of early developments on later ones in terms of the “developmental cascade”.

Saliency in robotics: 
In robotics vision, a set of feature extraction methods typically can be applied to derive a saliency map \cite{Itti01}, i.e. the identification of the parts of the image that are important for the robot behaviour. It can be created, e.g., by combining a set of feature extraction method for color, motion, orientation and brightness. These features are combined to generate the whole saliency  map used by the model to fixate the objects.

The authors explore the properties of "Deep Averaging Networks" on text classification problems, specifically sentiment and question answering tasks. DANs extend neural bag of words models, starting with a document representation that is the average of the word embeddings in that document, but extending to multiple feed-forward layers. The authors argue that these models are much simpler and faster to train than syntax and composition-based RNNs, while obtaining similar performance. Since this paper is actually arguing for simpler models, there is little technically here to understand, so the real contribution of the paper are the interesting experiments exploring how the DANs represent various phenomena. They show that differences between graded sentiment words (awesome, cool, ok, underwhelming, the worst) are magnified as layers are added. This shows the benefit of depth relative to a neural bag of words. Then they compare against RNNs with examples containing negation and contrastive conjunctions (e.g., but), which are traditionally modeled syntactically. They show that existing methods that we think can represent syntax/composition in fact are not strong enough. Something like "not bad" fully exposes the DAN -- it doubles the negation. But while the RNN-based models can learn not to simply double the negation, they are not powerful enough to reverse the polarity and get the example correct.

Finally, the authors introduce one novel mechanism for improving training, "word dropout." Similar to standard dropout, they randomly sample a subset of words at the input layer that are not used as part of the document representation. This gives the network multiple looks at each example with part of its feature space removed. Another way to think of this is data augmentation where new training instances are created by sampling feature vectors from existing data points with some features missing.

Novak et al. study the relationship between neural network sensitivity and generalization. Here, sensitivity is measured in terms of the Frobenius gradient of the network’s probabilities (resulting in a Jacobian matrix, not depending on the true label) or based on a coding scheme of activations. The latter is intended to quantify transitions between linear regions of the piece-wise linear model. To this end, all activations are assigned either $0$ or $1$ depending on their ReLU output. Based on a path between two or more input examples, the difference in this coding scheme is an estimator of how many linear regions have been “traversed”. Both metrics are illustrated in Figure 1, showing that they are low for test and training examples, or in regions within the same class, and high otherwise. The second metric is also illustrated in Figure 2. Based on these metrics, the authors show that these metrics correlate with the generalization gap, meaning that the sensitivity of the network and its generalization performance seem to be inherently connected.

https://i.imgur.com/iRt3ADe.jpg
Figure 1: For a network trained on MNIST, illustrations of a possible trajectory (left) and the corresponding sensitivity metrics (middle and right). I refer to the paper for details.

https://i.imgur.com/0G8su3K.jpg
Figure 2: Linear regions for a random 2-dimensional slice of the pre-logit space before and after training.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

The overall goal of the paper is measure how similar different layer activation profiles are to one another, in hopes of being able to quantify the similarity of the representations that different layers are learning. If you had a measure that captured this, you could ask questions like: “how similar are the representations that are learned by different networks on the same task”, and “what is the dynamic of representational change in a given layer throughout training”? 

Canonical Correlation Analysis is one way of approaching this question, and the way taken by this paper. The premise of CCA is that you have two multidimensional variable sets, where each set is made up of vectors representing dimensions within that variable set. Concretely, in this paper, the sets under examination are the activation profiles of two layers (either the same layer at different points in training, or different layers in the same network, or layers in different networks). An activation profile is thought of in terms of multiple vectors, where each vector represents a given neuron’s activation value, evaluated over some observation set X. Importantly, for the two layers that you’re comparing, the set of observations X needs to be of the same length, but the layers can have different number of neurons (and, consequently, different numbers of vectors making up that layer’s multivariate set). 

Given this setup, the goal of CCA is to find vectors that are linear combinations of the basis vectors of each set, to satisfy some constraint. In that broad sense, this is similar to the project of PCA, which also constructs linear-combination principal components to better represent the underlying data space. However, in PCA, the constraints that define these combinations are based on one multidimensional feature space, not two. In CCA, instead of generating k principal components, you generate k *pairs* of canonical correlates. Each canonical correlate pair, (U1, V1) is a linear combination of the activation vectors of sets L1 and L2  respectively, and is chosen with the goal of minimizing the the angle (cosine) distance between the correlates in each pair. 

If you think about L1 and L2 each only having two activations (that is: if you think about them as being two-dimensional spaces) then the goal of CCA is to find the cosine distance between the planes defined by the two activation spaces. An important intuition here is that in this framing, vector sets that are just linear transformations of one another (scalings, rotations, swaps in the arbitrary order of activations) will look the same, which wouldn’t be the case if you just looked at raw correlations between the individual activations. This is connected to the linear algebra idea that, if you have two vectors, and a third that is just a linear combination of the first two, the span of those vectors is still just that two-dimensional space. This property is important for the analysis of neural network representations because it means it will be able to capture similarities between representational spaces that have fundamental geometric similarities, even if they’re different on a more surface level. 

In prior papers, CCA had been used by calculating the CCA vectors between varying sets of layers, and then taking the mean CCA value over all of the pairs of vectors. This paper argues against that approach, on the theory that network layers are probably not using the full representational capacity of their activation dimensions (think, as analogy: a matrix with three columns, that only actually spans two), and so including in your average very low-order correlations is mostly adding uninformative noise to your similarity measure. Instead, this paper weights the correlation coefficients according to the magnitudes of the correlate vectors in the pair; as best I can tell, this is roughly analogous to weighting according to eigenvalues, in a PCA setting. 

Using this weighted-average similarity measure, the authors do some really interesting investigations into learning dynamics. These include: 
* Comparing the intermediate-layer representations learned by networks that achieve low train error via memorization vs via actually-generalizing solutions, and show that, during training, the intermediate representations of generalizing networks are more similar to one another than memorizing networks are to one another. Intuitively, this aligns with the idea that there are many ways to noisily memorize, but a more constrained number of ways to actually learn meaningful information about a dataset. A super interesting implication of this is the idea that representational similarity *on the training set* across multiple bootstrapped or randomized trainings could be used as a proxy for test set performance, which could be particularly valuable in contexts where test data is limited 

https://i.imgur.com/JwyHFmN.png

* Across networks, lower layers tend to be more similar to one another than layers closer to the output; said another way, the very simple (e.g. edge detectors) tend to be quite similar across networks, but the higher level representations are more divergent and influenceable by smaller quirks of the training set. 
* Within a given dataset, you can cluster learned internal representations across many training sets and recover groups trained with the same learning rate, even though the final layer softmax is inherently similar across models that achieve the same training error. This implies that metrics like this can give us some idea of the different minima that the optimization algorithm finds, as a function of different learning rates. 

Overall, I found this paper a great example of a straightforward idea used to clearly answer important and interesting questions, which is always refreshing amidst a sea of “tiny hack for an extra 0.05 accuracy”.