Meta learning is an area sparking a lot of research curiosity these days. It’s framed in different ways: models that can adapt, models that learn to learn, models that can learn a new task quickly. This paper uses a somewhat different lens: that of neural plasticity, and argues that applying the concept to modern neural networks will give us an effective, and biologically inspired way of building adaptable models. The basic premise of plasticity from a neurobiology perspective (at least how it was framed in the paper: I’m not a neuroscientist myself, and may be misunderstanding) is that plasticity performs a kind of gating function on the strength of a neural link being upregulated by experience. The most plastic a connection is, the more quickly it can get modified by new data; the less plastic, the more fixed it is. 

In concrete terms, this is implemented by subdividing the weight on each connection in the network into two parts: the “fixed” component, and the “plastic” component. (see picture). The fixed component acts like a typical weight: it gets modified during training, but stays fixed once training is done. The plastic component is composed of an alpha weight, multiplied by a term H. H is basically a decaying running average of the past input*output activations of this weight. Activations that are high in magnitude, and the same sign, for both the input and the output will lead to H being pushed higher. Note that that this H can continue to be updated even after the model is done training, because it builds up information whenever you pass a new input X through the network. The plastic component’s learned weight, alpha, controls how strong the influence of this is on the model. If alpha is near zero, then the connection behaves basically identically to a “typical” neural network, with weights that don’t change as a function of activation values. If alpha is positive, that means that strong co-activation within H will tend to make the connection weight higher. If alpha is negative, the opposite is true, and strong co-activation will make the connection weight more negative. (As an aside, I’d be really interested to see the distribution over alpha values in a trained model, relative to the weight values, and look at how often they go in the same direction as the weights, and increase magnitude, and how often they have the opposite direction and attenuate the weight towards zero).

These models are trained by running them for fixed size “episodes” during which the H value gets iteratively changed, and then the alpha parameters of H get updated in the way that would have reduced error over the episode.  One area in which they seem to show strong performance is that of memorization (where the network is shown an image once, and needs to reconstruct it later). The theory for why this is true is that the weights are able to store short-term information about which pixels are in the images it sees by temporarily boosting themselves higher for inputs and activations they’ve recently seen. 

There are definitely some intuitional gaps for me in this paper. The core one is: this framework just makes weights able to update themselves as a function of the values of their activations, not as a function of an actual loss function. That is to say: it seems like a potentially better analogy to neural plasticity is just a network that periodically gets more training data, and has some amount of connection plasticity to update as a result of that. 

Main Purpose:
* The main goal of the proposed method is to exploit a global perception
mechanism, known as figure-ground segregation and Boolean Map Theory of
visual attention to compute saliency map.

Drawbacks of previous works:
* Most of the previous works do not exploit the topological structures of an image to
saliency calculation. Thus this paper aims to exploit the topological structure of a
scene in saliency calculation.

Main Idea:
* Relying on Boolean Map Theory of visual attention, an observer’s momentary
conscious awareness of a scene can be represented by a Boolean map. Given an
input image, a set of Boolean maps can be generated by randomly thresholding its
feature channels, e.g. color channel. When the set of Boolean maps is generated,
some concepts from figure-ground segregation, introduced by Gestalt
psychological studies, can be utilized to create a saliency map. As Gestalt
psychological studies suggest, figures are more likely to be attended than to
background elements. But how the figures are characterized? There is some factors
that are likely to influence figure-ground segregation such as size, surroundedness,
symmetry and etc. The proposed method uses the surroundedness factor to form a
saliency map. This factor implies that figures tends to be surrounded, that is,
they’re likely to have a closed outer contour. Therefore, this factor can be
evaluated over Boolean maps and each region which has the closed outer contour,
not connected to the image borders, gets higher attention value. So each Boolean
map result in an attention map. Then, attentions maps are summed up lineally to
form a saliency map.
In brief, the contribution of the proposed method is twofold: 1) It uses Boolean
maps to characterizes an image. 2) It exploits the figure-ground segregation
concepts to form a saliency map.

Implementation Details:
* Given an input image, a set of Boolean maps are generated by thresholding the
color channels (Lab) with uniformly selected thresholds from 0 to 255. Then, each
Boolean map is evaluated by surroundedness. Those regions which are surrounded
(closed outer contour) get value 1 and others 0. These maps are called attention
maps. After some normalization steps, the attention maps (per Boolean map) are
summed up linearly to form a mean attention map. Again, some normalization and
post-processing steps are applied and the final saliency map is obtained.

* Conclusion
In this work, a novel Boolean Map based Saliency model is proposed to leverage
the surroundedness cue that helps in figure-ground segregation. The model
borrows the concept of Boolean map from the Boolean Map Theory of visual
attention and characterizes an image by a set of Boolean maps. This representation
leads to an efficient algorithm for saliency detection. BMS is the only model that
consistently achieves state-of-the-art performance on five benchmark eye tracking
datasets, and it is also shown to be useful in salient object detection.

This paper aims to learn a sparse semantic representation of texts and images instead of a dense representation trained by CLIP or ALIGN. 

The sparse embeddings are achieved by:

(1) For an input (image or text), extract it to a feature (using Transformer) $h$ where $h_{j}$ corresponds to the $jth$ word in the input.

(2) Each $j$ word embedding will be transformed to $p(h_{j})$ in vocabulary space $V$ by using a mapping function (in this paper, this is BERT Masked Language Model MLM). So each $p(h_{j})$ is a token in a vocabulary space $V$.

(3) A max pooling layer will be applied to $p(h_{j})$ to get a value denoted for that token. So in the end, we will have a sparse vector living in V-dimensional space.

https://i.imgur.com/BTvndLR.png

Training:
To achieve two goals (1) aligning text and images in the sparse embedding and (2) grounding the sparse vector with the human-understandable word in the vocabulary, they proposed 3-stage training:

Stage 1: Training image embedding with masked tokens. In the first stage, they co-trained both the image and text encoders and apply a binary mask on the text embedding. By matching with the masked text embedding, the image encoder is learned to ground its image embedding on the tokens from the pairing text. Therefore, after the stage 1 training, the image embedding is living in the vocabulary’s interpretable space.

Stage 2: Training with frozen image encoder. In this stage, they focus on grounding the text embedding to the same interpretable space where the image embedding is trained to reside in from stage 1. The key idea is to let the image encoder teach the text encoder as a teacher model. After stage 2 training, both image and text embeddings are in the same human-interpretable embedding space constructed by the vocabulary.

Stage 3: Fine-tuning both encoders, they boosted the image-text matching performance by finetuning both encoders jointly.

https://i.imgur.com/PWrEbkk.png

To further encourage the sparsity, they proposed to use FLOPs regularization loss such that only a small number of token embeddings in V are non-zeros.

Dumont et al. Compare different adversarial transformation attacks (including rotations and translations) against common as well as rotation-invariant convolutional neural networks. On MNIST, CIFAR-10 and ImageNet, they consider translations, rotations as well as the attack of [1] based on spatial transformer networks. Additionally, they consider rotation-invariant convolutional neural networks – however, both the attacks and the networks are not discussed/introduced in detail.  The results are interesting because translation- and rotation-based attacks are significantly more successful on CIFAR-10 compared to MNIST and ImageNet. The authors, however, do not give a satisfying explanation of this observation.

[1] C. Xiao, J.-Y. Zhu, B. Li, W. H, M. Liu, D. Song. Spatially-Transformed Adversarial Examples. ICLR, 2018.

Also find this summary at [davidstutz.de](https://davidstutz.de/category/reading/).

Image registration has been well studied problem in medical image analysis community, with rigid registration taking much of the spotlight. In addition to rigid registration, non-rigid registration is of great interest due to it's applications in inter-patient modality registration where deformations of organs are highly pronounced. However non-rigid registration is an ill-posed problem with numerous degrees of freedom, which makes finding the best transformation from source to final image very difficult. To counter this, some methods were proposed which constraint the non-rigid transformation $T$ to be within certain bounds, which is not always ideal. 

To this end, Tang et al. propose a novel framework which encapsulates the problem of non-rigid image registration into a graph-cut framework, which guarantees a global maxima (or minima) under certain conditions. The formulation requires that each pixel in source image has a displacement label (whcih is a vector) indicating its corresponding position in the floating image, according to an objective function. A smoothness constraint is also added to ensure that the values of the transformation function $T$ are meaningful and stay within natural limits (no large displacement should occur between absolute neighbouring pixels). The authors propose the following formulation as their objective function, which they then solve using graph-cuts methods:

$D^* = argmin_{D} \sum_{x\in X}||I(x) - J(x + D(x))|| + \lambda \sum_{(x,y)\in \mathcal{N}}||D(x) - D(y)||$

The equation above is not fully discretized, in the sense that $D$ is still unbounded and can vary from $[-\infty, \infty]$. To allow for optimization using graph-cuts, the transformation function $D$ is mapped to a finite set $\mathcal{W} = \{0, \pm s, \pm 2s...\pm ws\}^d$. Using this discretization, the equation above can be solved using graph-cuts via a sequence of alpha-expansion. $\alpha$-expansion is a two label problem where the cost of assigning a label $\alpha$ is calculated on the basis of the previous label of the pixel. Different costs are assigned to different scenarios where the previous label may keep it's original label, or change to new label $\alpha$. This is important since it is imperative that the cost conditions satisifes the inequality given by Kolmogorov \& Zabih which then guarantees a global optima. 

The method was tested on on MR data from BrainWeb dataset which were affinely pre-registered and intensity normalized to be within 0 and 255. The method demonstrated good qualitative results when compared two state-of-art methods DEMONS and FFD, where the average intensity differences for the proposed method was much lower than the competition, while the tissue overlap was higher.