Point pattern matching problem have been an active research topic in computation geometry and pattern recognition communities. These point sets typically arise in a variety of applications, where the problem lies in registration of these point sets which is encountered in stereo matching, feature based image registration and so on. Mathematically, the problem of registering two point sets translates to the following: Let $\{\mathcal{M}, \mathcal{S}\}$ be two finite set points which need to be registered, where $\mathcal{M}$ is the "moving" point set and $\mathcal{S}$ is the "fixed" or scene set. A transformation $\mathcal{T}$ is then calculated which can transform points from $\mathcal{M}$ to $\mathcal{S}$. 

To this end, Jian and Vemuri propose to use a Gaussian Mixture Model based representation of point sets which are registered together my minimizing a cost function using a slightly modified version of the Iterative Closest Point (ICP) algorithm. In this setting, the problem of point set registration becomes as that of aligning two Gaussian mixture models by minimizing discrepancy between two Gaussian mixtures. The cost function for this optimization is chosen as a closed-form version of the $L_2$ distance between Gaussian mixtures, which allows the algorithm to be computationally efficient. 

The main reason behind choosing Gaussian mixture models to represent discrete point sets was that it directly translates to interpreting point sets as randomly sampled data from a distribution of random point locations. This models the uncertainty of point sets well during feature extraction. The second reason was that hard discrete optimization problems that are encountered in point matching literature become tractable continuous optimization problems. 

The probability density function of a general Gaussian mixture is defined as follows:

\begin{equation}
    p(x) = \sum_{i=1}^{k}w_i \phi(x|\mu_i, \sigma_i)
\end{equation}

where:

$\phi(x|\mu_i, \sigma_i) = \dfrac{exp\left[-\frac{1}{2}(x - \mu_i)\sigma_{i}^{-1}(x-\mu_i)\right]}{\sqrt{(2\pi)^d |det(\sigma_i)|}}$
The GMM from the given point set is constructed as follows: i) the number of GMM components is equal to the number of points in the point set, and every component is weighted equally, ii) every component's mean vectors is represented by the spatial location of each point, iii) all components have same spherical covariance matrix. 

An intuitive reformulation of the point set registration problem is to solve an optimization problem such that a certain dissimilarity measure between the Gaussian mixtures constructed from the transformed model set and the fixed scene set is minimized. For this method, L2 distance is chosen as the dissimilarity measure for measuring similarity between two Gaussian mixtures. The objective can then be minimized by either a closed-form numerical method (in case the function is convex) or an iterative gradient based method. The method was applied and tested on both rigid and non-rigid point set registration problems. 

multi layer RNN in which first layer is LSTM, following layers $l$ have $t$,$c$ gates that control whether the state of the layer is carried from previous state or transferred previous layer:
$s_l^{[t]} = h_l^{[t]} \cdot t_l^{[t]} + s_{l-1}^{[t]} \cdot c
_l^{[t]}$

## **Introduction**
This paper presents a stochastic active contour model for image segmentation from cardiac MR images. The proposed algorithms aims to minimize an energy functional by the level set method while incorporating stochastic region based and edge based information as well as shape priors of the heart and local contour properties. Moreover, the algorithm also uses a parameter annealing component to dynamically balance the weightage of the components of the energy functional.


## **Method**
The paper locates a contour $C$ in a cardiac MR image that segments the image into two groups - the heart and the background. The corresponding objective energy functional can be represented by
$$
    J(C) = \lambda_1 J_1(C) + \lambda_2 J_2(C) + \lambda_3 J_3(C) + \lambda_4 J_4(C)
$$

Looking at these individual components, we have
* _Region Based Term: Model Matching_ **$J_1(\phi)$**: In order to segment the image into 2 regions, let the two regions inside and outside the contour C be represented by $\Omega_1$ and $\Omega_2$ respectively. For each region, consider a stochastic model to describe the pixel statistics of that region. Assuming that the pixel intensities of all the pixels in each region are statistically independent, the objective is to minimize the negative log-likelihood  of pixels belonging to the correct regions.
*  _Edge Based Term_ **$J_2(\phi)$**: In order for the contour C to be aligned to the prominent edges in the image, the edge map of the image (which can be obtained by various image smoothing methods such as Gaussian kernel blurring, edge-preserving anisotropic diffusions, Min/Max flow algorithms, etc.) has to be minimized.
* _Heart Shape Prior Term_ **$J_3(\phi)$**: In order to distinguish between similar looking tissues in the foreground and the background, an elliptical heart shaped prior is used. An ellipse can be described with 5 parameters with certain constraints in the conic equation. 
* _Contour Smoothing Term_ **$J_4(\phi)$**: In order to obtain a smooth contour of the segmented heart, the total Euclidean arc length of the contour C should be minimized.

The parameters ($\lambda_1$, $\lambda_2$, and $\lambda_3$) of the energy functional $J(C)$ need to be dynamically updated during the energy minimization. For example, the region-based and the edge-based terms should have a higher weightage in $J(C)$ during the initial steps of the segmentation, and at the later stages, their weightage should be reduced and that of the shape prior should be increased in order to keep the segmented output similar to the desired shape.

## **Results**

The two metrics used for assessing the performance were Area Similarity and Shape Similarity. The algorithm was tested on 48 images covering 143 contours, including manually annotated contours by an expert on six rat cardiac sequences of eight frames each, and the results indicated excellent segmentation agreement with the manually traced contours.

## **Discussion and Shortcomings**
Since STACS uses stochastic models instead of deterministic models, it can be applied to a large variety of images, and is especially helpful when distinguishing between visually similar adjacent regions.

Since STACS incorporates both region-based and edge-based information in its energy functional, this makes it more robust to noise as well as reduces the susceptibility to curve initialization.

Perhaps the most highlighting feature of STACS that distinguishes it from other active contour based models is that it incorporates shape based priors into the energy functional. This helps segment the heart from the chest wall, which is especially difficult since the two regions share similar texture.

Moreover, the scheduled parameter annealing adjusts the weights of the components of the energy functional, which helps dynamically vary the importance to different components at different stages of the segmentation process.

__Edit__: The paper has a newer version at https://openreview.net/pdf?id=B1YfAfcgl, where I also copied this review over with some changes.

__Note__: An earlier version of the review (almost identical to the present one) for an earlier version of the paper (available on arXiV) can be found here: http://www.shortscience.org/paper?bibtexKey=journals/corr/1611.01838#csaba
The only change concerns relation to previous work.

__Problem__: The problem considered is to derive an improved version of SGD for training neural networks (or minimize empirical loss) by modifying the loss optimized to that the solution found is more likely to end up in the vicinity of a minimum where the loss changes slowly ("flat minima" as in the paper of Hochreiter and Schmidhuber from 1997). 

__Motivation__: It is hypothetised that flat minima "generalize" better. 

__Algorithmic approach__: Let $f$ be the (empirical) loss to be minimized. Modify this to $$\overline{f}(x) = \rho f(x) - \log \int \exp(-\frac{\gamma}{2}\||z\||^2-f(x+z))\,dz$$ with some $\rho,\gamma>0$ tunable parameters. For $\||z\||^2\gg \frac{1}{\gamma}$, the term $\exp(-\frac{\gamma}{2}\||z\||^2)$ becomes very small, so effectively the second term is close to a constant times the integral of $f$ over a ball centered at $x$ and having a radius of $\propto \gamma^{-1/2}$. This is a smoothened version of $f$, hence one expects that by making this term more important then the first term, a procedure minimizing $\bar f$ will be more likely to end up at a flat minima of $f$. Since the gradient is somewhat complicated, an MCMC algorithm is proposed ("stochastic gradient Langevin dynamics" from Welling and Teh, 2011).

__Results__: There is a theoretical result that quantifies the increased smoothness of $\overline{f}$, which is connected to stability and ultimately to generalization through citing a result of Hardt et al. (2015). Empirical results show better validation error on two datasets: MNIST and CIFAR-10 (the respective networks are LeNet and All-CNN-C). The improvement is in terms of reaching the same validation error as with an "original SGD" but with fewer "epochs".

__Soundness, significance__: The proof of the __theoretical result__ relies on an arbitrary assumption that there exists some $c>0$ such that no eigenvalue of the hessian of $f$ lies in the set $[-2\gamma-c,c]$ (the reason for the assumption is because otherwise a uniform improvement cannot be shown). For $\rho=0$ the improvement of the smoothness (first and second order) is a factor of $1/(1+c/\gamma)$. The proof uses Laplace's method and is more a sketch than a rigorous proof (error terms are dropped; it would be good to make this clear in the statement of the result).

In the experiments the modified procedure did not consistently reach a smaller validation error. The authors did not present running times, hence it is unclear whether the procedure's increased computation cost is offset by the faster convergence. 

__Evaluation__: It is puzzling why a simpler smoothing, e.g., $$\overline{f}(x) = \int f(x+z) g(z) dz$$ (with $g$ being the density of a centered probability distribution) is not considered. The authors note that something "like this" may be infeasible in "deep neural networks" (the note is somewhat vague). However, under mild conditions, $\frac{\partial}{\partial x} \overline{f}(x) = \int \frac{\partial}{\partial x} f(x+z) g(z) dz$, hence, for $Z\sim g(\cdot)$, $\frac{\partial}{\partial x} f(x+Z)$ is an unbiased estimate of $\frac{\partial}{\partial x} \overline{f}(x)$, whose calculation is as cheap as that of vanilla SGD. Also, how much the smoothness of $f$ changes when using this approach is quite well understood.

__Related work__: Hochreiter and Schmidhuber argued a long time ago for the benefits of "flat minima" in an  identically titled Neural computation paper that appeared in 1997. This reviewer may not agree with the arguments in this paper, but the paper is highly relevant and should be cited. It is also strange that the specific modification that appears in this paper was proposed by others (Baldassi et al.), whom this paper also cites, but without giving these authors credit for introducing local entropy as a smoothing technique. 

This paper proposes an approach to measure motion similarity between human-human and human-object interaction. The authors claim that human activities are usually defined by the interaction between individual characters, such as a high-five interaction.

As the interaction datasets are not available authors provide multiple small-scale interaction datasets:
https://i.imgur.com/P815TYu.png
where:
- 2C = a Character-Character (2C) database using kick-boxing motions
- CRC = Character-Retargeted Character where the size of characters is adjusted while maintaining the nature of the interaction
https://i.imgur.com/XX1WNpO.png
- HOI = Human-Object Interaction where a Chair is used as an object
https://i.imgur.com/Z6cxd7R.png
- 2PB = 2 People Boxing 
https://i.imgur.com/yUxmpY5.png
- 2PD = 2 People Daily Interaction where people represented as a surface point cloud
https://i.imgur.com/EzyELg3.png


**Methodology**
- *Customized Interaction Mesh Structure*. An interaction mesh is created by generating a volumetric mesh using Delaunay Tetrahedralization. Interaction is therefore represented by a series of interaction meshes. To reduce the bias of unequal number of points per human body part, synthetic points (shown in blue) are derived from available skeleton structure (shown in red):
 https://i.imgur.com/jtlrH49.png

 The edges after Delaunay Tetrahedralization are filtered in a way that all edges connecting to the same character are removed, as they do not contribute to the interaction.
 https://i.imgur.com/nWeUNl1.png
 The temporal sequence of interaction is a series of interaction meshes.

- *Distance between interaction meshes*.
   - Distance between two interaction meshes of two-character interactions:

   $d(e_i, e_j) = (|e_{i1} - e_{j1}| + |e_{i2} - e_{j2}|) \times \frac{1}{2} (1 - cos\theta),$
   https://i.imgur.com/jMHGx3o.png
   where $e_{i1}$ and $e_{i2}$ are two endpoints of an edge.

   - *Earth Mover's Distance*. Earth Mover’s Distance (EMD) is used to find the best correspondence between the input interaction meshes to achieve the comparison of two interactions with different semantic meaning:

    $EMD(E_I^{t_I}, E_J^{tJ}) = \frac{D(E_I^{t_I}, E_J^{tJ})}{\sum_{i=1}^{m} \sum_{j=1}^{n} f_{i,j}^*},$

    where $D(E_I^{t_I}, E_J^{tJ}) = \sum_{i=1}^{m} \sum_{j=1}^{n} d(e_i, e_j) f_{i,j}^*$ represents the minimum distance between two interaction meshes and $f_{i,j}^*$ is the optimal set of flow values returned by the mass transport solver that finds the optimal edge-level correspondence between two interaction meshes. The concept of the mass transport solver is visualized below:
    https://i.imgur.com/wmRhcxP.png


 - *Distance between interactions sequences*.
    - spatial normalization - removing its pelvis translation and its horizontal facing angle in each frame

    - temporal sampling - non-linear sampling strategy based on the frame distance measured by EMD. The sampling algorithm samples fewer in temporal regions with high similarity, which contribute less to the context of the interaction.

    - temporal alignment - keyframes are aligned using Dynamic Time Warping (DTW)

Possible functionality with the proposed method:
- Interaction motion similarity analysis
- Interaction motion retrieval

Notice:
>The pre-process took 1.5 hours, 0.5 hour and 4 hours for the 2C, CRC and HOI databases respectively. Given the meshes, computing the distance between two interactions took 0.2 seconds on average.

Discussion:
- the algorithm focuses on boxing/kickboxing as they have clear rules. Extending the proposed algorithm for measuring motion similarity for daily activities would require careful annotation.
- While the method, in theory, can be applied to single human activities, it quite clear that it would perform worse than other baselines for the task. This implies that it is better not applied for use cases where a comparison between two independent movements having no interaction with each other (dancing, yoga) is required.
- The method works best for close interaction activities as the edges in the interaction mesh would tend to have a similar structure (e.g. edge length) in case of distant interacting objects. 
- Application-wise, the algorithm is not suitable for online real-time use.