Actor and Action Video Segmentation from a Sentence
Cees G. M. Snoek
arXiv e-Print archive - 2018 via Local arXiv
First published: 2018/03/20 (5 years ago) Abstract: This paper strives for pixel-level segmentation of actors and their actions
in video content. Different from existing works, which all learn to segment
from a fixed vocabulary of actor and action pairs, we infer the segmentation
from a natural language input sentence. This allows to distinguish between
fine-grained actors in the same super-category, identify actor and action
instances, and segment pairs that are outside of the actor and action
vocabulary. We propose a fully-convolutional model for pixel-level actor and
action segmentation using an encoder-decoder architecture optimized for video.
To show the potential of actor and action video segmentation from a sentence,
we extend two popular actor and action datasets with more than 7,500 natural
language descriptions. Experiments demonstrate the quality of the
sentence-guided segmentations, the generalization ability of our model, and its
advantage for traditional actor and action segmentation compared to the
This paper performs pixel-wise segmentation of the object of interest which is specified by a sentence. The model is composed of three main components: a **textual encoder**, a **video encoder**, and a **decoder**.https://i.imgur.com/gjbHNqs.png
- **Textual encoder** is word2vec pre-trained model followed by 1D CNN.
- **Video encoder** is a 3D CNN to obtain a visual representation of the video (can be combined with optical flow to obtain motion information).
- **Decoder**. Given a sentence representation $T$ a separate filter $f^r = tanh(W^r_fT + b^r_f)$ is created
to match each feature map in the video frame decoder and combined with visual features as $S^r_t = f^r * V^r_t$, for each $r$esolution at $t$imestep. The decoder is composed of sequence of transpose convolution layers to get the response map of the same size as the input video frame.