Skip to Main content Skip to Navigation
Journal articles

Learning Event Representations for Temporal Segmentation of Image Sequences by Dynamic Graph Embedding

Abstract : Recently, self-supervised learning has proved to be effective to learn representations of events suitable for temporal segmentation in image sequences, where events are understood as sets of temporally adjacent images that are semantically perceived as a whole. However, although this approach does not require expensive manual annotations, it is data hungry and suffers from domain adaptation problems. As an alternative, in this work, we propose a novel approach for learning event representations named Dynamic Graph Embedding (DGE). The assumption underlying our model is that a sequence of images can be represented by a graph that encodes both semantic and temporal similarity. The key novelty of DGE is to learn jointly the graph and its graph embedding. At its core, DGE works by iterating over two steps: 1) updating the graph representing the semantic and temporal similarity of the data based on the current data representation, and 2) updating the data representation to take into account the current data graph structure. The main advantage of DGE over state-of-the-art self-supervised approaches is that it does not require any training set, but instead learns iteratively from the data itself a low-dimensional embedding that reflects their temporal and semantic similarity. Experimental results on two benchmark datasets of real image sequences captured at regular time intervals demonstrate that the proposed DGE leads to event representations effective for temporal segmentation. In particular, it achieves robust temporal segmentation on the EDUBSeg and EDUBSeg-Desc benchmark datasets, outperforming the state of the art. Additional experiments on two Human Motion Segmentation benchmark datasets demonstrate the generalization capabilities of the proposed DGE.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03109960
Contributor : Herwig Wendt <>
Submitted on : Thursday, January 14, 2021 - 10:58:44 AM
Last modification on : Wednesday, June 9, 2021 - 10:00:33 AM
Long-term archiving on: : Thursday, April 15, 2021 - 6:34:47 PM

File

DGE-MANUSCRIPT-Arxiv-V4.pdf
Files produced by the author(s)

Identifiers

Citation

Mariella Dimiccoli, Herwig Wendt. Learning Event Representations for Temporal Segmentation of Image Sequences by Dynamic Graph Embedding. IEEE Transactions on Image Processing, Institute of Electrical and Electronics Engineers, 2021, 30, pp.1476-1486. ⟨10.1109/TIP.2020.3044448⟩. ⟨hal-03109960⟩

Share

Metrics

Record views

97

Files downloads

150