Volume Sweeping: Learning Photoconsistency for Multi-View Shape Reconstruction

Vincent Leroy (1, 2) , Jean-Sébastien Franco (2) , Edmond Boyer (2)

1 Naver Labs Europe [Meylan]
2 MORPHEO - Capture and Analysis of Shapes in Motion

We propose a full study and methodology for multi-view stereo reconstruction with performance capture data. Multi-view 3D reconstruction has largely been studied with general, high resolution and high texture content inputs, where classic low-level feature extraction and matching are generally successful. However in performance capture scenarios, texture content is limited by wider angle shots resulting in smaller subject projection areas, and intrinsically low image content of casual clothing. We present a dedicated pipeline, based on a per-camera depth map sweeping strategy, analyzing in particular how recent deep network advances allow to replace classic multi-view photoconsistency functions with one that is learned. We show that learning based on a volumetric receptive field around a 3D depth candidate improves over using per-view 2D windows, giving the photoconsistency inference more visibility over local 3D correlations in viewpoint color aggrega-tion. Despite being trained on a standard dataset of scanned static objects, the proposed method is shown to generalize and significantly outperform existing approaches on performance capture data, while achieving competitive results on recent benchmarks.

Mots clés: Multi View Stereo Reconstruction; Learned Photoconsistency; Performance Capture; Volume Sweeping

Domaines: Informatique [cs]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Vincent Leroy, Jean-Sébastien Franco, Edmond Boyer. Volume Sweeping: Learning Photoconsistency for Multi-View Shape Reconstruction. International Journal of Computer Vision, 2021, 129, pp.284-299. ⟨10.1007/s11263-020-01377-0⟩. ⟨hal-02975479⟩