Facial Geometric Detail Recovery via Implicit Representation

Xingyu Ren, Alexandros Lattas, Baris Gecer, Jiankang Deng, Chao Ma, Xiaokang Yang, Stefanos Zafeiriou
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University Imperial College London, Huawei CBG Huawei CBG
FG 2023
Paper arXiv Video Code
Teaser

We introduce a single-facial-image geometric detail recovery algorithm. Our method generates complete high-fidelity texture maps from occluded facial images, and employs implicit renderer and shape functions, to derive fine geometric details from decoupled detail normals. As a bonus, it disentangles the facial texture into ap- proximate diffuse albedo, diffuse and specular shading in a self-supervision manner. Input images are from FFHQ and CelebAMask-HQ.

Abstract

Learning a dense 3D model with fine-scale details from a single facial image is highly challenging and ill-posed. To address this problem, many approaches fit smooth geometries through facial prior while learning details as additional displacement maps or personalized basis. However, these techniques typically require vast datasets of paired multi-view data or 3D scans, whereas such datasets are scarce and expensive.

To alleviate heavy data dependency, we present a robust texture-guided geometric detail recovery approach using only a single in-the-wild facial image. Specifically, we inpaint occluded facial parts, generate complete textures, and build an accurate multi-view dataset of the target subject. In order to estimate the detailed geometry, we define an implicit signed distance function and employ a physically-based implicit renderer to reconstruct fine geometric details from the generated multiview images. Our method not only recovers accurate facial details but also decomposes the diffuse and specular albedo, normals and shading components in a self-supervised way. Finally, we register the implicit shape details to a 3D Morphable Model template, which can be used in traditional modeling and rendering pipelines. Extensive experiments demonstrate that the proposed approach can reconstruct impressive facial details from a single image, especially when compared with state-of-the-art methods trained on large datasets.

Pipeline

Pipeline
Overview of our method. Left: we perform a robust texture completion, utilizing StyleGANv2 to repair both externally- and self-occluded regions as preprocessing. Right: Based on a coarse 3DMM fitting, we render a multi-view dataset and use an implicit signed distance function and a novel physically-based implicit differentiable renderer (PBIDR) to improve the coarse shape. Based on the self-supervised decomposition of diffuse normal and specular normal, we obtain a detailed facial geometry as well as approximate diffuse albedo, diffuse shading and specular shading.

BibTeX

@inproceedings{ren2023facial,
      title={Facial geometric detail recovery via implicit representation},
      author={Ren, Xingyu and Lattas, Alexandros and Gecer, Baris and Deng, Jiankang and Ma, Chao and Yang, Xiaokang},
      booktitle={2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)},
      pages={1--8},
      year={2023},
      organization={IEEE}
    }