May 1, 2021

568 words 3 mins read

YadiraF/DECA

YadiraF/DECA

DECA: Detailed Expression Capture and Animation

repo name YadiraF/DECA
repo link https://github.com/YadiraF/DECA
homepage
language Python
size (curr.) 18911 kB
stars (curr.) 400
created 2020-04-08
license Other

DECA: Detailed Expression Capture and Animation

This is the official Pytorch implementation of DECA.

DECA reconstructs a 3D head model with detailed facial geometry from a single input image. The resulting 3D head model can be easily animated. Please refer to the arXiv paper for more details.

The main features:

  • Reconstruction: produces head pose, shape, detailed face geometry, and lighting information from a single image.
  • Animation: animate the face with realistic wrinkle deformations.
  • Robustness: tested on facial images in unconstrained conditions. Our method is robust to various poses, illuminations and occlusions.
  • Accurate: state-of-the-art 3D face shape reconstruction on the NoW Challenge benchmark dataset.

Getting Started

Clone the repo:

git clone https://github.com/YadiraF/DECA
cd DECA

Requirements

  • Python 3.7 (numpy, skimage, scipy, opencv)
  • PyTorch >= 1.6 (pytorch3d)
  • face-alignment (Optional for detecting face)
    You can run
    pip install -r requirements.txt
    

    Or use virtual environment by runing

    bash install_pip.sh
    

    Then follow the instruction to install pytorch3d.

Usage

  1. Prepare data
    a. download FLAME model, choose FLAME 2020 and unzip it, copy ‘generic_model.pkl’ into ./data
    b. download DECA trained model, and put it in ./data (no unzip required)
    c. (Optional) follow the instructions for the Albedo model to get ‘FLAME_albedo_from_BFM.npz’, put it into ./data

  2. Run demos
    a. reconstruction

    python demos/demo_reconstruct.py -i TestSamples/examples --saveDepth True --saveObj True
    

    to visualize the predicted 2D landmanks, 3D landmarks (red means non-visible points), coarse geometry, detailed geometry, and depth.

    Please run python demos/demo_reconstruct.py --help for more details.

    b. expression transfer

    python demos/demo_transfer.py
    

    Given an image, you can reconstruct its 3D face, then animate it by tranfering expressions from other images. Using Meshlab to open the detailed mesh obj file, you can see something like that:

    Note that, you need to set ‘–useTex True’ to get full texture.

    c. for the teaser gif (reposing and animation)

    python demos/demo_teaser.py 
    

    More demos and training code coming soon.

Evaluation

DECA (ours) achieves 9% lower mean shape reconstruction error on the NoW Challenge dataset compared to the previous state-of-the-art method.
The left figure compares the cumulative error of our approach and other recent methods (RingNet and Deng et al. have nearly identitical performance, so their curves overlap each other). Here we use point-to-surface distance as the error metric, following the NoW Challenge.

For more details of the evaluation, please check our arXiv paper.

Citation

If you find our work useful to your research, please consider citing:

@inproceedings{DECA:Siggraph2021,
  title={Learning an Animatable Detailed {3D} Face Model from In-The-Wild Images},
  author={Feng, Yao and Feng, Haiwen and Black, Michael J. and Bolkart, Timo},
  journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH)}, 
  volume = {40}, 
  number = {8}, 
  year = {2021}, 
  url = {https://doi.org/10.1145/3450626.3459936} 
}

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Acknowledgements

For functions or scripts that are based on external sources, we acknowledge the origin individually in each file.
Here are some great resources we benefit:

We would also like to thank other recent public 3D face reconstruction works that allow us to easily perform quantitative and qualitative comparisons :)
RingNet, Deep3DFaceReconstruction, Nonlinear_Face_3DMM, 3DDFA-v2, extreme_3d_faces, facescape

comments powered by Disqus