This repository provides the official PyTorch implementation for the following paper:
ReliTalk: Relightable Talking Portrait Generation from a Single Video
Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Wayne Wu, Xiangyu Fan, Lei Yang, and Ziwei Liu
From MMLab@NTU affiliated with S-Lab, Nanyang Technological University and SenseTime Research.
Video Data: HDTF
- Clone this repo:
git clone --recursive git@github.com:arthur-qiu/ReliTalk.git
- Create a conda environment
conda env create -f environment.yml
and activateconda activate ReliTalk
- We use
libmise
to extract 3D meshes, buildlibmise
by runningcd code; python setup.py install
- Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into
./code/flame/FLAME2020
Prepare the dataset following intructions in ./preprocess/README.md
.
Link the dataset folder to ./data/datasets
. Link the experiment output folder to ./data/experiments
.
cd code
python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama.conf
python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama_test.conf --is_eval --checkpoint [epoch1]
mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_eval/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_eval/
mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_train/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_train/
mv ../data/experiments/Obama ../data/experiments/Obama_store
python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_light.conf
python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_test_light.conf --is_eval --checkpoint [epoch2]
If you find this work useful for your research, please consider citing our paper:
@misc{qiu2023relitalk,
title={ReliTalk: Relightable Talking Portrait Generation from a Single Video},
author={Haonan Qiu and Zhaoxi Chen and Yuming Jiang and Hang Zhou and Xiangyu Fan and Lei Yang and Wayne Wu and Ziwei Liu},
year={2023},
eprint={2309.02434},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
This code borrows heavily from IMavatar.