PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes.

Model Architecture

Prerequisites

• Windows10 or Linux
• Python 3.7
• NVIDIA GPU + CUDA CuDNN
• PyTorch 1.7.1 (or higher)

Installation

• Clone this repo:

git clone https://github.com/VCL3D/PanoDR.git
cd PanoDR

• We recommend setting up a virtual environment (follow the virtualenv documentation). Once your environment is set up and activated, install the vcl3datlantis package:

cd src/utils
pip install -e .

Dataset

We use Structured3D dataset. To train a model on the dataset please download the dataset from the official website. We follow the official training, validation, and testing splits as defined by the authors. After downloading the dataset, split the scenes into training train, validation and test folders. The folders should have the following format:

Structured3D/
    train/
        scene_00000
        scene_00001
        ...
    test/
        scene_03250
        scene_03251
        ...
    validation/
        scene_03000
        scene_03001
        ...

In order to estimate the dense layout maps, specify the path to train and test folders and run:

python src\utils\vcl3datlantis\dataset\precompute_structure_semantics.py 

Structured3D provides 3 different room configurations, empty, simple and full, which,in theory, enables this dataset for Diminished Reality applications. In practice, this statement doesn't hold since the rooms are rendered with ray-tracing, fact that affects the global illumination and texture, so replacing a region from the full configuration with the corresponding empty one does not create photo-consistent results.

Full room configuration with annotated the object for diminsing	Full room configuration with replaced region from empty room configuration

It is obvious that the diminished scene has large photometric insconsistency at the diminished region, which is not suitable for deep learning algorithms. To overcome this barrier, we start to augment the empty rooms with objects from the corresponding full room configurations, so that replacing a region is photo-consistent.

Empty room configuration with augmented objects	Augmented room with diminished table

The only issue of this approach of creating samples is the abscence of shadows, which makes the whole scene less realistic, but still the gain of this method is greater for Diminished Reality applications.

Training

More info regarding the training of the model will be available soon!

Inference

You can download the pre-trained models from here and specify the arguments --eval_chkpnt_folder and --segmentation_model_chkpnt, respectively. Assuming the input image and mask are in the format as in the input folder run:

python src/train/test.py --inference --eval_path input/

Citation

If you use this code for your research, please cite the following:

@inproceedings{gkitsas2021panodr,
  title={PanoDR: Spherical Panorama Diminished Reality for Indoor Scenes},
  author={Gkitsas, Vasileios and Sterzentsenko, Vladimiros and Zioulis, Nikolaos and Albanis, Georgios and Zarpalas, Dimitrios},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3716--3726},
  year={2021}
}

Acknowledgements

This project has received funding from the European Union's Horizon 2020 innovation programme ATLANTIS under grant agreement No 951900.

Our code borrows from SEAN and deepfillv2.