Introduction

An unofficial implementation of DreamScene360. Our project supports both text and image to scene generation.

demo.mp4

The.Forbidden.City.webm

Install

git clone  --recursive https://github.com/TingtingLiao/dreamscene360.git 
cd dreamscene360
conda create -n dreamscene360 python=3.10 
conda activate dreamscene360 

# torch2.3.0+cu12.1 
pip install torch torchvision torchaudio
 
# requirements
pip install -r requirements.txt

# xformers  
pip install -U xformers --index-url https://download.pytorch.org/whl/cu121
 
# diffusion360 
pip install git+https://github.com/archerfmy/sd-t2i-360panoimage
huggingface-cli download archerfmy0831/sd-t2i-360panoimage
 
pip install -e submodules/diff-surfel-rasterization 
pip install -e submodules/simple-knn
 
sudo apt-get install libgtest-dev libeigen3-dev libboost-all-dev libopencv-dev libatlas-base-dev
sudo apt-get install liblapack-dev libsuitesparse-dev libcxsparse3 libgflags-dev libgoogle-glog-dev libgtest-dev
conda install -c conda-forge libstdcxx-ng=12  

# pybind11
cd submodules/360monodepth/code/cpp/3rd_party
git clone https://github.com/pybind/pybind11.git 
cd pybind11 && mkdir build && cd build
cmake .. && make -j8 
sudo make install
cd ../../ 

# ceres-solver
git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver && mkdir build && cd build
cmake .. && make -j8 
sudo make install 
cd ../../../  

# instaOmniDepth
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release  ..
make -j8 
cd ../python
python setup.py build
python setup.py bdist_wheel 
pip install dist/instaOmniDepth-0.1.0-cp310-cp310-linux_x86_64.whl # if failed, please check your file version in dist/ 

Usage

# text-to-scene generation 
python main.py prompt="a cozy living room"  upscale=True 

# (for <16GB gpu, set upscale=False) 
python main.py prompt="a cozy living room" upscale=False 

# with gui 
python main.py prompt="a cozy living room" upscale=True gui=True 

# image-to-scene 
python main.py input=data/i2p-image.jpg prompt="an office room" upscale=True 

Gallery

The.Great.Wall.mp4

The.Forbidden.City.mp4

Dubai.City.mp4

An.anime.style.Japanese.city.street.mp4

A.bedroom.mp4

training_process.webm

Acknowledgement

Special thanks to the projects and their contributors:

• DreamScene360
• Diffusion360
• 360monodepth
• 2DGS
• Equirec2Perspec