PREF

novel view sythesis	image regression	neural field editing

PREF is an efficient frequency-based neural scene representation that allows:

1. super fast training and rendering novel views. (fast and approximated solvers, FFT, exist.)
2. gloabl optimization behaviour. (optimizing a local point lead global change.)
3. editing optimized neural fields through frequency-domain manipulations. (convolutional kernels as Fourier multipliers.)

TL;DR

We learn frequency representations (blue) for neural signals (red).

You can also view it as a dense variant of Position Encoding or Fourier Feature Mapping by rearranging their parameters in the first linear layer into a phasor volume.

Installation

Tested on Ubuntu 20.04 + Pytorch 1.11.0 (It seems important to install Pytorh version higher than 1.11.0 for complex tensor optimization).

Using PREF

Replacing tiny PE encoder + deep MLP with PREF encoder (dense PE) + shallow MLP to accelerate neural signal reconstruction. This is a preliminary release, without being carefully tested.

Example

import torch
import torch.nn as nn
import torch.nn.functional as F
from encoder.pref_encoder import PREF

class NeuralField(nn.Module):
  def __init__(self):
    super().__init__()
    self.encoder = PREF(linear_freqs=[128]*3, reduced_freqs=[1]*3, feature_dim=16)
    
    input_dim = self.encoder.output_dim
    hidden_dim = 64
    self.mlp = torch.nn.Sequential(
      torch.nn.Linear(input_dim, hidden_dim), 
      torch.nn.ReLU(inplace=True), 
      torch.nn.Linear(hidden_dim, 1))

  def forward(self, x):
    x = self.encoder(x)
    x = self.mlp(x)
    return x

NeRF with PREF

Installation

Install environment:

conda create -n PhasoRF python=3.8
conda activate PhasoRF
pip install torch torchvision
pip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard plyfile

Dataset

• Synthetic-NeRF
• Synthetic-NSVF
• Tanks&Temples
• Forward-facing

Quick Start

The training script is in train.py, to train a NeRF with PREF:

python train.py --config configs/hotdog.txt

You need to set --render_test 1/--render_path 1 if you want to render testing views or path after training.

More options refer to the opt.py.

We provide a few examples in the configuration folder, note that:

dataset_name, choices = ['blender', 'llff', 'nsvf', 'tankstemple'].

Notice: We have tested parameters for 'blender' dataset, and you should get about $36.4$ psnr for the hotdog scene if correctly installing the repo.

Acknowledgement

NeRF with PREF extensively uses code from TensoRF.

Citation

If you find our code or paper helpful in your research, please consider citing

@article{Huang2022PREF,
  author = {Huang, Binbin and Yan, Xinhao and Chen, Anpei and Gao,Shenghua and Yu, Jingyi},
  title = {PREF: Phasorial Embedding Fields for Compact Neural Representations},
  publisher = {arXiv},
  year = {2022},
}

Please also consider citing

@article{tensorf,
  title={TensoRF: Tensorial Radiance Fields},
  author={Chen, Anpei and Xu, Zexiang and Geiger, Andreas and Yu, Jingyi and Su, Hao},
  journal={arXiv preprint arXiv:2203.09517},
  year={2022}
}