Incremental NeRF SLAM

Project Page

Bridging explicit and implicit representations for SLAM

Broad Idea

Recent techniques (like iMAP) adopt a novel view of SLAM -- that of online learning. The goal of such systems is to build a representation (a map) of the environment that is suited for navigation and relocalization, fully online (i.e., as a robot explores a new environment).

The iMAP paper uses a fully implicit map, i.e., a NeRF is the sole map representation used for mapping and localization. This imposes operational constraints (2 Hz map update rate, 8 Hz localization update rate).

The idea is to build a SLAM system that, akin to ORB-SLAM, brings together components that work well in practice (but in the context of implicit representations for SLAM).

Road Map

1. Implement major parts of the NeRF-SLAM pipeline
   1. Data loader (load in videos)
   2. RGB-D NeRF pipeline (i.e., replace RGB image rendering loss with losses proposed in iMAP)
   3. Active image sampling
   4. Keyframe management logic - -- until this point, assume localization is GT
   5. Localization pipeline using NeRF
2. Replace localization pipeline with traditional RGB-D odometry

Related Papers

1. iMAP: implicit mapping and positioning in real-time [Project] [PDF]
2. iNeRF: inverting neural radiance fields for pose estimation [Project] [PDF]
3. Nerf--: Neural radiance fields without known camera parameters [Project] [PDF]
4. NeRF: Representing scenes as neural radiance fields for view synthesis [Project] [PDF]
5. Fourier features let networks learn high frequency functions in low dimensional domains [Project] [PDF]
6. SIREN: Implicit neural representations with periodic activation functions [Project] [PDF]
7. KiloNeRF: Speeding up neural radiant

Useful External Links

• Blogs
  • Understanding TUM dataset (https://www.programmersought.com/article/51194902722/)
  • Toolkit for TUM dataset (https://vision.in.tum.de/data/datasets/rgbd-dataset/tools)
• Papers

Code details

Unofficial reimplementation of Implicit Mapping and Positioning in Real-Time (link). The project page can be found here This repo uses (inverse-NeRF) for incremental camera pose estimation and simulateously building the map of the scene encoded and decoded by NeRF modules.

| Update: The official implementation has been released and can be found inside the (repo)

Code structure

main-repository/
│
├── train.py - incremental localization and mapping on TUM dataset
├── train_nerf.py - static NeRF training of a scene
├── train_inv.py - used inverse-nerf for camera pose estimation of a scene
│
├── utils/ - helper functions for camera pose estimation 
│   ├── pose_utils.py - borrowed from inverse-nerf implementation [(here)](https://github.com/salykovaa/inerf)
│   ├── align_trajectory.py - helper function to align trajectories and finding scale using Umeya's method
│   ├── base_dataset.py - All the data augmentations are implemented here
│   └── base_trainer.py
│
├── models/ - contains implementation of nerf encoder and decoder
│   ├── lie_group_helpers.py - borrowed from inverse-nerf implementation [(here)](https://github.com/salykovaa/inerf)
│   ├── nerf_origin.py - original nerf implementation
│   ├── rendering.py - contains the decoder of the nerf
│   └── base_trainer.py
|
└── datasets/ - helper functions for camera pose estimation 
    ├── TUM_inc.py - dataset/dataloader for incremental localization and mapping on TUM dataset
    ├── llff_TUM.py - dataset/dataloader for static NeRF training of a scene