add gaussianSplat application

docs/jupyter/visualization/3d_gaussian_splatting.ipynb

@@ -0,0 +1,148 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import open3d as o3d\n",
+    "from open3d.web_visualizer import draw"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 3D Gaussian Splatting\n",
+    "\n",
+    "Open3D supports basic operations with 3D Gaussian Splats. You can read and write Gaussian Splats to and from the [`ply`](https://github.com/graphdeco-inria/gaussian-splatting/blob/main/scene/gaussian_model.py) and [`splat`](https://github.com/antimatter15/splat/blob/main/convert.py) file formats. In addition, we have partial rendering support, i.e. you can visualize a Gaussian Splat as a point cloud with color. In the future we will add complete rendering support as well as editing support. This tutorial will guide you through the process of loading, visualizing, and manipulating 3D Gaussian splats using Open3D.\n",
+    "\n",
+    "To begin with, let's download some sample 3DGS data. We will use a cropped version of the garden scene from the [Mip NeRF 360 dataset](https://jonbarron.info/mipnerf360/). Lets download the data to get started:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.ply\n",
+    "!curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.splat"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Reading, writing and converting Gaussian Splats\n",
+    "\n",
+    "Now we can read these files into Open3D. The data format for the 3DGS representation is the following:\n",
+    "* `pcd.point[‘positions’]` (N,3) – (x,y,z) for each splat.\n",
+    "* `pcd.point[‘opacity’]` – (N,1) opacity\n",
+    "* `pcd.point[‘rot’]` (N, 4) - quaternion rotation of Gaussian.\n",
+    "* `pcd.point[‘scale']` (N, 3) - x, y, z scales for Gaussian.\n",
+    "* `pcd.point[‘f_dc’]` (N, 3) – DC components for RGB colors.\n",
+    "* `pcd.point[‘f_rest’]` (N, Nc, 3) – SH coeffs for RGB colors. (Nc = 3, 8 or 15)\n",
+    "\n",
+    "`f_rest` will not be present for the `.splat` format files. `opacity`, `f_dc` and `rot` (quaternions) are also quantized to 8 bit to reduce storage space in this format."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ply_3dgs = o3d.t.io.read_point_cloud('mipnerf360_garden_crop_table.ply')\n",
+    "print(ply_3dgs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "splat_3dgs = o3d.t.io.read_point_cloud('mipnerf360_garden_crop_table.splat')\n",
+    "print(splat_3dgs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can convert `.splat` files to `.ply` and vice versa."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Compress .ply format 3DGS by saving to .splat format\n",
+    "o3d.t.io.write_point_cloud('mipnerf360_garden_crop_table_from_ply.splat', ply_3dgs)\n",
+    "# Convert .splat format 3DGS to .ply format for wider compatibility\n",
+    "o3d.t.io.write_point_cloud('mipnerf360_garden_crop_table_from_splat.ply', splat_3dgs)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualizing Gaussian Splats\n",
+    "\n",
+    "We can visualize Gaussian Splats similar to Point Clouds. Full 3DGS visualization support is in progress and at the moment, we can only visualize the colors in the Gaussian centers as colored points. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Visualize splat format 3DGS as a point cloud (no view-dependent effects)\n",
+    "draw(splat_3dgs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Visualize full ply format 3DGS as a point cloud.\n",
+    "draw(ply_3dgs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/cpp/CMakeLists.txt

@@ -77,6 +77,7 @@ endif()
 
 if (BUILD_GUI)
     open3d_add_example(Draw)
+    open3d_add_example(GaussianSplat)
     open3d_add_example(MultipleWindows       Open3D::3rdparty_threads)
     open3d_add_example(OffscreenRendering)
     open3d_add_example(TICPSequential        Open3D::3rdparty_threads)

examples/cpp/GaussianSplat.cpp

@@ -0,0 +1,41 @@
+// ----------------------------------------------------------------------------
+// -                        Open3D: www.open3d.org                            -
+// ----------------------------------------------------------------------------
+// Copyright (c) 2018-2024 www.open3d.org
+// SPDX-License-Identifier: MIT
+// ----------------------------------------------------------------------------
+
+// To run this example, download some sample Gaussian Splats. You can use these
+// samples to get started:
+// clang-format off
+// curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.ply
+// curl -O https://github.com/isl-org/open3d_downloads/releases/download/3dgs-1/mipnerf360_garden_crop_table.splat
+// clang-format on
+
+#include <cstdlib>
+
+#include "open3d/Open3D.h"
+
+using namespace open3d;
+
+void PrintUsage() {
+    utility::LogInfo("Visualize Gaussian Splat from PLY or SPLAT file.");
+    utility::LogInfo("Usage: GaussianSplat <filename.[ply|splat]>");
+}
+
+int main(int argc, char **argv) {
+    utility::SetVerbosityLevel(utility::VerbosityLevel::Debug);
+    if (argc != 2 ||
+        utility::ProgramOptionExistsAny(argc, argv, {"-h", "--help"})) {
+        PrintUsage();
+        return 1;
+    }
+    std::shared_ptr<t::geometry::PointCloud> gsplat =
+            std::make_shared<t::geometry::PointCloud>();
+    if (!t::io::ReadPointCloud(argv[1], *gsplat)) {
+        utility::LogWarning("Failed to read file {}", argv[1]);
+        return 1;
+    }
+    visualization::Draw({visualization::DrawObject(argv[1], gsplat)},
+                        "Gaussian Splat");
+}