Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Objectron: add blueprint info + minor cleaning #5810

Merged
merged 1 commit into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 24 additions & 9 deletions examples/python/objectron/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<!--[metadata]
title = "Objectron"
tags = ["2D", "3D", "Object detection", "Pinhole camera"]
tags = ["2D", "3D", "Object detection", "Pinhole camera", "Blueprint"]
description = "Example of using the Rerun SDK to log the Google Research Objectron dataset."
thumbnail = "https://static.rerun.io/objectron/b645ef3c8eff33fbeaefa6d37e0f9711be15b202/480w.png"
thumbnail_dimensions = [480, 480]
Expand All @@ -20,20 +20,20 @@ build_args = ["--frames=150"]

Visualize the [Google Research Objectron](https://github.com/google-research-datasets/Objectron) dataset including camera poses, sparse point-clouds and surfaces characterization.

# Used Rerun types
## Used Rerun types
[`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d), [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), [`Image`](https://ref.rerun.io/docs/python/0.14.1/common/image_helpers/#rerun.ImageEncoded)<sup>*</sup>, [`Transform3D`](https://www.rerun.io/docs/reference/types/archetypes/transform3d), [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole)

# Background
## Background

This example visualizes the Objectron database, a rich collection of object-centric video clips accompanied by AR session metadata.
With high-resolution images, object pose, camera pose, point-cloud, and surface plane information available for each sample, the visualization offers a comprehensive view of the object from various angles.
Additionally, the dataset provides manually annotated 3D bounding boxes, enabling precise object localization and orientation.

# Logging and visualizing with Rerun
## Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

## Timelines
### Timelines

For each processed frame, all data sent to Rerun is associated with the two [`timelines`](https://www.rerun.io/docs/concepts/timelines) `time` and `frame_idx`.

Expand All @@ -42,7 +42,7 @@ rr.set_time_sequence("frame", sample.index)
rr.set_time_seconds("time", sample.timestamp)
```

## Video
### Video

Pinhole camera is utilized for achieving a 3D view and camera perspective through the use of the [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole) and [`Transform3D`](https://www.rerun.io/docs/reference/types/archetypes/transform3d) archetypes.

Expand All @@ -68,15 +68,15 @@ The input video is logged as a sequence of [`ImageEncoded`](https://ref.rerun.io
rr.log("world/camera", rr.ImageEncoded(path=sample.image_path))
```

## Sparse point clouds
### Sparse point clouds

Sparse point clouds from `ARFrame` are logged as [`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d) archetype to the `world/points` entity.

```python
rr.log("world/points", rr.Points3D(positions, colors=[255, 255, 255, 255]))
```

## Annotated bounding boxes
### Annotated bounding boxes

Bounding boxes annotated from `ARFrame` are logged as [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), containing details such as object position, sizes, center and rotation.

Expand All @@ -94,7 +94,22 @@ rr.log(
)
```

# Run the code
### Setting up the default blueprint

The default blueprint is configured with the following code:

```python
blueprint = rrb.Horizontal(
rrb.Spatial3DView(origin="/world", name="World"),
rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["/world/**"]),
)
```

In particular, we want to reproject the points and the 3D annotation box in the 2D camera view corresponding to the pinhole logged at `"/world/camera"`. This is achieved by setting the view's contents to the entire `"/world/**"` subtree, which include both the pinhole transform and the image data, as well as the point cloud and the 3D annotation box.



## Run the code
To run this example, make sure you have Python version at least 3.9, the Rerun repository checked out and the latest SDK installed:
```bash
# Setup
Expand Down
9 changes: 5 additions & 4 deletions examples/python/objectron/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,13 +215,14 @@ def main() -> None:
rr.script_add_args(parser)
args = parser.parse_args()

blueprint = rrb.Horizontal(
rrb.Spatial3DView(origin="/world", name="World"),
rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["/world/**"]),
)
rr.script_setup(
args,
"rerun_example_objectron",
default_blueprint=rrb.Horizontal(
rrb.Spatial3DView(origin="/world", name="World"),
rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["+ $origin/**", "+ /world/**"]),
),
default_blueprint=blueprint,
)

dir = ensure_recording_available(args.recording, args.dataset_dir, args.force_reprocess_video)
Expand Down
Loading