From 5a8ac729d46c042372b701256cb7df69b5f77df6 Mon Sep 17 00:00:00 2001 From: "Masson, Lola" Date: Tue, 23 Jul 2024 13:51:48 +0000 Subject: [PATCH] Document the available buffer ids and image types --- ChangeLog.md | 2 + doc/sphinx/content/manip_images/buffer_ids.md | 173 ++++++++++++++++++ doc/sphinx/content/manip_images/index.md | 8 + .../content/{ => manip_images}/stlimage.md | 28 --- doc/sphinx/index.md | 2 +- 5 files changed, 184 insertions(+), 29 deletions(-) create mode 100644 doc/sphinx/content/manip_images/buffer_ids.md create mode 100644 doc/sphinx/content/manip_images/index.md rename doc/sphinx/content/{ => manip_images}/stlimage.md (65%) diff --git a/ChangeLog.md b/ChangeLog.md index 1e15dcf4..bbfade7a 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -5,6 +5,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added +- The documentation about the different available `buffer_id` has been updated to provide the complete list for each compatible device. ## 1.6.0 - 2024-06-10 ### Changed diff --git a/doc/sphinx/content/manip_images/buffer_ids.md b/doc/sphinx/content/manip_images/buffer_ids.md new file mode 100644 index 00000000..383d05d3 --- /dev/null +++ b/doc/sphinx/content/manip_images/buffer_ids.md @@ -0,0 +1,173 @@ +# Buffer types + +`ifm3d` supports multiple device families (O3R, O3D3xx, O3X1xx). + +Each device natively provides different types of data, and the `ifm3d` library adds additional buffers for convenience. + +The tables below list all available buffer types and which device they apply to. +We also specify if the data is provided directly by the device, meaning it is streamed from the embedded device, or calculated by `ifm3d` API on the receiver device. +This is important information to be considered when trying to work around bandwidth limitations and delays. + +In this tables, the NA abbreviation stands for "Not Applicable." + +## O3R Available buffers + +| `buffer_id` | Description | Native or calculated in ifm3d | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | +| `ALGO_DEBUG` | Proprietary data used for troubleshooting by ifm. For internal use only. | Native | +| `CONFIDENCE_IMAGE` | 2D image containing information about each pixel status. Only available for the 3D ports. | Native | +| `EXTRINSIC_CALIB` | Current values of the optical system to user coordinate transformation for the port currently targeted by the `FrameGrabber`. Only available for the 3D port.
Can also be retrieved with the `TOF_INFO` buffer. | Calculated | +| `INTRINSIC_CALIB` | The intrinsic parameters for the port or camera currently targeted by the`FrameGrabber`. Only available for the 3D port.
Can also be retrieved with the `TOF_INFO` buffer. | Native | +| `INVERSE_INTRINSIC_CALIBRATION` | The inverse intrinsic parameters for the port or camera currently targeted by the`FrameGrabber`.
Only available for the 3D port.
Can also be retrieved with the `TOF_INFO` | Native | +| `JPEG_IMAGE` | The 2D RGB image. | Native | +| `NORM_AMPLITUDE_IMAGE` | The normalized amplitude image. | Native | +| `O3R_ODS_INFO` | The zones output of ODS. Only available for devices with an ODS license. | Native | +| `O3R_ODS_OCCUPANCY_GRID` | The occupancy grid output of ODS. Only available for devices with an ODS license. | Native | +| `O3R_RESULT_ARRAY2D` | Contains the result for O3R applications like PDS and MCC. | Native | +| `O3R_RESULT_IMU` | The IMU data for the OVP on-board IMU at port 6. | Native | +| `O3R_RESULT_JSON` | Contains the result for O3R applications like PDS and MCC. | Native | +| `RADIAL_DISTANCE_IMAGE` | 2D image containing the radial distance value for each pixel. | Native | +| `RADIAL_DISTANCE_NOISE` | 2D image containing the distance noise value for each pixel. | Native | +| `REFLECTIVITY` | 2D image containing the estimated IR reflectivity (in percent). Only available for the 3D ports. | Native | +| `RGB_INFO` | Contains various information about the RGB camera and the latest RGB frame (see[`RGBInfoV1`](https://api.ifm3d.com/stable/_autosummary/ifm3dpy.deserialize.RGBInfoV1.html)) | Native | +| `TOF_INFO` | Contains various information about the TOF camera and the latest TOF frame (see[`TOFInfoV4`](https://api.ifm3d.com/stable/_autosummary/ifm3dpy.deserialize.TOFInfoV4.html)) | Native | +| `XYZ` | The point cloud in Cartesian coordinates. | Calculated | + + +## O3D Available buffers + +| `buffer_id` | Description | Native or calculated in ifm3d | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | +| `AMPLITUDE_IMAGE` | 2D image containing amplitude information for each pixel. | Native | +| `CARTESIAN_X_COMPONENT` | Cartesian coordinate along X axis. | Calculated | +| `CARTESIAN_Y_COMPONENT` | Cartesian coordinate along Y axis. | Calculated | +| `CARTESIAN_Z_COMPONENT` | Cartesian coordinate along Z axis. | Calculated | +| `CONFIDENCE_IMAGE` | 2D image containing information about each pixel status. | Native | +| `DIAGNOSTIC` | Provides some diagnostic information. See [details below](#diagnostic-buffer). | Native | +| `EXPOSURE_TIME` | Exposure times used for the last frame. | Calculated | +| `EXTRINSIC_CALIB` | Current values of the optical system to user coordinate transformation for the port currently targeted by the `FrameGrabber`. | Calculated | +| `INTRINSIC_CALIB` | The intrinsic parameters for the port or camera currently targeted by the`FrameGrabber`. | Native | +| `INVERSE_INTRINSIC_CALIBRATION` | The inverse intrinsic parameters for the port or camera currently targeted by the`FrameGrabber`. | Native | +| `JSON_DIAGNOSTIC` | Diagnostic information in JSON format. | Native | +| `JSON_MODEL` | For internal use only. | ?? | +| `NORM_AMPLITUDE_IMAGE` | The normalized amplitude image. | Native | +| `RADIAL_DISTANCE_IMAGE` | 2D image containing the radial distance value for each pixel. | Native | +| `UNIT_VECTOR_ALL` | Unit vectors that can be used to calculate the point cloud directly from the distance image. | Native | +| `XYZ` | The point cloud in Cartesian coordinates. | Calculated | + +:::{note} +The `CARTESIAN_ALL` buffer is deprecated, and the `XYZ` buffer should be used instead. +::: + + +### `DIAGNOSTIC` buffer + +This buffer provides diagnostic information such as temperature, frame duration and framerate. + +To unpack the information in the `DIAGNOSTIC` buffer, use the following structure: + + +| Content | Type | Size | +| ------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------- | +| Illumination temperature | 32-bit signed int | 4 bytes | +| Frontend Temperature 1 | 32-bit signed int | 4 bytes | +| Frontend Temperature 2 | 32-bit signed int | 4 bytes | +| i.mx6 Temperature | 32-bit signed int | 4 bytes | +| Frame duration: time between the last trigger and the readiness for a new trigger. Only relevant in triggered mode. | 32-bit unsigned integer | 4 bytes | +| Framerate. Only relevant in continuous mode (also called free running mode). | 32-bit unsigned integer | 4 bytes | + +:::{note} +All the temperatures are displayed in 0.1 °C. +Invalid temperatures have the value 0x7FFF (32767). +Some temperatures are only measured for specific part numbers, which might be why they are invalid. +::: + +In Python, the `DIAGNOSTIC` data can be unpacked with: + +```python +# Necessary imports +import struct +import time +from ifm3dpy.device import O3D +from ifm3dpy.framegrabber import FrameGrabber, buffer_id + +# Create the objects +o3d = O3D() #EDIT here when using a non-default IP address +fg = FrameGrabber(o3d) + +#Collect a frame +fg.start([buffer_id.DIAGNOSTIC,]) +time.sleep(5) # Grace period after initialization of the data stream +[ok, frame] = fg.wait_for_frame().wait_for(500) + +# Get the data from the collected frame +data = frame.get_buffer(buffer_id.DIAGNOSTIC) +# Unpack the data +unpacked_data = struct.unpack('<4i2I', data) + +fg.stop() +``` + +## O3X Available buffers + +| `buffer_id` | Description | Native or calculated in ifm3d | +| ----------------------- | ------------------------------------------------------------- | ----------------------------- | +| `CARTESIAN_X_COMPONENT` | Cartesian coordinate along X axis. | Calculated | +| `CARTESIAN_Y_COMPONENT` | Cartesian coordinate along Y axis. | Calculated | +| `CARTESIAN_Z_COMPONENT` | Cartesian coordinate along Z axis. | Calculated | +| `CONFIDENCE_IMAGE` | 2D image containing information about each pixel status. | Native | +| `NORM_AMPLITUDE_IMAGE` | The normalized amplitude image. | Native | +| `RADIAL_DISTANCE_IMAGE` | 2D image containing the radial distance value for each pixel. | Native | +| `RADIAL_DISTANCE_NOISE` | 2D image containing the distance noise value for each pixel. | Native | +| `XYZ` | The point cloud in Cartesian coordinates. | Calculated | + +:::{note} +Even though the `GRAYSCALE_IMAGE` buffer is shown to be available in the O3X JSON configuration, no acquisition mode currently provides this buffer, so we have not included it in the list. +::: + +## Data formats + +`ifm3d` supports multiple devices. These devices support multiple buffers and data types which can have different data formats on different devices. + +We recommend to retrieve the format of a specific `buffer_id` programmatically. +One can use the code below (adapt to the specific `buffer_id`): +:::::{tabs} +::::{group-tab} C++ + +```cpp +ifm3d::Buffer xyz = frame->GetBuffer(ifm3d::buffer_id::XYZ); + +// Query the data format and channel. +std::cout << xyz.nchannels() << std::endl; +std::cout << static_cast(xyz.dataFormat()) << std::endl; +``` + +:::: +::::{group-tab} Python + +```python +print(type(frame.get_buffer(buffer_id.XYZ))) +print(np.shape(frame.get_buffer(buffer_id.XYZ))) +print(frame.get_buffer(buffer_id.RADIAL_DISTANCE_NOISE).dtype) +``` + +:::: +::::: + +In C++, one has to refer to the pixel format correspondence defined below: + +``` +FORMAT_8U = 0, +FORMAT_8S = 1, +FORMAT_16U = 2, +FORMAT_16S = 3, +FORMAT_32U = 4, +FORMAT_32S = 5, +FORMAT_32F = 6, +FORMAT_64U = 7, +FORMAT_64F = 8, +FORMAT_16U2 = 9, +FORMAT_32F3 = 10 +``` + +For instance, the format returned for `ifm3d::buffer_id::XYZ` is `2`, which corresponds to `FORMAT_32F` (32 bit float). diff --git a/doc/sphinx/content/manip_images/index.md b/doc/sphinx/content/manip_images/index.md new file mode 100644 index 00000000..c6f4184c --- /dev/null +++ b/doc/sphinx/content/manip_images/index.md @@ -0,0 +1,8 @@ +# Images and buffers manipulation + +:::{toctree} +:titlesonly: +Buffer types +Accessing pixel values +::: + diff --git a/doc/sphinx/content/stlimage.md b/doc/sphinx/content/manip_images/stlimage.md similarity index 65% rename from doc/sphinx/content/stlimage.md rename to doc/sphinx/content/manip_images/stlimage.md index d0d0b8ad..f007984e 100644 --- a/doc/sphinx/content/stlimage.md +++ b/doc/sphinx/content/manip_images/stlimage.md @@ -231,31 +231,3 @@ int main(int argc, const char** argv) } ``` -## Device, Schema and data types for ifm3d::Buffer - -```ifm3d``` supports multiple devices. These devices support multiple buffers/data types which can have different data formats on different devices. -The following table summarizes the buffers formats, along with their [ifm3d::buffer_id](../../../modules/framegrabber/include/ifm3d/fg#L22), so a ```std::set``` needs to be pass in ```FrameGrabber::Start``` method to enable the corresponding buffer for grabbing. - - -| Data/Image | ifm3d schema | O3D3XX | O3X | O3R | -|-----------------------------------------|-------------------------------------------------|--------------------|---------------|--------------------| -| Radial distance image | ifm3d::buffer_id::RADIAL_DISTANCE | std::uint16_t | float | float | -| Amplitude image | ifm3d::buffer_id::AMPLITUDE | std::uint16_t | float | float | -| Raw amplitude image | ifm3d::buffer_id::RAW_AMPLITUDE | std::uint16_t | float | NA | -| Cartesian coordinate | ifm3d::buffer_id::XYZ | std::int16_t | float | float | -| Unit vector | ifm3d::buffer_id::UNIT_VECTOR_ALL | float | float | NA | -| Image grayscale | ifm3d::buffer_id::GRAY | std::uint16_t | float | NA | -| Distance Noise Image | ifm3d::buffer_id::DISTANCE_NOISE | NA | std::uint16_t | std::uint16_t | -| Confidence Image | ifm3d::buffer_id::CONFIDENCE | std::uint8_t | std::uint8_t | std::unit16_t | -| JPEG | ifm3d::buffer_id::IMG_JPEG | NA | NA | std::unit8_t | - -Along with above data ifm3d also supports [ifm3d::buffer_id](../../../modules/framegrabber/include/ifm3d/fg#L22) values, which is used to obtain data that are obtained in standard STL containers or C++ default types. - -| Data/Image | ifm3d schema | O3D3XX | O3X | O3R | -|-----------------------------------------|-------------------------------------------------|----------------------------------|----------------------------------|----------------------------------| -| JSON Model Data | ifm3d::buffer_id::JSON_MODEL | std::string | NA | NA | -| JSON Model Data | ifm3d::buffer_id::JSON_MODEL | std::string | NA | NA | -| Intrinsic calibration parameter | ifm3d::buffer_id::INTRINSIC_CALIBRATION | std::vector | NA | std::vector | -| inverse Intrinsic calibration parameter | ifm3d::buffer_id::INVERSE_INTRINSIC_CALIBRATION | std::vector | NA | std::vector | -| Illumination temperature | ifm3d::buffer_id::ILLUMINATION_TEMP | float | NA | NA | -| Exposure time | ifm3d::buffer_id::EXPOSURE_TIME | std::vector\ | std::vector\ | std::vector\ | diff --git a/doc/sphinx/index.md b/doc/sphinx/index.md index b19a91d2..4d4ec6e1 100644 --- a/doc/sphinx/index.md +++ b/doc/sphinx/index.md @@ -9,7 +9,7 @@ Installation instructions Migration Guide Basic library usage Command Line tool -Manipulating images +Manipulating images Python API C++ API Examples