Pose Skeleton Preview without Camera Image underneath

[saurabhchalke]

Hi, I am trying to preview only the skeleton without a background instead of the default skeleton overlay over the Camera image.
I have tried removing the input_stream: "IMAGE_GPU:input_image" from the pose_renderer_gpu.pbtxt file, but still seem to get the camera image output.

Can you please suggest the necessary modifications here?

My pose_renderer_gpu.pbtxt file:
`# MediaPipe pose landmarks rendering subgraph.

type: "PoseRendererGpu"

GPU image. (GpuBuffer)

input_stream: "IMAGE:input_image"

Pose landmarks. (NormalizedLandmarkList)

input_stream: "LANDMARKS:pose_landmarks"

Region of interest calculated based on landmarks. (NormalizedRect)

input_stream: "ROI:roi"

Detected pose. (Detection)

input_stream: "DETECTION:detection"

GPU image with rendered data. (GpuBuffer)

output_stream: "IMAGE:output_image"

node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_image"
output_stream: "SIZE:image_size"
}

Calculates rendering scale based on the pose roi.

node {
calculator: "RectToRenderScaleCalculator"
input_stream: "NORM_RECT:roi"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "RENDER_SCALE:render_scale"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderScaleCalculatorOptions] {
multiplier: 0.0012
}
}
}

Converts detections to drawing primitives for annotation overlay.

node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTION:detection"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}

node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "pose_landmarks"
output_stream: "visible_pose_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 25 }
}
}
}

Converts landmarks to drawing primitives for annotation overlay.

node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:pose_landmarks"
input_stream: "RENDER_SCALE:render_scale"
output_stream: "RENDER_DATA:landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 7
landmark_connections: 0
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 11
landmark_connections: 12
landmark_connections: 11
landmark_connections: 13
landmark_connections: 13
landmark_connections: 15
landmark_connections: 15
landmark_connections: 17
landmark_connections: 15
landmark_connections: 19
landmark_connections: 15
landmark_connections: 21
landmark_connections: 17
landmark_connections: 19
landmark_connections: 12
landmark_connections: 14
landmark_connections: 14
landmark_connections: 16
landmark_connections: 16
landmark_connections: 18
landmark_connections: 16
landmark_connections: 20
landmark_connections: 16
landmark_connections: 22
landmark_connections: 18
landmark_connections: 20
landmark_connections: 11
landmark_connections: 23
landmark_connections: 12
landmark_connections: 24
landmark_connections: 23
landmark_connections: 24
landmark_connections: 23
landmark_connections: 25
landmark_connections: 24
landmark_connections: 26
landmark_connections: 25
landmark_connections: 27
landmark_connections: 26
landmark_connections: 28
landmark_connections: 27
landmark_connections: 29
landmark_connections: 28
landmark_connections: 30
landmark_connections: 29
landmark_connections: 31
landmark_connections: 30
landmark_connections: 32
landmark_connections: 27
landmark_connections: 31
landmark_connections: 28
landmark_connections: 32

  landmark_color { r: 255 g: 255 b: 255 }
  connection_color { r: 255 g: 255 b: 255 }
  thickness: 3.0
  visualize_landmark_depth: false
  utilize_visibility: true
  visibility_threshold: 0.5
}

}
}

Take left pose landmarks.

node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "pose_landmarks"
output_stream: "landmarks_left_side"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 1 end: 4 }
ranges: { begin: 7 end: 8 }
ranges: { begin: 9 end: 10 }
ranges: { begin: 11 end: 12 }
ranges: { begin: 13 end: 14 }
ranges: { begin: 15 end: 16 }
ranges: { begin: 17 end: 18 }
ranges: { begin: 19 end: 20 }
ranges: { begin: 21 end: 22 }
ranges: { begin: 23 end: 24 }

  combine_outputs: true
}

}
}

Take right pose landmarks.

node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "pose_landmarks"
output_stream: "landmarks_right_side"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 4 end: 7 }
ranges: { begin: 8 end: 9 }
ranges: { begin: 10 end: 11 }
ranges: { begin: 12 end: 13 }
ranges: { begin: 14 end: 15 }
ranges: { begin: 16 end: 17 }
ranges: { begin: 18 end: 19 }
ranges: { begin: 20 end: 21 }
ranges: { begin: 22 end: 23 }
ranges: { begin: 24 end: 25 }

  combine_outputs: true
}

}
}

Render pose joints as big white circles.

node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:visible_pose_landmarks"
input_stream: "RENDER_SCALE:render_scale"
output_stream: "RENDER_DATA:landmarks_background_joints_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_color { r: 255 g: 255 b: 255 }
connection_color { r: 255 g: 255 b: 255 }
thickness: 5.0
visualize_landmark_depth: false
utilize_visibility: true
visibility_threshold: 0.5
}
}
}

Render pose left side joints as orange circles (inside white ones).

node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:landmarks_left_side"
input_stream: "RENDER_SCALE:render_scale"
output_stream: "RENDER_DATA:landmarks_left_joints_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_color { r: 255 g: 138 b: 0 }
connection_color { r: 255 g: 138 b: 0 }
thickness: 3.0
visualize_landmark_depth: false
utilize_visibility: true
visibility_threshold: 0.5
}
}
}

Render pose right side joints as cyan circles (inside white ones).

node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:landmarks_right_side"
input_stream: "RENDER_SCALE:render_scale"
output_stream: "RENDER_DATA:landmarks_right_joints_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_color { r: 0 g: 217 b: 231 }
connection_color { r: 0 g: 217 b: 231 }
thickness: 3.0
visualize_landmark_depth: false
utilize_visibility: true
visibility_threshold: 0.5
}
}
}

Converts normalized rects to drawing primitives for annotation overlay.

node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:roi"
output_stream: "RENDER_DATA:roi_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}

Draws annotations and overlays them on top of the input images.

node {
calculator: "AnnotationOverlayCalculator"
input_stream: "detection_render_data"
input_stream: "landmarks_render_data"
input_stream: "landmarks_background_joints_render_data"
input_stream: "landmarks_left_joints_render_data"
input_stream: "landmarks_right_joints_render_data"
input_stream: "roi_render_data"
output_stream: "IMAGE_GPU:output_image"
}
`

[sgowroji]

@saurabhchalke Can you help me with what platform are you referring. Thanks

[saurabhchalke]

Hi @sgowroji, I am using Pose Tracking on the Android platform, and running it on GPU.
I am using the default graphs present in the mediapipe/graphs/pose_tracking.

I tried updating the pose_renderer_gpu.pbtxt subgraph to remove the input_stream: "IMAGE_GPU:input_image" in the AnnotationOverlayCalculator step at the end(line 266), but still see the image preview returned along with the pose skeleton render in the output image stream.

My current understanding is that tweaking the output stream (eg. removing the image preview, changing the colors of pose joints/skeletons etc.) can be accomplished by updating the calculator nodes in the pose_renderer_gpu.pbtxt file. Please let me know if understanding is wrong here?

[jiuqiant]

@gkarpiak, please take a look at this pose renderer gpu question. Thanks.

[gkarpiak]

@saurabhchalke the idea of overlay is that you overlay over some image which input_stream: "IMAGE_GPU:input_image" in the AnnotationOverlayCalculator, so what you need to is to provide there, not the camera image, but some image which is a solid color.

You can do it in multiple ways: send it to the graph as another input or maybe loading from the assets, or maybe writing your own calculator that generates such an image based on dimensions of the original camera image.

[saurabhchalke]

Thanks for the suggestion @gkarpiak, but I am trying to eliminate any background whatsoever (not even a solid color). Is that possible with the current Calculators?
I am just trying to return the pose skeleton with a transparent background.

[gkarpiak]

@saurabhchalke have you tried to pass a transparent input into input_stream: "IMAGE_GPU:input_image"? (I mentioned multiple ways you can try to do that in my previous reply.)

Also, if you need just skeleton - you may consider returning skeleton landmarks and render them in any way you need.

[saurabhchalke]

Thanks for your inputs @gkarpiak. I have one last question, I have eliminated the image_frame from the final GpuBuffer output, and trying to superimpose the Skeleton over my Android view, but the output has a black background with the Skelton instead.

Would converting the GpuBuffer to ImageFrame solve the issue here? If so are there any way using any existing calculators that can be used to produce output which would have transparent backgrounds, and can directly be used over Android screens.

Thanks for your patience.

[sgowroji]

Hi @saurabhchalke, Did you try the workaround suggested in the above comment. Please update the status. Thanks!

[sgowroji]

Hi @saurabhchalke, Could you please respond to my above comment. Thanks!

[sgowroji]

We are closing this ticket as of now due to lack of activity.