MediaPipe to openFrameworks

Notes on how to connect Google's MediaPipe ML Framework to openFrameworks

MediaPipe is a cross-platform framework for building multimodal applied machine learning pipelines. I want to be able to use it in external applications.

This tutorial walks through how to stream MediaPipe data out over UDP, so any external app and receive and use the data.

I show how to modify the mediapipe example mediapipe/examples/desktop/hand_tracking to add in a new node that recieves hand tracking data as input, broadcasts that data over UDP on port 8080, and then passes the tracking data on to the rest of the graph as output.

Tested on macOS Mojave (10.14.6) and openFrameworks 0.10.1

---

Beigin by installing MediaPipe on your system using google's instructions.

Then install and setup Google Protobufs for openFrameworks using my previous tutorial.

If you've never used Bazel before, the build system and organization of Mediapipe can be really confusing. I try to go through step-by-step below, but you can find more information in the MediaPipe Docs.

---

We're going to modify MediaPipe's desktop hand tracking example to stream out landmarks and bounding rectangles over UDP.

1. Update the Graph Definition to Include our new PassThrough Calculator

Modify mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

# Add New Node
node {
  calculator: "MyPassThroughCalculator"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "NORM_RECT:hand_rect"
  input_stream: "DETECTIONS:palm_detections"
  output_stream: "LANDMARKS:hand_landmarks_out"
  output_stream: "NORM_RECT:hand_rect_out"
  output_stream: "DETECTIONS:palm_detections_out"
}

# Modify input_stream names of next node

# Subgraph that renders annotations and overlays them on top of the input
# images (see renderer_cpu.pbtxt).
node {
  calculator: "RendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "LANDMARKS:hand_landmarks_out"
  input_stream: "NORM_RECT:hand_rect_out"
  input_stream: "DETECTIONS:palm_detections_out"
  output_stream: "IMAGE:output_video"
}

NOTE You can visualize the graph to test that inputs and outpus match up at https://viz.mediapipe.dev/

When you add in the custom MyPassThroughCalculator your graph should look like this:

2. Making a Custom Calculator

Adding UDP, Detections, Landmarks, and Hand Rectangles to the PassThrough Calculator

1. Copy the file src/mediapipe/my_pass_though_calculator.cc to your Mediapipe calculators directory mediapipe/calculators/core.

2. The main differences between my_pass_though_calculator.cc and the original pass_though_calculator.cc are that it adds UDP streaming, and uses Landmark, Rect, and Detection protobufs in the ::mediapipe::Status Process() function. Next we need to modify the graph file to declare the Tag of the calculators input and stream.

3. Modifying Calculators BUILD file

1. Add the following in mediapipe/calculators/core/BUILD to include my_pass_through_calculator dependencies:

cc_library(
    name = "my_pass_through_calculator",
    srcs = ["my_pass_through_calculator.cc"],
    visibility = [
        "//visibility:public",
    ],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework/formats:landmark_cc_proto",
        "//mediapipe/framework/formats:rect_cc_proto",
        "//mediapipe/framework/formats:detection_cc_proto",
        "//mediapipe/framework/formats:wrapper_hand_tracking_cc_proto",
    ],
    alwayslink = 1,
)

4. Modify the Graphs BUILD file

1. Lastly, you need to modify the mediapipe/graphs/hand_tracking/BUILD file to add the new calculator as a dependency:

cc_library(
    name = "desktop_offline_calculators",
    deps = [
        "//mediapipe/calculators/core:flow_limiter_calculator",
        "//mediapipe/calculators/core:gate_calculator",
        "//mediapipe/calculators/core:immediate_mux_calculator",
        "//mediapipe/calculators/core:packet_inner_join_calculator",
        "//mediapipe/calculators/core:previous_loopback_calculator",
        "//mediapipe/calculators/video:opencv_video_decoder_calculator",
        "//mediapipe/calculators/video:opencv_video_encoder_calculator",
        "//mediapipe/calculators/core:my_pass_through_calculator"
    ],
)

5. Add the wrapper proto

1. Copy the wrapper_hand_tracker.proto in this repo's /mediapipe directory into mediapipe/framework/formats directory.

2. Add it and its dependencies to mediapipe/framework/formats/BUILD:

mediapipe_proto_library(
    name = "wrapper_hand_tracking_proto",
    srcs = ["wrapper_hand_tracking.proto"],
    visibility = ["//visibility:public"],
    deps = [
      "//mediapipe/framework/formats:landmark_proto",
      "//mediapipe/framework/formats:detection_proto",
      "//mediapipe/framework/formats:rect_proto",
    ],
)

You should be able to build and run the mediapipe hand_tracking_desktop_live example now without any errors. In your mediapipe root directory, run:

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1     mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu     --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

Sending Multiple Protobuf Messages over UDP

There's no good way to send multiple messages (like a LandmarkList, Rect, and DetectionList) in proto2. Therefore, you have to wrap those messages into a new protobuf.

I called mine wrapper_hand_tracking.proto and added it and its dependencies to mediapipe/framework/formats/BUILD.

Here's a roadblock I hit:

• MediaPipe uses protobuf-3.11.4 and is built with Bazel, but I'm using protobuf-3.6.1 for openFrameworks, and it builds with CMake.
• So if I link the wrapper_hand_tracking.proto generated with Bazel to the openFrameworks project, there are compatibility issues with the protobuf static library I've already built and linked.

So the hacky workaround I found was to build wrapper_hand_tracking.proto with the MediaPipe build system, and then rebuild it with my system-wide protobuf installation. Here's what I did ... in the top-level /mediapipe directory:

1. First build with the .proto with bazel.
   • Most of wrapper_hand_tracking.proto file is commented out, just leaving the import calls to link to dependent protos.
2. Second, uncomment the .proto and rebuild with protoc.
   • For openFrameworks, I've altread built a separate static protobuf library for openFrameworks using my previous tutorial. I need to generate the .pb.h and .pb.cc files from wrapper_hand_tracking.proto using this library.
   • In wrapper_hand_tracking.proto, comment out the import calls at the top of the file, and uncomment the body of the proto (protoc doesn't link dependencies, so I just copied all the necessary protobufs into this one file).
   • Go into the mediapipes\framework\formats directory and run protoc --cpp_out=. wrapper_hand_tracking.proto
   • Move those wrapper_hand_tracking.pb.h and wrapper_hand_tracking.pb.cc files over to your openFrameworks src folder.
   • Drag and drop these two files into your openFrameworks project (be sure to check "Add to Project").
   • There shouldn't be any errors now when you add #include "wrapper_hand_tracking.pb.h" to ofApp.h

Running the Example

When you run the MediaPipe example hand_tracking_desktop_live, it broadcasts any hand landmarks and rectangles on port localhost:8080. The openFrameworks example example-protobuf-udp is listening for those protobufs on port 8080.

Run the MediaPipe Example

1. From your MediaPipe root directory, run hand_tracking_desktop_live:

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu     --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

You should see a video feed of yourself, with hand landmarks and bounding rectangle overlaid.

Run the openFrameworks Example

1. Build example-protobuf-udp in openFramework's Project Generator

2. Drag and drop the /libs directory into the Xcode Project Pane and select Add to Target.

3. Drag and drop your newly generated wrapper_hand_tracking.pb.cc and wrapper_hand_tracking.pb.h files into the /src directory in the Project Pane and select Add to Target.

4. Link the libprotobuf.a static library from /libs/protobuf/lib/osx in Project Settings > General > Linked Frameworks and Libraries.

5. In Project Settings > Build Settings, add the following to Header Search Paths:

$(PROJECT_DIR)/libs/protobuf/include
$(PROJECT_DIR)/libs/protobuf/include/google
$(PROJECT_DIR)/libs/protobuf/include/google/compiler
$(PROJECT_DIR)/libs/protobuf/include/google/compiler/cpp
$(PROJECT_DIR)/libs/protobuf/include/google/io
$(PROJECT_DIR)/libs/protobuf/include/google/stubs
$(PROJECT_DIR)/libs/protobuf/include/google/util

6. Hit ⌘-r to Run.

You should see the numbered landmarks and bounding rectangle on a white screen.

Press 'SPACE' to use your hand to swat around some particles.

Note: The example runs on the cpu, so it's a little slow. But the framerate improves a bit once a hand is detected.