polling multi_face_landmarks packet from the face_mesh crashes the code

[vamsee-asu2019]

Hi,
I am trying to access the landmarks from the face_mesh as shown in the comments of #1739.
But now when I go out of frame or there are no facial landmarks detected the code crashes. I tried to put all the code block in try and catch, even that didn`t work.
Can anyone help me with this?

[sgowroji]

Hi @vamsee-asu2019, Could you share the crash log here. And also mention what all did you try ?

[vamsee-asu2019]

Hi @sgowroji, there is no crash log . But the visualization freezes when there are no facial landmarks detected. All the change I made was to access the multi_face_landmarks output from face_mesh using a different poller.

[sgowroji]

@vamsee-asu2019 Can you share the graph to understand the issue much better. Thanks!

[vamsee-asu2019]

@sgowroji I am using the same graph from the pbtxt file here face_mesh_desktop_live_gpu.pbtxt

And I am trying to access the multi_face_landmarks as follows in the demo_run_graph_main.cc

ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller poller_detection,
                  graph.AddOutputStreamPoller("multi_face_landmarks"));

Inside the while loop for accessing each frame of videocapture

mediapipe::Packet detection_packet;
if (poller_detection.Next(&detection_packet))
       {
         //break;
        auto &output_landmarks = detection_packet.Get<std::vector<::mediapipe::NormalizedLandmarkList>>();
        if(!output_landmarks.empty()){
        ::mediapipe::NormalizedLandmarkList landmarks_1 = output_landmarks[0];
     }

So when there are no landmarks from the detection packet, it crashes. I tried to put the entire code inside a try-catch block, but even then it crashes. It would be great if you could answer :

Is this behaviour expected when accessing the multi_face_landmarks?
What can be done to overcome this behaviour?

[vamsee-asu2019]

Hello @ivan-grishchenko,
any updates on this ? I am still facing the issue.

[vamsee-asu2019]

Is anyone looking into this?

[vamsee-asu2019]

@ivan-grishchenko
I have narrowed down the issue. The problem is in the graph_output_stream.cc at https://github.com/google/mediapipe/blob/a92cff7a60031f5c3097b06e74416732d85b5011/mediapipe/framework/graph_output_stream.cc#L144 .
Seems that the lock is not being released. I have debugged it by using the print statements before and after this line. Also it only crashes when I look at extreme angles.

[vamsee-asu2019]

@jiuqiant Can you help me with this? I am stuck with this issue for more than 10 days now.

[vamsee-asu2019]

I have changed the https://github.com/google/mediapipe/blob/a92cff7a60031f5c3097b06e74416732d85b5011/mediapipe/framework/graph_output_stream.cc#L144 to handler_condvar_.WaitWithTimeout(&mutex_,duration); . But now , it crahes in Get method of the poller that fetches the packet. I tried adding a try catch around that Get() method , even then it is crashing saysing that the packet is empty . Error log:

F20210325 17:52:27.421635 65368 packet.h:768] Packet::Get() failed: Expected a Packet of type: class std::vector<class mediapipe::NormalizedLandmarkList,class std::allocator<class mediapipe::NormalizedLandmarkList> >, but received an empty Packet. *** Check failure stack trace: *** @ 00007FF6FD48EBBB public: void __cdecl google::LogMessage::Flush(void) __ptr64 @ 00007FF6FD48D959 public: __cdecl google::LogMessageFatal::~LogMessageFatal(void) __ptr64 @ 00007FF6FD158C8B public: void __cdecl google::LogMessage::LogStream::set_ctr(int) __ptr64 @ 00007FF6FD4A7489 bool __cdecl google::Demangle(char const * __ptr64,char * __ptr64,int) @ 00007FF6FD4A8DC8 bool __cdecl google::Demangle(char const * __ptr64,char * __ptr64,int) @ 00007FF6FD4A3EDC bool __cdecl google::Demangle(char const * __ptr64,char * __ptr64,int) @ 00007FFF17A77C24 BaseThreadInitThunk @ 00007FFF183ED721 RtlUserThreadStart

[vamsee-asu2019]

Hi @ivan-grishchenko , @sgowroji and @jiuqiant ,
I was able to solve this issue by adding hanging the following line https://github.com/google/mediapipe/blob/a92cff7a60031f5c3097b06e74416732d85b5011/mediapipe/framework/graph_output_stream.cc#L144 to handler_condvar_.WaitWithTimeout(&mutex_,duration); and adding a count variable inside the while loop : https://github.com/google/mediapipe/blob/a92cff7a60031f5c3097b06e74416732d85b5011/mediapipe/framework/graph_output_stream.cc#L138 , with a condition that if (count > 3){ mutex_.Unlock(); return false; }

And then adding a try-catch when we call the .Next() method to get the multi_face_landmarks packet.

[sgowroji]

We are glad to see you resolved. And thank you so much for sharing your code, which will be useful for other users.

[Hrituraj202]

I am facing the same issue. The console shows no error but screen just freezes giving no response at all ... Please help me

[sgowroji]

Hi @Hrituraj202, Did this resolution #1747 (comment) helped ?. If not can you share complete details in a new issue. Thanks!