Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Objectron]How to get estimated coordinates #1647

Closed
Nekokojpn opened this issue Feb 23, 2021 · 4 comments
Closed

[Objectron]How to get estimated coordinates #1647

Nekokojpn opened this issue Feb 23, 2021 · 4 comments
Assignees
@ahmadyan
Labels
legacy:objectron Objectron related issues platform:android Issues with Android as Platform

Comments

@Nekokojpn
Copy link

Hello. I'm sorry for not well English.
I want to make an app to try virtual shoe using Objectron and ARCore.
First, I think need estimated coordinates on Java.
for that purpose, I tried add output stream named "coordinates" using PassThroughCalclator.
Do I need to do anything special to get the coordinates in Java?
Here is code I added to mediapipe/example/android/src/com/google/mediapipe/apps/objectdetection3d/MainActiviy.java: onCreate()

processor.addPacketCallback("coordinates",
        new PacketCallback() {
            @Override
            public void process(Packet packet) {
                
              List<Sequence> list = PacketGetter.getProtoVector(packet, Sequence.parser());
              for(Sequence seq : list) {
                
                java.util.List<FrameAnnotation> annotations = seq.getFrameAnnotationsList();
                
                for(FrameAnnotation frame : annotations) {
                    List<ObjectAnnotation> objs = frame.getAnnotationsList();
                    for(ObjectAnnotation obj : objs) {
                        List<AnnotatedKeyPoint> keypoints = obj.getKeypointsList();
                        for(AnnotatedKeyPoint keypoint : keypoints) {
                            Point3D p = keypoint.getPoint3D();
                            Log.e(TAG, "x: " + p.getX() + "y: " + p.getY() + "z: " + p.getZ());
                        }
                    }
                }
            }
            }
        });

But, when I run it on Android, the app will crash.
Here is log:

02-23 16:38:51.761: E/native(19094): E20210223 16:38:51.761590 19160 filter_detection_calculator.cc:261] Pass detection with score 0.580293
02-23 16:38:51.761: E/native(19094): E20210223 16:38:51.761891 19160 filter_detection_calculator.cc:261] Pass detection with score 0.513783
02-23 16:38:51.800: A/libc(19094): Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 19164 (Thread-8), pid 19094 (etection3d_shoe)
02-23 16:38:51.883: A/DEBUG(19176): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
02-23 16:38:51.883: A/DEBUG(19176): Build fingerprint: 'google/sunfish/sunfish:11/RQ1A.210105.002/6985033:user/release-keys'
02-23 16:38:51.883: A/DEBUG(19176): Revision: 'MP1.0'
02-23 16:38:51.883: A/DEBUG(19176): ABI: 'arm64'
02-23 16:38:51.884: A/DEBUG(19176): Timestamp: 2021-02-23 16:38:51+0900
02-23 16:38:51.884: A/DEBUG(19176): pid: 19094, tid: 19164, name: Thread-8 >>> com.google.mediapipe.apps.objectdetection3d_shoe <<<
02-23 16:38:51.884: A/DEBUG(19176): uid: 10356
02-23 16:38:51.884: A/DEBUG(19176): signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
02-23 16:38:51.884: A/DEBUG(19176): x0 0000000000000000 x1 0000000000004adc x2 0000000000000006 x3 000000793b3db320
02-23 16:38:51.884: A/DEBUG(19176): x4 747265766e6f6320 x5 747265766e6f6320 x6 747265766e6f6320 x7 206f7420656c6269
02-23 16:38:51.884: A/DEBUG(19176): x8 00000000000000f0 x9 c5550b29897a077d x10 0000000000000000 x11 ffffffc0fffffbdf
02-23 16:38:51.884: A/DEBUG(19176): x12 0000000000000001 x13 76206f7420656c62 x14 0000007c485dbd22 x15 00000000fffffffe
02-23 16:38:51.884: A/DEBUG(19176): x16 0000007c485d8c80 x17 0000007c485ba870 x18 eb9df7111f86a5ca x19 0000000000004a96
02-23 16:38:51.884: A/DEBUG(19176): x20 0000000000004adc x21 00000000ffffffff x22 000000794da6eafa x23 000000793b3dd000
02-23 16:38:51.884: A/DEBUG(19176): x24 0000007c4a3f6bf0 x25 000000793b3dd000 x26 000000793b3dd000 x27 000000000000002e
02-23 16:38:51.884: A/DEBUG(19176): x28 0000000000000006 x29 000000793b3db3a0
02-23 16:38:51.884: A/DEBUG(19176): lr 0000007c4856e2a0 sp 000000793b3db300 pc 0000007c4856e2cc pst 0000000000000000
02-23 16:38:51.904: E/native(19094): E20210223 16:38:51.904356 19163 filter_detection_calculator.cc:261] Pass detection with score 0.569673
02-23 16:38:51.905: E/native(19094): E20210223 16:38:51.904963 19163 filter_detection_calculator.cc:261] Pass detection with score 0.514522
02-23 16:38:52.029: E/native(19094): E20210223 16:38:52.028869 19162 filter_detection_calculator.cc:261] Pass detection with score 0.508491
02-23 16:38:52.038: A/DEBUG(19176): backtrace:
02-23 16:38:52.038: A/DEBUG(19176): #00 pc 000000000004e2cc /apex/com.android.runtime/lib64/bionic/libc.so (abort+164) (BuildId: 95f68a34fdf7b31f42d3f811d5d88f58)
02-23 16:38:52.038: A/DEBUG(19176): #1 pc 0000000000c15d30 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (google::logging_fail()+8)
02-23 16:38:52.038: A/DEBUG(19176): #2 pc 0000000000c14a08 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (google::LogMessage::SendToLog()+908)
02-23 16:38:52.038: A/DEBUG(19176): #3 pc 0000000000c14fb8 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (google::LogMessage::Flush()+220)
02-23 16:38:52.038: A/DEBUG(19176): #4 pc 0000000000c196e8 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (google::LogMessageFatal::~LogMessageFatal()+16)
02-23 16:38:52.038: A/DEBUG(19176): #5 pc 0000000000a816ac /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::internal_statusor::Helper::Crash(absl::lts_2020_09_23::Status const&)+76)
02-23 16:38:52.038: A/DEBUG(19176): #6 pc 00000000009a2b6c /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (Java_com_google_mediapipe_framework_PacketGetter_nativeGetProtoVector+584)
02-23 16:38:52.038: A/DEBUG(19176): #7 pc 0000000000044b88 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/oat/arm64/base.odex (art_jni_trampoline+152)
02-23 16:38:52.038: A/DEBUG(19176): #8 pc 00000000001337e8 /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+568) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #9 pc 00000000001a8a94 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+228) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #10 pc 000000000031830c /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+376) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #11 pc 000000000030e638 /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+996) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #12 pc 000000000067e070 /apex/com.android.art/lib64/libart.so (MterpInvokeStatic+548) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #13 pc 000000000012d994 /apex/com.android.art/lib64/libart.so (mterp_op_invoke_static+20) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.038: A/DEBUG(19176): #14 pc 0000000000245a86 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/oat/arm64/base.vdex (com.google.mediapipe.framework.PacketGetter.getProtoVector+8)
02-23 16:38:52.039: A/DEBUG(19176): #15 pc 0000000000305c34 /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.4023174618862212076)+268) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #16 pc 000000000030dc24 /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+200) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #17 pc 000000000030e61c /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+968) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #18 pc 000000000067e070 /apex/com.android.art/lib64/libart.so (MterpInvokeStatic+548) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #19 pc 000000000012d994 /apex/com.android.art/lib64/libart.so (mterp_op_invoke_static+20) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #20 pc 00000000002bdd5e /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/oat/arm64/base.vdex (com.google.mediapipe.apps.objectdetection3d.MainActivity$1.process+12)
02-23 16:38:52.039: A/DEBUG(19176): #21 pc 0000000000305c34 /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.4023174618862212076)+268) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #22 pc 0000000000669e20 /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+780) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #23 pc 000000000013cff8 /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #24 pc 0000000000133564 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #25 pc 00000000001a8a78 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #26 pc 0000000000553bfc /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithVarArgsart::ArtMethod*(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, std::__va_list)+468) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #27 pc 0000000000553d9c /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+92) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #28 pc 00000000003ab850 /apex/com.android.art/lib64/libart.so (art::JNI::CallVoidMethodV(_JNIEnv*, _jobject*, _jmethodID*, std::__va_list)+660) (BuildId: c820164b0c64ea40f1abc53aaf588b44)
02-23 16:38:52.039: A/DEBUG(19176): #29 pc 000000000099220c /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (_JNIEnv::CallVoidMethod(_jobject*, _jmethodID*, ...)+116)
02-23 16:38:52.039: A/DEBUG(19176): #30 pc 0000000000991fb4 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::android::Graph::CallbackToJava(_JNIEnv*, _jobject*, mediapipe::Packet const&)+760)
02-23 16:38:52.039: A/DEBUG(19176): #31 pc 00000000009b8ba0 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::tool::CallbackCalculator::Process(mediapipe::CalculatorContext*)+108)
02-23 16:38:52.039: A/DEBUG(19176): #32 pc 00000000009faa70 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::CalculatorNode::ProcessNode(mediapipe::CalculatorContext*)+1056)
02-23 16:38:52.039: A/DEBUG(19176): #33 pc 00000000009e1920 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunCalculatorNode(mediapipe::CalculatorNode*, mediapipe::CalculatorContext*)+312)
02-23 16:38:52.039: A/DEBUG(19176): #34 pc 00000000009e1374 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunNextTask()+256)
02-23 16:38:52.039: A/DEBUG(19176): #35 pc 0000000000a17c54 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::RunWorker()+392)
02-23 16:38:52.039: A/DEBUG(19176): #36 pc 0000000000a17884 /data/app/~~D9nigWbiK1AiczhahPVeoA==/com.google.mediapipe.apps.objectdetection3d_shoe-PKU4bqx1hBrNtBe2bTKLCw==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::WorkerThread::ThreadBody(void*)+1616)
02-23 16:38:52.039: A/DEBUG(19176): #37 pc 00000000000afd4c /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64) (BuildId: 95f68a34fdf7b31f42d3f811d5d88f58)
02-23 16:38:52.039: A/DEBUG(19176): #38 pc 0000000000050288 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 95f68a34fdf7b31f42d3f811d5d88f58)
02-23 16:38:52.168: E/native(19094): E20210223 16:38:52.168073 19163 filter_detection_calculator.cc:261] Pass detection with score 0.501851
02-23 16:38:52.294: E/native(19094): E20210223 16:38:52.294701 19163 filter_detection_calculator.cc:261] Pass detection with score 0.501268

Does anyone know about this?
Thank you.
@sgowroji sgowroji self-assigned this Feb 23, 2021
@sgowroji sgowroji added the platform:android Issues with Android as Platform label Feb 23, 2021
@sgowroji sgowroji added the legacy:objectron Objectron related issues label Mar 3, 2021
@sgowroji sgowroji assigned jiuqiant and unassigned sgowroji Mar 4, 2021
@jiuqiant jiuqiant assigned jianingwei and unassigned jiuqiant Mar 4, 2021
@jianingwei jianingwei assigned ahmadyan and unassigned jianingwei Mar 9, 2021
@ahmadyan
Copy link

ahmadyan commented Mar 9, 2021

Your debug dump is not very informative, so I can't guess what is going wrong here. It looks like you are pinging filter_detection_calculator.cc, which is for the 2D crop and not what you are looking for. Maybe 2D detector is not detecting any shoe, so the 3D detector is not even firing up.

What you need is the lifted_objects packet. You can define a new output stream for the graph, and read that packet from Java side. If your goal is implement a try-on, it would be much easier to do the rendering by the Mediapipe in the graph as well. For starter, you can use the GlAnimationOverlayCalculator and pass the shoe asset (instead of a 3D bounding box) to the renderer:

  input_side_packet: "TEXTURE:obj_texture"
  input_side_packet: "ANIMATION_ASSET:obj_asset_name"

@sgowroji sgowroji added the stat:awaiting response Waiting for user response label Mar 9, 2021
@sgowroji
Copy link

sgowroji commented Mar 9, 2021

Hi @Nekokojpn , Please go through the above comment and share the status.

@sgowroji
Copy link

Hi @Nekokojpn, Assuming you got resolved and lack of activity we are closing this issue!

@sgowroji sgowroji removed the stat:awaiting response Waiting for user response label Mar 16, 2021
@betrisey
Copy link

What you need is the lifted_objects packet. You can define a new output stream for the graph, and read that packet from Java side.

Hello @ahmadyan,
I'd like to get the coordinates of the detected objects from Java, so I think that I need to use the lifted_objects packet you suggested but I don't know how to parse the protobuf packet.
I haven't found which parser to use or if there exists one in Java yet.
Do you have any suggestions or know where to look please?

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahmadyan ahmadyan

Labels
legacy:objectron Objectron related issues platform:android Issues with Android as Platform
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ahmadyan @betrisey @jiuqiant @Nekokojpn @sgowroji @jianingwei