Add PerceptionFeatures library and implement ArucoDetector class #159

prashanthr05 · 2020-11-29T14:43:17Z

This PR implements a PerceptionFeatures (Perception -> Features (Image features) open to renaming suggestions) library and a ArcuoDetector class (requires OpenCV as dependency).

The ArucoDetector class is implemented as an Advanceable gives as an output a container of ArucoMarkerData which more or less has the same structure as a ContactBase class (Landmark features from images seen by a camera can be seen analogous to contacts made by the robot).

Currently only implements individual markers detection.

cc @HosameldinMohamed @gabrielenava

S-Dafarra · 2021-01-05T16:10:31Z

There seem to be two issues in the CI:

a memory leak in the tests in Ubuntu
it cannot find the file opencv2/aruco.hpp on Windows

Do you have any idea? Does the memory leak occurs also in your system?

traversaro · 2021-01-05T17:29:19Z

it cannot find the file opencv2/aruco.hpp on Windows

Probably we need to install opencv4[contrib] in vcpkg, not just opencv4 .

traversaro · 2021-01-05T17:33:31Z

a memory leak in the tests in Ubuntu

It seems that the valgrind error is not a memory leak, but rather a use of not initialized memory in the opencv 3.2 version available in Ubuntu. Probably we can just add that error to a valgrind suppression file, or add one if we don't have it (see robotology/idyntree#352 for an example of how to add it).

traversaro · 2021-01-05T17:47:06Z

a memory leak in the tests in Ubuntu

It seems that the valgrind error is not a memory leak, but rather a use of not initialized memory in the opencv 3.2 version available in Ubuntu. Probably we can just add that error to a valgrind suppression file, or add one if we don't have it (see robotology/idyntree#352 for an example of how to add it).

We already have it in https://github.com/dic-iit/bipedal-locomotion-framework/blob/e43cc1827fa1e2062f117fc37f1bf986cf6b5fc3/cmake/valgrind-linux.supp , so we just need to add the appropriate suppressions there.

prashanthr05 · 2021-03-12T14:44:19Z

it cannot find the file opencv2/aruco.hpp on Windows

Probably we need to install opencv4[contrib] in vcpkg, not just opencv4 .

How do we choose that in the CI? because I noticed in vcpkg opencv port that it is not part of the default features.

Will this work?

./vcpkg.exe install opencv[contrib]:x64-windows

UPDATE Seems like the windows CI passes? The only failing CI is Ubuntu for the valgrind memory leak for which we need to suppress the valgrind check.

traversaro · 2021-03-12T14:55:43Z

it cannot find the file opencv2/aruco.hpp on Windows

Probably we need to install opencv4[contrib] in vcpkg, not just opencv4 .

How do we choose that in the CI? because I noticed in vcpkg opencv port that it is not part of the default features.

Will this work?
./vcpkg.exe install opencv[contrib]:x64-windows

That will probably work, but it takes a lot of time, and we need to propagate that eventually to https://github.com/robotology/robotology-vcpkg-ports . As it seems to be that conda-forge's OpenCV has instead aruco, can we just add the install of opencv in the conda-forge based CI, and disable this component in the vcpkg-based Windows build?

prashanthr05 · 2021-03-12T14:58:34Z

Note that in the C++ CI windows and conda-forge CI, the ArucoDetectorTest is not being called. Probably, it's not being compiled due a missing FRAMEWORK_COMPILE_XXX flag in the CMakeLists or the missing opencv dependency.

traversaro · 2021-03-12T14:58:40Z

it cannot find the file opencv2/aruco.hpp on Windows

Probably we need to install opencv4[contrib] in vcpkg, not just opencv4 .

How do we choose that in the CI? because I noticed in vcpkg opencv port that it is not part of the default features.
Will this work?
./vcpkg.exe install opencv[contrib]:x64-windows
That will probably work, but it takes a lot of time, and we need to propagate that eventually to https://github.com/robotology/robotology-vcpkg-ports . As it seems to be that conda-forge's OpenCV has instead aruco, can we just add the install of opencv in the conda-forge based CI, and disable this component in the vcpkg-based Windows build?

Damn, librealsense on conda-forge on the other hand does not support Windows (see conda-forge/librealsense-feedstock#3).

Probably it is just easier to not test this stuff on Windows for the time being.

prashanthr05 · 2021-03-12T15:27:47Z

a memory leak in the tests in Ubuntu

It seems that the valgrind error is not a memory leak, but rather a use of not initialized memory in the opencv 3.2 version available in Ubuntu. Probably we can just add that error to a valgrind suppression file, or add one if we don't have it (see robotology/idyntree#352 for an example of how to add it).

We already have it in https://github.com/dic-iit/bipedal-locomotion-framework/blob/e43cc1827fa1e2062f117fc37f1bf986cf6b5fc3/cmake/valgrind-linux.supp , so we just need to add the appropriate suppressions there.

I skimmed through https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto to understand better the changes.

Added the suppression in f8c02f3

src/Perception/include/BipedalLocomotion/Perception/Features/ArucoDetector.h

src/Perception/src/ArucoDetector.cpp

prashanthr05 · 2021-03-23T10:49:36Z

CI status related to ArucoDetectorTest,

does not run in conda-forge workflow (windows, macos, ubuntu 20.04). This is because I have not added the dependency installation of opencv in conda-forge-ci.yml.
failing in regular workflow on ubuntu 20.04 (valgrind suppression rules seems not to be working)
does not run in regular workflow windows and macOS

traversaro · 2021-03-23T10:58:39Z

The CI failure is:

==26059== 512 bytes in 1 blocks are possibly lost in loss record 3,445 of 3,529
==26059==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26059==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==26059==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==26059==    by 0x79A0322: allocate_stack (allocatestack.c:622)
==26059==    by 0x79A0322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==26059==    by 0x79F5750: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x79FE279: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x4C04FBB: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x7A01544: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x7A0180E: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x79FEB67: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x4C05795: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x4C057FE: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x79FBE65: tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)

while the suppression is:

{
   <ubuntu-20.04-opencv-tbb>
   Memcheck:Leak
   fun:malloc
   fun:_ZN3tbb8internal23allocate_via_handler_v3Em
}

As I don't see any tbb::internal::allocate_via_handler_v3(unsigned long) (demangling of _ZN3tbb8internal23allocate_via_handler_v3Em) in the failure, probably we are missing a suppression? For me we can also skip directly the valgrind for this test and add it later if it is faster.

prashanthr05 · 2021-03-23T11:13:26Z

The CI failure is:

==26059== 512 bytes in 1 blocks are possibly lost in loss record 3,445 of 3,529
==26059==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26059==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==26059==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==26059==    by 0x79A0322: allocate_stack (allocatestack.c:622)
==26059==    by 0x79A0322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==26059==    by 0x79F5750: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x79FE279: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x4C04FBB: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x7A01544: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x7A0180E: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x79FEB67: ??? (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)
==26059==    by 0x4C05795: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x4C057FE: ??? (in /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0)
==26059==    by 0x79FBE65: tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const (in /usr/lib/x86_64-linux-gnu/libtbb.so.2)

while the suppression is:

{
   <ubuntu-20.04-opencv-tbb>
   Memcheck:Leak
   fun:malloc
   fun:_ZN3tbb8internal23allocate_via_handler_v3Em
}

As I don't see any tbb::internal::allocate_via_handler_v3(unsigned long) (demangling of _ZN3tbb8internal23allocate_via_handler_v3Em) in the failure, probably we are missing a suppression? For me we can also skip directly the valgrind for this test and add it later if it is faster.

I added the failure related suppression in 4924ce2 and it didn't seem to work

{
   <ubuntu-20.04-opencv-aruco>
   Memcheck:Leak
   match-leak-kinds: possible
   fun:calloc
   fun:allocate_dtv
   fun:_dl_allocate_tls
   fun:allocate_stack
   fun:pthread_create@@GLIBC_2.2.5
   fun:_ZN2cv12WorkerThreadC1ERNS_10ThreadPoolEj
   fun:_ZN2cv10ThreadPool12reconfigure_Ej
   fun:_ZN2cv21parallel_for_pthreadsERKNS_5RangeERKNS_16ParallelLoopBodyEd
   fun:_ZN2cv13parallel_for_ERKNS_5RangeERKNS_16ParallelLoopBodyEd
   fun:_ZN2cv3hal8opt_AVX212cvtBGRtoGrayEPKhmPhmiiiib
   fun:_ZN2cv3hal12cvtBGRtoGrayEPKhmPhmiiiib
   fun:_ZN2cv16cvtColorBGR2GrayERKNS_11_InputArrayERKNS_12_OutputArrayEb
   fun:_ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii
   fun:_ZN2cv5arucoL14_convertToGreyERKNS_11_InputArrayERKNS_12_OutputArrayE
}

I don't understand how ??? should be handled :(

traversaro · 2021-03-23T11:15:41Z

I added the failure related suppression in 4924ce2 and it didn't seem to work

How are you generating this suppression files?

traversaro · 2021-03-23T11:16:35Z

I added the failure related suppression in 4924ce2 and it didn't seem to work

How are you generating this suppression files?

Are you using https://wiki.wxwidgets.org/Valgrind_Suppression_File_Howto#How_to_make_a_suppression_file ? Probably we can add --gen-suppressions=all to all our tests so that it is easier to suppress when spurious failures appear.

prashanthr05 · 2021-03-23T11:21:03Z

How are you generating this suppression files?

Compiled in Release

$ valgrind --leak-check=full --gen-suppressions=all -s ./build/bin/ArucoDetectorTestUnitTests 
==17955== Memcheck, a memory error detector
==17955== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==17955== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==17955== Command: ./build/bin/ArucoDetectorTestUnitTests
==17955== 
===============================================================================
All tests passed (6 assertions in 1 test case)

==17955== 
==17955== HEAP SUMMARY:
==17955==     in use at exit: 7,048 bytes in 52 blocks
==17955==   total heap usage: 15,300 allocs, 15,248 frees, 23,776,868 bytes allocated
==17955== 
==17955== 3,344 bytes in 11 blocks are possibly lost in loss record 22 of 22
==17955==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==17955==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==17955==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==17955==    by 0x8571322: allocate_stack (allocatestack.c:622)
==17955==    by 0x8571322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==17955==    by 0x53D667C: cv::WorkerThread::WorkerThread(cv::ThreadPool&, unsigned int) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_core.so.4.2.0)
==17955==    by 0x53D6F8F: cv::ThreadPool::reconfigure_(unsigned int) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_core.so.4.2.0)
==17955==    by 0x53D7C5F: cv::parallel_for_pthreads(cv::Range const&, cv::ParallelLoopBody const&, double) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_core.so.4.2.0)
==17955==    by 0x53D4623: cv::parallel_for_(cv::Range const&, cv::ParallelLoopBody const&, double) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_core.so.4.2.0)
==17955==    by 0x6FC48E3: cv::hal::opt_AVX2::cvtBGRtoGray(unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, int, int, bool) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_imgproc.so.4.2.0)
==17955==    by 0x6C4B931: cv::hal::cvtBGRtoGray(unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, int, int, bool) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_imgproc.so.4.2.0)
==17955==    by 0x6C4DC6F: cv::cvtColorBGR2Gray(cv::_InputArray const&, cv::_OutputArray const&, bool) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_imgproc.so.4.2.0)
==17955==    by 0x6C10F87: cv::cvtColor(cv::_InputArray const&, cv::_OutputArray const&, int, int) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_imgproc.so.4.2.0)
==17955==    by 0x67AD389: cv::aruco::_convertToGrey(cv::_InputArray const&, cv::_OutputArray const&) (in /home/pramadoss/iit_ws/scrapyard/opencv/install/lib/libopencv_aruco.so.4.2.0)
==17955== 
{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: possible
   fun:calloc
   fun:allocate_dtv
   fun:_dl_allocate_tls
   fun:allocate_stack
   fun:pthread_create@@GLIBC_2.2.5
   fun:_ZN2cv12WorkerThreadC1ERNS_10ThreadPoolEj
   fun:_ZN2cv10ThreadPool12reconfigure_Ej
   fun:_ZN2cv21parallel_for_pthreadsERKNS_5RangeERKNS_16ParallelLoopBodyEd
   fun:_ZN2cv13parallel_for_ERKNS_5RangeERKNS_16ParallelLoopBodyEd
   fun:_ZN2cv3hal8opt_AVX212cvtBGRtoGrayEPKhmPhmiiiib
   fun:_ZN2cv3hal12cvtBGRtoGrayEPKhmPhmiiiib
   fun:_ZN2cv16cvtColorBGR2GrayERKNS_11_InputArrayERKNS_12_OutputArrayEb
   fun:_ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii
   fun:_ZN2cv5arucoL14_convertToGreyERKNS_11_InputArrayERKNS_12_OutputArrayE
}
==17955== LEAK SUMMARY:
==17955==    definitely lost: 0 bytes in 0 blocks
==17955==    indirectly lost: 0 bytes in 0 blocks
==17955==      possibly lost: 3,344 bytes in 11 blocks
==17955==    still reachable: 3,704 bytes in 41 blocks
==17955==         suppressed: 0 bytes in 0 blocks
==17955== Reachable blocks (those to which a pointer was found) are not shown.
==17955== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==17955== 
==17955== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

traversaro · 2021-03-23T11:23:09Z

How are you generating this suppression files?

Compiled in Release

This failure is different from the CI one. I would quickly add that command to CI to get the suppression corresponding to CI.

prashanthr05 · 2021-03-23T12:26:25Z

How are you generating this suppression files?

Compiled in Release

This failure is different from the CI one. I would quickly add that command to CI to get the suppression corresponding to CI.

Solid advice (I will remember this). Thanks @traversaro. Looks like it worked.

CI status related to ArucoDetectorTest,

does not run in conda-forge workflow (windows, macos, ubuntu 20.04). This is because I have not added the dependency installation of opencv in conda-forge-ci.yml.

failing in regular workflow on ubuntu 20.04 (valgrind suppression rules seems not to be working)

does not run in regular workflow windows and macOS

Ubuntu regular CI passed running both non-valgrind and valgrind tests.
macOS CI passed running non-valgrind tests (valgrind disabled by default).
Should we worry about the windows CI versions not attempting to run the tests (windows TESTING seems to be ON in ci.yml)?

traversaro · 2021-03-23T13:33:42Z

Should we worry about the windows CI versions not attempting to run the tests (windows TESTING seems to be ON in ci.yml)?

As far as I remember we disable Windows compilation of this component due to the fact that by default opencv in vcpkg does not have the contrib component with the aruco stuff?

prashanthr05 · 2021-03-23T13:41:44Z

Should we worry about the windows CI versions not attempting to run the tests (windows TESTING seems to be ON in ci.yml)?

As far as I remember we disable Windows compilation of this component due to the fact that by default opencv in vcpkg does not have the contrib component with the aruco stuff?

Ah okay, yeah. So it is automatically taken care of by framework_dependent_option cmake, I suppose. Understood.

prashanthr05 · 2021-03-23T17:33:33Z

@GiulioRomualdi could you take a look at this PR and merge it if you are ok with the changes?

GiulioRomualdi

Just a minor comment

src/Perception/src/ArucoDetector.cpp

prashanthr05 requested review from GiulioRomualdi and S-Dafarra November 29, 2020 14:43

prashanthr05 self-assigned this Nov 29, 2020

prashanthr05 force-pushed the perception/aruco branch from 0f66887 to 24dc1ff Compare November 29, 2020 14:50

prashanthr05 force-pushed the perception/aruco branch from 5662cbd to 8fda15e Compare March 12, 2021 14:29

prashanthr05 changed the title ~~Add Perception pipeline and implement ArucoDetector class~~ Add PerceptionFeatures library and implement ArucoDetector class Mar 12, 2021

S-Dafarra reviewed Mar 15, 2021

View reviewed changes

prashanthr05 force-pushed the perception/aruco branch from a6d0a96 to daa3310 Compare March 23, 2021 10:01

S-Dafarra approved these changes Mar 23, 2021

View reviewed changes

prashanthr05 force-pushed the perception/aruco branch from c8c700f to 07263b4 Compare March 23, 2021 17:32

prashanthr05 added 4 commits March 26, 2021 16:42

Add ArucoDetector class to PerceptionFeatures library

93b5780

[Perception] Add ArucoDetectorTest and bugfix ArucoDetector

8229baf

Update README and CHANGELOG

69ed7ab

[Perception] remove support for AprilTag to fix ci compilation

71b4d6b

prashanthr05 force-pushed the perception/aruco branch from 07263b4 to d67154e Compare March 26, 2021 15:43

prashanthr05 added 3 commits March 26, 2021 16:46

Add valgrind suppression opencv ubuntu20.04

c34bf0a

[cmake] add gen-suppressions to unit test valgrind command

482ceed

[Perception] avoid memcpy in ArucoDetector and fix docs

c552e59

prashanthr05 force-pushed the perception/aruco branch from d67154e to c552e59 Compare March 26, 2021 15:47

GiulioRomualdi reviewed Mar 26, 2021

View reviewed changes

src/Perception/src/ArucoDetector.cpp Outdated Show resolved Hide resolved

prashanthr05 and others added 2 commits March 26, 2021 17:06

[Perception][ArucoDetector] fix debug print

81b92b5

Merge branch 'master' into perception/aruco

c6f21c6

GiulioRomualdi merged commit 3ed2ec2 into ami-iit:master Mar 26, 2021

prashanthr05 deleted the perception/aruco branch March 29, 2021 07:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PerceptionFeatures library and implement ArucoDetector class #159

Add PerceptionFeatures library and implement ArucoDetector class #159

prashanthr05 commented Nov 29, 2020 •

edited

Loading

S-Dafarra commented Jan 5, 2021

traversaro commented Jan 5, 2021

traversaro commented Jan 5, 2021

traversaro commented Jan 5, 2021

prashanthr05 commented Mar 12, 2021 •

edited

Loading

traversaro commented Mar 12, 2021

prashanthr05 commented Mar 12, 2021 •

edited

Loading

traversaro commented Mar 12, 2021 •

edited

Loading

prashanthr05 commented Mar 12, 2021

prashanthr05 commented Mar 23, 2021

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 •

edited

Loading

traversaro commented Mar 23, 2021

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 •

edited

Loading

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 •

edited

Loading

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021

GiulioRomualdi left a comment

Add PerceptionFeatures library and implement ArucoDetector class #159

Add PerceptionFeatures library and implement ArucoDetector class #159

Conversation

prashanthr05 commented Nov 29, 2020 • edited Loading

S-Dafarra commented Jan 5, 2021

traversaro commented Jan 5, 2021

traversaro commented Jan 5, 2021

traversaro commented Jan 5, 2021

prashanthr05 commented Mar 12, 2021 • edited Loading

traversaro commented Mar 12, 2021

prashanthr05 commented Mar 12, 2021 • edited Loading

traversaro commented Mar 12, 2021 • edited Loading

prashanthr05 commented Mar 12, 2021

prashanthr05 commented Mar 23, 2021

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 • edited Loading

traversaro commented Mar 23, 2021

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 • edited Loading

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021 • edited Loading

traversaro commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021

prashanthr05 commented Mar 23, 2021

GiulioRomualdi left a comment

Choose a reason for hiding this comment

prashanthr05 commented Nov 29, 2020 •

edited

Loading

prashanthr05 commented Mar 12, 2021 •

edited

Loading

prashanthr05 commented Mar 12, 2021 •

edited

Loading

traversaro commented Mar 12, 2021 •

edited

Loading

prashanthr05 commented Mar 23, 2021 •

edited

Loading

prashanthr05 commented Mar 23, 2021 •

edited

Loading

prashanthr05 commented Mar 23, 2021 •

edited

Loading