GSOC17 - Facemark API #1257

kurnianggoro · 2017-06-27T13:58:54Z

Merge with extra: opencv/opencv_extra#365

Project: API for Facial Landmark Detector
Mentor: Delia Passalacqua

This pull request contains the implementation of Facemark API as well as two facemark algorithm, AAM and LBF.

video: https://www.youtube.com/watch?v=B7WGyhl2zm8
report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc

Currently, there are several interfaces provided in the Facemark base class including:

load training data
set custom face detector
face detection
training the model
saving a trained model
loading a trained model
fitting

For the AAM algorithm, all of its training functionalities are already coded.
Several related functionalities are listed as follows:

Procrustes analysis
PCA projection
customized Delaunay triangulation
warp image
Gram-Schmidth orthonormalization
feature extraction

sergboec · 2017-07-14T08:56:49Z

@kurnianggoro Do you have plans to add implementation of algorithm from dlib ?
edit: sorry, i suppose this is the answer #1199

alalek · 2017-09-28T10:06:47Z

@kurnianggoro What is the license of images from opencv_extra patch?

kurnianggoro · 2017-09-28T13:23:06Z

hi @alalek , I just found them on google, will it be problem?
also, now this pull request produce error on the building bot,
the error is from test stereo, seems like I should fetch the latest update...

alalek · 2017-09-28T14:04:12Z

test_stereo has sporadic failures - build restarted.

All images should have known proper usage rights. It is better to reuse existed OpenCV images, or use Google filtering (but anyway double check is required).

Initial structure of the facemark API and AAM header

kurnianggoro · 2017-10-02T08:27:48Z

@alalek hi, i have changed the file and provide the source links (the license is available,you can check), they are from wikipedia, so i hope it will not be problem

vpisarev · 2017-10-02T14:36:33Z

@kurnianggoro, let's not put faces of public people (or not public, it does not matter) to OpenCV. lena is the exception, 'cause it's considered a standard test image.

alalek

Thank you for contribution!

Please take a look on comments below.

alalek · 2017-10-02T14:54:14Z

modules/face/src/facemarkLBF.cpp

+
+    FacemarkLBF::Params::Params(){
+
+        cascade_face = "../data/haarcascade_frontalface_alt.xml";


Hardcoded paths should be avoided in sources.
Required parameters should be passed explicitly.

alalek · 2017-10-02T14:54:36Z

modules/face/src/facemarkLBF.cpp

+        tree_n=6;
+        tree_depth=5;
+        bagging_overlap = 0.4;
+        model_filename = "LBF.model";


this is just the default filename for the saving file, is it also problem? if it were for reading file like the cascade face then for sure it will be problem

This is minor problem (at least there is no relative path to subdirectory).

alalek · 2017-10-02T15:04:38Z

modules/face/src/facemarkLBF.cpp

+        }
+
+        FILE *fd = fopen(s.c_str(), "rb");
+        regressor.read(fd, params);


OpenCV modules should not read/write files directly, especially via old "C-API" (there are a lot of missing error checks at least; there is no in-memory storage support; cross-platform issues).
Why FileStorage doesn't work here?

ok, i will try to change to filestorage.

Well, there is similar PR #1199 with same problem. Probably we should keep this as is if there is some problem with cv::FileStorage.
I believe it is fine for the first version. So consider this problem as minor too.

alalek · 2017-10-02T15:06:00Z

modules/face/src/facemarkLBF.cpp

+        regressor.trainRegressor(imgs, gt_shapes, current_shapes, bboxes, mean_shape, 0, params);
+
+        FILE *fd = fopen(params.model_filename.c_str(), "wb");
+        assert(fd);


assert() should not be used in OpenCV, use CV_Assert() instead.

alalek · 2017-10-02T15:07:05Z

modules/face/src/facemarkLBF.cpp

+
+        // random shuffle
+        unsigned int seed = (unsigned int)std::time(0);
+        std::srand(seed);


rand() is not thread-safe - can't provide reproducible results (for testing).
Consider to use cv::rand() , cv::theRNG() instead.

according to the documentation http://docs.opencv.org/3.2.0/d2/de8/group__core__array.html#ga75843061d150ad6564b5447e38e57722 theRNG function will use different random generator for each thread, but in this case the program only needs to randomize the order of training sample, which should produce same random order for each sample category (i.e. the randomized order or image sample should corresponds to the randomized order of the annotation). If reproducible results is expected, then i think we just need to set the seed value to a fixed number, or just remove this shuffling part (i don't think it will produce significantly different results)

cv::theRNG is thread-safe (maintains own state for each thread).
Seed initialization from time() should be removed from algorithm implementations - seed should be controlled externally.

Regarding this exact algorithm, I believe it should work without shuffling of input data.

alalek · 2017-10-02T15:18:35Z

modules/face/src/liblinear.hpp

+void cdotc_(fcomplex *dotval, int *n, fcomplex *cx, int *incx,
+            fcomplex *cy, int *incy);
+
+void cdotu_(fcomplex *dotval, int *n, fcomplex *cx, int *incx,


Using of external libraries should be avoided. Especially in case of duplicated functionality.
Why does cv::ml::LogisticRegression not work for you?

I tried to remove the liblinear dependency but was not successful (using gradient descent regression and simple regression w^T x = y). The LogisticRegression result is in range 0..1, isn't it? I considered to use this function before but worried that the result will be different to the original version (i need to normalize the target values, i.e. the landmark points, into 0..1 range,maybe by dividing the values by width and height of input image, and not sure about the result) and the deadline was approaching, so i did not test LogisticRegression. But let me try now...

OK. Thank you for information.

alalek · 2017-10-02T15:32:03Z

modules/face/test/test_facemark_lbf.cpp

+    string model_filename = "LBF.model";
+
+    Ptr<Facemark> facemark = FacemarkLBF::create();
+    EXPECT_NO_THROW(facemark->loadModel(model_filename.c_str()));


Each test should be standalone and don't rely on results of other tests.
Also tests should not write any data (models) if it is not required.

Probably we should put model somewhere or we can merge train/load tests into one.

merging the train-load test is easier, i can put a flag for not saving the model.

kurnianggoro · 2017-10-03T02:06:25Z

@vpisarev the test function needs at least 2 images, so i put Mark Zuckerberg and Dwayne Johnson photos from wikipedia (the license allows it, CCA 2.0 and CCA 3.0), will that be a problem? also regarding to the lena image, in other library it was removed due to copyright matter, i am not sure why opencv still have it. check it here torch/nn#854 also in wikipedia it is stated that lena is non free image https://en.wikipedia.org/wiki/File:Lenna.png

alalek · 2017-10-04T11:17:08Z

AAM tests are crashed in the "Debug" build. Please take a look.

kurnianggoro · 2017-10-06T03:01:05Z

modules/face/src/facemarkAAM.cpp

@@ -560,8 +560,8 @@ namespace face {
        reduce(Ys,sumYs, 0, CV_REDUCE_SUM);

        //calculate the normrnd
-        double normX = sqrt(sumXs.at<float>(0)+sumXs.at<float>(1));
-        double normY = sqrt(sumYs.at<float>(0)+sumYs.at<float>(1));
+        double normX = sqrt(Mat(sumXs.reshape(1)).at<float>(0)+Mat(sumXs.reshape(1)).at<float>(1));


@alalek the error in build debug is fixed here, but why it is not checked anymore?

This build is optional, it is not triggered automatically.
Debug build is passed, thanks!

alalek

Thank you for update.
Please take a look on comments below.

alalek · 2017-10-09T15:43:25Z

modules/face/test/test_facemark_aam.cpp

+using namespace cv::face;
+
+CascadeClassifier face_detector;
+bool customDetector( InputArray image, OutputArray ROIs, void * config = 0 ){


Make this function local to prevent conflicts via static qualifier or anonymous namespace.

alalek · 2017-10-09T15:48:50Z

modules/face/src/facemarkAAM.cpp

+        char x[256];
+        for(int i=0;i< (int)AAM.scales.size();i++){
+            sprintf(x,"scale%i_max_m",i);
+            fs << x << AAM.textures[i].max_m;


sprintf is unsafe in general and banned by static code analyzers. consider to use cv::format() instead

alalek · 2017-10-09T15:50:12Z

modules/face/src/facemarkAAM.cpp

+    }
+
+    void FacemarkAAMImpl::saveModel(String s){
+        FileStorage fs(s.c_str(),FileStorage::WRITE);


Did you try WRITE_BASE64 (should reduce the model size)?

alalek · 2017-10-09T20:09:07Z

modules/face/test/test_facemark_aam.cpp

+    EXPECT_NO_THROW(facemark->training());
+}
+
+TEST(CV_Face_FacemarkAAM, can_load_model) {


The same note about standalone tests.

kurnianggoro · 2017-10-12T21:25:25Z

Hi @alalek , I already fixed all the problems you've mentioned, but the building bot failed due to stereo module, can you please restart the building? also regarding the liblinear depency, I've tried to use opencv's logistric regression and neural network but both of them are not compatible. For the log-res, it seems like that the algorithm automatically assign categories to the training data and the number of resulted categories are inconsistent across different training data. As for the neural network, it do not support linear activation, I tried to train but the results are nan. So, I grab the support vector regression code from the liblinear, it just one function, not too long. I've tested it and the result is good. You can try to train using the Helen dataset if you want. The tutorial is here http://pullrequest.opencv.org/buildbot/export/pr_contrib/1257/docs/d7/dec/tutorial_facemark_usage.html but make sure your system has around 9GB of RAM and just wait 45in~1hour for the for the training. I've tried to train on smaller dataset but the results are not good.

alalek · 2017-10-18T11:21:53Z

@kurnianggoro Thank you! Update looks good to me.

alalek · 2017-10-18T11:43:14Z

The current issue is related to test images. We can't accept such risk.
Could you please replace these images? For example, from free datasets (but the most part of datasets is not free/restricted/missing license), or very old photos - like Einstein / Kennedy or similar (not alive persons).

alalek · 2017-10-18T12:04:13Z

modules/face/src/facemarkLBF.cpp

+            Gnorm1_new = 0;
+
+            for(i=0; i<active_size; i++){
+                int j = i+rand()%(active_size-i);


Please avoid using of rand() - it doesn't provide reproducible results.

Like "stereo" test (it uses rand() too), which fails time-to-time.

kurnianggoro · 2017-10-25T03:08:47Z

hi @alalek i have changed the test images using frames taken from the tracking module test video, I hope it will not be a problem now.

alalek · 2017-10-27T13:42:19Z

Thank you! Looks good to me 👍

Macbull · 2018-06-03T22:35:04Z

Is there a python wrapper of this API in any release version?
Thanks

kurnianggoro force-pushed the facelandmark branch from 1abc204 to 843ef4b Compare July 7, 2017 16:22

kurnianggoro force-pushed the facelandmark branch 23 times, most recently from e284e30 to 491f554 Compare September 1, 2017 08:18

kurnianggoro force-pushed the facelandmark branch from 967d07d to 28f7894 Compare September 6, 2017 07:26

Initial commit of facemark API

a9cf839

Initial structure of the facemark API and AAM header

kurnianggoro added 5 commits October 2, 2017 14:42

Allow custom parameters

060ee50

Cleaning up

eee7ac8

Custom parameters for default detector, training, and get custom data

3f1f886

AAM scales

bc5a784

minor fixes , update the opencv_extra files

7fc15de

kurnianggoro force-pushed the facelandmark branch from 28f7894 to 7fc15de Compare October 2, 2017 07:55

alalek added 2 commits October 2, 2017 12:52

change path to lbp cascade

367fc87

face: avoid memory leaks

0716019

alalek reviewed Oct 2, 2017

View reviewed changes

utilize the filestorage for the model, fixing some minor issues

41c708c

kurnianggoro commented Oct 6, 2017

View reviewed changes

alalek reviewed Oct 9, 2017

View reviewed changes

remove the liblinear dependency

9f90785

kurnianggoro force-pushed the facelandmark branch from a8025ff to 9f90785 Compare October 12, 2017 07:06

fix the aam test, avoiding to write any files

bb63240

alalek reviewed Oct 18, 2017

View reviewed changes

use RNG and changes the test files

ae84821

alalek merged commit e85a802 into opencv:master Oct 27, 2017

alalek mentioned this pull request Oct 30, 2017

PR #1257 refactoring #1428

Merged

terfendail mentioned this pull request Jan 22, 2018

Suggestions for future GSoCs opencv/opencv#7263

Open


		FacemarkLBF::Params::Params(){

		cascade_face = "../data/haarcascade_frontalface_alt.xml";

GSOC17 - Facemark API #1257

GSOC17 - Facemark API #1257

Conversation

kurnianggoro commented Jun 27, 2017 • edited Loading

sergboec commented Jul 14, 2017 • edited Loading

alalek commented Sep 28, 2017

kurnianggoro commented Sep 28, 2017 • edited Loading

alalek commented Sep 28, 2017

kurnianggoro commented Oct 2, 2017

vpisarev commented Oct 2, 2017

alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kurnianggoro Oct 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kurnianggoro commented Oct 3, 2017 • edited Loading

alalek commented Oct 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alalek Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kurnianggoro commented Oct 12, 2017

alalek commented Oct 18, 2017

alalek commented Oct 18, 2017 • edited Loading

Choose a reason for hiding this comment

kurnianggoro commented Oct 25, 2017

alalek commented Oct 27, 2017

Macbull commented Jun 3, 2018

kurnianggoro commented Jun 27, 2017 •

edited

Loading

sergboec commented Jul 14, 2017 •

edited

Loading

kurnianggoro commented Sep 28, 2017 •

edited

Loading

kurnianggoro Oct 3, 2017 •

edited

Loading

kurnianggoro commented Oct 3, 2017 •

edited

Loading

alalek Oct 9, 2017 •

edited

Loading

alalek commented Oct 18, 2017 •

edited

Loading