Distributing LightGBM with OpenMP

[AlbertoEAF]

Hello,

Problem
Whilst creating a LightGBM provider in Java, I'm having the issue that OpenMP is required to train models (more specifically libgomp), even when I set num_threads=1.

For Ubuntu systems that have OpenMP installed this is no problem, but for other uncontrolled Linux amd64 environments this is not the case.

I thought of shipping libgomp.so with LightGBM and dynamically loading it, but it doesn't work as its version is not the correct one.

Taking care of versions
If I ship libgomp.so, it has to be compatible with the system's libc version, and as there's forward-compatibility to libc versions, to target as many possible machines as possible I should use a version that is as old as possible of libgomp (where libc>=2.14 according to LightGBM's docs).

Need help:
I was thus interested in finding out what were the versions with which the last release was generated.

Do you build the releases using the .travis.yml file in the repo?

If so, for Linux it seems to use Ubuntu bionic (18.04), but I'm not aware of which docker image is actually being used. It seems to be a custom one, as my Ubuntu 18.04 has an older version of clang (6.0.0) than that image (7.0 according to their documentation).

Can you help? I know nothing about Travis. Thanks!

[StrikerRUS]

Hi @AlbertoEAF !

I'm having the issue that OpenMP is required to train models (more specifically libgomp), even when I set num_threads=1.

You can build LightGBM without OpenMP: https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-threadless-version-not-recommended.

Do you build the releases using the .travis.yml file in the repo?

All our artifacts are come from Azure Pipelines builds.
https://github.com/microsoft/LightGBM/blob/master/.vsts-ci.yml
And here is the docker image for Linux (ubuntu-14.04) builds.
https://hub.docker.com/r/lightgbm/vsts-agent/dockerfile

Example of test logs if you need them: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=5974&view=results

-- The C compiler identification is GNU 4.8.4
-- The CXX compiler identification is GNU 4.8.4
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "3.1") 
-- Found OpenMP_CXX: -fopenmp (found version "3.1") 
-- Found OpenMP: TRUE (found version "3.1")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /__w/1/s/build

[StrikerRUS]

Also, you may find this script in which we check libraries' versions at CI side helpful to install appropriate dependencies.
https://github.com/microsoft/LightGBM/blob/master/helpers/check_dynamic_dependencies.py

[AlbertoEAF]

Thank you so much for those quick and informative replies @StrikerRUS !

I'll check those out, and I'm sure I'll have more questions :)

[AlbertoEAF]

Thanks @StrikerRUS , I tried using those images but I have no account user nor token for the agents image, thus I copied the dockerfiles and modified them so I could have a working "copy" of lightgbm docker image from a raw ubuntu14.04.

Now I want to copy the /usr/local/lib/libomp.so from llvm and the lightgbm build, my question now is, how do I know that a release 2.3.150 matches the commit X in this repo? (ping @imatiach-msft as you seemed to do the releases :): https://mvnrepository.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib/2.3.150)

I ask this for I only find tag 2.3.1 in this repo.

Thanks again guys!

[StrikerRUS]

@AlbertoEAF

Now I want to copy the /usr/local/lib/libomp.so from llvm ...

Please note that all our Linux releases are built with gcc, not clang. We use clang only from time to time to check that everything is working with it.

We haven't changed the core content of the docker image from early begging. And we have no plans to change it in the near future (refer to #2760 (comment)).

If you need, I think I can extract the needed files from our docker for you.

[AlbertoEAF]

Thank you so much @StrikerRUS, there's no need to pull files, and thanks for correcting me on gcc vs llvm - I'd be making a big mistake. I already have the image running with the Dockerfile modifications - making sure I build using your environment was the biggest priority :).

I think I just need to know now which commit it was that the 2.3.150 release https://mvnrepository.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib/2.3.150 originated from @imatiach-msft - can you explain how each .jar release matches lighgbm's commits / releases?

[StrikerRUS]

@AlbertoEAF OK, fine!

One more thing you may want to know is that we include jars (separately for each platform) in our artifacts. So you can grab the latest ones from this link.

[imatiach-msft]

@AlbertoEAF it looks like I created 2.3.150 on Dec 9. Note that while I don't keep track of the exact commit I do copy all of the source over to the jar when I release it, although it's only limited to the *.java files which may not be as useful:

drwxrwxr-x 2 ilya ilya 4.0K Aug 22 2018 DeployLightGBM-2.1.200
drwxrwxr-x 2 ilya ilya 4.0K Aug 26 2018 DeployLightGBM-2.1.250
drwxrwxr-x 2 ilya ilya 4.0K Dec 13 2018 DeployLightGBM-2.2.200
drwxrwxr-x 2 ilya ilya 4.0K Mar 20 2019 DeployLightGBM-2.2.300
drwxrwxr-x 2 ilya ilya 4.0K Apr 30 2019 DeployLightGBM-2.2.350
drwxrwxr-x 2 ilya ilya 4.0K Aug 29 2019 DeployLightGBM-2.2.400
drwxrwxr-x 2 ilya ilya 4.0K Oct 8 2019 DeployLightGBM-2.3.100
drwxrwxr-x 2 ilya ilya 4.0K Dec 9 01:05 DeployLightGBM-2.3.150
drwxrwxr-x 2 ilya ilya 4.0K Mar 29 15:16 DeployLightGBM-2.3.160
drwxrwxr-x 2 ilya ilya 4.0K Apr 3 00:59 DeployLightGBM-2.3.170

I think changeset b6aa5ee might have been it, but I'm not 100% sure.

As long as you have this fix you should be fine:

[imatiach-msft]

@AlbertoEAF yep, I do those maven releases, they are more for mmlspark than an independent java package. Sorry I don't have the exact commit, will try to track them better moving forward. I usually get one of the builds from master with my latest changes (usually the version bump includes an important bugfix). I agree this is not a good process and I probably should start doing a more regular release cycle, but at this time I am still trying to improve the code reliability with major fixes/enhancements. You could probably just take the release date and look at the latest merged commit in master for any of the released jars, it should correspond to that change.

[AlbertoEAF]

Thank you @imatiach-msft !

Just for feedback I don't think they need to be regular, but knowing the LightGBM commit when a new release comes out would be nice to have.

[StrikerRUS]

@AlbertoEAF Hey! Are there any other points we can help you with?

[AlbertoEAF]

Nope sorry!

I just figured out today how to get a build that works with OpenMP across many distros thanks to your help!

Thank you! I'll close the issue now :)

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.