LightGBM model locale fix #53

AlbertoEAF · 2020-09-16T09:15:37Z

This implements the fix for model locale with our own patch to LightGBM v3.0.0.
Such patch currently lives at feedzai/LightGBM as it wasn't yet merged to Microsoft's mainline code.

…rsion

codecov · 2020-09-25T14:40:03Z

Codecov Report

Merging #53 into master will increase coverage by 0.78%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #53      +/-   ##
============================================
+ Coverage     78.46%   79.25%   +0.78%     
- Complexity      369      382      +13     
============================================
  Files            39       39              
  Lines          1333     1364      +31     
  Branches        123      127       +4     
============================================
+ Hits           1046     1081      +35     
+ Misses          221      215       -6     
- Partials         66       68       +2

Impacted Files	Coverage Δ	Complexity Δ
...eedzai/openml/provider/lightgbm/SWIGResources.java	`83.54% <0.00%> (-1.75%)`	`19.00% <0.00%> (+2.00%)`	⬇️
...er/lightgbm/LightGBMBinaryClassificationModel.java	`59.09% <0.00%> (+1.94%)`	`9.00% <0.00%> (+1.00%)`
...openml/provider/lightgbm/LightGBMModelCreator.java	`87.09% <0.00%> (+3.09%)`	`30.00% <0.00%> (+8.00%)`
...feedzai/openml/provider/lightgbm/LightGBMSWIG.java	`81.25% <0.00%> (+15.29%)`	`12.00% <0.00%> (+2.00%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c752c68...c603135. Read the comment docs.

openml-lightgbm/lightgbm-provider/pom.xml

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java

openml-lightgbm/lightgbm-builder/pom.xml

AlbertoEAF · 2020-09-25T16:25:05Z

I actually have one question @shenggwang @paulojrp , I added the folder standard_code_models with two model files.
Should we:

Use git lfs to store them?
Move this folder to the test resources as well? Right now it sits randomly

Finally, the tests generate a new folder: new_code_models, should we erase it or move it somewhere else somehow? It's ugly where it sits after you run tests. Put it in tmp? If the test fails we should not erase that folder or we can't diagnose what's wrong right?

openml-lightgbm/lightgbm-builder/make.sh

openml-lightgbm/lightgbm-provider/pom.xml

openml-lightgbm/lightgbm-provider/diff_models.py

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java

openml-lightgbm/lightgbm-provider/standard_code_models/extend_model.py

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java

openml-lightgbm/lightgbm-provider/standard_code_models/extend_model.py

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java

openml-lightgbm/lightgbm-builder/pom.xml

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java

AlbertoEAF · 2020-09-29T15:39:34Z

@shenggwang do we need the dummy lightgbm-builder/javadoc/README.md in the lightgbm-builder or can I delete it?

openml-lightgbm/lightgbm-provider/pom.xml

shengwangsw · 2020-09-29T16:40:08Z

@shenggwang do we need the dummy lightgbm-builder/javadoc/README.md in the lightgbm-builder or can I delete it?

There is no javadoc/README.md
I think you are outdated 😄

See https://github.com/feedzai/feedzai-openml-java/tree/master/openml-lightgbm/lightgbm-builder

This implements the fix for model locale with Feedzai's patch to LightGBM v3.0.0. Such patch currently lives at feedzai/LightGBM as it wasn't yet merged to Microsoft's mainline code. Also, it integrates the latest version of make-lightgbm with support for build caches to speed up repeated builds of LightGBM.

Alberto Ferreira added 7 commits September 11, 2020 20:13

LGBM prototype with build for LGBM which now fails with the latest ve…

8289649

…rsion

Test using the custom PoC code to mock read/writes in C++

58d2a6d

Use the latest make-lightgbm with caching

6975a2f

Use new LightGBM with fmt

cc62077

Use merged make-lightgbm with submodules support

d1d5bd8

Use latest make-lightgbm version

a8cabc2

Add model rewrite functional tests

f48d293

AlbertoEAF mentioned this pull request Sep 16, 2020

[fix] Locale independent model load/save microsoft/LightGBM#3267

Closed

Model writes use fmt library

c26f3ed

AlbertoEAF mentioned this pull request Sep 23, 2020

Fix model locale issue and improve model R/W performance. microsoft/LightGBM#3405

Merged

Alberto Ferreira added 4 commits September 23, 2020 19:58

Patched LightGBM version passes all tests independently of locale

1448d51

Test model diff with Java and do optional Python report

1c953f3

Update python scripts

5359eb3

Update python usage

a3f4b53

Alberto Ferreira added 3 commits September 25, 2020 16:10

Update test code for Codacy

742ee32

Update extend_model.py for Codacy

fb88ca7

Add docstrings for extend_model.py

36b2637

shengwangsw reviewed Sep 25, 2020

View reviewed changes

Alberto Ferreira added 2 commits September 25, 2020 17:17

Update pom descriptions

5b5c1a8

Address review comments from Sheng

96ee811

AlbertoEAF marked this pull request as ready for review September 25, 2020 16:22

paulojrp reviewed Sep 25, 2020

View reviewed changes

...rc/test/java/com/feedzai/openml/provider/lightgbm/LightGBMBinaryClassificationModelTest.java Outdated Show resolved Hide resolved

shengwangsw reviewed Sep 28, 2020

View reviewed changes

openml-lightgbm/lightgbm-provider/standard_code_models/extend_model.py Outdated Show resolved Hide resolved

Alberto Ferreira added 4 commits September 28, 2020 11:53

Add final keywords

1113b4e

Move python scripts to test resources

69dc536

Remove LC_ALL test configuration

98edf63

Update docstring

6579fe5

Alberto Ferreira added 3 commits September 28, 2020 12:45

Delete temporary test outputs

3191bee

Please Codacy random cryptographic checks

a6d64c1

Move reference models to test resources

9a0ed8f

shengwangsw reviewed Sep 28, 2020

View reviewed changes

Address Sheng's comments

a49c274

paulojrp reviewed Sep 28, 2020

View reviewed changes

Alberto Ferreira added 5 commits September 29, 2020 11:19

Update docstrings

aa83d29

Remove python diff_utils.py from test. Now it's manual.

f367470

Delete temporary dir in test

38df851

Cleanup unused imports

c835fba

Remove model logger from test

c299c51

AlbertoEAF mentioned this pull request Sep 29, 2020

Fix LightGBM models locale sensitivity and improve R/W performance. feedzai/LightGBM#3

Closed

Update pom.xml to use tag

fb66fc2

Upgrade make-lightgbm to have fix-cache patch.

c603135

AlbertoEAF commented Sep 29, 2020

View reviewed changes

openml-lightgbm/lightgbm-provider/pom.xml Show resolved Hide resolved

paulojrp approved these changes Sep 30, 2020

View reviewed changes

shengwangsw approved these changes Sep 30, 2020

View reviewed changes

AlbertoEAF merged commit 83ae7a2 into master Sep 30, 2020

AlbertoEAF deleted the ft-af-lgbm-model-locale-fix branch September 30, 2020 11:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LightGBM model locale fix #53

LightGBM model locale fix #53

AlbertoEAF commented Sep 16, 2020 •

edited

Loading

codecov bot commented Sep 25, 2020 •

edited

Loading

AlbertoEAF commented Sep 25, 2020 •

edited

Loading

AlbertoEAF commented Sep 29, 2020 •

edited

Loading

shengwangsw commented Sep 29, 2020 •

edited

Loading

LightGBM model locale fix #53

LightGBM model locale fix #53

Conversation

AlbertoEAF commented Sep 16, 2020 • edited Loading

codecov bot commented Sep 25, 2020 • edited Loading

Codecov Report

AlbertoEAF commented Sep 25, 2020 • edited Loading

AlbertoEAF commented Sep 29, 2020 • edited Loading

shengwangsw commented Sep 29, 2020 • edited Loading

AlbertoEAF commented Sep 16, 2020 •

edited

Loading

codecov bot commented Sep 25, 2020 •

edited

Loading

AlbertoEAF commented Sep 25, 2020 •

edited

Loading

AlbertoEAF commented Sep 29, 2020 •

edited

Loading

shengwangsw commented Sep 29, 2020 •

edited

Loading