[MGPU] python-gpu/test_large_sizes.py failed.

[trivialfis]

During test, weight_ and labels_ inside Metainfo are resharded with a different and non-empty distribution, causing the CHECK inside reshard to fail.

The cause is pretty simple, the test first run on single GPU:

xgboost/tests/python-gpu/test_large_sizes.py

Line 100 in 516457f

eprint("gpu_hist updater 1 gpu")

Then later run on multi-gpu:

xgboost/tests/python-gpu/test_large_sizes.py

Line 106 in 516457f

eprint("gpu_hist updater all gpus")

While running multi-gpu test, the distribution of weigth_ and label_ remains unchanged. Hence, to make the test pass we simply have to remove one of these two tests.

This issue is a reminder for me to sort out a better approach for multi-gpu.

[hcho3]

@trivialfis Does #3738 solve this issue?

[trivialfis]

No, this one needs be solved by handling changed parameter.

[hcho3]

@trivialfis I'm confused, don't lines 100 and 106 create Booster object from scratch? Which objects are being shared?

[hcho3]

Also, should I add this as a blocking issue?

[trivialfis]

@hcho3 The weight and label from DMatrix are not resharded. To fix this we need to come up with a solution handling changing parameters, in this case, the number of GPUs . I think we can solve it along with the callback issue, so a known bug describing users currently can't change the number of GPUs in between two training/predict sessions might be more appropriate than a quick fix (reshard everything).

[hcho3]

So add an item to "known issue" for the next release, then?

[trivialfis]

Yes, please.

[hcho3]

@trivialfis What would be the work-around for this issue? Should I delete the DMatrix object and re-load?

[trivialfis]

Should work, but let me test it.

[trivialfis]

@hcho3 Tested.
Inserting:

            ag_dtrain = xgb.DMatrix(X, y, nthread=40)

In between two training sessions in /test_large_sizes.py will let the test pass. The above line essentially recreated DMatrix. I will focus on #3825 and #3795 next. After that we can try to find out a solution to these problems. Feel free to let me know if you have any suggestion. :)

[hcho3]

Got it. The release note for 0.81 will contain a short description of the work-around. Thanks!

[pseudotensor]

How does this work-around apply to us using the sklearn interface?

I mention because h2o4gpu testing hasn't failed with xgboost errors in a while, but after recent updates we get the distribution is empty issue for 1 test.