Fix perf gap in thread safe prediction

[ShvetsKS]

As in #6648 local RegTree::FVec storage was introduced to preserve thread safety each training iteration requires buffer initialization in case of subsampling.

stages (santander dataset)	before #6648	thread safe #6648 and master	current PR
full training	136s	165s	140s
PredictRaw	55s	78s	59s

this PR introduces threading for local buffers initialization.
But as still some gaps are exist is it possible to have not thread safe PredictDMatrix call during training?

[trivialfis]

I thought we have prediction cache enabled?

[ShvetsKS]

I thought we have prediction cache enabled?

For subsampling case we still don't have prediction caching (seems it's still in progress: #6683, major complexity is related to multiclass classification case as for each group own indices subset is generated.)

[trivialfis]

@ShvetsKS Would you like to take a look into the GPU implementation of subsampling? Or my WIP rewrite for CPU hist? The subsampling doesn't have to conflict with cache.

[trivialfis]

Last time your found it to be slower than master branch, which is expected as I don't have time to incooperate many recent optimization into it. But I believe my rewrite can at least offer some insight on how to implement some of the features in a different way.

[ShvetsKS]

Last time your found it to be slower than master branch, which is expected as I don't have time to incooperate many recent optimization into it. But I believe my rewrite can at least offer some insight on how to implement some of the features in a different way.

Current changes are applicable even for inference stage as it's better to initialize RegTree::FVec thread locally anyway.

[codecov-io]

Codecov Report

Merging #6696 (198747f) into master (9b267a4) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6696   +/-   ##
=======================================
  Coverage   81.56%   81.56%           
=======================================
  Files          13       13           
  Lines        3759     3759           
=======================================
  Hits         3066     3066           
  Misses        693      693           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b267a4...198747f. Read the comment docs.

[ShvetsKS]

problem with continuous-integration/travis-ci/pr is related with server node
await ServerNode.close(self)
seems restarting should help.

[trivialfis]

Sorry for the long wait. Still on vacation, but will try to look into it soon as possible.