Feature interaction constraint for GPU Hist.

[trivialfis]

This is still WIP, my approach is to just drop all features that don't comply to constraints before entering split evaluation. Not sure about how to merge it with CPU SplitEvaluator yet.

GPU Exact will be left to another PR. Initially I tried to implement a kernel side evaluator that rejects features during split evaluation, see: 70a5936 , which is much more complicated and now replaced, but might still be useful for GPU EXACT.

related to #4169

@RAMitchell

[RAMitchell]

I don't really care about gpu_exact and am willing to deprecate the algorithm in future. Probably after we do some work to improve the performance of sparse data sets for gpu_hist.

[trivialfis]

I'm not sure about why VC failed to build generated host stub from host_device_vector.cu.

[hcho3]

@trivialfis I found https://issues.jenkins-ci.org/plugins/servlet/mobile#issue/JENKINS-9104. Looks like we have issues running multiple MSBuild jobs running in parallel.

[trivialfis]

@hcho3 Could you help working around it? From what Andreas said, it seems to be a matter of setting envs.

[hcho3]

Let me see if I can improve stability of Windows tests

[trivialfis]

@RAMitchell

Probably after we do some work to improve the performance of sparse data sets for gpu_hist.

I have been thinking about that too. Perhaps the feature bundling from lgb would help, but it has memory usage issue as mentioned by @hcho3 in #4354 (comment)

[trivialfis]

This PR uses SplitEvaluator to enable feature interaction support for GPU Hist. This avoids creating another interaction constraint implementation, but still we need to think about how to merge the code path of other split evaluators. One thing worth noting is that, interaction constraint is different from other split evaluators in two ways, it has dynamic size buffer to track tree splits, and it doesn't have to be used during actual split evaluation. So in the future we might want to distinguish it from others.

[RAMitchell]

I think we need to build a DeviceSplitEvaluator class that shadows the functionality of SplitEvaluator but uses device memory internally and may be updated on the device (e.g. when registering new splits I do not want to copy memory up and down, this is a very large performance penalty). Do you think this is possible?

[trivialfis]

@RAMitchell

For feature interaction constraint (FIC), it won't be necessary since feature sampling has already been using host memory, FIC does not do any extra host device copying.

For other split evaluators, I don't think the current abstract interface of SplitEvaluator is useful, it's just making things more complicated than necessary. There are only 3 of them, technically FIC is better used for feature selection than gradient gain evaluation. So 2 ifs are all we need ...

[RAMitchell]

Its not strictly true that its already doing a copy - I think if column sampling is 1.0 the column sampler always returns the same HostDeviceVector and no copy occurs. The cost of introducing a single small memcpy between host and device for each node may be about 30% increased runtime.

[trivialfis]

@RAMitchell Sorry for the ambiguity. I'm aware of this. The correct term is: no more than using feature sampling.

The difficulties of implementing this on device are:

• It needs to use set. Children merge their parents' constraint set during split. Ordering doesn't matter, but no duplication is allowed. Situations like [[0, 2], [1, 3, 4], [5, 6], [3, 5]] where 3 appears in two different set can happen. Say root splits at 1, and its left child splits at 5. During split the two set [1, 3, 4], [3, 5] needs to be merged into child because children should be able to interact with existing parent's split (also this is what the current implementation do).

• FIC can come in with varying size, from single element [[0]] to all features.

• Also device memory allocation is inevitable since every node needs to have its own set. Combined with first difficulty, to avoid copying, we need to 1. compute the size of set on device (the information is not copied to host). 2. Allocate memory on host. 3 Come back to device and compute the actual set. I assume this is not much better than copying memory. Also, combined with difficulty 2, the computation won't be very efficient.

Currently I don't have any good idea for how to meet your requirement yet. But suggestions are welcomed. If it's any consolation, I plan to bring back feature grouping to reduce sparsity #4501, which should bring some nice improvement for GPU Hist after we support it.

[RAMitchell]

To implement a set I would use a boolean (bit?) vector of length n_features. You should know how much memory to allocate ahead of time because the maximum number of nodes is always constrained in the GPU algorithm and we know how many interaction sets there are. Should be fun to try and implement :)

Looking at monotone constraints and feature interaction constraints at a higher level, it seems to me that there is a desire from users to directly influence the optimisation process of tree construction. I wonder if there is a more general way of specifying this as an interface? Probably not but interesting to think about.

[trivialfis]

@RAMitchell Let me wrap up other PRs first so I can focus on this.

[trivialfis]

Closing for now, will open a new PR when it's ready.