Initial support for column-wise data split

[rongou]

Support splitting data by column for in-memory DMatrix. When loading data, first construct the full dmatrix, and then slice the columns based on rank and world_size. MetaInfo is kept as is except for the num_nonzero_ field.

Part of #8424

[rongou]

@trivialfis @hcho3

One question I have is what happens with sparse data, when a row may come up empty for the column slice. Do we support having multiple row pointers pointing to the same Entry? I guess we can also say this only supports dense dmatrix for now.

[hcho3]

@rongou XGBoost currently allows empty rows in a CSR matrix.

[rongou]

@hcho3 good to know! Removed the empty row comment.

[trivialfis]

Exciting feature!

A couple questions:

• Do we need to assume every participant has complete access to the data? Otherwise, we can't split them.
• We can use GetBatch<CSCPage>.
• The MetaInfo::Copy can be implemented based on Extend.

[rongou]

• Do we need to assume every participant has complete access to the data? Otherwise, we can't split them.

Here we assume we're doing distributed training, so all data is available to all the workers. For vertical federated learning, data is already "split", so this is not needed.

• We can use GetBatch<CSCPage>.

This just does a transpose of the CSR page, right? Then once we slice the columns, we'd have to transpose it back to CSR since most of the code uses that. That seems less efficient.

• The MetaInfo::Copy can be implemented based on Extend.

Done.

[trivialfis]

apologies for the ambiguity, I meant MetaInfo::Extend.

This just does a transpose of the CSR page, right? Then once we slice the columns, we'd have to transpose it back to CSR since most of the code uses that. That seems less efficient.

Yeah, you are right that we need to get the CSR back anyway. I just thought using CSC might be simpler in code. Also, the SparsePage::GetTranspose is implemented in parallel. I will leave the decision to you.

so all data is available to all the workers.

Hmm.. I'm confused by this assumption. Why is it necessary for all data to be available to all workers? How's it different from federated learning?

[rongou]

apologies for the ambiguity, I meant MetaInfo::Extend.

Done.

This just does a transpose of the CSR page, right? Then once we slice the columns, we'd have to transpose it back to CSR since most of the code uses that. That seems less efficient.

Yeah, you are right that we need to get the CSR back anyway. I just thought using CSC might be simpler in code. Also, the SparsePage::GetTranspose is implemented in parallel. I will leave the decision to you.

Yeah not sure about the running time, but this is probably more memory efficient. Anyway, I think we just need correctness here, it probably doesn't make too much sense for column split if the data fits in memory. It gets more interesting when we add support for external memory to train on super wide datasets.

so all data is available to all the workers.

Hmm.. I'm confused by this assumption. Why is it necessary for all data to be available to all workers? How's it different from federated learning?

Right this is just the assumption for distributed training. We make the same assumption for row split. I guess we could add an option to not split the data if the use has already done some preprocessing to split the data beforehand, but I don't believe this is currently supported.

For federated learning, it's the opposite, the data is already split and cannot be combined or exchanged.

[trivialfis]

For federated learning, it's the opposite, the data is already split and cannot be combined or exchanged.

I thought in the case of distributed learning it's the same? The distributed framework/user would split the features and XGBoost just train on those input?

[rongou]

For federated learning, it's the opposite, the data is already split and cannot be combined or exchanged.

I thought in the case of distributed learning it's the same? The distributed framework/user would split the features and XGBoost just train on those input?

Not sure about dask or spark, but if we just using python, loading data in the distributed setting automatically triggers splitting: https://github.com/dmlc/xgboost/blob/master/src/c_api/c_api.cc#L213

[rongou]

Can this be merged? Thanks!

[trivialfis]

apologies, triggered the CI. Will merge it once it's finished.

[trivialfis]

@rongou We need to slice the feature_weights/names/types accordingly as they have the length of n_features.

[rongou]

@trivialfis hmm we are not changing the metadata, so I'm not sure these need to be changed. Slicing a dmatrix only slices the feature values stored for each row so that we may reduce memory usage during boosting, but the metadata is kept as is.