Ensure that training and testing data align

[mrocklin]

Currently if you provide training and testing data that have the same number of partitions, but a different number of rows per partition then the user will get a non-informative error.

Given that we need to have all the data in memory anyway, we could just fix this for the user and balance partitions for them.

cc @jrbourbeau

[jrbourbeau]

Thanks for opening up this issue @mrocklin. Below is an example to reproduce the issue

import numpy as np
import dask.dataframe as dd
from dask.distributed import Client
from dask_xgboost import XGBClassifier
    
client = Client(processes=True,
                n_workers=2,
                threads_per_worker=1,
                memory_limit='3GB')

# Create dataset
np.random.seed(2)
a = np.random.rand(100, 10)
df = pd.DataFrame(a, columns=[f'feature_{i}' for i in range(a.shape[1])])
X = dd.from_pandas(df.iloc[:, :-1], chunksize=50)
y = dd.from_pandas(df.iloc[:, -1], chunksize=51)

# Print out length of each parition for X and y
print(X.map_partitions(len).compute())
print(y.map_partitions(len).compute())

# Fit dask-xgboost classifier
clf = XGBClassifier()
clf.fit(X, y)

Running this example will output

0    50
1    50
dtype: int64

for the X partition lengths, and

0    51
1    49
dtype: int64

for the y partition lengths (note they are different). And then an error message.

Traceback details

distributed.worker - WARNING -  Compute Failed
Function:  train_part
args:      ({'DMLC_NUM_WORKER': 2, 'DMLC_TRACKER_URI': '127.0.0.1', 'DMLC_TRACKER_PORT': 9091}, {'base_score': 0.5, 'booster': 'gbtree', 'colsample_bylevel': 1, 'colsample_bytree': 1, 'gamma': 0, 'learning_rate': 0.1, 'max_delta_step': 0, 'max_depth': 3, 'min_child_weight': 1, 'missing': None, 'n_estimators': 100, 'nthread': 1, 'objective': 'multi:softprob', 'reg_alpha': 0, 'reg_lambda': 1, 'scale_pos_weight': 1, 'seed': 0, 'silent': 1, 'subsample': 1, 'num_class': 100}, [(    feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  feature_6  feature_7  feature_8
0    0.435995   0.025926   0.549662   0.435322   0.420368   0.330335   0.204649   0.619271   0.299655
1    0.621134   0.529142   0.134580   0.513578   0.184440   0.785335   0.853975   0.494237   0.846561
2    0.505246   0.065287   0.428122   0.096531   0.127160   0.596745   0.226012   0.106946   0.220306
3    0.467787   0.201743   0.640407   0.483070   0.505237   0.386893   0.793637   0.580004   0.162299
4    0.964551   0.50000
kwargs:    {'dmatrix_kwargs': {'feature_names': Index(['feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_4',
       'feature_5', 'feature_6', 'feature_7', 'feature_8'],
      dtype='object')}, 'num_boost_round': 100}
Exception: XGBoostError('Long error message', "b'[11:49:52] src/objective/multiclass_obj.cc:43: Check failed: preds->Size() == (static_cast<size_t>(param_.num_class) * info.labels_.size()) SoftmaxMultiClassObj: label size and pred size does not match\\n\\nStack trace returned 7 entries:\\n[bt] (0) 0   libxgboost.dylib                    0x0000001c1c95aa51 dmlc::StackTrace() + 305\\n[bt] (1) 1   libxgboost.dylib                    0x0000001c1c95a7df dmlc::LogMessageFatal::~LogMessageFatal() + 47\\n[bt] (2) 2   libxgboost.dylib                    0x0000001c1c9d249e xgboost::obj::SoftmaxMultiClassObj::GetGradient(xgboost::HostDeviceVector<float>*, xgboost::MetaInfo const&, int, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*) + 430\\n[bt] (3) 3   libxgboost.dylib                    0x0000001c1c956b66 xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) + 1014\\n[bt] (4) 4   libxgboost.dylib             0x0000001c1c9742f4 XGBoosterUpdateOneIter + 164\\n[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x0")
distributed.worker - WARNING -  Compute Failed
Function:  train_part
args:      ({'DMLC_NUM_WORKER': 2, 'DMLC_TRACKER_URI': '127.0.0.1', 'DMLC_TRACKER_PORT': 9091}, {'base_score': 0.5, 'booster': 'gbtree', 'colsample_bylevel': 1, 'colsample_bytree': 1, 'gamma': 0, 'learning_rate': 0.1, 'max_delta_step': 0, 'max_depth': 3, 'min_child_weight': 1, 'missing': None, 'n_estimators': 100, 'nthread': 1, 'objective': 'multi:softprob', 'reg_alpha': 0, 'reg_lambda': 1, 'scale_pos_weight': 1, 'seed': 0, 'silent': 1, 'subsample': 1, 'num_class': 100}, [(    feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  feature_6  feature_7  feature_8
50   0.266590   0.203876   0.296205   0.8
41868   0.924015   0.978584   0.414331   0.773187   0.168053
51   0.424206   0.844092   0.197020   0.818916   0.072325   0.846108   0.423649   0.140741   0.305917
52   0.446285   0.233264   0.238765   0.693173   0.647769   0.063925   0.812518   0.140184   0.240564
53   0.506255   0.850932   0.929737   0.463310   0.102749   0.474305   0.290458   0.309397   0.242959
54   0.397410   0.00357
kwargs:    {'dmatrix_kwargs': {'feature_names': Index(['feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_4',
       'feature_5', 'feature_6', 'feature_7', 'feature_8'],
      dtype='object')}, 'num_boost_round': 100}
Exception: XGBoostError('Long error message', "b'[11:49:52] src/objective/multiclass_obj.cc:43: Check failed: preds->Size() == (static_cast<size_t>(param_.num_class) * info.labels_.size()) SoftmaxMultiClassObj: label size and pred size does not match\\n\\nStack trace returned 7 entries:\\n[bt] (0) 0   libxgboost.dylib                    0x0000001c1c95aa51 dmlc::StackTrace() + 305\\n[bt] (1) 1   libxgboost.dylib                    0x0000001c1c95a7df dmlc::LogMessageFatal::~LogMessageFatal() + 47\\n[bt] (2) 2   libxgboost.dylib                    0x0000001c1c9d249e xgboost::obj::SoftmaxMultiClassObj::GetGradient(xgboost::HostDeviceVector<float>*, xgboost::MetaInfo const&, int, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*) + 430\\n[bt] (3) 3   libxgboost.dylib                    0x0000001c1c956b66 xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) + 1014\\n[bt] (4) 4   libxgboost.dylib             0x0000001c1c9742f4 XGBoosterUpdateOneIter + 164\\n[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x0")

Traceback (most recent call last):
  File "example.py", line 34, in <module>
    clf.fit(X, y)
  File "/Users/jbourbeau/quansight/dask-xgboost/dask_xgboost/core.py", line 353, in fit
    num_boost_round=self.n_estimators)
  File "/Users/jbourbeau/quansight/dask-xgboost/dask_xgboost/core.py", line 193, in train
    labels, dmatrix_kwargs, **kwargs)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/distributed/client.py", line 670, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync
    six.reraise(*error[0])
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/distributed/utils.py", line 262, in f
    result[0] = yield future
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/jbourbeau/quansight/dask-xgboost/dask_xgboost/core.py", line 155, in _train
    results = yield client._gather(futures)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/distributed/client.py", line 1497, in _gather
    traceback)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/Users/jbourbeau/quansight/dask-xgboost/dask_xgboost/core.py", line 90, in train_part
    bst = xgb.train(param, dtrain, **kwargs)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/xgboost/training.py", line 204, in train
    xgb_model=xgb_model, callbacks=callbacks)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/xgboost/training.py", line 74, in _train_internal
    bst.update(dtrain, i, obj)
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/xgboost/core.py", line 1021, in update
    dtrain.handle))
  File "/Users/jbourbeau/miniconda/envs/quansight/lib/python3.6/site-packages/xgboost/core.py", line 151, in _check_call
    raise XGBoostError(_LIB.XGBGetLastError())
xgboost.core.XGBoostError: ('Long error message', "b'[11:49:52] src/objective/multiclass_obj.cc:43: Check failed: preds->Size() == (static_cast<size_t>(param_.num_class) * info.labels_.size()) SoftmaxMultiClassObj: label size and pred size does not match\\n\\nStack trace returned 7 entries:\\n[bt] (0) 0   libxgboost.dylib                    0x0000001c1c95aa51 dmlc::StackTrace() + 305\\n[bt] (1) 1   libxgboost.dylib                    0x0000001c1c95a7df dmlc::LogMessageFatal::~LogMessageFatal() + 47\\n[bt] (2) 2   libxgboost.dylib                    0x0000001c1c9d249e xgboost::obj::SoftmaxMultiClassObj::GetGradient(xgboost::HostDeviceVector<float>*, xgboost::MetaInfo const&, int, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*) + 430\\n[bt] (3) 3   libxgboost.dylib                    0x0000001c1c956b66 xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) + 1014\\n[bt] (4) 4   libxgboost.dylib                 0x0000001c1c9742f4 XGBoosterUpdateOneIter + 164\\n[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x0")

The relevant portion of the error message is label size and pred size does not match.

[jrbourbeau]

Given that we need to have all the data in memory anyway, we could just fix this for the user and balance partitions for them.

I'm in favor of this idea. I'll open up a PR with a proposal to balance the partitions for input data.

[ghost]

Is there a work around that will ensure training and testing data align?

I am reading several CSVs into Dask Dataframes

train_data = dd.read_csv()
labels = dd.read_csv()

I confirmed that the lengths are the same: 502732

Then run this:

est = xgb.XGBClassifier()
est.fit(train_data, labels)

Then I get this error:

Check failed: preds.Size() == info.labels_.Size() (17398 vs. 213797) labels are not correctly provided

[mrocklin]

Currently there is no easy workaround. It would be good to provide one though. I would think that this work would happen upstream in dask.dataframe , but I don't have a concrete plan here. Help would be welcome.