[jvm-packages] group data is only set for training set and is set incorrectly

[gaofan0905]

When creating a watch, input data is split into trainMatrix and testMatrix randomly. But the input groupData is set only to trainMatrix. And the groupData param is for the original data set, it does fit for the split trainMatrix any more.

https://github.com/dmlc/xgboost/blob/master/jvm-packages/xgboost4j-spark/src/main/scala/ml/dmlc/xgboost4j/scala/spark/XGBoost.scala#L520

[superbobry]

Yes, this is a known limitation of the current group data support. The "right" way to fix this is to make the group explicitly available for each row in the input data frame, e.g. via #2749.

[gaofan0905]

seems #2749 is merged, just wondering is there any example on how to set the group id in dataframe and pass it to XGBoost?

[superbobry]

No, it has not been merged yet. Exposing this in the JVM wrapper would require a little bit of work as well.

[gaofan0905]

I made some local changes like following, would that work?

And also when looking at the results, I found for some records, the prediction result is different for the same input. Is it the nature of distributed training? That only a random subset of all trees will be used?

`
private object Watches {
def apply(
params: Map[String, Any],
labeledPoints: Iterator[XGBLabeledPoint],
baseMarginsOpt: Option[Array[Float]],
cacheDirName: Option[String]): Watches = {
val trainTestRatio = params.get("trainTestRatio").map(_.toString.toDouble).getOrElse(1.0)
if (params.contains("groupData") && params("groupData") != null) {
val groups = params("groupData").asInstanceOfSeq[Seq[Int]].toArray
val total = groups.sum
var cnt = 0
var index = 0
breakable {
for (i <- groups.indices) {
index = i
cnt += groups(i)
if (1.0 * cnt / total >= trainTestRatio) break
}
}

  val (trainGroups, testGroups) = groups.splitAt(index + 1)

  val trainPoints = mutable.ArrayBuffer.empty[XGBLabeledPoint]
  val testPoints = mutable.ArrayBuffer.empty[XGBLabeledPoint]

  while (labeledPoints.hasNext) {
    val p = labeledPoints.next()
    if (trainPoints.size < cnt) {
      trainPoints += p
    } else {
      testPoints += p
    }
  }

  val trainMatrix = new DMatrix(trainPoints.iterator, cacheDirName.map(_ + "/train").orNull)
  val testMatrix = new DMatrix(testPoints.iterator, cacheDirName.map(_ + "/test").orNull)

  for (baseMargins <- baseMarginsOpt) {
    val (trainMargin, testMargin) = baseMargins.splitAt(cnt)
    trainMatrix.setBaseMargin(trainMargin)
    testMatrix.setBaseMargin(testMargin)
  }

  trainMatrix.setGroup(trainGroups)
  testMatrix.setGroup(testGroups)
  new Watches(trainMatrix, testMatrix, cacheDirName)

} else {
  val seed = params.get("seed").map(_.toString.toLong).getOrElse(System.nanoTime())
  val r = new Random(seed)
  val testPoints = mutable.ArrayBuffer.empty[XGBLabeledPoint]
  val trainPoints = labeledPoints.filter { labeledPoint =>
    val accepted = r.nextDouble() <= trainTestRatio
    if (!accepted) {
      testPoints += labeledPoint
    }

    accepted
  }
  val trainMatrix = new DMatrix(trainPoints, cacheDirName.map(_ + "/train").orNull)
  val testMatrix = new DMatrix(testPoints.iterator, cacheDirName.map(_ + "/test").orNull)
  r.setSeed(seed)
  for (baseMargins <- baseMarginsOpt) {
    val (trainMargin, testMargin) = baseMargins.partition(_ => r.nextDouble() <= trainTestRatio)
    trainMatrix.setBaseMargin(trainMargin)
    testMatrix.setBaseMargin(testMargin)
  }

  new Watches(trainMatrix, testMatrix, cacheDirName)
}

}
`

[hcho3]

#2749 has been merged now. I think additional work would be needed to enable grouping in test set.

All feature requests are now consolidated to #3439. This issue should be re-opened if someone decides to actively work on implementing this feature.

[CodingCat]

in the master branch of xgboost, we have allowed the user to have per-instance group info (like qid), check #3369