StreamingHistogram reports incorrect bin counts

[metasim]

Unless I'm misunderstanding the purpose of binCounts, it appears to be working incorrectly.

Test code:

val tileSize = 5
def rndTile = {
  val data = Array.fill(tileSize * tileSize)(scala.util.Random.nextGaussian())
  ArrayTile(data, tileSize, tileSize): Tile
}

val rdd = spark.sparkContext.makeRDD(Seq((1, rndTile), (2, rndTile), (3, rndTile)))
val h = rdd.histogram()
println(h.totalCount()) // 75
println(h.binCounts().map(_._2).sum) // 1200
println(h.asInstanceOf[StreamingHistogram].buckets().map(_._2).sum) // 75

[pomadchin]

Hey @metasim can you provide more detailed information on how to reproduce it?

I implemented this unit test (see numbers in comments):

import geotrellis.spark._
import geotrellis.spark.testkit._
import geotrellis.raster._
import geotrellis.raster.histogram._
import org.scalatest._

class HistogramSpec extends FunSpec
  with Matchers
  with TestEnvironment {
  describe("HistogramSpec") {
    it("should count bins number correct") {
      val tileSize = 5
      def rndTile = {
        val data = Array.fill(tileSize * tileSize)(scala.util.Random.nextGaussian())
        ArrayTile(data, tileSize, tileSize): Tile
      }

      val rdd = sc.makeRDD(Seq((1, rndTile), (2, rndTile), (3, rndTile)))
      val h = rdd.histogram()
      println(h.totalCount()) // 75
      println(h.binCounts().map(_._2).sum) // 75
      println(h.asInstanceOf[StreamingHistogram].buckets().map(_._2).sum) // 75
    }
  }
}

[metasim]

That's really odd. For println(h.binCounts().map(_._2).sum) we're definitely not getting the same number. We're using GT 1.2.1. What version did you use?

[pomadchin]

@metasim current master; look into this issue: #2590

[moradology]

Looks to me like this issue is resolved on master and the upcoming 2.0 release. Closing for now