Build is unstable under some circumstances

[tdunning]

Building and testing on my laptop is pretty stable but building and testing on travis has >50% failure rate. Building on a large AWS node also shows significant failure rates.

The build command on travis differs from the one I normally use. Travis uses:

 mvn -B clean install -DskipTests
 mvn -B test

I emulated this on the large AWS node and got the following results:

baseline
rm -rf ~/.m2 ; mvn -B clean install -DskipTests ; mvn -B test
      7 [INFO] BUILD FAILURE
     13 [INFO] BUILD SUCCESS

no-rm-baseline
mvn -B clean install -DskipTests ; mvn -B test
      6 [INFO] BUILD FAILURE
     14 [INFO] BUILD SUCCESS

no-clean-no-rm-baseline
mvn -B install -DskipTests ; mvn -B test
      5 [INFO] BUILD FAILURE
     15 [INFO] BUILD SUCCESS

bare-test-batch-baseline
mvn -B test
      3 [INFO] BUILD FAILURE
      7 [INFO] BUILD SUCCESS

no-batch-baseline
mvn test
      2 [INFO] BUILD FAILURE
      8 [INFO] BUILD SUCCESS

I see no difference in failure rates here. I also looked at the environment and was unable to discern any difference due to -B.

The pom has the following surefire configuration:

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.18.1</version>
            <configuration>
                <parallel>methods</parallel>
                <perCoreThreadCount>true</perCoreThreadCount>
                <threadCount>1</threadCount>
            </configuration>
        </plugin>

Commenting out the configuration elements here doesn't seem to change anything either.

[tdunning]

The final upshot here was a subtle-ish bug in the ArrayDigest where adding a data point to a centroid with an identical mean as others could cause a problem. This only occurred when all but one of the collocated centroids were full which fooled the code into thinking that there was a unique nearest centroid. In addition, it seems that the @afterclass method in TestDigest is called under surprising circumstances (which probably means something else is wrong in the way that logs are being kept for tests)

[tdunning]

I think that this has been dealt with.