Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

[Heron-3733] CI Pipeline Failures Involving the <stmgr_unittest> #3735

Merged
merged 14 commits into from
Nov 15, 2021
Merged

[Heron-3733] CI Pipeline Failures Involving the <stmgr_unittest> #3735

merged 14 commits into from
Nov 15, 2021

Conversation

surahman
Copy link
Member

@surahman surahman commented Nov 13, 2021 •
edited
Loading

Issue #3733: CI Pipeline Failures Involving the stmgr_unittest

There is an issue with running the stmgr_unittest on thread constrained systems. TravisCI also runs into this issue because the tier we run on provides us with 2 cores.

The initial part of this PR is supposed to test to see if the offending tests have all been isolated. On a local system with reduced cores the following tests are stuck waiting in the following busy loop:

while (dummy_stmgr->NumStartBPMsgs() == 0) sleep(1);

In these tests:

incubator-heron/heron/stmgr/tests/cpp/server/stmgr_unittest.cpp

Lines 1030 to 1031 in c32c09a

// Wait till we get the back pressure notification
while (dummy_stmgr->NumStartBPMsgs() == 0) sleep(1);

incubator-heron/heron/stmgr/tests/cpp/server/stmgr_unittest.cpp

Lines 1140 to 1141 in c32c09a

// Wait till we get the back pressure notification
while (dummy_stmgr->NumStartBPMsgs() == 0) sleep(1);

incubator-heron/heron/stmgr/tests/cpp/server/stmgr_unittest.cpp

Lines 1397 to 1398 in c32c09a

// Wait till we get the back pressure notification
while (dummy_stmgr->NumStartBPMsgs() == 0) sleep(1);

Here are the tests on a 4-core Ubuntu system now passing where they were failing before:

Test results 
INFO: Elapsed time: 881.986s, Critical Path: 157.75s
INFO: 2697 processes: 1222 internal, 1475 local.
INFO: Build completed successfully, 2697 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.0s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.5s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.5s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.3s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 0.7s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 3.0s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 0.9s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 0.5s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 0.7s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.4s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 0.9s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 0.6s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 0.7s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 0.3s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.4s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.5s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.5s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 0.8s
//heron/api/tests/scala:api-scala-test                                   PASSED in 1.2s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (1.2s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 1.0s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.0s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.0s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.0s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.0s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.0s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.1s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.3s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.1s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.0s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.3s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.1s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 0.3s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.3s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 0.4s
//heron/common/tests/java:EchoTest                                       PASSED in 1.0s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 1.3s
//heron/common/tests/java:HeronServerTest                                PASSED in 1.6s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.3s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 5.5s
//heron/common/tests/java:SystemConfigTest                               PASSED in 0.4s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.4s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.4s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 1.1s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 1.1s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 0.4s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.4s
//heron/executor/tests/python:executor_unittest                          PASSED in 1.5s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 0.7s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 0.8s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 0.7s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 0.7s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 0.7s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 0.7s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 0.9s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 0.5s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 0.7s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 0.8s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 0.9s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 0.7s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 0.8s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 0.8s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 0.8s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 0.9s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 0.7s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 0.5s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 2.7s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.8s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 0.7s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 0.9s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 0.6s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.4s
//heron/instance/tests/java:HandleReadTest                               PASSED in 0.7s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 5.7s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.3s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 2.7s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 2.6s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 3.1s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.0s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.2s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.2s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.1s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 1.3s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.1s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 1.1s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 1.4s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.0s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.1s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 1.1s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.2s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.1s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 1.1s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 1.1s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 0.6s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 0.5s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 0.5s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 0.4s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 0.5s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 9.5s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 0.8s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 0.4s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 0.4s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 0.5s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.5s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 0.5s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 1.2s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.4s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.4s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 0.8s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 0.7s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.3s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 1.7s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 0.5s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 1.4s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 2.4s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.5s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 3.8s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 3.4s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 2.1s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 4.9s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 0.5s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 1.3s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 2.3s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 1.7s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 2.4s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 12.2s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 0.5s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 0.3s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 0.9s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 3.9s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 1.6s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 2.6s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.5s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.4s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 1.0s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 0.9s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 0.4s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 0.5s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 1.2s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 0.6s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 1.6s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 0.9s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 0.5s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 0.6s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 0.6s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 0.8s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 3.2s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 1.5s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 1.1s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 1.8s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.4s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 0.3s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 2.8s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.3s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 0.7s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 0.5s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.3s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 0.4s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 1.0s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 0.5s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 0.4s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.3s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.3s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 0.4s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 0.4s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.3s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 0.5s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 2.2s
//heron/spi/tests/java:ConfigTest                                        PASSED in 1.2s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.3s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 0.3s
//heron/spi/tests/java:KeysTest                                          PASSED in 0.3s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.3s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.3s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 2.1s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.3s
//heron/spi/tests/java:ResourceTest                                      PASSED in 0.3s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 1.6s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.3s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 0.7s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 2.5s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 2.5s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 1.4s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.0s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 0.8s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 1.2s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 1.6s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.0s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 1.2s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.1s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.4s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.4s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.1s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stmgr_unittest                            PASSED in 20.1s
  WARNING: //heron/stmgr/tests/cpp/server:stmgr_unittest: Test execution time (20.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.0s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.6s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.0s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.1s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.0s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.0s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 36.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.5s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 0.7s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.4s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.1s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.3s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.6s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.6s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.5s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 1.5s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 0.7s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.5s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 0.5s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 0.8s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.3s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.3s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 0.4s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 1.1s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 0.5s

INFO: Build completed successfully, 2697 total actions

@surahman surahman force-pushed the 3733-CI-Stmgr-Unittest branch from 99c9e63 to 89929f8 Compare November 13, 2021 03:20
@surahman
Copy link
Member Author

New commits from other branches have broken the pipeline:

Traceback (most recent call last):
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 679, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/.bootstrap/pex/pex.py", line 691, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/integration_test/src/python/local_test_runner/main.py", line 180, in <module>
    main()
  File "/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/integration_test/src/python/local_test_runner/main.py", line 115, in main
    conf_string = pkgutil.get_data(__name__, conf_file).decode()
  File "/usr/lib/python3.6/pkgutil.py", line 634, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 832, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/.pex/unzipped_pexes/9aaea117cab7d477701d8d2e3495171bd049d9bd/integration_test/src/python/local_test_runner/integration_test/src/python/local_test_runner/resources/test.conf'
The command "scripts/travis/ci.sh" exited with 1.

I have dropped down to the Pod Template PR merge.

@surahman surahman force-pushed the 3733-CI-Stmgr-Unittest branch from 89929f8 to eaba3bc Compare November 13, 2021 05:52
@surahman
[Stmgr] <Unit Test>
3ccf83e 
Disabling tests to isolate CI issues:
(StMgr, test_back_pressure_instance)
@surahman
[Stmgr] <Unit Test>
8c10c2d 
Disabling tests to isolate CI issues:
(StMgr, test_back_pressure_instance)
(StMgr, test_spout_death_under_backpressure)
@surahman
[Stmgr] <Unit Test>
721175d 
Disabling tests to isolate CI issues:
(StMgr, test_back_pressure_instance)
(StMgr, test_spout_death_under_backpressure)
(StMgr, test_back_pressure_stmgr_reconnect)
@surahman
Merge branch '3733-CI-Stmgr-Unittest-dev' into 3733-CI-Stmgr-Unittest
dbf2455
@surahman surahman force-pushed the 3733-CI-Stmgr-Unittest branch from eaba3bc to dbf2455 Compare November 13, 2021 16:11
@surahman
[Stmgr] <Unit Test>
bbb031b 
Potential fix for (StMgr, test_back_pressure_stmgr_reconnect) by setting high and low watermarks for back pressure.
@surahman
[Stmgr] <Unit Test>
36bbee8 
Potential fix for (StMgr, test_back_pressure_instance) by setting high and low watermarks for back pressure and increasing Spouts to ramp up pressure faster.
@surahman
Copy link
Member Author

surahman commented Nov 13, 2021 •
edited
Loading

After a lot of code digging and multithreaded debugging on the command line in GDB I have isolated and fixed the bug with the failing test on a 4 core machine. The test is also passing on a machine with many more cores.

The issue is with backpressure buildup in the three problematic tests. The bolt threads are killed and the spouts are allowed to build backpressure to set up the scenario for the "event" in the test. I have tweaked the high/low watermarks for the backpressure as well as tweaked the spout counts to get the tests to work faster.

There are some weird PEX errors that are coming up but if the CI runtime passes the 100-105min mark we can safely assume the stmgr_unittest is now passing in entirety in the CI pipeline as well.

Edit: It can be flaky but will pass locally. It will pass by the 3rd automatic run attempt for the unit test.

First run 
INFO: Elapsed time: 379.150s, Critical Path: 353.90s
INFO: 235 processes: 236 local.
INFO: Build completed successfully, 235 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.1s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.5s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.3s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.3s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 0.5s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 2.0s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 0.5s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 0.4s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 0.4s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.4s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 0.5s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 0.4s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 0.4s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 0.3s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.5s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.3s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.3s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 0.5s
//heron/api/tests/scala:api-scala-test                                   PASSED in 0.9s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (0.9s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 0.6s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.0s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.0s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.0s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.0s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.0s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.0s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.4s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.1s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.2s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.0s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 0.3s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.3s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 0.4s
//heron/common/tests/java:EchoTest                                       PASSED in 0.6s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 1.1s
//heron/common/tests/java:HeronServerTest                                PASSED in 1.5s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.3s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 5.1s
//heron/common/tests/java:SystemConfigTest                               PASSED in 0.4s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.4s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.3s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 0.9s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 1.0s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 0.4s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.4s
//heron/executor/tests/python:executor_unittest                          PASSED in 1.3s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 0.9s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 1.0s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 0.8s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 0.6s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 0.6s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 0.5s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 0.9s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 0.5s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 0.6s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 0.7s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 0.6s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 0.6s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 0.7s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 0.6s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 0.5s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 0.5s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 0.5s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 0.5s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 2.6s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.6s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 0.7s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 0.6s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 0.7s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.3s
//heron/instance/tests/java:HandleReadTest                               PASSED in 0.6s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 5.6s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.3s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 2.7s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 2.5s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 3.1s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.0s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.1s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.2s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.1s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 1.2s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.2s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 1.1s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 1.2s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.0s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.1s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 1.1s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.1s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.1s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 1.1s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 1.1s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 0.7s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 0.5s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 0.4s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 0.5s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 9.5s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 0.5s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 0.4s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 0.5s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.6s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 0.6s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 0.8s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.5s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.4s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 0.6s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 0.5s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.3s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 1.2s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 0.4s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 1.2s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 1.9s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.4s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 2.8s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 2.4s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 1.4s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 3.6s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 0.5s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 1.4s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 1.5s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 1.2s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 1.7s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 11.8s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 0.4s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 0.4s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 0.8s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 2.9s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 1.4s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 1.8s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.4s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.3s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 0.9s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 0.8s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 0.4s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 0.5s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 0.9s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 0.6s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 1.3s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 0.8s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 0.5s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 0.7s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 0.6s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 0.7s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 2.1s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 1.0s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 1.0s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 1.2s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.3s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 0.3s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 1.7s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.4s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 0.7s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 0.5s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.3s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 0.8s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 0.5s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.3s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.3s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 0.5s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 0.4s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.3s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 0.5s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 1.4s
//heron/spi/tests/java:ConfigTest                                        PASSED in 1.0s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.3s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 0.2s
//heron/spi/tests/java:KeysTest                                          PASSED in 0.3s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.3s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.2s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 1.6s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.3s
//heron/spi/tests/java:ResourceTest                                      PASSED in 0.3s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 1.5s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.3s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 0.4s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 2.0s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 2.0s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 1.1s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.0s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 0.6s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 1.3s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 1.1s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.1s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 1.2s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.1s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.4s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.4s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.1s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.0s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.7s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.1s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.0s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.1s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.0s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 26.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.4s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 0.7s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.2s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.0s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.2s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.5s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.4s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.3s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 1.5s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 0.7s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.4s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 0.5s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 0.6s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.3s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.3s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 0.4s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 1.6s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 0.4s
//heron/stmgr/tests/cpp/server:stmgr_unittest                             FLAKY, failed in 1 out of 2 in 315.1s
  Stats over 2 runs: max = 315.1s, min = 37.0s, avg = 176.0s, dev = 139.0s
  /root/.cache/bazel/_bazel_root/f4ab758bd53020512013f7dfa13b6902/execroot/org_apache_heron/bazel-out/k8-fastbuild/testlogs/heron/stmgr/tests/cpp/server/stmgr_unittest/test_attempts/attempt_1.log
Second run 
INFO: 235 processes: 1 internal, 234 local.
INFO: Build completed successfully, 235 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.1s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.4s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.5s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.3s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 0.5s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 2.1s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 0.6s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 0.5s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 0.5s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.4s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 0.5s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 0.4s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 0.8s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 1.0s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.3s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.4s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.4s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 0.6s
//heron/api/tests/scala:api-scala-test                                   PASSED in 1.3s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (1.3s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 1.3s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.0s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.1s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.0s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.0s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.0s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.1s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.3s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.1s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.2s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.1s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 0.4s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.4s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 0.5s
//heron/common/tests/java:EchoTest                                       PASSED in 0.9s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 1.8s
//heron/common/tests/java:HeronServerTest                                PASSED in 1.9s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.3s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 5.6s
//heron/common/tests/java:SystemConfigTest                               PASSED in 0.4s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.4s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.4s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 1.0s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 1.0s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 0.4s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.5s
//heron/executor/tests/python:executor_unittest                          PASSED in 1.6s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 1.6s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 0.7s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 0.6s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 0.8s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 1.1s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 0.8s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 1.9s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 0.5s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 1.2s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 1.3s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 0.6s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 1.3s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 0.7s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 0.7s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 0.8s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 0.7s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 0.7s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 0.8s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 2.8s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.6s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 0.6s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 1.4s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 1.7s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.3s
//heron/instance/tests/java:HandleReadTest                               PASSED in 0.9s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 5.8s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.3s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 2.7s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 2.6s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 3.2s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.2s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.2s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.2s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.2s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 1.4s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.3s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 1.2s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 1.3s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.1s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.3s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 1.2s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.3s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.7s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 1.6s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 1.5s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 0.8s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 0.5s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 0.5s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 0.5s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 0.5s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 9.6s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 0.4s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 0.5s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 0.7s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.6s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 0.6s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 0.6s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.5s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.4s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 1.7s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 1.1s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.5s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 1.2s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 0.6s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 1.3s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 2.2s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.5s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 2.8s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 2.7s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 2.2s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 6.0s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 1.2s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 2.4s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 1.8s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 2.8s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 3.3s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 12.5s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 1.1s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 0.6s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 1.8s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 3.8s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 2.7s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 3.6s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.5s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.6s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 1.8s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 1.2s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 0.6s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 0.8s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 1.1s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 0.9s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 1.9s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 1.0s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 0.7s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 0.8s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 0.6s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 0.7s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 4.0s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 2.4s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 1.5s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 2.8s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.3s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 0.4s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 2.1s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.3s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 0.8s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 0.7s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.4s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 0.4s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 0.9s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 1.2s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.3s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.3s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 0.6s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 0.5s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.4s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 0.9s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 2.4s
//heron/spi/tests/java:ConfigTest                                        PASSED in 1.8s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.5s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 0.2s
//heron/spi/tests/java:KeysTest                                          PASSED in 0.3s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.5s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.2s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 2.8s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.3s
//heron/spi/tests/java:ResourceTest                                      PASSED in 0.7s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 2.0s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.4s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 0.7s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 3.0s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 2.8s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 1.6s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.0s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 0.8s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 2.5s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 1.2s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.3s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 1.1s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.2s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.9s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.9s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stmgr_unittest                            PASSED in 34.4s
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.0s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.7s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.1s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.0s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.0s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.0s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 26.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.6s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 1.3s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.5s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.1s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.4s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.8s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.8s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.4s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 1.5s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 1.7s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.6s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 0.5s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 0.6s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.4s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.9s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 0.4s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 2.5s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 0.5s

INFO: Build completed successfully, 235 total actions
Third run 
INFO: 235 processes: 1 internal, 234 local.
INFO: Build completed successfully, 235 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.1s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.7s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.3s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.4s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 1.2s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 2.8s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 0.7s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 0.8s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 0.9s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.3s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 0.6s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 0.4s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 0.4s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 0.3s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.3s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.5s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.3s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 0.5s
//heron/api/tests/scala:api-scala-test                                   PASSED in 1.5s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (1.5s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 1.2s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.1s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.1s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.1s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.1s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.1s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.5s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.0s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.1s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.2s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.1s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 0.5s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.3s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 0.4s
//heron/common/tests/java:EchoTest                                       PASSED in 0.8s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 1.5s
//heron/common/tests/java:HeronServerTest                                PASSED in 1.6s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.3s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 5.2s
//heron/common/tests/java:SystemConfigTest                               PASSED in 0.5s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.5s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.3s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 1.0s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 1.0s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 0.5s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.5s
//heron/executor/tests/python:executor_unittest                          PASSED in 1.5s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 1.0s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 0.7s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 0.8s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 0.8s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 1.2s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 1.1s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 1.1s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 1.0s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 0.8s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 1.5s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 1.6s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 1.6s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 1.3s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 1.3s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 0.8s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 0.8s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 0.8s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 0.9s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 2.6s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.6s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 0.6s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 0.6s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 0.6s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.3s
//heron/instance/tests/java:HandleReadTest                               PASSED in 1.0s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 5.6s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.3s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 2.8s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 2.5s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 3.2s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.3s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.2s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.2s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.1s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 1.3s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.2s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 1.2s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 1.3s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.2s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.4s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 1.5s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.1s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.1s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 1.2s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 1.1s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 0.6s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 0.6s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 0.5s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 0.5s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.7s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 9.5s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 0.6s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 0.5s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 0.6s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 0.6s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.6s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 0.9s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 0.8s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.6s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.5s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 0.9s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 0.8s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.3s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 1.4s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 0.7s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 1.8s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 2.8s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.8s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 3.4s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 2.9s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 2.2s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 6.9s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 1.4s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 2.3s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 1.9s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 2.3s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 3.6s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 13.0s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 1.2s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 0.7s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 2.1s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 5.2s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 2.3s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 3.3s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.6s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.6s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 1.7s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 1.5s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 0.9s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 0.7s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 1.7s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 0.5s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 2.1s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 1.6s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 0.9s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 1.0s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 0.8s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 1.2s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 3.5s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 1.9s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 1.7s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 1.4s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.4s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 0.6s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 2.2s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.4s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 0.8s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 0.8s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.5s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 1.1s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 1.2s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 0.5s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.3s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.4s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 0.5s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 0.9s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.3s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 1.0s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 3.5s
//heron/spi/tests/java:ConfigTest                                        PASSED in 2.5s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.4s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 0.4s
//heron/spi/tests/java:KeysTest                                          PASSED in 0.3s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.4s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.3s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 3.2s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.3s
//heron/spi/tests/java:ResourceTest                                      PASSED in 0.8s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 1.6s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.3s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 0.6s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 4.2s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 2.3s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 2.0s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.0s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 0.7s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 2.1s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 1.2s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.2s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 1.2s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.2s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.7s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.7s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stmgr_unittest                            PASSED in 36.1s
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.0s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.7s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.1s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.0s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.0s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.1s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 26.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.5s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 1.6s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.3s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.1s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.3s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.6s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.5s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.4s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 2.0s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 1.0s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.7s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 0.5s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 0.7s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.5s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.5s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 0.4s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 1.4s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 0.5s

INFO: Build completed successfully, 235 total actions
Fourth run (clean build and test) 
INFO: Elapsed time: 1935.743s, Critical Path: 148.47s
INFO: 5628 processes: 2424 internal, 3204 local.
INFO: Build completed successfully, 5628 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.0s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.4s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.3s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.3s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 0.6s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 2.2s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 0.5s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 0.5s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 0.4s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.3s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 0.5s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 0.5s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 0.4s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 0.3s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.4s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.4s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.5s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 0.7s
//heron/api/tests/scala:api-scala-test                                   PASSED in 1.2s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (1.2s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 0.7s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.0s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.0s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.0s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.1s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.0s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.1s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.0s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.0s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.3s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.1s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.0s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.3s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.0s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 0.4s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.3s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 0.7s
//heron/common/tests/java:EchoTest                                       PASSED in 0.7s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 2.0s
//heron/common/tests/java:HeronServerTest                                PASSED in 2.5s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.5s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 5.1s
//heron/common/tests/java:SystemConfigTest                               PASSED in 0.4s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.4s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.3s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 1.1s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 1.3s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 0.5s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.5s
//heron/executor/tests/python:executor_unittest                          PASSED in 1.9s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 0.6s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 0.6s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 0.5s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 0.8s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 0.5s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 0.8s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 1.0s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 0.5s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 0.6s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 0.8s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 0.8s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 0.7s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 0.7s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 0.8s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 0.7s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 0.7s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 0.5s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 0.8s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 2.5s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.6s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 1.2s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 0.7s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 0.6s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.5s
//heron/instance/tests/java:HandleReadTest                               PASSED in 0.7s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 5.6s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.3s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 2.7s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 2.6s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 3.3s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.0s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.2s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.3s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.1s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 1.1s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.3s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 1.2s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 1.1s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.0s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.2s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 1.1s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.1s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.1s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 1.3s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 1.1s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 0.6s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 0.5s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 0.4s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 0.4s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 0.7s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.5s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 9.5s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 0.8s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 0.5s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 0.5s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 0.6s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.6s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 0.8s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 0.9s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.4s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.4s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 0.9s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 0.5s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.3s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 2.6s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 0.5s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 1.4s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 1.8s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.4s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 3.8s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 4.3s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 2.1s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 3.7s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 0.5s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 2.1s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 2.3s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 1.6s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 1.7s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 12.8s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 0.7s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 0.3s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 0.8s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 4.2s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 1.7s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 1.9s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.4s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.5s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 1.2s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 0.8s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 0.5s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 0.8s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 1.5s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 1.1s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 2.4s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 1.1s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 0.6s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 1.0s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 0.4s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 1.0s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 2.9s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 1.4s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 1.1s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 1.6s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.4s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 0.4s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 1.9s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.6s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 0.7s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 0.5s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.5s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 0.9s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 0.6s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 0.3s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.3s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.6s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 0.5s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 0.8s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.3s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 0.6s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 1.5s
//heron/spi/tests/java:ConfigTest                                        PASSED in 1.6s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.4s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 0.3s
//heron/spi/tests/java:KeysTest                                          PASSED in 0.3s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.3s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.3s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 2.5s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.6s
//heron/spi/tests/java:ResourceTest                                      PASSED in 0.4s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 1.9s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.5s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 0.8s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 3.0s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 1.9s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 1.7s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.0s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 0.6s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 1.3s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 1.5s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.1s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 1.1s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.0s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.3s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.3s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stmgr_unittest                            PASSED in 35.3s
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.0s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.0s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.0s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.7s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.0s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.0s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.1s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.0s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 36.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.5s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 0.7s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.7s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.1s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.2s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.6s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.3s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.3s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 1.5s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 0.7s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.5s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 0.6s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 0.9s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.4s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.4s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 0.4s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 1.3s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 0.5s

@surahman
[Stmgr] <Unit Test>
9afe578 
Potential fix for (StMgr, test_spout_death_under_backpressure) by setting high and low watermarks for back pressure and increasing Spouts to ramp up pressure faster.
@surahman
Merge branch '3733-CI-Stmgr-Unittest-dev' into 3733-CI-Stmgr-Unittest
8ab8eee
@surahman surahman force-pushed the 3733-CI-Stmgr-Unittest branch from e100f9a to 8ab8eee Compare November 13, 2021 23:11
@thinker0
Copy link
Member

@surahman
After the Pex upgrade, the location of test.conf changes, so it seems to be necessary to change the location of the conf file in the test code.

@surahman
Copy link
Member Author

@thinker0 Thank you! I see the following two references to test.conf:

integration_test/src/python/local_test_runner/BUILD:        "resources/test.conf",
integration_test/src/python/local_test_runner/main.py:DEFAULT_TEST_CONF_FILE = "integration_test/src/python/local_test_runner/resources/test.conf"

Are we supposed to update to the following?

integration_test/src/python/local_test_runner/main.py:DEFAULT_TEST_CONF_FILE = "resources/test.conf"

@thinker0
Copy link
Member

It worked fine in package, but it seems that the code and resource location are different when pex is executed.

 % ./bazel-bin/integration_test/src/python/local_test_runner/local-test-runner
Traceback (most recent call last):
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 679, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/.bootstrap/pex/pex.py", line 691, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/local/Cellar/python@3.8/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/Cellar/python@3.8/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/Cellar/python@3.8/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/integration_test/src/python/local_test_runner/main.py", line 180, in <module>
    main()
  File "/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/integration_test/src/python/local_test_runner/main.py", line 115, in main
    conf_string = pkgutil.get_data(__name__, conf_file).decode()
  File "/usr/local/Cellar/python@3.8/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pkgutil.py", line 638, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 1032, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/Users/thinker0/.pex/unzipped_pexes/711d1b3191bfc9248ac2f22a0e8bd8a6c5687d01/integration_test/src/python/local_test_runner/integration_test/src/python/local_test_runner/resources/test.conf'
thinker0@house3s-iMac heron % find /Users/thinker0/.pex/ -name test.conf -ls
13001283648        8 -rw-r-----    1 thinker0         staff                 471 Nov 14 13:12 /Users/thinker0/.pex//user_code/068ce22b8f5f9a55a36f8b5366021bc93c943fcd/integration_test/src/python/local_test_runner/resources/test.conf

@nicknezis
Copy link
Contributor

This is super impressive. I'd love to see how you setup/used gdb for the debugging.

@surahman
Copy link
Member Author

surahman commented Nov 14, 2021 •
edited
Loading

Thank you for your kind words. The updates do not appear to be aggressive enough for Travis CI and it is now a calibration task. I am restricting the CPU in Docker to 1.5 to see if I can isolate it locally.

My advice would be to use Eclipse because it supports pretty much every language. I am not going to install Eclipse and go through the process of configuring everything.

  • Configure the build with the -g debug flag in config/configure.ac.
  • Launch the heron-dev docker container.
  • docker ps and then docker exec -it to open a second terminal into the heron-dev.
  • Install gdb attach and set your breakpoint -b and watchpoints in your tests.
  • Run the stmg_unittest in isolation.
  • Good bug hunting.

EDIT: I do not like referencing Stackoverflow but this too.

Docker 1.5 CPU build and test 
INFO: Elapsed time: 3726.921s, Critical Path: 514.36s
INFO: 5243 processes: 2403 internal, 2840 local.
INFO: Build completed successfully, 5243 total actions
//heron/api/tests/cpp:serialization_unittest                             PASSED in 0.0s
//heron/api/tests/java:BaseWindowedBoltTest                              PASSED in 0.8s
//heron/api/tests/java:ConfigTest                                        PASSED in 0.5s
//heron/api/tests/java:CountStatAndMetricTest                            PASSED in 0.3s
//heron/api/tests/java:GeneralReduceByKeyAndWindowOperatorTest           PASSED in 1.2s
//heron/api/tests/java:HeronSubmitterTest                                PASSED in 6.4s
//heron/api/tests/java:JoinOperatorTest                                  PASSED in 2.4s
//heron/api/tests/java:KVStreamletShadowTest                             PASSED in 1.0s
//heron/api/tests/java:KeyByOperatorTest                                 PASSED in 1.1s
//heron/api/tests/java:LatencyStatAndMetricTest                          PASSED in 0.6s
//heron/api/tests/java:ReduceByKeyAndWindowOperatorTest                  PASSED in 2.3s
//heron/api/tests/java:StreamletImplTest                                 PASSED in 1.8s
//heron/api/tests/java:StreamletShadowTest                               PASSED in 1.0s
//heron/api/tests/java:StreamletUtilsTest                                PASSED in 1.5s
//heron/api/tests/java:UtilsTest                                         PASSED in 0.7s
//heron/api/tests/java:WaterMarkEventGeneratorTest                       PASSED in 0.9s
//heron/api/tests/java:WindowManagerTest                                 PASSED in 0.6s
//heron/api/tests/java:WindowedBoltExecutorTest                          PASSED in 2.2s
//heron/api/tests/scala:api-scala-test                                   PASSED in 4.9s
  WARNING: //heron/api/tests/scala:api-scala-test: Test execution time (4.9s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/ckptmgr/tests/java:CheckpointManagerServerTest                   PASSED in 1.5s
//heron/common/tests/cpp/basics:fileutils_unittest                       PASSED in 0.1s
//heron/common/tests/cpp/basics:rid_unittest                             PASSED in 0.1s
//heron/common/tests/cpp/basics:strutils_unittest                        PASSED in 0.0s
//heron/common/tests/cpp/basics:utils_unittest                           PASSED in 0.1s
//heron/common/tests/cpp/config:topology-config-helper_unittest          PASSED in 0.0s
//heron/common/tests/cpp/errors:errors_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/errors:module_unittest                          PASSED in 0.2s
//heron/common/tests/cpp/errors:syserrs_unittest                         PASSED in 0.1s
//heron/common/tests/cpp/metrics:count-metric_unittest                   PASSED in 0.0s
//heron/common/tests/cpp/metrics:mean-metric_unittest                    PASSED in 0.2s
//heron/common/tests/cpp/metrics:multi-count-metric_unittest             PASSED in 0.1s
//heron/common/tests/cpp/metrics:multi-mean-metric_unittest              PASSED in 0.1s
//heron/common/tests/cpp/metrics:time-spent-metric_unittest              PASSED in 1.4s
//heron/common/tests/cpp/network:http_unittest                           PASSED in 0.2s
//heron/common/tests/cpp/network:order_unittest                          PASSED in 0.1s
//heron/common/tests/cpp/network:packet_unittest                         PASSED in 0.1s
//heron/common/tests/cpp/network:piper_unittest                          PASSED in 2.0s
//heron/common/tests/cpp/network:rate_limit_unittest                     PASSED in 4.1s
//heron/common/tests/cpp/network:switch_unittest                         PASSED in 0.4s
//heron/common/tests/cpp/threads:spcountdownlatch_unittest               PASSED in 2.1s
//heron/common/tests/java:ByteAmountTest                                 PASSED in 1.3s
//heron/common/tests/java:CommunicatorTest                               PASSED in 0.5s
//heron/common/tests/java:ConfigReaderTest                               PASSED in 1.7s
//heron/common/tests/java:EchoTest                                       PASSED in 1.7s
//heron/common/tests/java:FileUtilsTest                                  PASSED in 4.0s
//heron/common/tests/java:HeronServerTest                                PASSED in 2.2s
//heron/common/tests/java:PackageTypeTest                                PASSED in 0.6s
//heron/common/tests/java:SysUtilsTest                                   PASSED in 7.0s
//heron/common/tests/java:SystemConfigTest                               PASSED in 1.0s
//heron/common/tests/java:TopologyUtilsTest                              PASSED in 0.7s
//heron/common/tests/java:WakeableLooperTest                             PASSED in 1.6s
//heron/common/tests/python/pex_loader:pex_loader_unittest               PASSED in 1.4s
//heron/downloaders/tests/java:DLDownloaderTest                          PASSED in 3.1s
//heron/downloaders/tests/java:ExtractorTests                            PASSED in 1.1s
//heron/downloaders/tests/java:RegistryTest                              PASSED in 0.7s
//heron/executor/tests/python:executor_unittest                          PASSED in 2.6s
//heron/healthmgr/tests/java:BackPressureDetectorTest                    PASSED in 3.7s
//heron/healthmgr/tests/java:BackPressureSensorTest                      PASSED in 1.7s
//heron/healthmgr/tests/java:BufferSizeSensorTest                        PASSED in 1.4s
//heron/healthmgr/tests/java:DataSkewDiagnoserTest                       PASSED in 3.6s
//heron/healthmgr/tests/java:ExecuteCountSensorTest                      PASSED in 1.4s
//heron/healthmgr/tests/java:GrowingWaitQueueDetectorTest                PASSED in 1.7s
//heron/healthmgr/tests/java:HealthManagerTest                           PASSED in 2.7s
//heron/healthmgr/tests/java:HealthPolicyConfigReaderTest                PASSED in 1.1s
//heron/healthmgr/tests/java:LargeWaitQueueDetectorTest                  PASSED in 0.9s
//heron/healthmgr/tests/java:MetricsCacheMetricsProviderTest             PASSED in 1.3s
//heron/healthmgr/tests/java:PackingPlanProviderTest                     PASSED in 2.0s
//heron/healthmgr/tests/java:ProcessingRateSkewDetectorTest              PASSED in 1.4s
//heron/healthmgr/tests/java:ScaleUpResolverTest                         PASSED in 1.8s
//heron/healthmgr/tests/java:SlowInstanceDiagnoserTest                   PASSED in 1.9s
//heron/healthmgr/tests/java:UnderProvisioningDiagnoserTest              PASSED in 1.3s
//heron/healthmgr/tests/java:WaitQueueSkewDetectorTest                   PASSED in 2.1s
//heron/instance/tests/java:ActivateDeactivateTest                       PASSED in 1.0s
//heron/instance/tests/java:BoltInstanceTest                             PASSED in 1.0s
//heron/instance/tests/java:BoltStatefulInstanceTest                     PASSED in 3.1s
//heron/instance/tests/java:ConnectTest                                  PASSED in 0.8s
//heron/instance/tests/java:CustomGroupingTest                           PASSED in 1.8s
//heron/instance/tests/java:EmitDirectBoltTest                           PASSED in 1.7s
//heron/instance/tests/java:EmitDirectSpoutTest                          PASSED in 1.8s
//heron/instance/tests/java:GlobalMetricsTest                            PASSED in 0.6s
//heron/instance/tests/java:HandleReadTest                               PASSED in 1.3s
//heron/instance/tests/java:HandleWriteTest                              PASSED in 6.7s
//heron/instance/tests/java:MultiAssignableMetricTest                    PASSED in 0.5s
//heron/instance/tests/java:SpoutInstanceTest                            PASSED in 3.3s
//heron/instance/tests/java:SpoutStatefulInstanceTest                    PASSED in 3.3s
//heron/instance/tests/python/network:event_looper_unittest              PASSED in 4.7s
//heron/instance/tests/python/network:gateway_looper_unittest            PASSED in 11.5s
//heron/instance/tests/python/network:heron_client_unittest              PASSED in 1.4s
//heron/instance/tests/python/network:metricsmgr_client_unittest         PASSED in 1.4s
//heron/instance/tests/python/network:protocol_unittest                  PASSED in 1.8s
//heron/instance/tests/python/network:st_stmgrcli_unittest               PASSED in 2.6s
//heron/instance/tests/python/utils:communicator_unittest                PASSED in 1.6s
//heron/instance/tests/python/utils:custom_grouping_unittest             PASSED in 2.0s
//heron/instance/tests/python/utils:global_metrics_unittest              PASSED in 2.2s
//heron/instance/tests/python/utils:log_unittest                         PASSED in 1.7s
//heron/instance/tests/python/utils:metrics_helper_unittest              PASSED in 1.4s
//heron/instance/tests/python/utils:outgoing_tuple_helper_unittest       PASSED in 2.6s
//heron/instance/tests/python/utils:pplan_helper_unittest                PASSED in 1.4s
//heron/instance/tests/python/utils:py_metrics_unittest                  PASSED in 1.4s
//heron/instance/tests/python/utils:topology_context_impl_unittest       PASSED in 2.7s
//heron/instance/tests/python/utils:tuple_helper_unittest                PASSED in 2.4s
//heron/io/dlog/tests/java:DLInputStreamTest                             PASSED in 1.2s
//heron/io/dlog/tests/java:DLOutputStreamTest                            PASSED in 1.3s
//heron/metricscachemgr/tests/java:CacheCoreTest                         PASSED in 1.4s
//heron/metricscachemgr/tests/java:MetricsCacheQueryUtilsTest            PASSED in 1.0s
//heron/metricscachemgr/tests/java:MetricsCacheTest                      PASSED in 1.2s
//heron/metricsmgr/tests/java:FileSinkTest                               PASSED in 1.0s
//heron/metricsmgr/tests/java:HandleTManagerLocationTest                 PASSED in 0.8s
//heron/metricsmgr/tests/java:MetricsCacheSinkTest                       PASSED in 10.0s
//heron/metricsmgr/tests/java:MetricsManagerServerTest                   PASSED in 1.9s
//heron/metricsmgr/tests/java:MetricsUtilTests                           PASSED in 1.0s
//heron/metricsmgr/tests/java:PrometheusSinkTests                        PASSED in 1.9s
//heron/metricsmgr/tests/java:SinkExecutorTest                           PASSED in 1.0s
//heron/metricsmgr/tests/java:TManagerSinkTest                           PASSED in 9.9s
//heron/metricsmgr/tests/java:WebSinkTest                                PASSED in 1.2s
//heron/packing/tests/java:FirstFitDecreasingPackingTest                 PASSED in 1.9s
//heron/packing/tests/java:PackingPlanBuilderTest                        PASSED in 0.6s
//heron/packing/tests/java:PackingUtilsTest                              PASSED in 0.7s
//heron/packing/tests/java:ResourceCompliantRRPackingTest                PASSED in 4.1s
//heron/packing/tests/java:RoundRobinPackingTest                         PASSED in 2.1s
//heron/packing/tests/java:ScorerTest                                    PASSED in 0.6s
//heron/scheduler-core/tests/java:HttpServiceSchedulerClientTest         PASSED in 4.6s
//heron/scheduler-core/tests/java:JsonFormatterUtilsTest                 PASSED in 1.0s
//heron/scheduler-core/tests/java:LaunchRunnerTest                       PASSED in 4.4s
//heron/scheduler-core/tests/java:LauncherUtilsTest                      PASSED in 7.0s
//heron/scheduler-core/tests/java:LibrarySchedulerClientTest             PASSED in 0.9s
//heron/scheduler-core/tests/java:RuntimeManagerMainTest                 PASSED in 9.9s
//heron/scheduler-core/tests/java:RuntimeManagerRunnerTest               PASSED in 7.8s
//heron/scheduler-core/tests/java:SchedulerClientFactoryTest             PASSED in 3.4s
//heron/scheduler-core/tests/java:SchedulerMainTest                      PASSED in 14.0s
//heron/scheduler-core/tests/java:SchedulerServerTest                    PASSED in 0.9s
//heron/scheduler-core/tests/java:SchedulerUtilsTest                     PASSED in 5.2s
//heron/scheduler-core/tests/java:SubmitDryRunRenderTest                 PASSED in 8.9s
//heron/scheduler-core/tests/java:SubmitterMainTest                      PASSED in 7.1s
//heron/scheduler-core/tests/java:UpdateDryRunRenderTest                 PASSED in 7.1s
//heron/scheduler-core/tests/java:UpdateTopologyManagerTest              PASSED in 15.7s
//heron/schedulers/tests/java:AuroraCLIControllerTest                    PASSED in 0.9s
//heron/schedulers/tests/java:AuroraContextTest                          PASSED in 1.1s
//heron/schedulers/tests/java:AuroraLauncherTest                         PASSED in 1.9s
//heron/schedulers/tests/java:AuroraSchedulerTest                        PASSED in 7.5s
//heron/schedulers/tests/java:HeronExecutorTaskTest                      PASSED in 4.4s
//heron/schedulers/tests/java:HeronMasterDriverTest                      PASSED in 7.1s
//heron/schedulers/tests/java:KubernetesContextTest                      PASSED in 0.7s
//heron/schedulers/tests/java:KubernetesControllerTest                   PASSED in 0.8s
//heron/schedulers/tests/java:KubernetesLauncherTest                     PASSED in 2.3s
//heron/schedulers/tests/java:KubernetesSchedulerTest                    PASSED in 1.5s
//heron/schedulers/tests/java:KubernetesUtilsTest                        PASSED in 1.2s
//heron/schedulers/tests/java:LaunchableTaskTest                         PASSED in 1.3s
//heron/schedulers/tests/java:LocalLauncherTest                          PASSED in 5.6s
//heron/schedulers/tests/java:LocalSchedulerTest                         PASSED in 1.2s
//heron/schedulers/tests/java:MarathonControllerTest                     PASSED in 6.5s
//heron/schedulers/tests/java:MarathonLauncherTest                       PASSED in 2.1s
//heron/schedulers/tests/java:MarathonSchedulerTest                      PASSED in 1.2s
//heron/schedulers/tests/java:MesosFrameworkTest                         PASSED in 2.5s
//heron/schedulers/tests/java:MesosLauncherTest                          PASSED in 1.5s
//heron/schedulers/tests/java:MesosSchedulerTest                         PASSED in 2.3s
//heron/schedulers/tests/java:NomadSchedulerTest                         PASSED in 10.2s
//heron/schedulers/tests/java:SlurmControllerTest                        PASSED in 4.0s
//heron/schedulers/tests/java:SlurmLauncherTest                          PASSED in 7.0s
//heron/schedulers/tests/java:SlurmSchedulerTest                         PASSED in 5.2s
//heron/schedulers/tests/java:TaskResourcesTest                          PASSED in 0.8s
//heron/schedulers/tests/java:TaskUtilsTest                              PASSED in 1.1s
//heron/schedulers/tests/java:V1ControllerTest                           PASSED in 6.2s
//heron/schedulers/tests/java:VolumesTests                               PASSED in 0.7s
//heron/schedulers/tests/java:YarnLauncherTest                           PASSED in 3.2s
//heron/schedulers/tests/java:YarnSchedulerTest                          PASSED in 6.1s
//heron/simulator/tests/java:AllGroupingTest                             PASSED in 0.7s
//heron/simulator/tests/java:CustomGroupingTest                          PASSED in 1.2s
//heron/simulator/tests/java:FieldsGroupingTest                          PASSED in 4.0s
//heron/simulator/tests/java:InstanceExecutorTest                        PASSED in 2.5s
//heron/simulator/tests/java:LowestGroupingTest                          PASSED in 1.1s
//heron/simulator/tests/java:RotatingMapTest                             PASSED in 0.5s
//heron/simulator/tests/java:ShuffleGroupingTest                         PASSED in 0.8s
//heron/simulator/tests/java:SimulatorTest                               PASSED in 3.0s
//heron/simulator/tests/java:TopologyManagerTest                         PASSED in 1.7s
//heron/simulator/tests/java:TupleCacheTest                              PASSED in 0.7s
//heron/simulator/tests/java:XORManagerTest                              PASSED in 1.4s
//heron/spi/tests/java:ConfigLoaderTest                                  PASSED in 5.9s
//heron/spi/tests/java:ConfigTest                                        PASSED in 3.2s
//heron/spi/tests/java:ContextTest                                       PASSED in 0.8s
//heron/spi/tests/java:ExceptionInfoTest                                 PASSED in 1.1s
//heron/spi/tests/java:KeysTest                                          PASSED in 1.2s
//heron/spi/tests/java:MetricsInfoTest                                   PASSED in 0.8s
//heron/spi/tests/java:MetricsRecordTest                                 PASSED in 0.5s
//heron/spi/tests/java:NetworkUtilsTest                                  PASSED in 5.2s
//heron/spi/tests/java:PackingPlanTest                                   PASSED in 0.8s
//heron/spi/tests/java:ResourceTest                                      PASSED in 1.1s
//heron/spi/tests/java:ShellUtilsTest                                    PASSED in 6.3s
//heron/spi/tests/java:TokenSubTest                                      PASSED in 0.6s
//heron/spi/tests/java:UploaderUtilsTest                                 PASSED in 2.0s
//heron/statefulstorages/tests/java:DlogStorageTest                      PASSED in 10.2s
//heron/statefulstorages/tests/java:HDFSStorageTest                      PASSED in 7.6s
//heron/statefulstorages/tests/java:LocalFileSystemStorageTest           PASSED in 4.1s
//heron/statemgrs/tests/cpp:zk-statemgr_unittest                         PASSED in 0.2s
//heron/statemgrs/tests/java:CuratorStateManagerTest                     PASSED in 2.4s
//heron/statemgrs/tests/java:LocalFileSystemStateManagerTest             PASSED in 3.3s
//heron/statemgrs/tests/java:ZkUtilsTest                                 PASSED in 7.2s
//heron/statemgrs/tests/python:configloader_unittest                     PASSED in 1.3s
//heron/statemgrs/tests/python:statemanagerfactory_unittest              PASSED in 2.5s
//heron/statemgrs/tests/python:zkstatemanager_unittest                   PASSED in 1.5s
//heron/stmgr/tests/cpp/grouping:all-grouping_unittest                   PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:custom-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:fields-grouping_unittest                PASSED in 0.1s
//heron/stmgr/tests/cpp/grouping:lowest-grouping_unittest                PASSED in 0.0s
//heron/stmgr/tests/cpp/grouping:shuffle-grouping_unittest               PASSED in 0.0s
//heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest               PASSED in 1.7s
  WARNING: //heron/stmgr/tests/cpp/server:checkpoint-gateway_unittest: Test execution time (1.7s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stateful-restorer_unittest                PASSED in 0.2s
  WARNING: //heron/stmgr/tests/cpp/server:stateful-restorer_unittest: Test execution time (0.2s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/server:stmgr_unittest                            PASSED in 43.6s
//heron/stmgr/tests/cpp/util:neighbour_calculator_unittest               PASSED in 0.1s
  WARNING: //heron/stmgr/tests/cpp/util:neighbour_calculator_unittest: Test execution time (0.1s excluding execution overhead) outside of range for MODERATE tests. Consider setting timeout="short" or size="small".
//heron/stmgr/tests/cpp/util:rotating-map_unittest                       PASSED in 0.1s
//heron/stmgr/tests/cpp/util:tuple-cache_unittest                        PASSED in 3.7s
//heron/stmgr/tests/cpp/util:xor-manager_unittest                        PASSED in 4.0s
//heron/tmanager/tests/cpp/server:stateful_checkpointer_unittest         PASSED in 0.1s
//heron/tmanager/tests/cpp/server:stateful_restorer_unittest             PASSED in 5.1s
//heron/tmanager/tests/cpp/server:tcontroller_unittest                   PASSED in 0.1s
//heron/tmanager/tests/cpp/server:tmanager_unittest                      PASSED in 27.2s
//heron/tools/apiserver/tests/java:ConfigUtilsTests                      PASSED in 0.9s
//heron/tools/apiserver/tests/java:TopologyResourceTests                 PASSED in 1.8s
//heron/tools/cli/tests/python:client_command_unittest                   PASSED in 1.6s
//heron/tools/cli/tests/python:opts_unittest                             PASSED in 1.3s
//heron/tools/explorer/tests/python:explorer_unittest                    PASSED in 1.8s
//heron/tools/tracker/tests/python:query_operator_unittest               PASSED in 1.9s
//heron/tools/tracker/tests/python:query_unittest                        PASSED in 1.7s
//heron/tools/tracker/tests/python:topology_unittest                     PASSED in 1.6s
//heron/tools/tracker/tests/python:tracker_unittest                      PASSED in 2.5s
//heron/uploaders/tests/java:DlogUploaderTest                            PASSED in 1.6s
//heron/uploaders/tests/java:GcsUploaderTests                            PASSED in 0.9s
//heron/uploaders/tests/java:HdfsUploaderTest                            PASSED in 1.1s
//heron/uploaders/tests/java:HttpUploaderTest                            PASSED in 2.0s
//heron/uploaders/tests/java:LocalFileSystemConfigTest                   PASSED in 0.6s
//heron/uploaders/tests/java:LocalFileSystemContextTest                  PASSED in 0.6s
//heron/uploaders/tests/java:LocalFileSystemUploaderTest                 PASSED in 1.6s
//heron/uploaders/tests/java:S3UploaderTest                              PASSED in 3.1s
//heron/uploaders/tests/java:ScpUploaderTest                             PASSED in 1.5s

@surahman
Copy link
Member Author

surahman commented Nov 14, 2021 •
edited
Loading

@thinker0 we have more PEX related errors, any insights? I have no experience with PEX or this part of the code base.

===> Starting heron integration_test scala at 2021-11-14 17:55:18
[2021-11-14 17:55:19 +0000] [INFO]: Starting server at port 8080
Traceback (most recent call last):
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 679, in execute_entry
    return self.execute_module(entry_point, alter_sys)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/.bootstrap/pex/pex.py", line 691, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/integration_test/src/python/test_runner/main.py", line 479, in <module>
    main()
  File "/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/integration_test/src/python/test_runner/main.py", line 428, in main
    conf_string = pkgutil.get_data(__name__, conf_file).decode()
  File "/usr/lib/python3.6/pkgutil.py", line 634, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 832, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/.pex/unzipped_pexes/d5bd60b574982804743c6708dcbf1c059d9b38f3/integration_test/src/python/test_runner/integration_test/src/python/test_runner/resources/test.json'

From this it appears to be conf related:

    conf_string = pkgutil.get_data(__name__, conf_file).decode()
  File "/usr/lib/python3.6/pkgutil.py", line 634, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 832, in get_data
FileNotFoundError: [Errno 2] No such file or directory:

On the upside, it seems to have run for 119mins (~2hrs) so maybe the settings for the stmgr_unittest are more aggressive enough... Let us see.

Edit: I think I got it.

integration_test/src/python/test_runner/main.py:DEFAULT_TEST_CONF_FILE = "integration_test/src/python/test_runner/resources/test.json"
integration_test/src/python/topology_test_runner/main.py:DEFAULT_TEST_CONF_FILE = "integration_test/src/python/topology_test_runner/resources/test.json"

@surahman
[Integration Test] <Python>
c19c300 
Updating <DEFAULT_TEST_CONF_FILE> to new path.
@surahman
Copy link
Member Author

Tally-ho folks. I think I have fixed the stmgr_unittest.

Current build failure issue:

[2021-11-14 21:02:56 +0000] [INFO]: Total integration topology test time = 731s
[2021-11-14 21:02:56 +0000] [ERROR]: FAILURE: 1/6 tests failed:
[2021-11-14 21:02:56 +0000] [ERROR]:   - [504s]  : 20211114205045_IntegrationTopologyTest_FieldsGrouping_7e1ba576-9264-4df2-bfda-57af1da884db
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

@surahman surahman force-pushed the 3733-CI-Stmgr-Unittest branch from dee6001 to 69772e7 Compare November 14, 2021 21:39
@surahman
Copy link
Member Author

@joshfischer1108 could you please review and merge?

Both parallel runs 1 and 2 have passed and the CI pipeline is now fixed 🎆.

I do not have permission to link issues to PRs but issue #3733 is the issue referencing this PR.

@thinker0
Copy link
Member

+1

@surahman
Copy link
Member Author

@thinker0 thank you for your help with the PEX issue 😄

@nicknezis nicknezis self-requested a review November 15, 2021 06:35
@nicknezis nicknezis linked an issue Nov 15, 2021 that may be closed by this pull request
CI Pipeline Failures Involving the <stmgr_unittest>. #3733
Closed
@nicknezis nicknezis merged commit b295bf5 into apache:master Nov 15, 2021
@surahman
Copy link
Member Author

Thank you @nicknezis! 😄

surahman added a commit to surahman/incubator-heron that referenced this pull request Nov 15, 2021
@surahman
Fixed CI Pipeline Failures Involving the <stmgr_unittest> (apache#3735)
d454585
surahman added a commit to surahman/incubator-heron that referenced this pull request Nov 15, 2021
@surahman
Fixed CI Pipeline Failures Involving the <stmgr_unittest> (apache#3735)
6c91f7e
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@nicknezis nicknezis nicknezis approved these changes

Assignees
No one assigned
Labels
stmgr
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CI Pipeline Failures Involving the <stmgr_unittest>.
3 participants
@surahman @thinker0 @nicknezis