Intermittent failure on AuTest (lua_states_stats)

[masaori335]

https://ci.trafficserver.apache.org/job/autest-github/13379/console

Test: lua_states_stats: Failed
    File: lua_states_stats.test.py
    Directory: /var/jenkins/workspace/autest-github/src/tests/gold_tests/pluginTest/lua
   Starting Test lua_states_stats : No issues found - Passed
      Reason: Started!
   Process: server: Passed
     Setting up : Lamda - Passed
     Setting up : MakeDir - Passed
     Test : Checking that any test passes - Passed
        Reason: One or more tests passed
     
       Test : Checking that ReturnCode == None - Failed
          Reason: Returned Value 0 != None
       
     
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       
   Process: ts: Passed
     Setting up : MakeDir - Passed
     Setting up : Copy - Passed
     Setting up : MakeDir - Passed
     Setting up : Chown - Passed
     Setting up : Chown - Passed
     Setting up : CopyAs - Passed
     Setting up : CopyAs - Passed
     Setting up : CopyAs - Passed
     Setting up : Copy - Passed
     Setting up : MakeDir - Passed
     Setting up : Copy - Passed
     Setting up : MakeDir - Passed
     Setting up : Chown - Passed
     Setting up : Chown - Passed
     Setting up : MakeDir - Passed
     Setting up : Chown - Passed
     Setting up : Chown - Passed
     Setting up : MakeDir - Passed
     Setting up : MakeDir - Passed
     Setting up : Chown - Passed
     Setting up : Chown - Passed
     Setting up : MakeDir - Passed
     Setting up : Chown - Passed
     Setting up : Chown - Passed
     Setting up : Lamda - Passed
     Setting up : Lamda - Passed
     Setting up : Lamda - Passed
     Setting up : Lamda - Passed
     Test : Checking that ReturnCode == 2 - Passed
        Reason: Returned Value: 2 == 2
     file /var/tmp/ausb-6844.13379/lua_states_stats/_output/lua_states_stats-ts/stream.stderr.txt : Checking that /var/tmp/ausb-6844.13379/lua_states_stats/_output/lua_states_stats-ts/stream.stderr.txt matches gold/lifecycle.stderr.gold - Passed
        Reason: Values match
     file /var/tmp/ausb-6844.13379/lua_states_stats/ts/log/diags.log : diags.log should not contain errors - Passed
        Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/ts/log/diags.log excludes expression
     file /var/tmp/ausb-6844.13379/lua_states_stats/ts/log/diags.log : diags.log should not contain errors - Passed
        Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/ts/log/diags.log excludes expression
     Appending File /var/tmp/ausb-6844.13379/lua_states_stats/ts/config/records.config : Writing config file records.config - Passed
        Reason: Success
     Appending File /var/tmp/ausb-6844.13379/lua_states_stats/ts/config/remap.config : Appended file /var/tmp/ausb-6844.13379/lua_states_stats/ts/config/remap.config - Passed
        Reason: Success
     Appending File /var/tmp/ausb-6844.13379/lua_states_stats/ts/config/plugin.config : Appended file /var/tmp/ausb-6844.13379/lua_states_stats/ts/config/plugin.config - Passed
        Reason: Success
   Process: Default: Skipped
      Reason: Was not started
   Process: Default: Skipped
      Reason: Was not started
   Run: Lua states: Passed
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/0-tr-Default/stream.stdout.txt : expected 4 states - Passed
          Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/_output/0-tr-Default/stream.stdout.txt contained expression
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/0-tr-Default/stream.stdout.txt : expected 4 states - Passed
          Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/_output/0-tr-Default/stream.stdout.txt contained expression
   Run: Lua hello: Passed
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       Test : Checking that TotalRunTime < 5.0 - Passed
          Reason: Returned value: 3.021519422531128 < 5.0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/1-tr-Default/stream.stderr.txt : hello world content - Passed
          Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/_output/1-tr-Default/stream.stderr.txt contained expression
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       Test : Checking that TotalRunTime < 5.0 - Passed
          Reason: Returned value: 3.021519422531128 < 5.0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/1-tr-Default/stream.stderr.txt : hello world content - Passed
          Reason: Contents of /var/tmp/ausb-6844.13379/lua_states_stats/_output/1-tr-Default/stream.stderr.txt contained expression
   Run: Check for metrics: Failed
     Test : Checking that True == _isRunningAfter - Passed
        Reason: Returned Value: True == _isRunningAfter
     Starting TestRun 2-tr : No Issues found - Passed
        Reason: Started!
     Process: Default: Failed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt : Checking that /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt matches gold/metrics.stdout.gold - Failed
          Reason: File differences
           Gold File : /var/jenkins/workspace/autest-github/src/tests/gold_tests/pluginTest/lua/gold/metrics.stdout.gold
           Data File : /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt
           - plugin.lua.global.states ``
           - plugin.lua.global.gc_bytes ``
           - plugin.lua.global.threads ``
           - plugin.lua.remap.states ``
           - plugin.lua.remap.gc_bytes ``
           - plugin.lua.remap.threads ``
     Process: Default: Failed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
       file /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt : Checking that /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt matches gold/metrics.stdout.gold - Failed
          Reason: File differences
           Gold File : /var/jenkins/workspace/autest-github/src/tests/gold_tests/pluginTest/lua/gold/metrics.stdout.gold
           Data File : /var/tmp/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.stdout.txt
           - plugin.lua.global.states ``
           - plugin.lua.global.gc_bytes ``
           - plugin.lua.global.threads ``
           - plugin.lua.remap.states ``
           - plugin.lua.remap.gc_bytes ``
           - plugin.lua.remap.threads ``
   Run: Check for lifecycle stats: Passed
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0
     Process: Default: Passed
       Test : Checking that ReturnCode == 0 - Passed
          Reason: Returned Value: 0 == 0

FWIW, stream.all.txt of the 2-tr is empty.
https://ci.trafficserver.apache.org/autest/ausb-6844.13379/lua_states_stats/_output/2-tr-Default/stream.all.txt

[shinrich]

Failed again here https://ci.trafficserver.apache.org/job/autest-github/13557/console

Looks like the problem is that the metrics aren't present yet. Sometimes the 15 second delay is not sufficient. I don't think we currently have a means to detect that the stats has been pushed out for sure. Even waiting for the stat to show up may not be sufficient, since the state may be there but the final value may not be pushed yet. I disabled the remap_stats test a while back for this reason. I'll push up a PR to do the same here until the we figure out this technology.

[shinrich]

Looks like @traeak created this test, so I assigned him.

[masaori335]

8.1.x branch has the same issue.
https://ci.trafficserver.apache.org/view/8.1.x/job/autest-8.1.x/69/