Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to Apply iDFlakies on apache/metron (Junit5) #22

Open
luoos opened this issue Dec 13, 2020 · 0 comments
Open

Fail to Apply iDFlakies on apache/metron (Junit5) #22

luoos opened this issue Dec 13, 2020 · 0 comments

Comments

@luoos
Copy link
Contributor

luoos commented Dec 13, 2020

apache/metron uses JUnit 5. iDFlakies/Testrunner fail to run most of its tests (some tests are successful).

Sample exceptions:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-parsers:0.7.2
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 131 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.parsers.bro.BasicBroParserTest#testX509BroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testConnBroMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testFourDayFutureBecomesPast(), org.apache.metron.parsers.bro.BasicBroParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.cef.CEFParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.SnortParserTest#testGoodMessage(), org.apache.metron.parsers.cef.CEFParserTest#testTimestampPriority(), org.apache.metron.parsers.SnortParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFdevTimeWithNoCustomFormat(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.YafParserTest#test(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0(), org.apache.metron.parsers.asa.BasicAsaParserTest#testCISCOFW106023(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#testParse(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureDefault(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogHostAndProg(), org.apache.metron.parsers.cef.CEFParserTest#testMissingYearFromDate(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testWeirdBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testNoPatternForTag(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseInvalidLogTypeMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSoftwareBroMessage(), org.apache.metron.parsers.SnortParserTest#throws_exception_on_bad_config_date_format(), org.apache.metron.parsers.bro.BasicBroParserTest#testDhcpBroMessage(), org.apache.metron.filters.FiltersTest#testSingleQueryFilter(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFcustomdevTimeFormat(), org.apache.metron.parsers.bro.BasicBroParserTest#testKnownDevicesBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedOtherLine(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserCyberArk(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig61CustomFields(), org.apache.metron.parsers.ise.BasicIseParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFMultiLine(), org.apache.metron.parsers.asa.BasicAsaParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic71(), org.apache.metron.parsers.asa.BasicAsaParserTest#testIp6Addr(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroMessage(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat80(), org.apache.metron.parsers.asa.BasicAsaParserTest#testCISCOFW106006(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseLogoutLine(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat71(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig70And80NoCustomFields(), org.apache.metron.parsers.bro.BasicBroParserTest#testBadMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testMissingYearFromDate(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseRBMLine(), org.apache.metron.parsers.cef.CEFParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.asa.BasicAsaParserTest#testUnexpectedMessage(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testRadiusBroMessageSuccess(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserAdallom(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic60(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#testParse(), org.apache.metron.parsers.leef.LEEFParserTest#testTimestampPriority(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureTimeZoneOffset(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedLogoutLine(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_1_0_versionIncluded(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0_delimiterSpecified(), org.apache.metron.parsers.bro.BasicBroParserTest#testFtpBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testUnwrappedBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testDpdBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testRfbBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testNoticeBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedRBMLine(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig70And80CustomFields(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#testParse(), org.apache.metron.parsers.cef.CEFParserTest#testPaloAltoCEF(), org.apache.metron.parsers.bro.BasicBroParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroMessageWithZeroDecimalTruncation(), org.apache.metron.parsers.SnortParserTest#throws_exception_on_bad_config_timezone(), org.apache.metron.parsers.bro.BasicBroParserTest#testSoftwareBroMessage2(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic80(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseConfig61NoCustomFields(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseTraffic70(), org.apache.metron.parsers.bro.BasicBroParserTest#testRadiusBroMessageFailed(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.bro.BasicBroParserTest#testFilesBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testDnsBroMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseOtherLine(), org.apache.metron.parsers.bro.BasicBroParserTest#testDecimalFormatAssumptions(), org.apache.metron.parsers.bro.BasicBroParserTest#testSmtpBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogIpHost(), org.apache.metron.parsers.lancope.BasicLancopeParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.bro.BasicBroParserTest#testHttpBroDecimalMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#testConfigureTimeZoneText(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat70(), org.apache.metron.parsers.bro.BasicBroParserTest#testSshBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSslBroMessage(), org.apache.metron.parsers.asa.BasicAsaParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseMalformedLoginLine(), org.apache.metron.parsers.asa.BasicAsaParserTest#testShortTimestamp(), org.apache.metron.parsers.sourcefire.BasicSourcefireParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.SnortParserTest#parses_timestamp_as_local_zone_by_default(), org.apache.metron.parsers.bro.BasicBroParserTest#testCaptureLossBroMessage(), org.apache.metron.parsers.leef.LEEFParserTest#testInvalid(), org.apache.metron.parsers.bro.BasicBroParserTest#testStatsBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testSipBroMessage(), org.apache.metron.parsers.cef.CEFParserTest#testEscaping(), org.apache.metron.parsers.cef.CEFParserTest#testFourDayFutureBecomesPast(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_2_0_delimiterUsedIncorrectly(), org.apache.metron.parsers.cef.CEFParserTest#testBasicHeader(), org.apache.metron.parsers.cef.CEFParserTest#testCustomLabelWithSpace(), org.apache.metron.parsers.SnortParserTest#testBadMessage(), org.apache.metron.parsers.websphere.GrokWebSphereParserTest#testParseLoginLine(), org.apache.metron.parsers.cef.CEFParserTest#testRtValueAsEpochTimestamp(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseSystem61(), org.apache.metron.parsers.asa.BasicAsaParserTest#testSyslogHost(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#getsReadCharsetFromDefault(), org.apache.metron.parsers.cef.CEFParserTest#testCEFParserWAF(), org.apache.metron.parsers.bro.BasicBroParserTest#testProtocolKeyCleanedUp(), org.apache.metron.parsers.SnortParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseThreat60(), org.apache.metron.parsers.ise.BasicIseParserTest#testParse(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEFParserSample(), org.apache.metron.parsers.leef.LEEFParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.asa.BasicAsaParserTest#testInvalidIpAddr(), org.apache.metron.parsers.ise.BasicIseParserTest#getsReadCharsetFromConfig(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseInvalidVersionMessage(), org.apache.metron.parsers.SnortParserTest#uses_configuration_to_parse(), org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest#testParseSystem80(), org.apache.metron.parsers.cef.CEFParserTest#testBasicExtensions(), org.apache.metron.parsers.leef.LEEFParserTest#testLEEF_CEFlikeSample(), org.apache.metron.parsers.cef.CEFParserTest#testInvalid(), org.apache.metron.parsers.SquidParserTest#test(), org.apache.metron.parsers.fireeye.BasicFireEyeParserTest#testTimestampParsing(), org.apache.metron.filters.FiltersTest#testDefault(), org.apache.metron.parsers.cef.CEFParserTest#testSuccessfulWhenCEFContainsJSON(), org.apache.metron.parsers.bro.BasicBroParserTest#testKnownCertsBroMessage(), org.apache.metron.parsers.bro.BasicBroParserTest#testBadMessageNonJson())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-pcap-backend:0.7.2
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 24 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#date_range_filters_out_all_results(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_pcaps_by_start_end_ns_with_fixed_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#fixed_filter_returns_all_results_for_full_date_range(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#protocol_filters_out_all_results_with_fixed_filter(), org.apache.metron.pcap.query.PcapCliTest#missing_start_time_arg_prints_error_and_help(), org.apache.metron.pcap.query.PcapCliTest#runs_query_pcap_filter_job_with_default_argument_list(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_pcaps_by_start_end_ns_with_empty_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_range_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#invalid_fixed_filter_arg_prints_help(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#query_filter_returns_all_results_for_full_date_range(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#ip_address_filters_out_all_results_with_fixed_filter(), org.apache.metron.spout.pcap.deserializer.FromKeyDeserializerTest#empty_or_null_key_throws_illegal_argument_exception(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_BYTEARRAY_MATCHER_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#invalid_query_filter_arg_prints_help(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_greater_than_value_with_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_with_fixed_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#testTimestampInPacket(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_default_argument_list(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#filters_results_by_dst_port_with_query_filter(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#ip_address_filters_out_all_results_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_full_argument_list(), org.apache.metron.pcap.query.PcapCliTest#runs_fixed_pcap_filter_job_with_full_argument_list_and_default_dateformat(), org.apache.metron.pcap.integration.PcapTopologyIntegrationTest#protocol_filters_out_all_results_with_query_filter(), org.apache.metron.pcap.query.PcapCliTest#runs_query_pcap_filter_job_with_full_argument_list())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)

ERROR (WRITE_ERROR_STDOUT_THROWABLE): org.apache.metron:metron-solr-storm:0.7.2
ava.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(org.apache.metron.indexing.integration.SolrIndexingIntegrationTest#test())
	at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
	at scala.util.Success.flatMap(Try.scala:247)
	at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
	at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
	at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
	at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
	at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
	at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.<init>(RandomDetector.java:26)
	at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@luoos