[BUG] When using timeout option(-t) of qualification tool, it does not print anything in output after timeout.

[viadea]

Describe the bug
A clear and concise description of what the bug is.
When using timeout option(-t) of qualification tool, it does not print anything in output after timeout.

ERROR Qualification: Processing log files took longer then 20 seconds, stopping processing any more event logs
INFO ToolTextFileWriter: CSV output location: /tmp/testqualification/rapids_4_spark_qualification_output/rapids_4_spark_qualification_output.csv
==========================================================
|App ID|App Duration|SQL DF Duration|Problematic Duration|
==========================================================
==========================================================

Steps/Code to reproduce bug
Please provide a list of steps or a code sample to reproduce the issue.
Avoid posting private or sensitive data.

export SPARK_HOME=/xxx/spark
java -cp /Users/xxx/github/spark-rapids/tools/target/rapids-4-spark-tools_2.12-21.08.0-SNAPSHOT.jar:$SPARK_HOME/jars/* com.nvidia.spark.rapids.tool.qualification.QualificationMain \
--num-threads 8 \
--order desc \
-o /tmp/testqualification/ \
-t 20 \
/Users/xxx/eventlog/tpcds/49/49

Expected behavior
A clear and concise description of what you expected to happen.
I expect after timeout, the output of the qualification tool can print the output for whatever logs processed.

Environment details (please complete the following information)

• Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
• Spark configuration settings related to the issue

21.08 snapshot qualification tool.
Spark 3.1.1

Additional context
Add any other context about the problem here.

[viadea]

Close this issue as per my tests. This is just slower execution of the qualification tool than last version.

[tgravescs]

using hthis to look into why its slower

[tgravescs]

the problem is here: bb8baab#diff-8190fc4fb61da36bdcb2aaff10beba7ed17a5bd08a5157d4b981765b9f36e605R92

Where we changed a foreach over list to be a while. The while loop is just 4-5x slower then doing hte list.foreach {}

I'm looking at changing back to a List operation