Organize the benchmark output folder

[gavinlichn]

Description

For each benchmark test case, save benchmark result in same folder.

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Describe the tests that you ran to verify your changes.

[Zhenzhong1]

Why do we save benchmark result in same folder For each benchmark test case, ?

How do we know which outcome belongs to which cases?

I guess may affect the CD workflow when calculating the average latency for each test case. Please help to check~
@daisy-ycguo @chensuyue

[gavinlichn]

Why do we save benchmark result in same folder For each benchmark test case, ?

How do we know which outcome belongs to which cases?

For each case, will create one folder. For next case, will create another folder. So, we can identify the test case of the folder.

For current code, for each test case, will create MULTIPLE folders, one folder for each "user_queries" item.
For example, if plan to test the chatqna for 4, 8, 16, 32 queries, it will create 4 folders.

After applied my PR, it will create one folder, and put 4 queries' results into this folder, make it easy to compare and report.

I guess may affect the CD workflow when calculating the average latency for each test case. Please help to check~ @daisy-ycguo @chensuyue

[gavinlichn]

@chensuyue Can you help to confirm please? Does the CD flow depend on current folder structure or now.

[lvliang-intel]

@chensuyue Can you help to confirm please? Does the CD flow depend on current folder structure or now.

@chensuyue, please help to confirm.

[mkbhanda]

Concerned that the CICD or benchmark test machine will run out of space and what if we clean up the machine before copying over the results.
What if we pass in also a remove file storage location to move the benchmark results, one by one, after completion. In this scenario we would be creating temp files in the benchmark directory on the CICD or other test machine and deleting once we have copied over. If this remove location is passed in as an environment variable with access credentials, then that remove location could even be a github location with versioning.

[chensuyue]

The new benchmark workflow didn't use this evals/benchmark/benchmark.py anymore, I can merge this PR, but this will not covered by CD test.