Adds support for multi-options execution. #235
Conversation
| # Note. For future multi-execution options, initialize them as below and added them into option_configuration. | ||
| # initialize options that might show up multiple times | ||
| api = [*set(arguments['--api'])] if arguments['--api'] else [API.DEFAULT.value] | ||
| format_option = [*set(arguments['--format'])] if arguments['--format'] else [Format.DEFAULT.value] | ||
| # option_configuration is used for tracking multi-execution options | ||
| option_configuration = [api, format_option] | ||
| option_configuration_combination = list(itertools.product(*option_configuration)) |
There was a problem hiding this comment.
For these options that might show up multiple times, do two things then their combination will be calculated and executed
- Initialize them as line 359,
- Add them into list
option_configuration
That's it!
|
|
||
|
|
||
| # Insert a benchmark result row into the benchmark report (table) | ||
| def insert_into_report_table(table, row): |
There was a problem hiding this comment.
We now create a report table according to the execution command first, and then insert each benchmark result row into the table after each execution.
| assert file_size == getsize(file) / BYTES_TO_MB | ||
| assert result_with_gc > 0 | ||
| assert write_memory_usage_peak >= 0 | ||
| execution_with_command(['write', file]) | ||
|
|
||
|
|
||
| def test_option_read(file=generate_test_path('integers.ion')): |
There was a problem hiding this comment.
Quick note that for all unit tests (not only this one),
They will only make sure the execution works and don't throw any execution. However, it won't check the benchmark results. E.g., the unit test won't test if the c extension must be faster than pure python implementation or a sample file size must be X MB.
There was a problem hiding this comment.
Is there a way to at least test that the correct options combinations are executed?
There was a problem hiding this comment.
The new commit adds it.
I returned raw table data from the execution, store them in a sorted list and compare it to desired options.
| def read_micro_benchmark_event(iterations, warmups, c_extension, file, memory_profiling, iterator=False): | ||
| return 0, 0, 0 |
cheqianh
changed the title
| # option_configuration is used for tracking options may show up multiple times. | ||
| option_configuration = [api, format_option] | ||
| option_configuration_combination = list(itertools.product(*option_configuration)) |
| assert file_size == getsize(file) / BYTES_TO_MB | ||
| assert result_with_gc > 0 | ||
| assert write_memory_usage_peak >= 0 | ||
| execution_with_command(['write', file]) | ||
|
|
||
|
|
||
| def test_option_read(file=generate_test_path('integers.ion')): |
There was a problem hiding this comment.
Is there a way to at least test that the correct options combinations are executed?
Description:
This PR adds support for multi-options execution - #226.
For example, user provides
--api simple_ion,--api event,--format ion_textand--format ion_binary. the CLI tool will generates combinations of all these two options (--apiand--format) and execute each of them.E.g. the report should be something like
--api simple_ion,--format ion_binary--api simple_ion,--format ion_text--api event,--format ion_binary--api event,--format ion_textIn addition, this PR fixed few typo, modified unit test. I also left comments to highlight some changes.
Test
Unit test, 204/204 passed
Example
Command
Configuration Options
Result
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.