Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement better testing framework #248

Closed
nuald opened this issue Apr 16, 2020 · 2 comments
Closed

Implement better testing framework #248

nuald opened this issue Apr 16, 2020 · 2 comments
Assignees

Comments

@nuald
Copy link
Collaborator

nuald commented Apr 16, 2020

Right now the JIT and lazy initialization may interfere with the results, and to avoid it we can implement the following:

  • run the tested code in the loop and as soon as the previous result is faster than the current, stop the loop (surely, it could be a fluctuation caused by the external reason, but running all tests X times would take forever to execute)
  • extending the test runner communication protocol to support several notifications from within one test (and write to the results.log file only the best result).

Unfortunately, it requires changes in all tests.

@eregon
Copy link

eregon commented Apr 18, 2020

That sounds good, running the actual benchmark workload 2+ times seems better than a warmup with a different workflow.

Another way as I suggested in #246 is to not warmup, just avoid counting startup. Lazy initialization that takes 0.6s seems excessive and arguably an issue of the library used.

@nuald
Copy link
Collaborator Author

nuald commented Jun 18, 2020

JIT warming up has been removed, and lazy initialization issues are addressed (not finished yet though) with the testing of the code to work properly before the benchmark.

The proposed idea has been tested, and dismissed. While the 2nd or 3rd iteration could be faster, it's not always happening as GC could affect the results quite drastically. Fine tuning is required, and it's out of scope of that project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants