-
Notifications
You must be signed in to change notification settings - Fork 304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add hello-world benchmark #231
Conversation
This is actually a very cool idea. One of the big complains outlined in this issue was that this test suite includes startup times, so it would be super cool just to see what the cost actually is for all of them! |
Is this ready to merge, or do you want to work on it more first? |
Good question! I was thinking to just show you what I was up to, but I do also think it can be merged and we can take it from there. Maybe other people will want to add languages and help with the CI setup. The thing to keep in mind is that it's not all put together in CI. So there are languages that have hello-world, that won't have a tool chain set up in CI. But I guess that's fine as long as you are aware. |
Ok great. I'll just go ahead and merge this as a start, and others will be welcome to add! |
Indeed. Seeing the startup times was my main driver for this. Let's see if it gives us any ideas how to mitigate. Hello World is not dealing with the JIT aspects, though. But maybe we can figure that out too, eventually. |
I had kind-of hoped people wouldn't just repeat the same mistakes: :but would do better. |
So measure it ? https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/startup.html |
Hello,
I started to add a hello-world benchmark. Two drivers for me:
When it comes to toolchains I at some point decided I wanted CI to run the benchmark. Making all these toolchains co-exist in a Github Actions workflow is VERY time consuming, though, so it put a bit of a stop in my implementing of Hello Worlds. But it is probably good to have one fast-running benchmark that can test the clean-compile-run of all languages reliably.
Anyway, @bddicken, before I prioritize this PR too much, I'd like your feedback on the idea and the way I am going at it. Is it something you would even consider merging? I'll have use for it even if you don't so don't hesitate to tell me the truth. 😄
As for the state of this PR, I have added some 15 languages so far, maybe less, maybe more. And CI runs some of them. I think my next steps will be to add more languages and then see how many of them I can get to run in CI.