Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measure also PyPy and GraalPy in benchmarking-public ? #705

Open
paugier opened this issue Nov 15, 2024 · 2 comments
Open

Measure also PyPy and GraalPy in benchmarking-public ? #705

paugier opened this issue Nov 15, 2024 · 2 comments

Comments

@paugier
Copy link

paugier commented Nov 15, 2024

I realize that this project is about CPython, but it seems to me that it would be useful for the Python community to also include in https://github.com/faster-cpython/benchmarking-public few performance measurements of stable releases of PyPy and GraalPy.

Currently it's quite difficult to compare the numbers given by the different teams. PyPy and GraalPy both state that they are approximately 4 times faster than CPython. But it would be nice to get numbers measured using your measurement method. I guess that PyPy and GraalPy developers would be available to help for the setup if needed so it would not add too much work for the Faster CPython team.

For the Faster CPython project, it would give good ideas of what can be gained.

And for the scientific/data Python community it would be a good indication of how interesting it would be to go towards an ecosystem more gentle with alternative interpreters , using in particular C API like HPy (mentioned also in https://github.com/faster-cpython/ideas/blob/main/3.14/README.md).

So, my question is: would it be possible to also include few measurements using PyPy and GraalPy in https://github.com/faster-cpython/benchmarking-public ?

@mdboom
Copy link
Contributor

mdboom commented Nov 21, 2024

I like the suggestion, and I would like to see it. I think speed.python.org had this as a goal at one time. However, it's a pretty expensive proposition, and I think we'd need more resources in order to be able to support it.

I feel very strongly about keeping the tools to reproduce these results open source and as a point of collaboration. Specifically, pyperf (measurement and hardware tuning fundamentals), pyperformance (a set of benchmarks and benchmark-running harness), python-macrobenchmarks (additional macrobenchmarks) and bench-runner (infrastructure for running all of the above on Github Actions). From this, alternative implementations can see exactly what and how we are measuring and reproduce themselves. I see this as sort of the principles of "doing good science".

@markshannon
Copy link
Member

The best available comparison of GraalPython, PyPy and CPython from a performance perspective is https://onlinelibrary.wiley.com/doi/10.1002/spe.3267. I don't know if there is a non-paywalled version anywhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants