Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Do not merge] Compare 2.2.0 release benchmarks against 2.3 master branch #32086

Closed
wants to merge 1 commit into from
Closed

[Do not merge] Compare 2.2.0 release benchmarks against 2.3 master branch #32086

wants to merge 1 commit into from

Conversation

cadedaniel
Copy link
Member

@cadedaniel cadedaniel commented Jan 31, 2023 •
edited
Loading

This PR compares latest performance numbers, as of commit 664c844, with the Ray 2.2.0 release. I'll create another PR after branch cut; this is just to catch things earlier.

@cadedaniel cadedaniel force-pushed the releases/2.3-prerelease branch from f1e7979 to d649c6d Compare January 31, 2023 00:05
@cadedaniel cadedaniel changed the title [Draft] 2.3 release performance PR [Do not merge] Compare 2.2.0 release benchmarks against 2.3 master branch Jan 31, 2023
cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/benchmarks/many_actors.json
"actors_per_second": 600.3905147006428,
"_dashboard_memory_usage_mb": 427.753472,
"_dashboard_test_success": true,
"_peak_memory": 6.35,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created #32090 for peak memory increase

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/benchmarks/many_nodes.json
"_peak_process_memory": "PID\tMEM\tCOMMAND\n291\t0.2GiB\t/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n3002\t0.18GiB\tpython distributed/test_many_tasks.py --num-tasks=1000\n415\t0.17GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/dashboa\n47\t0.09GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/anyscale session web_terminal_server --deploy\n3204\t0.07GiB\tray::StateAPIGeneratorActor.start\n3139\t0.06GiB\tray::MemoryMonitorActor.run\n798\t0.06GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/agen\n44\t0.06GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-notebook --NotebookApp.token=agh0_Ckg\n342\t0.05GiB\t/home/ray/anaconda3/bin/python -m ray.util.client.server --address=172.31.151.143:9031 --host=0.0.0.\n615\t0.05GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/log_m",
"_dashboard_memory_usage_mb": 233.955328,
"_dashboard_test_success": true,
"_peak_memory": 4.43,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created #32090 for peak memory increase

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/benchmarks/many_pgs.json
"_peak_process_memory": "PID\tMEM\tCOMMAND\n291\t0.58GiB\t/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n3057\t0.31GiB\tpython distributed/test_many_pgs.py\n415\t0.13GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/dashboa\n47\t0.09GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/anyscale session web_terminal_server --deploy\n542\t0.08GiB\t/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=\n716\t0.06GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/agen\n3192\t0.06GiB\tray::MemoryMonitorActor.run\n44\t0.06GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-notebook --NotebookApp.token=aph0_Ckc\n342\t0.05GiB\t/home/ray/anaconda3/bin/python -m ray.util.client.server --address=172.31.238.30:9031 --host=0.0.0.0\n615\t0.05GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/log_m",
"_dashboard_memory_usage_mb": 186.888192,
"_dashboard_test_success": true,
"_peak_memory": 4.38,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created #32090 for peak memory increase

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/benchmarks/many_tasks.json
"_peak_process_memory": "PID\tMEM\tCOMMAND\n291\t1.11GiB\t/home/ray/anaconda3/lib/python3.7/site-packages/ray/core/src/ray/gcs/gcs_server --log_dir=/tmp/ray/s\n2033\t0.74GiB\tpython distributed/test_many_tasks.py --num-tasks=10000\n415\t0.45GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/dashboa\n47\t0.09GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/anyscale session web_terminal_server --deploy\n2240\t0.07GiB\tray::StateAPIGeneratorActor.start\n717\t0.06GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/dashboard/agen\n2169\t0.06GiB\tray::MemoryMonitorActor.run\n44\t0.06GiB\t/home/ray/anaconda3/bin/python /home/ray/anaconda3/bin/jupyter-notebook --NotebookApp.token=aph0_Ckg\n342\t0.05GiB\t/home/ray/anaconda3/bin/python -m ray.util.client.server --address=172.31.171.196:9031 --host=0.0.0.\n615\t0.05GiB\t/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/log_m",
"_dashboard_memory_usage_mb": 446.6688,
"_dashboard_test_success": true,
"_peak_memory": 6.4,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created #32090 for peak memory increase

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "single_client_tasks_and_get_batch",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 11.24301668594478
"perf_metric_value": 9.84148376315661
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

12% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "single_client_tasks_sync",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 1294.2667282228672
"perf_metric_value": 1200.8853386364904
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "single_client_tasks_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 10904.790007336374
"perf_metric_value": 9992.216255226114
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "multi_client_tasks_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 32133.367075183483
"perf_metric_value": 24894.461452735395
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

23% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "1_n_actor_calls_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 11646.429573331836
"perf_metric_value": 10264.35850171556
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

12% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "n_n_actor_calls_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 35151.93038362862
"perf_metric_value": 29179.56691260244
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

17% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "1_n_async_actor_calls_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 10613.303070856366
"perf_metric_value": 9558.899002308037
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/microbenchmark.json
},
{
"perf_metric_name": "n_n_async_actor_calls_async",
"perf_metric_type": "THROUGHPUT",
"perf_metric_value": 28665.946308284096
"perf_metric_value": 24203.919719797126
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

16% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/scalability/single_node.json
"large_object_size": 107374182400,
"large_object_time": 255.3529769390001,
"large_object_time": 338.213855138,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32% regression

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is highly variable due to EBS #30711

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/scalability/single_node.json
},
{
"perf_metric_name": "1000000_queued_time",
"perf_metric_type": "LATENCY",
"perf_metric_value": 186.92522427999995
"perf_metric_value": 205.44312294999997
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/stress_tests/stress_test_dead_actors.json
"avg_iteration_time": 1.9442666888237,
"max_iteration_time": 9.263823747634888,
"min_iteration_time": 0.8118352890014648,
"avg_iteration_time": 2.066628179550171,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/stress_tests/stress_test_dead_actors.json
"max_iteration_time": 9.263823747634888,
"min_iteration_time": 0.8118352890014648,
"avg_iteration_time": 2.066628179550171,
"max_iteration_time": 13.97415018081665,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

51% regression

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a max value, so high variance. I don't think we need to investigate since we don't know how widespread this is.

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/stress_tests/stress_test_many_tasks.json
@@ -3,45 +3,45 @@
{
"perf_metric_name": "stage_0_time",
"perf_metric_type": "LATENCY",
"perf_metric_value": 5.958704233169556
"perf_metric_value": 9.163766622543335
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

54% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/stress_tests/stress_test_many_tasks.json
},
{
"perf_metric_name": "stage_3_time",
"perf_metric_type": "LATENCY",
"perf_metric_value": 2336.072838306427
"perf_metric_value": 2730.786130428314
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

17% regression

cadedaniel
cadedaniel commented Jan 31, 2023
View reviewed changes
release/release_logs/2.2.0/stress_tests/stress_test_placement_group.json
@@ -1,16 +1,16 @@
{
"avg_pg_create_time_ms": 0.8335708558568591,
"avg_pg_remove_time_ms": 0.847586638137635,
"avg_pg_create_time_ms": 0.8819333648672524,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6% regression

@cadedaniel cadedaniel mentioned this pull request Feb 7, 2023
Closed
@cadedaniel
Copy link
Member Author

Doing this again with tip of release here #32274

@cadedaniel cadedaniel closed this Feb 7, 2023
@cadedaniel cadedaniel deleted the releases/2.3-prerelease branch February 7, 2023 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cadedaniel