diff --git a/debezium/perf/README.md b/debezium/perf/README.md index 909975caaf9..3b5e2d475ec 100644 --- a/debezium/perf/README.md +++ b/debezium/perf/README.md @@ -1,5 +1,4 @@ -Debezium - Kafka Perf -===================== +# Debezium - Kafka Perf The docker compose file in this directory is similar to the one in the ../demo directory, with additional @@ -19,8 +18,130 @@ analysis oriented, it has considerably larger requirements than our other feature-oriented demos. -Once the compose is running -=========================== +## Additional building steps + +On top of what is required for `../demo` (see +`../demo/README.md`), the automated testing +requires building the Deephaven Java client examples. +At the toplevel directory of your git clone (`../..`), run: + +``` +./gradlew java-client-session-examples:installDist +``` + +## Automated testing + +The script `run_experiment.sh` in this directory performs a +full test for one engine (either Deephaven or Materialize). +It will: + +- Start the containers required for a particular run (and only those). +- Ensure container logs are preserved for the run. +- Load the demo code in the respective engine and sample update delays to a log file. +- Set the given pageviews per second rate, and wait a fixed amount of time thereafter for processing to settle. +- Take multiple samples for CPU and memory utilization over a defined period. + Output from top in batch mode is sent to a log file and later post-processed. +- Stop and "reset" the containers. + +The example + +``` +cd debezium/perf +./run_experiment.sh dh 5000 20 10 1.0 +``` + +will run an experiment for Deephaven (tag `dh`; use tag `mz` for Materialize) with a target rate of 5,000 pageviews per second. +It will wait 20 seconds after setting the target rate to begin sampling CPU and memory utilization using `top` in batch mode. +10 samples will be obtained, with a delay between samples of 1.0 seconds. + +Example output from a run: + +``` +cfs@erke 12:18:20 ~/dh/oss3/deephaven-core/debezium/perf +$ ./run_experiment.sh dh 5000 20 10 1.0 +About to run an experiment for dh with 5000 pageviews/s. + +Actions that will be performed in this run: +1. Start compose services required for for dh. +2. Execute demo in dh and setup update delay logging. +3. Set 5000 pageviews per second rate. +4. Wait 20 seconds. +5. Take 10 samples for mem and CPU utilization, 1.0 seconds between samples. +6. Stop and 'reset' (down) compose. + +Running experiment. + +1. Starting compose. +PERF_TAG=2022.03.22.16.18.41_UTC_dh_5000 + +Logs are being saved to logs/2022.03.22.16.18.41_UTC_dh_5000. + +2. Running demo in dh and sampling delays. +1 compiler directives added +Table users = +Table items = +Table purchases = +Table pageviews = +Table pageviews_stg = +Table purchases_by_item = +Table pageviews_by_item = +Table item_summary = +Table top_viewed_items = +Table top_converting_items = +Table profile_views_per_minute_last_10 = +Table profile_views = +Table profile_views_enriched = +Table dd_flagged_profiles = +Table dd_flagged_profile_view = +Table high_value_users = +Table hvu_test = +Table pageviews_summary = + +1 compiler directives added +No displayable variables updated + + +3. Setting pageviews per second +LOADGEN Connected. +Setting pageviews_per_second: old value was 50, new value is 5000. +Goodbye. + +4. Waiting for 20 seconds. + +5. Sampling top. +name=redpanda, tag=CPU_PCT, mean=84.14, samples=80.0, 84.2, 85.0, 87.0, 85.0, 82.0, 85.0, 84.0, 84.2, 85.0 +name=redpanda, tag=RES_GiB, mean=0.77, samples=0.7678, 0.7698, 0.7698, 0.7698, 0.7718, 0.7718, 0.7718, 0.7718, 0.7718, 0.7776 +name=deephaven, tag=CPU_PCT, mean=35.21, samples=66.7, 31.7, 28.0, 31.0, 27.0, 23.0, 46.0, 47.0, 25.7, 26.0 +name=deephaven, tag=RES_GiB, mean=2.40, samples=2.4, 2.4, 2.4, 2.4, 2.4, 2.4, 2.4, 2.4, 2.4, 2.4 + +6. Stopping and 'reset' (down) compose. +Stopping core-debezium-perf_envoy_1 ... done +Stopping core-debezium-perf_grpc-proxy_1 ... done +Stopping core-debezium-perf_loadgen_1 ... done +Stopping core-debezium-perf_debezium_1 ... done +Stopping core-debezium-perf_server_1 ... done +Stopping core-debezium-perf_redpanda_1 ... done +Stopping core-debezium-perf_mysql_1 ... done +Stopping core-debezium-perf_web_1 ... done +Removing core-debezium-perf_envoy_1 ... done +Removing core-debezium-perf_grpc-proxy_1 ... done +Removing core-debezium-perf_loadgen_1 ... done +Removing core-debezium-perf_debezium_1 ... done +Removing core-debezium-perf_server_1 ... done +Removing core-debezium-perf_redpanda_1 ... done +Removing core-debezium-perf_mysql_1 ... done +Removing core-debezium-perf_web_1 ... done +Removing network core-debezium-perf_default + +Experiment finished. +``` + +The CPU and memory utilization samples are shown on stdout and also saved to a file in the +new directory under `logs/`, in this case `logs/2022.03.22.16.18.41_UTC_dh_5000.` + +## Manual testing + +### Once the compose is running Both Materialize and Deephaven are running. We now can make them execute their respective demo scripts. @@ -54,8 +175,7 @@ a command socket interface for loadgen; see `../demo/README.md` for instructions. -Tracking the last processed pageview timestamp -============================================== +### Tracking the last processed pageview timestamp * In DH, the `pageviews_summary` table can help track the last pageview seen. @@ -72,8 +192,7 @@ Tracking the last processed pageview timestamp FROM pageviews_summary;' -U materialize -h localhost -p 6875 ``` -Memory and CPU requirements -=========================== +## Memory and CPU requirements The parameters used for images in the docker compose file in this directory are geared towards high message throughput. While Deephaven diff --git a/debezium/perf/run_experiment.sh b/debezium/perf/run_experiment.sh index 41873a0c3eb..0f846f0fdac 100755 --- a/debezium/perf/run_experiment.sh +++ b/debezium/perf/run_experiment.sh @@ -13,12 +13,26 @@ wait_s="$3" top_samples="$4" top_delay="$5" -echo "Starting compose and waiting for simulation to start." +echo "About to run an experiment for ${engine} with ${rate_per_s} pageviews/s." +echo +echo "Actions that will be performed in this run:" +echo "1. Start compose services required for for ${engine}." +echo "2. Execute demo in ${engine} and setup update delay logging." +echo "3. Set ${rate_per_s} pageviews per second rate." +echo "4. Wait ${wait_s} seconds." +echo "5. Take ${top_samples} samples for mem and CPU utilization, ${top_delay} seconds between samples." +echo "6. Stop and 'reset' (down) compose." +echo +echo "Running experiment." +echo +echo "1. Starting compose." export PERF_TAG=$(./start_perf_run.sh "$engine" "$rate_per_s") -echo "Compose start done." echo "PERF_TAG=${PERF_TAG}" +echo +echo "Logs are being saved to logs/$PERF_TAG." +echo -echo "Running demo in engine and sampling delays." +echo "2. Running demo in ${engine} and sampling delays." if [ "$engine" = "mz" ]; then ./mz_run_demo.sh ./mz_sample_dt.sh @@ -29,20 +43,24 @@ else echo "$0: Internal error, aborting." 1>&2 exit 1 fi +echo -echo "Setting pageviews per second" +echo "3. Setting pageviews per second" ./set_pageviews_per_second.sh $rate_per_s +echo -echo "Waiting for $wait_s seconds..." +echo "4. Waiting for $wait_s seconds." sleep "$wait_s" +echo -echo "Sampling top." +echo "5. Sampling top." ./sample_top.sh "$engine" "$top_samples" "$top_delay" +echo -echo "Stopping compose." +echo "6. Stopping and 'reset' (down) compose." ./stop_all.sh - +echo echo "Experiment finished." exit 0