diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000000..c17926ce55 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,73 @@ +# Benchmarks + +## NYC Taxi Monthly Trips with Precipitation + +For this benchmark, we adapt [an example data science workload](https://github.com/toddwschneider/nyc-taxi-data/blob/c65ad8332a44f49770644b11576c0529b40bbc76/citibike_comparison/analysis/analysis_queries.sql#L1) into a pandas workload that reads from a public S3 bucket and calculates the average trip duration and number of trips based on features like weather conditions, pickup and dropoff location, month, and whether the trip was on a weekday. + +### Dataset + +The New York City Taxi and Limousine Commission's [For Hire Vehicle High Volume dataset](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) (FHVHV) consists of over one billion trips taken by "for hire vehicles" including Uber and Lyft. To get the weather on a given day, we use a separate dataset of [Central Park weather observations](https://github.com/toddwschneider/nyc-taxi-data/blob/c65ad8332a44f49770644b11576c0529b40bbc76/data/central_park_weather.csv). The For Hire Vehicle High Volume dataset consists of 1,036,465,968 rows and 24 columns. The Central Park Weather dataset consists of 5,538 rows and 9 columns. + +### Setting + +For this benchmark, we use the full FHVHV dataset stored in Parquet files on S3. The total size of this dataset is 24.7 GiB. The Central Park Weather data ia stored in a single CSV file on S3 and its total size is 514 KiB. + +We compared Bodo's performance on this workload to other systems including [Dask](https://www.dask.org/), [Modin on Ray](https://docs.ray.io/en/latest/ray-more-libs/modin/index.html), and [PySpark](https://spark.apache.org/docs/latest/api/python/index.html) and observed a speedup of 20-240x. The implementations for all of these systems can be found in [`nyc_taxi`](./nyc_taxi/). Versions of the packages used are summarized below. + +| Package | Version | +|----------------|----------------| +| bodo | 2024.10 | +| dask | 2024.9.1 | +| dask-cloudprovider | 2024.9.1 | +| modin | 0.32.0 | +| ray | 2.40.0 | +| spark | 3.5.2 | + +For cluster creation and configuration, we use the [Bodo SDK](https://docs.bodo.ai/2024.12/guides/using_bodo_platform/bodo_platform_sdk_guide/) for Bodo, Dask Cloud Provider for Dask, Ray for Modin, and AWS EMR for Spark. Scripts to configure and launch clusters for each system can be found in the same directory as the implementation. + +Each benchmark is collected on a cluster containing 4 worker instances and 128 physical cores. Dask, Modin, and Spark use 4 `r6i.16xlarge` instances, each consisting of 32 physical cores and 256 GiB of memory. Dask Cloud Provider also allocates an additional `c6i.xlarge` instance for the distributed scheduler which contains 2 cores. Bodo is run on 4 `c6i.16xlarge` instances, each consisting of 32 physical cores and 64 GiB of memory. + +### Results + +The graph below summarizes the total execution time of each system (averaged over 3 runs). Results were last collected on December 12th, 2024. + +Monthly High Volume for Hire Vehicle Trips with Precipitation Benchmark Execution Time + +## Local Benchmark + +You can start to see the benefits of using Bodo from your laptop by running a smaller version of our benchmark locally. To set up, install the required packages using pip: + +``` shell +pip install bodo==2024.12.1 +pip install "dask[dataframe]"==2024.12.0 +pip install "modin[all]"==0.32.0 +pip install pyspark==3.5.3 +pip install boto3 # for S3 download +``` + +To run the entire benchmarks as a script + +``` shell +# From the benchmarks/ directory +./nyc_taxi/run_local.sh +``` + +We use a smaller subset of the [For Hire Vehicle High Volume dataset](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) to allow the workload to run locally on an Apple M2 Macbook Pro with 10 cores and 16 GB memory. Even at this smaller scale, Bodo shows a roughly 3x improvement over the next best system (Dask). The results below were collected December 17th, 2024. Note that these numbers might differ based on your specific hardware and operating system. + +| System | Total Execution Time (s) | +|----------------|----------------| +| Bodo | 1.007 | +| Dask | 3.091 | +| Modin/Ray | 13.65 | +| PySpark | 27.27 | + +To see an even bigger difference, try increasing the number of rows read by specifying a different parquet file such as `s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/fhvhv_tripdata_2019-02.parquet`. On this size (~20 million rows), Dask runs out of memory whereas Bodo continue to run: + +``` shell +# From the benchmarks/ directory. +# Run Dask on first parquet file (~20 million rows) +python -m nyc_taxi.local_versions -s dask -d nyc-taxi/fhvhv_tripdata/fhvhv_tripdata_2019-02.parquet + +# Run Bodo on first parquet file (~20 million rows) +python -m nyc_taxi.local_versions -s bodo -d nyc-taxi/fhvhv_tripdata/fhvhv_tripdata_2019-02.parquet +``` \ No newline at end of file diff --git a/benchmarks/__init__.py b/benchmarks/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/benchmarks/img/nyc-taxi-benchmark.png b/benchmarks/img/nyc-taxi-benchmark.png index 3ceb43fc1c..0434992213 100644 Binary files a/benchmarks/img/nyc-taxi-benchmark.png and b/benchmarks/img/nyc-taxi-benchmark.png differ diff --git a/benchmarks/nyc_taxi/bodo/nyc_taxi_precipitation.py b/benchmarks/nyc_taxi/bodo/nyc_taxi_precipitation.py index ef5ad8d668..524ff4a6f0 100644 --- a/benchmarks/nyc_taxi/bodo/nyc_taxi_precipitation.py +++ b/benchmarks/nyc_taxi/bodo/nyc_taxi_precipitation.py @@ -13,19 +13,20 @@ @bodo.jit(cache=True) -def get_monthly_travels_weather(): - start = time.time() +def get_monthly_travels_weather(weather_dataset, hvfhv_dataset): + start_read = time.time() central_park_weather_observations = pd.read_csv( - "s3://bodo-example-data/nyc-taxi/central_park_weather.csv", parse_dates=["DATE"] + weather_dataset, + parse_dates=["DATE"], ) central_park_weather_observations = central_park_weather_observations.rename( columns={"DATE": "date", "PRCP": "precipitation"}, copy=False ) - fhvhv_tripdata = pd.read_parquet("s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/") + fhvhv_tripdata = pd.read_parquet(hvfhv_dataset) end = time.time() - print("Reading Time: ", (end - start)) + print("Reading Time: ", (end - start_read)) - start = time.time() + start_compute = time.time() central_park_weather_observations["date"] = central_park_weather_observations[ "date" @@ -88,14 +89,17 @@ def get_time_bucket(t): copy=False, ) end = time.time() - print("Monthly Taxi Travel Times Computation Time: ", end - start) + print("Monthly Taxi Travel Times Computation Time: ", end - start_compute) - start = time.time() + start_write = time.time() monthly_trips_weather.to_parquet("monthly_trips_weather.pq") end = time.time() - print("Writing time:", (end - start)) + print("Writing time:", (end - start_write)) + print("Total E2E time:", (end - start_read)) return monthly_trips_weather if __name__ == "__main__": - get_monthly_travels_weather() + weather_dataset = "s3://bodo-example-data/nyc-taxi/central_park_weather.csv" + hvfhv_dataset = "s3://bodo-example-data/nyc-taxi/fhvhv/" + get_monthly_travels_weather(weather_dataset, hvfhv_dataset) diff --git a/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.ipynb b/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.ipynb index 5dbcfc1d41..f4aeaf0f5e 100644 --- a/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.ipynb +++ b/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.ipynb @@ -32,9 +32,317 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 34, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating scheduler instance\n", + "\n", + "Cloud init\n", + "==========\n", + "\n", + "\n", + "#cloud-config\n", + "\n", + "\n", + "# Bootstrap\n", + "packages:\n", + " - apt-transport-https\n", + " - ca-certificates\n", + " - curl\n", + " - gnupg-agent\n", + " - software-properties-common\n", + " - ubuntu-drivers-common\n", + "\n", + "# Enable ipv4 forwarding, required on CIS hardened machines\n", + "write_files:\n", + " - path: /etc/sysctl.d/enabled_ipv4_forwarding.conf\n", + " content: |\n", + " net.ipv4.conf.all.forwarding=1\n", + "\n", + "# create the docker group\n", + "groups:\n", + " - docker\n", + "\n", + "# Add default auto created user to docker group\n", + "system_info:\n", + " default_user:\n", + " groups: [docker]\n", + "\n", + "\n", + "runcmd:\n", + " \n", + " # Install Docker\n", + " - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -\n", + " - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"\n", + " - apt-get update -y\n", + " - apt-get install -y docker-ce docker-ce-cli containerd.io\n", + " - systemctl start docker\n", + " - systemctl enable docker\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " # Run container\n", + " - 'docker run --net=host -e EXTRA_CONDA_PACKAGES=\"s3fs==2024.10.0\" daskdev/dask:latest env DASK_INTERNAL_INHERIT_CONFIG=\"eyJ0ZW1wb3JhcnktZGlyZWN0b3J5IjogbnVsbCwgInZpc3VhbGl6YXRpb24iOiB7ImVuZ2luZSI6IG51bGx9LCAidG9rZW5pemUiOiB7ImVuc3VyZS1kZXRlcm1pbmlzdGljIjogZmFsc2V9LCAiZGF0YWZyYW1lIjogeyJiYWNrZW5kIjogInBhbmRhcyIsICJzaHVmZmxlIjogeyJtZXRob2QiOiBudWxsLCAiY29tcHJlc3Npb24iOiBudWxsfSwgInBhcnF1ZXQiOiB7Im1ldGFkYXRhLXRhc2stc2l6ZS1sb2NhbCI6IDUxMiwgIm1ldGFkYXRhLXRhc2stc2l6ZS1yZW1vdGUiOiAxLCAibWluaW11bS1wYXJ0aXRpb24tc2l6ZSI6IDc1MDAwMDAwfSwgImNvbnZlcnQtc3RyaW5nIjogbnVsbCwgInF1ZXJ5LXBsYW5uaW5nIjogbnVsbH0sICJhcnJheSI6IHsiYmFja2VuZCI6ICJudW1weSIsICJjaHVuay1zaXplIjogIjEyOE1pQiIsICJjaHVuay1zaXplLXRvbGVyYW5jZSI6IDEuMjUsICJyZWNodW5rIjogeyJtZXRob2QiOiBudWxsLCAidGhyZXNob2xkIjogNH0sICJzdmciOiB7InNpemUiOiAxMjB9LCAic2xpY2luZyI6IHsic3BsaXQtbGFyZ2UtY2h1bmtzIjogbnVsbH0sICJxdWVyeS1wbGFubmluZyI6IG51bGx9LCAib3B0aW1pemF0aW9uIjogeyJhbm5vdGF0aW9ucyI6IHsiZnVzZSI6IHRydWV9LCAiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJhZG1pbiI6IHsidHJhY2ViYWNrIjogeyJzaG9ydGVuIjogWyJjb25jdXJyZW50W1xcXFxcXC9dZnV0dXJlc1tcXFxcXFwvXSIsICJkYXNrW1xcXFxcXC9dKGJhc2V8Y29yZXxsb2NhbHxtdWx0aXByb2Nlc3Npbmd8b3B0aW1pemF0aW9ufHRocmVhZGVkfHV0aWxzKVxcLnB5IiwgImRhc2tbXFxcXFxcL11hcnJheVtcXFxcXFwvXWNvcmVcXC5weSIsICJkYXNrW1xcXFxcXC9dZGF0YWZyYW1lW1xcXFxcXC9dKGNvcmV8bWV0aG9kcylcXC5weSIsICJkaXN0cmlidXRlZFtcXFxcXFwvXShjbGllbnR8c2NoZWR1bGVyfHV0aWxzfHdvcmtlcilcXC5weSIsICJ0b3JuYWRvW1xcXFxcXC9dZ2VuXFwucHkiLCAicGFuZGFzW1xcXFxcXC9dY29yZVtcXFxcXFwvXSJdfX0sICJkaXN0cmlidXRlZCI6IHsidmVyc2lvbiI6IDIsICJzY2hlZHVsZXIiOiB7ImFsbG93ZWQtZmFpbHVyZXMiOiAzLCAiYmFuZHdpZHRoIjogMTAwMDAwMDAwLCAiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAiY29udGFjdC1hZGRyZXNzIjogbnVsbCwgImRlZmF1bHQtZGF0YS1zaXplIjogIjFraUIiLCAiZXZlbnRzLWNsZWFudXAtZGVsYXkiOiAiMWgiLCAiaWRsZS10aW1lb3V0IjogbnVsbCwgIm5vLXdvcmtlcnMtdGltZW91dCI6IG51bGwsICJ3b3JrLXN0ZWFsaW5nIjogdHJ1ZSwgIndvcmstc3RlYWxpbmctaW50ZXJ2YWwiOiAiMTAwbXMiLCAid29ya2VyLXNhdHVyYXRpb24iOiAxLjEsICJ3b3JrZXItdHRsIjogIjUgbWludXRlcyIsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyIsICJzcGxpdC10YXNrc2h1ZmZsZSI6ICIxdXMiLCAic3BsaXQtc3RhZ2UiOiAiMXVzIn0sICJ2YWxpZGF0ZSI6IGZhbHNlLCAiZGFzaGJvYXJkIjogeyJzdGF0dXMiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDB9LCAidGFza3MiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDAwMH0sICJ0bHMiOiB7ImNhLWZpbGUiOiBudWxsLCAia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImJva2VoLWFwcGxpY2F0aW9uIjogeyJhbGxvd193ZWJzb2NrZXRfb3JpZ2luIjogWyIqIl0sICJrZWVwX2FsaXZlX21pbGxpc2Vjb25kcyI6IDUwMCwgImNoZWNrX3VudXNlZF9zZXNzaW9uc19taWxsaXNlY29uZHMiOiA1MDB9fSwgImxvY2tzIjogeyJsZWFzZS12YWxpZGF0aW9uLWludGVydmFsIjogIjEwcyIsICJsZWFzZS10aW1lb3V0IjogIjMwcyJ9LCAiaHR0cCI6IHsicm91dGVzIjogWyJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5wcm9tZXRoZXVzIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmluZm8iLCAiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIuanNvbiIsICJkaXN0cmlidXRlZC5odHRwLmhlYWx0aCIsICJkaXN0cmlidXRlZC5odHRwLnByb3h5IiwgImRpc3RyaWJ1dGVkLmh0dHAuc3RhdGljcyJdfSwgImFsbG93ZWQtaW1wb3J0cyI6IFsiZGFzayIsICJkaXN0cmlidXRlZCJdLCAiYWN0aXZlLW1lbW9yeS1tYW5hZ2VyIjogeyJzdGFydCI6IHRydWUsICJpbnRlcnZhbCI6ICIycyIsICJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAicG9saWNpZXMiOiBbeyJjbGFzcyI6ICJkaXN0cmlidXRlZC5hY3RpdmVfbWVtb3J5X21hbmFnZXIuUmVkdWNlUmVwbGljYXMifV19fSwgIndvcmtlciI6IHsiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAibXVsdGlwcm9jZXNzaW5nLW1ldGhvZCI6ICJzcGF3biIsICJ1c2UtZmlsZS1sb2NraW5nIjogdHJ1ZSwgInRyYW5zZmVyIjogeyJtZXNzYWdlLWJ5dGVzLWxpbWl0IjogIjUwTUIifSwgImNvbm5lY3Rpb25zIjogeyJvdXRnb2luZyI6IDUwLCAiaW5jb21pbmciOiAxMH0sICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImRhZW1vbiI6IHRydWUsICJ2YWxpZGF0ZSI6IGZhbHNlLCAicmVzb3VyY2VzIjoge30sICJsaWZldGltZSI6IHsiZHVyYXRpb24iOiBudWxsLCAic3RhZ2dlciI6ICIwIHNlY29uZHMiLCAicmVzdGFydCI6IGZhbHNlfSwgInByb2ZpbGUiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMTBtcyIsICJjeWNsZSI6ICIxMDAwbXMiLCAibG93LWxldmVsIjogZmFsc2V9LCAibWVtb3J5IjogeyJyZWNlbnQtdG8tb2xkLXRpbWUiOiAiMzBzIiwgInJlYmFsYW5jZSI6IHsibWVhc3VyZSI6ICJvcHRpbWlzdGljIiwgInNlbmRlci1taW4iOiAwLjMsICJyZWNpcGllbnQtbWF4IjogMC42LCAic2VuZGVyLXJlY2lwaWVudC1nYXAiOiAwLjF9LCAidHJhbnNmZXIiOiAwLjEsICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NSwgIm1heC1zcGlsbCI6IGZhbHNlLCAic3BpbGwtY29tcHJlc3Npb24iOiAiYXV0byIsICJtb25pdG9yLWludGVydmFsIjogIjEwMG1zIn0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7fSwgInByZS1zcGF3bi1lbnZpcm9uIjogeyJNQUxMT0NfVFJJTV9USFJFU0hPTERfIjogNjU1MzYsICJPTVBfTlVNX1RIUkVBRFMiOiAxLCAiTUtMX05VTV9USFJFQURTIjogMSwgIk9QRU5CTEFTX05VTV9USFJFQURTIjogMX19LCAiY2xpZW50IjogeyJoZWFydGJlYXQiOiAiNXMiLCAic2NoZWR1bGVyLWluZm8taW50ZXJ2YWwiOiAiMnMiLCAic2VjdXJpdHktbG9hZGVyIjogbnVsbCwgInByZWxvYWQiOiBbXSwgInByZWxvYWQtYXJndiI6IFtdfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogZmFsc2UsICJzaGFyZCI6ICI2NE1pQiIsICJvZmZsb2FkIjogIjEwTWlCIiwgImRlZmF1bHQtc2NoZW1lIjogInRjcCIsICJzb2NrZXQtYmFja2xvZyI6IDIwNDgsICJ1Y3giOiB7ImN1ZGEtY29weSI6IG51bGwsICJ0Y3AiOiBudWxsLCAibnZsaW5rIjogbnVsbCwgImluZmluaWJhbmQiOiBudWxsLCAicmRtYWNtIjogbnVsbCwgImNyZWF0ZS1jdWRhLWNvbnRleHQiOiBudWxsLCAiZW52aXJvbm1lbnQiOiB7fX0sICJ6c3RkIjogeyJsZXZlbCI6IDMsICJ0aHJlYWRzIjogMH0sICJ0aW1lb3V0cyI6IHsiY29ubmVjdCI6ICIzMHMiLCAidGNwIjogIjMwcyJ9LCAicmVxdWlyZS1lbmNyeXB0aW9uIjogbnVsbCwgInRscyI6IHsiY2lwaGVycyI6IG51bGwsICJtaW4tdmVyc2lvbiI6IDEuMiwgIm1heC12ZXJzaW9uIjogbnVsbCwgImNhLWZpbGUiOiBudWxsLCAic2NoZWR1bGVyIjogeyJjZXJ0IjogbnVsbCwgImtleSI6IG51bGx9LCAid29ya2VyIjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9LCAiY2xpZW50IjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9fSwgIndlYnNvY2tldHMiOiB7InNoYXJkIjogIjhNaUIifX0sICJkaWFnbm9zdGljcyI6IHsibnZtbCI6IHRydWUsICJjdWRmIjogZmFsc2UsICJjb21wdXRhdGlvbnMiOiB7Im1heC1oaXN0b3J5IjogMTAwLCAibmZyYW1lcyI6IDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiYXN5bmNpbyIsICJmdW5jdG9vbHMiLCAidGhyZWFkaW5nIiwgImRhdGFzaGFkZXIiLCAiZGFzayIsICJkZWJ1Z3B5IiwgImRpc3RyaWJ1dGVkIiwgImNvaWxlZCIsICJjdWRmIiwgImN1bWwiLCAibWF0cGxvdGxpYiIsICJwbHVnZ3kiLCAicHJlZmVjdCIsICJyZWNodW5rZXIiLCAieGFycmF5IiwgInhnYm9vc3QiLCAieGRpc3QiLCAiX19jaGFubmVsZXhlY19fIiwgImV4ZWNuZXQiXSwgImlnbm9yZS1maWxlcyI6IFsicnVucHlcXC5weSIsICJweXRlc3QiLCAicHlcXC50ZXN0IiwgInB5dGVzdC1zY3JpcHRcXC5weSIsICJfcHl0ZXN0IiwgInB5Y2hhcm0iLCAidnNjb2RlX3B5dGVzdCIsICJnZXRfb3V0cHV0X3ZpYV9tYXJrZXJzXFwucHkiXX0sICJlcnJlZC10YXNrcyI6IHsibWF4LWhpc3RvcnkiOiAxMDB9fSwgInAycCI6IHsiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMTAsICJkZWxheSI6IHsibWluIjogIjFzIiwgIm1heCI6ICIzMHMifX19LCAiZGlzayI6IHRydWV9LCAiZGFzaGJvYXJkIjogeyJsaW5rIjogIntzY2hlbWV9Oi8ve2hvc3R9Ontwb3J0fS9zdGF0dXMiLCAiZXhwb3J0LXRvb2wiOiBmYWxzZSwgImdyYXBoLW1heC1pdGVtcyI6IDUwMDAsICJwcm9tZXRoZXVzIjogeyJuYW1lc3BhY2UiOiAiZGFzayJ9fSwgImFkbWluIjogeyJsYXJnZS1ncmFwaC13YXJuaW5nLXRocmVzaG9sZCI6ICIxME1CIiwgInRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MiLCAiY3ljbGUiOiAiMXMifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShhc2N0aW1lKXMgLSAlKG5hbWUpcyAtICUobGV2ZWxuYW1lKXMgLSAlKG1lc3NhZ2UpcyIsICJsb3ctbGV2ZWwtbG9nLWxlbmd0aCI6IDEwMDAsICJwZGItb24tZXJyIjogZmFsc2UsICJzeXN0ZW0tbW9uaXRvciI6IHsiaW50ZXJ2YWwiOiAiNTAwbXMiLCAibG9nLWxlbmd0aCI6IDcyMDAsICJkaXNrIjogdHJ1ZSwgImhvc3QtY3B1IjogZmFsc2UsICJnaWwiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMW1zIn19LCAiZXZlbnQtbG9vcCI6ICJ0b3JuYWRvIn0sICJybW0iOiB7InBvb2wtc2l6ZSI6IG51bGx9fSwgImNsb3VkcHJvdmlkZXIiOiB7ImVjcyI6IHsiZmFyZ2F0ZV9zY2hlZHVsZXIiOiBmYWxzZSwgImZhcmdhdGVfc3BvdCI6IGZhbHNlLCAiZmFyZ2F0ZV93b3JrZXJzIjogZmFsc2UsICJmYXJnYXRlX3VzZV9wcml2YXRlX2lwIjogZmFsc2UsICJzY2hlZHVsZXJfY3B1IjogMTAyNCwgInNjaGVkdWxlcl9tZW0iOiA0MDk2LCAid29ya2VyX2NwdSI6IDQwOTYsICJ3b3JrZXJfbWVtIjogMTYzODQsICJ3b3JrZXJfZ3B1IjogMCwgIm5fd29ya2VycyI6IDAsICJzY2hlZHVsZXJfdGltZW91dCI6ICI1IG1pbnV0ZXMiLCAiaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJjcHVfYXJjaGl0ZWN0dXJlIjogIlg4Nl82NCIsICJncHVfaW1hZ2UiOiAicmFwaWRzYWkvcmFwaWRzYWk6bGF0ZXN0IiwgImNsdXN0ZXJfbmFtZV90ZW1wbGF0ZSI6ICJkYXNrLXt1dWlkfSIsICJjbHVzdGVyX2FybiI6ICIiLCAiZXhlY3V0aW9uX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfcG9saWNpZXMiOiBbXSwgImNsb3Vkd2F0Y2hfbG9nc19ncm91cCI6ICIiLCAiY2xvdWR3YXRjaF9sb2dzX3N0cmVhbV9wcmVmaXgiOiAie2NsdXN0ZXJfbmFtZX0iLCAiY2xvdWR3YXRjaF9sb2dzX2RlZmF1bHRfcmV0ZW50aW9uIjogMzAsICJ2cGMiOiAiZGVmYXVsdCIsICJzdWJuZXRzIjogW10sICJzZWN1cml0eV9ncm91cHMiOiBbXSwgInRhZ3MiOiB7fSwgImVudmlyb25tZW50Ijoge30sICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJhbWkiOiBudWxsLCAiaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJzY2hlZHVsZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJ3b3JrZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2cGMiOiBudWxsLCAic3VibmV0X2lkIjogbnVsbCwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAiZmlsZXN5c3RlbV9zaXplIjogNDAsICJrZXlfbmFtZSI6IG51bGwsICJpYW1faW5zdGFuY2VfcHJvZmlsZSI6IHt9LCAiaW5zdGFuY2VfdGFncyI6IHsiY3JlYXRlZEJ5IjogImRhc2stY2xvdWRwcm92aWRlciJ9LCAidm9sdW1lX3RhZ3MiOiB7ImNyZWF0ZWRCeSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgImVuYWJsZV9kZXRhaWxlZF9tb25pdG9yaW5nIjogZmFsc2UsICJ1c2VfcHJpdmF0ZV9pcCI6IGZhbHNlfSwgImF6dXJlIjogeyJsb2NhdGlvbiI6IG51bGwsICJyZXNvdXJjZV9ncm91cCI6IG51bGwsICJzdWJzY3JpcHRpb25faWQiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzdWJuZXQiOiBudWxsLCAic2VjdXJpdHlfZ3JvdXAiOiBudWxsLCAicHVibGljX2luZ3Jlc3MiOiB0cnVlLCAidm1fc2l6ZSI6ICJTdGFuZGFyZF9EUzFfdjIiLCAiZGlza19zaXplIjogNTAsICJzY2hlZHVsZXJfdm1fc2l6ZSI6IG51bGwsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2bV9pbWFnZSI6IHsicHVibGlzaGVyIjogIkNhbm9uaWNhbCIsICJvZmZlciI6ICJVYnVudHVTZXJ2ZXIiLCAic2t1IjogIjE4LjA0LUxUUyIsICJ2ZXJzaW9uIjogImxhdGVzdCJ9LCAiYm9vdHN0cmFwIjogdHJ1ZSwgImF1dG9fc2h1dGRvd24iOiB0cnVlLCAibWFya2V0cGxhY2VfcGxhbiI6IG51bGwsICJleHRyYV9vcHRpb25zIjoge319fSwgImRpZ2l0YWxvY2VhbiI6IHsidG9rZW4iOiBudWxsLCAicmVnaW9uIjogIm55YzMiLCAic2l6ZSI6ICJzLTF2Y3B1LTFnYiIsICJpbWFnZSI6ICJ1YnVudHUtMjAtMDQteDY0In0sICJnY3AiOiB7InNvdXJjZV9pbWFnZSI6ICJwcm9qZWN0cy91YnVudHUtb3MtY2xvdWQvZ2xvYmFsL2ltYWdlcy91YnVudHUtbWluaW1hbC0xODA0LWJpb25pYy12MjAyMDEwMTQiLCAiem9uZSI6ICJ1cy1lYXN0MS1jIiwgIm5ldHdvcmsiOiAiZGVmYXVsdCIsICJuZXR3b3JrX3Byb2plY3RpZCI6IG51bGwsICJwcm9qZWN0aWQiOiAiIiwgIm9uX2hvc3RfbWFpbnRlbmFuY2UiOiAiVEVSTUlOQVRFIiwgIm1hY2hpbmVfdHlwZSI6ICJuMS1zdGFuZGFyZC0xIiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDUwLCAibmdwdXMiOiAiIiwgImdwdV90eXBlIjogIiIsICJkaXNrX3R5cGUiOiAicGQtc3RhbmRhcmQiLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QiLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJpbnN0YW5jZV9sYWJlbHMiOiB7ImNvbnRhaW5lcl92bSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgInNlcnZpY2VfYWNjb3VudCI6ICJkZWZhdWx0In0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGlvbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9LCAiaWJtIjogeyJhcGlfa2V5IjogbnVsbCwgImltYWdlIjogImdoY3IuaW8vZGFzay9kYXNrOmxhdGVzdCIsICJyZWdpb24iOiAidXMtZWFzdCIsICJwcm9qZWN0X2lkIjogbnVsbCwgInNjaGVkdWxlcl9jcHUiOiAiMS4wIiwgInNjaGVkdWxlcl9tZW0iOiAiNEciLCAic2NoZWR1bGVyX3RpbWVvdXQiOiA2MDAsICJ3b3JrZXJfY3B1IjogIjIuMCIsICJ3b3JrZXJfbWVtIjogIjhHIiwgIndvcmtlcl90aHJlYWRzIjogMX0sICJvcGVuc3RhY2siOiB7InJlZ2lvbiI6ICJSZWdpb25PbmUiLCAic2l6ZSI6IG51bGwsICJhdXRoX3VybCI6IG51bGwsICJhcHBsaWNhdGlvbl9jcmVkZW50aWFsX2lkIjogbnVsbCwgImFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfc2VjcmV0IjogbnVsbCwgImF1dGhfdHlwZSI6ICJ2M2FwcGxpY2F0aW9uY3JlZGVudGlhbCIsICJuZXR3b3JrX2lkIjogbnVsbCwgImltYWdlIjogbnVsbCwgImtleXBhaXJfbmFtZSI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJleHRlcm5hbF9uZXR3b3JrX2lkIjogbnVsbCwgImNyZWF0ZV9mbG9hdGluZ19pcCI6IGZhbHNlLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QifX19\" python -m distributed.cli.dask_scheduler'\n", + "\n", + " \n", + " # Shutdown when command is done\n", + " - shutdown -h now\n", + " \n", + "Created instance i-0311fa1be461bd2a9 as dask-d7369884-scheduler\n", + "Waiting for scheduler to run at 18.220.177.101:8786\n", + "Scheduler is running\n", + "Creating worker instance\n", + "Creating worker instance\n", + "Creating worker instance\n", + "Creating worker instance\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/scottroutledge/miniforge3/envs/benchmark_dask/lib/python3.10/contextlib.py:142: UserWarning: Creating your cluster is taking a surprisingly long time. This is likely due to pending resources. Hang tight! \n", + " next(self.gen)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Cloud init\n", + "==========\n", + "\n", + "\n", + "#cloud-config\n", + "\n", + "\n", + "# Bootstrap\n", + "packages:\n", + " - apt-transport-https\n", + " - ca-certificates\n", + " - curl\n", + " - gnupg-agent\n", + " - software-properties-common\n", + " - ubuntu-drivers-common\n", + "\n", + "# Enable ipv4 forwarding, required on CIS hardened machines\n", + "write_files:\n", + " - path: /etc/sysctl.d/enabled_ipv4_forwarding.conf\n", + " content: |\n", + " net.ipv4.conf.all.forwarding=1\n", + "\n", + "# create the docker group\n", + "groups:\n", + " - docker\n", + "\n", + "# Add default auto created user to docker group\n", + "system_info:\n", + " default_user:\n", + " groups: [docker]\n", + "\n", + "\n", + "runcmd:\n", + " \n", + " # Install Docker\n", + " - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -\n", + " - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"\n", + " - apt-get update -y\n", + " - apt-get install -y docker-ce docker-ce-cli containerd.io\n", + " - systemctl start docker\n", + " - systemctl enable docker\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " # Run container\n", + " - 'docker run --net=host -e EXTRA_CONDA_PACKAGES=\"s3fs==2024.10.0\" daskdev/dask:latest env DASK_INTERNAL_INHERIT_CONFIG=\"eyJ0ZW1wb3JhcnktZGlyZWN0b3J5IjogbnVsbCwgInZpc3VhbGl6YXRpb24iOiB7ImVuZ2luZSI6IG51bGx9LCAidG9rZW5pemUiOiB7ImVuc3VyZS1kZXRlcm1pbmlzdGljIjogZmFsc2V9LCAiZGF0YWZyYW1lIjogeyJiYWNrZW5kIjogInBhbmRhcyIsICJzaHVmZmxlIjogeyJtZXRob2QiOiBudWxsLCAiY29tcHJlc3Npb24iOiBudWxsfSwgInBhcnF1ZXQiOiB7Im1ldGFkYXRhLXRhc2stc2l6ZS1sb2NhbCI6IDUxMiwgIm1ldGFkYXRhLXRhc2stc2l6ZS1yZW1vdGUiOiAxLCAibWluaW11bS1wYXJ0aXRpb24tc2l6ZSI6IDc1MDAwMDAwfSwgImNvbnZlcnQtc3RyaW5nIjogbnVsbCwgInF1ZXJ5LXBsYW5uaW5nIjogbnVsbH0sICJhcnJheSI6IHsiYmFja2VuZCI6ICJudW1weSIsICJjaHVuay1zaXplIjogIjEyOE1pQiIsICJjaHVuay1zaXplLXRvbGVyYW5jZSI6IDEuMjUsICJyZWNodW5rIjogeyJtZXRob2QiOiBudWxsLCAidGhyZXNob2xkIjogNH0sICJzdmciOiB7InNpemUiOiAxMjB9LCAic2xpY2luZyI6IHsic3BsaXQtbGFyZ2UtY2h1bmtzIjogbnVsbH0sICJxdWVyeS1wbGFubmluZyI6IG51bGx9LCAib3B0aW1pemF0aW9uIjogeyJhbm5vdGF0aW9ucyI6IHsiZnVzZSI6IHRydWV9LCAiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJhZG1pbiI6IHsidHJhY2ViYWNrIjogeyJzaG9ydGVuIjogWyJjb25jdXJyZW50W1xcXFxcXC9dZnV0dXJlc1tcXFxcXFwvXSIsICJkYXNrW1xcXFxcXC9dKGJhc2V8Y29yZXxsb2NhbHxtdWx0aXByb2Nlc3Npbmd8b3B0aW1pemF0aW9ufHRocmVhZGVkfHV0aWxzKVxcLnB5IiwgImRhc2tbXFxcXFxcL11hcnJheVtcXFxcXFwvXWNvcmVcXC5weSIsICJkYXNrW1xcXFxcXC9dZGF0YWZyYW1lW1xcXFxcXC9dKGNvcmV8bWV0aG9kcylcXC5weSIsICJkaXN0cmlidXRlZFtcXFxcXFwvXShjbGllbnR8c2NoZWR1bGVyfHV0aWxzfHdvcmtlcilcXC5weSIsICJ0b3JuYWRvW1xcXFxcXC9dZ2VuXFwucHkiLCAicGFuZGFzW1xcXFxcXC9dY29yZVtcXFxcXFwvXSJdfX0sICJkaXN0cmlidXRlZCI6IHsidmVyc2lvbiI6IDIsICJzY2hlZHVsZXIiOiB7ImFsbG93ZWQtZmFpbHVyZXMiOiAzLCAiYmFuZHdpZHRoIjogMTAwMDAwMDAwLCAiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAiY29udGFjdC1hZGRyZXNzIjogbnVsbCwgImRlZmF1bHQtZGF0YS1zaXplIjogIjFraUIiLCAiZXZlbnRzLWNsZWFudXAtZGVsYXkiOiAiMWgiLCAiaWRsZS10aW1lb3V0IjogbnVsbCwgIm5vLXdvcmtlcnMtdGltZW91dCI6IG51bGwsICJ3b3JrLXN0ZWFsaW5nIjogdHJ1ZSwgIndvcmstc3RlYWxpbmctaW50ZXJ2YWwiOiAiMTAwbXMiLCAid29ya2VyLXNhdHVyYXRpb24iOiAxLjEsICJ3b3JrZXItdHRsIjogIjUgbWludXRlcyIsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyIsICJzcGxpdC10YXNrc2h1ZmZsZSI6ICIxdXMiLCAic3BsaXQtc3RhZ2UiOiAiMXVzIn0sICJ2YWxpZGF0ZSI6IGZhbHNlLCAiZGFzaGJvYXJkIjogeyJzdGF0dXMiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDB9LCAidGFza3MiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDAwMH0sICJ0bHMiOiB7ImNhLWZpbGUiOiBudWxsLCAia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImJva2VoLWFwcGxpY2F0aW9uIjogeyJhbGxvd193ZWJzb2NrZXRfb3JpZ2luIjogWyIqIl0sICJrZWVwX2FsaXZlX21pbGxpc2Vjb25kcyI6IDUwMCwgImNoZWNrX3VudXNlZF9zZXNzaW9uc19taWxsaXNlY29uZHMiOiA1MDB9fSwgImxvY2tzIjogeyJsZWFzZS12YWxpZGF0aW9uLWludGVydmFsIjogIjEwcyIsICJsZWFzZS10aW1lb3V0IjogIjMwcyJ9LCAiaHR0cCI6IHsicm91dGVzIjogWyJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5wcm9tZXRoZXVzIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmluZm8iLCAiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIuanNvbiIsICJkaXN0cmlidXRlZC5odHRwLmhlYWx0aCIsICJkaXN0cmlidXRlZC5odHRwLnByb3h5IiwgImRpc3RyaWJ1dGVkLmh0dHAuc3RhdGljcyJdfSwgImFsbG93ZWQtaW1wb3J0cyI6IFsiZGFzayIsICJkaXN0cmlidXRlZCJdLCAiYWN0aXZlLW1lbW9yeS1tYW5hZ2VyIjogeyJzdGFydCI6IHRydWUsICJpbnRlcnZhbCI6ICIycyIsICJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAicG9saWNpZXMiOiBbeyJjbGFzcyI6ICJkaXN0cmlidXRlZC5hY3RpdmVfbWVtb3J5X21hbmFnZXIuUmVkdWNlUmVwbGljYXMifV19fSwgIndvcmtlciI6IHsiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAibXVsdGlwcm9jZXNzaW5nLW1ldGhvZCI6ICJzcGF3biIsICJ1c2UtZmlsZS1sb2NraW5nIjogdHJ1ZSwgInRyYW5zZmVyIjogeyJtZXNzYWdlLWJ5dGVzLWxpbWl0IjogIjUwTUIifSwgImNvbm5lY3Rpb25zIjogeyJvdXRnb2luZyI6IDUwLCAiaW5jb21pbmciOiAxMH0sICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImRhZW1vbiI6IHRydWUsICJ2YWxpZGF0ZSI6IGZhbHNlLCAicmVzb3VyY2VzIjoge30sICJsaWZldGltZSI6IHsiZHVyYXRpb24iOiBudWxsLCAic3RhZ2dlciI6ICIwIHNlY29uZHMiLCAicmVzdGFydCI6IGZhbHNlfSwgInByb2ZpbGUiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMTBtcyIsICJjeWNsZSI6ICIxMDAwbXMiLCAibG93LWxldmVsIjogZmFsc2V9LCAibWVtb3J5IjogeyJyZWNlbnQtdG8tb2xkLXRpbWUiOiAiMzBzIiwgInJlYmFsYW5jZSI6IHsibWVhc3VyZSI6ICJvcHRpbWlzdGljIiwgInNlbmRlci1taW4iOiAwLjMsICJyZWNpcGllbnQtbWF4IjogMC42LCAic2VuZGVyLXJlY2lwaWVudC1nYXAiOiAwLjF9LCAidHJhbnNmZXIiOiAwLjEsICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NSwgIm1heC1zcGlsbCI6IGZhbHNlLCAic3BpbGwtY29tcHJlc3Npb24iOiAiYXV0byIsICJtb25pdG9yLWludGVydmFsIjogIjEwMG1zIn0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7fSwgInByZS1zcGF3bi1lbnZpcm9uIjogeyJNQUxMT0NfVFJJTV9USFJFU0hPTERfIjogNjU1MzYsICJPTVBfTlVNX1RIUkVBRFMiOiAxLCAiTUtMX05VTV9USFJFQURTIjogMSwgIk9QRU5CTEFTX05VTV9USFJFQURTIjogMX19LCAiY2xpZW50IjogeyJoZWFydGJlYXQiOiAiNXMiLCAic2NoZWR1bGVyLWluZm8taW50ZXJ2YWwiOiAiMnMiLCAic2VjdXJpdHktbG9hZGVyIjogbnVsbCwgInByZWxvYWQiOiBbXSwgInByZWxvYWQtYXJndiI6IFtdfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogZmFsc2UsICJzaGFyZCI6ICI2NE1pQiIsICJvZmZsb2FkIjogIjEwTWlCIiwgImRlZmF1bHQtc2NoZW1lIjogInRjcCIsICJzb2NrZXQtYmFja2xvZyI6IDIwNDgsICJ1Y3giOiB7ImN1ZGEtY29weSI6IG51bGwsICJ0Y3AiOiBudWxsLCAibnZsaW5rIjogbnVsbCwgImluZmluaWJhbmQiOiBudWxsLCAicmRtYWNtIjogbnVsbCwgImNyZWF0ZS1jdWRhLWNvbnRleHQiOiBudWxsLCAiZW52aXJvbm1lbnQiOiB7fX0sICJ6c3RkIjogeyJsZXZlbCI6IDMsICJ0aHJlYWRzIjogMH0sICJ0aW1lb3V0cyI6IHsiY29ubmVjdCI6ICIzMHMiLCAidGNwIjogIjMwcyJ9LCAicmVxdWlyZS1lbmNyeXB0aW9uIjogbnVsbCwgInRscyI6IHsiY2lwaGVycyI6IG51bGwsICJtaW4tdmVyc2lvbiI6IDEuMiwgIm1heC12ZXJzaW9uIjogbnVsbCwgImNhLWZpbGUiOiBudWxsLCAic2NoZWR1bGVyIjogeyJjZXJ0IjogbnVsbCwgImtleSI6IG51bGx9LCAid29ya2VyIjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9LCAiY2xpZW50IjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9fSwgIndlYnNvY2tldHMiOiB7InNoYXJkIjogIjhNaUIifX0sICJkaWFnbm9zdGljcyI6IHsibnZtbCI6IHRydWUsICJjdWRmIjogZmFsc2UsICJjb21wdXRhdGlvbnMiOiB7Im1heC1oaXN0b3J5IjogMTAwLCAibmZyYW1lcyI6IDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiYXN5bmNpbyIsICJmdW5jdG9vbHMiLCAidGhyZWFkaW5nIiwgImRhdGFzaGFkZXIiLCAiZGFzayIsICJkZWJ1Z3B5IiwgImRpc3RyaWJ1dGVkIiwgImNvaWxlZCIsICJjdWRmIiwgImN1bWwiLCAibWF0cGxvdGxpYiIsICJwbHVnZ3kiLCAicHJlZmVjdCIsICJyZWNodW5rZXIiLCAieGFycmF5IiwgInhnYm9vc3QiLCAieGRpc3QiLCAiX19jaGFubmVsZXhlY19fIiwgImV4ZWNuZXQiXSwgImlnbm9yZS1maWxlcyI6IFsicnVucHlcXC5weSIsICJweXRlc3QiLCAicHlcXC50ZXN0IiwgInB5dGVzdC1zY3JpcHRcXC5weSIsICJfcHl0ZXN0IiwgInB5Y2hhcm0iLCAidnNjb2RlX3B5dGVzdCIsICJnZXRfb3V0cHV0X3ZpYV9tYXJrZXJzXFwucHkiXX0sICJlcnJlZC10YXNrcyI6IHsibWF4LWhpc3RvcnkiOiAxMDB9fSwgInAycCI6IHsiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMTAsICJkZWxheSI6IHsibWluIjogIjFzIiwgIm1heCI6ICIzMHMifX19LCAiZGlzayI6IHRydWV9LCAiZGFzaGJvYXJkIjogeyJsaW5rIjogIntzY2hlbWV9Oi8ve2hvc3R9Ontwb3J0fS9zdGF0dXMiLCAiZXhwb3J0LXRvb2wiOiBmYWxzZSwgImdyYXBoLW1heC1pdGVtcyI6IDUwMDAsICJwcm9tZXRoZXVzIjogeyJuYW1lc3BhY2UiOiAiZGFzayJ9fSwgImFkbWluIjogeyJsYXJnZS1ncmFwaC13YXJuaW5nLXRocmVzaG9sZCI6ICIxME1CIiwgInRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MiLCAiY3ljbGUiOiAiMXMifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShhc2N0aW1lKXMgLSAlKG5hbWUpcyAtICUobGV2ZWxuYW1lKXMgLSAlKG1lc3NhZ2UpcyIsICJsb3ctbGV2ZWwtbG9nLWxlbmd0aCI6IDEwMDAsICJwZGItb24tZXJyIjogZmFsc2UsICJzeXN0ZW0tbW9uaXRvciI6IHsiaW50ZXJ2YWwiOiAiNTAwbXMiLCAibG9nLWxlbmd0aCI6IDcyMDAsICJkaXNrIjogdHJ1ZSwgImhvc3QtY3B1IjogZmFsc2UsICJnaWwiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMW1zIn19LCAiZXZlbnQtbG9vcCI6ICJ0b3JuYWRvIn0sICJybW0iOiB7InBvb2wtc2l6ZSI6IG51bGx9fSwgImNsb3VkcHJvdmlkZXIiOiB7ImVjcyI6IHsiZmFyZ2F0ZV9zY2hlZHVsZXIiOiBmYWxzZSwgImZhcmdhdGVfc3BvdCI6IGZhbHNlLCAiZmFyZ2F0ZV93b3JrZXJzIjogZmFsc2UsICJmYXJnYXRlX3VzZV9wcml2YXRlX2lwIjogZmFsc2UsICJzY2hlZHVsZXJfY3B1IjogMTAyNCwgInNjaGVkdWxlcl9tZW0iOiA0MDk2LCAid29ya2VyX2NwdSI6IDQwOTYsICJ3b3JrZXJfbWVtIjogMTYzODQsICJ3b3JrZXJfZ3B1IjogMCwgIm5fd29ya2VycyI6IDAsICJzY2hlZHVsZXJfdGltZW91dCI6ICI1IG1pbnV0ZXMiLCAiaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJjcHVfYXJjaGl0ZWN0dXJlIjogIlg4Nl82NCIsICJncHVfaW1hZ2UiOiAicmFwaWRzYWkvcmFwaWRzYWk6bGF0ZXN0IiwgImNsdXN0ZXJfbmFtZV90ZW1wbGF0ZSI6ICJkYXNrLXt1dWlkfSIsICJjbHVzdGVyX2FybiI6ICIiLCAiZXhlY3V0aW9uX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfcG9saWNpZXMiOiBbXSwgImNsb3Vkd2F0Y2hfbG9nc19ncm91cCI6ICIiLCAiY2xvdWR3YXRjaF9sb2dzX3N0cmVhbV9wcmVmaXgiOiAie2NsdXN0ZXJfbmFtZX0iLCAiY2xvdWR3YXRjaF9sb2dzX2RlZmF1bHRfcmV0ZW50aW9uIjogMzAsICJ2cGMiOiAiZGVmYXVsdCIsICJzdWJuZXRzIjogW10sICJzZWN1cml0eV9ncm91cHMiOiBbXSwgInRhZ3MiOiB7fSwgImVudmlyb25tZW50Ijoge30sICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJhbWkiOiBudWxsLCAiaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJzY2hlZHVsZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJ3b3JrZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2cGMiOiBudWxsLCAic3VibmV0X2lkIjogbnVsbCwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAiZmlsZXN5c3RlbV9zaXplIjogNDAsICJrZXlfbmFtZSI6IG51bGwsICJpYW1faW5zdGFuY2VfcHJvZmlsZSI6IHt9LCAiaW5zdGFuY2VfdGFncyI6IHsiY3JlYXRlZEJ5IjogImRhc2stY2xvdWRwcm92aWRlciJ9LCAidm9sdW1lX3RhZ3MiOiB7ImNyZWF0ZWRCeSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgImVuYWJsZV9kZXRhaWxlZF9tb25pdG9yaW5nIjogZmFsc2UsICJ1c2VfcHJpdmF0ZV9pcCI6IGZhbHNlfSwgImF6dXJlIjogeyJsb2NhdGlvbiI6IG51bGwsICJyZXNvdXJjZV9ncm91cCI6IG51bGwsICJzdWJzY3JpcHRpb25faWQiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzdWJuZXQiOiBudWxsLCAic2VjdXJpdHlfZ3JvdXAiOiBudWxsLCAicHVibGljX2luZ3Jlc3MiOiB0cnVlLCAidm1fc2l6ZSI6ICJTdGFuZGFyZF9EUzFfdjIiLCAiZGlza19zaXplIjogNTAsICJzY2hlZHVsZXJfdm1fc2l6ZSI6IG51bGwsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2bV9pbWFnZSI6IHsicHVibGlzaGVyIjogIkNhbm9uaWNhbCIsICJvZmZlciI6ICJVYnVudHVTZXJ2ZXIiLCAic2t1IjogIjE4LjA0LUxUUyIsICJ2ZXJzaW9uIjogImxhdGVzdCJ9LCAiYm9vdHN0cmFwIjogdHJ1ZSwgImF1dG9fc2h1dGRvd24iOiB0cnVlLCAibWFya2V0cGxhY2VfcGxhbiI6IG51bGwsICJleHRyYV9vcHRpb25zIjoge319fSwgImRpZ2l0YWxvY2VhbiI6IHsidG9rZW4iOiBudWxsLCAicmVnaW9uIjogIm55YzMiLCAic2l6ZSI6ICJzLTF2Y3B1LTFnYiIsICJpbWFnZSI6ICJ1YnVudHUtMjAtMDQteDY0In0sICJnY3AiOiB7InNvdXJjZV9pbWFnZSI6ICJwcm9qZWN0cy91YnVudHUtb3MtY2xvdWQvZ2xvYmFsL2ltYWdlcy91YnVudHUtbWluaW1hbC0xODA0LWJpb25pYy12MjAyMDEwMTQiLCAiem9uZSI6ICJ1cy1lYXN0MS1jIiwgIm5ldHdvcmsiOiAiZGVmYXVsdCIsICJuZXR3b3JrX3Byb2plY3RpZCI6IG51bGwsICJwcm9qZWN0aWQiOiAiIiwgIm9uX2hvc3RfbWFpbnRlbmFuY2UiOiAiVEVSTUlOQVRFIiwgIm1hY2hpbmVfdHlwZSI6ICJuMS1zdGFuZGFyZC0xIiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDUwLCAibmdwdXMiOiAiIiwgImdwdV90eXBlIjogIiIsICJkaXNrX3R5cGUiOiAicGQtc3RhbmRhcmQiLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QiLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJpbnN0YW5jZV9sYWJlbHMiOiB7ImNvbnRhaW5lcl92bSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgInNlcnZpY2VfYWNjb3VudCI6ICJkZWZhdWx0In0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGlvbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9LCAiaWJtIjogeyJhcGlfa2V5IjogbnVsbCwgImltYWdlIjogImdoY3IuaW8vZGFzay9kYXNrOmxhdGVzdCIsICJyZWdpb24iOiAidXMtZWFzdCIsICJwcm9qZWN0X2lkIjogbnVsbCwgInNjaGVkdWxlcl9jcHUiOiAiMS4wIiwgInNjaGVkdWxlcl9tZW0iOiAiNEciLCAic2NoZWR1bGVyX3RpbWVvdXQiOiA2MDAsICJ3b3JrZXJfY3B1IjogIjIuMCIsICJ3b3JrZXJfbWVtIjogIjhHIiwgIndvcmtlcl90aHJlYWRzIjogMX0sICJvcGVuc3RhY2siOiB7InJlZ2lvbiI6ICJSZWdpb25PbmUiLCAic2l6ZSI6IG51bGwsICJhdXRoX3VybCI6IG51bGwsICJhcHBsaWNhdGlvbl9jcmVkZW50aWFsX2lkIjogbnVsbCwgImFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfc2VjcmV0IjogbnVsbCwgImF1dGhfdHlwZSI6ICJ2M2FwcGxpY2F0aW9uY3JlZGVudGlhbCIsICJuZXR3b3JrX2lkIjogbnVsbCwgImltYWdlIjogbnVsbCwgImtleXBhaXJfbmFtZSI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJleHRlcm5hbF9uZXR3b3JrX2lkIjogbnVsbCwgImNyZWF0ZV9mbG9hdGluZ19pcCI6IGZhbHNlLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QifX19\" python -m distributed.cli.dask_spec tcp://18.220.177.101:8786 --spec ''{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"name\": \"dask-d7369884-worker-87177d09\"}}'''\n", + "\n", + " \n", + " # Shutdown when command is done\n", + " - shutdown -h now\n", + " \n", + "\n", + "Cloud init\n", + "==========\n", + "\n", + "\n", + "#cloud-config\n", + "\n", + "\n", + "# Bootstrap\n", + "packages:\n", + " - apt-transport-https\n", + " - ca-certificates\n", + " - curl\n", + " - gnupg-agent\n", + " - software-properties-common\n", + " - ubuntu-drivers-common\n", + "\n", + "# Enable ipv4 forwarding, required on CIS hardened machines\n", + "write_files:\n", + " - path: /etc/sysctl.d/enabled_ipv4_forwarding.conf\n", + " content: |\n", + " net.ipv4.conf.all.forwarding=1\n", + "\n", + "# create the docker group\n", + "groups:\n", + " - docker\n", + "\n", + "# Add default auto created user to docker group\n", + "system_info:\n", + " default_user:\n", + " groups: [docker]\n", + "\n", + "\n", + "runcmd:\n", + " \n", + " # Install Docker\n", + " - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -\n", + " - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"\n", + " - apt-get update -y\n", + " - apt-get install -y docker-ce docker-ce-cli containerd.io\n", + " - systemctl start docker\n", + " - systemctl enable docker\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " # Run container\n", + " - 'docker run --net=host -e EXTRA_CONDA_PACKAGES=\"s3fs==2024.10.0\" daskdev/dask:latest env DASK_INTERNAL_INHERIT_CONFIG=\"eyJ0ZW1wb3JhcnktZGlyZWN0b3J5IjogbnVsbCwgInZpc3VhbGl6YXRpb24iOiB7ImVuZ2luZSI6IG51bGx9LCAidG9rZW5pemUiOiB7ImVuc3VyZS1kZXRlcm1pbmlzdGljIjogZmFsc2V9LCAiZGF0YWZyYW1lIjogeyJiYWNrZW5kIjogInBhbmRhcyIsICJzaHVmZmxlIjogeyJtZXRob2QiOiBudWxsLCAiY29tcHJlc3Npb24iOiBudWxsfSwgInBhcnF1ZXQiOiB7Im1ldGFkYXRhLXRhc2stc2l6ZS1sb2NhbCI6IDUxMiwgIm1ldGFkYXRhLXRhc2stc2l6ZS1yZW1vdGUiOiAxLCAibWluaW11bS1wYXJ0aXRpb24tc2l6ZSI6IDc1MDAwMDAwfSwgImNvbnZlcnQtc3RyaW5nIjogbnVsbCwgInF1ZXJ5LXBsYW5uaW5nIjogbnVsbH0sICJhcnJheSI6IHsiYmFja2VuZCI6ICJudW1weSIsICJjaHVuay1zaXplIjogIjEyOE1pQiIsICJjaHVuay1zaXplLXRvbGVyYW5jZSI6IDEuMjUsICJyZWNodW5rIjogeyJtZXRob2QiOiBudWxsLCAidGhyZXNob2xkIjogNH0sICJzdmciOiB7InNpemUiOiAxMjB9LCAic2xpY2luZyI6IHsic3BsaXQtbGFyZ2UtY2h1bmtzIjogbnVsbH0sICJxdWVyeS1wbGFubmluZyI6IG51bGx9LCAib3B0aW1pemF0aW9uIjogeyJhbm5vdGF0aW9ucyI6IHsiZnVzZSI6IHRydWV9LCAiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJhZG1pbiI6IHsidHJhY2ViYWNrIjogeyJzaG9ydGVuIjogWyJjb25jdXJyZW50W1xcXFxcXC9dZnV0dXJlc1tcXFxcXFwvXSIsICJkYXNrW1xcXFxcXC9dKGJhc2V8Y29yZXxsb2NhbHxtdWx0aXByb2Nlc3Npbmd8b3B0aW1pemF0aW9ufHRocmVhZGVkfHV0aWxzKVxcLnB5IiwgImRhc2tbXFxcXFxcL11hcnJheVtcXFxcXFwvXWNvcmVcXC5weSIsICJkYXNrW1xcXFxcXC9dZGF0YWZyYW1lW1xcXFxcXC9dKGNvcmV8bWV0aG9kcylcXC5weSIsICJkaXN0cmlidXRlZFtcXFxcXFwvXShjbGllbnR8c2NoZWR1bGVyfHV0aWxzfHdvcmtlcilcXC5weSIsICJ0b3JuYWRvW1xcXFxcXC9dZ2VuXFwucHkiLCAicGFuZGFzW1xcXFxcXC9dY29yZVtcXFxcXFwvXSJdfX0sICJkaXN0cmlidXRlZCI6IHsidmVyc2lvbiI6IDIsICJzY2hlZHVsZXIiOiB7ImFsbG93ZWQtZmFpbHVyZXMiOiAzLCAiYmFuZHdpZHRoIjogMTAwMDAwMDAwLCAiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAiY29udGFjdC1hZGRyZXNzIjogbnVsbCwgImRlZmF1bHQtZGF0YS1zaXplIjogIjFraUIiLCAiZXZlbnRzLWNsZWFudXAtZGVsYXkiOiAiMWgiLCAiaWRsZS10aW1lb3V0IjogbnVsbCwgIm5vLXdvcmtlcnMtdGltZW91dCI6IG51bGwsICJ3b3JrLXN0ZWFsaW5nIjogdHJ1ZSwgIndvcmstc3RlYWxpbmctaW50ZXJ2YWwiOiAiMTAwbXMiLCAid29ya2VyLXNhdHVyYXRpb24iOiAxLjEsICJ3b3JrZXItdHRsIjogIjUgbWludXRlcyIsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyIsICJzcGxpdC10YXNrc2h1ZmZsZSI6ICIxdXMiLCAic3BsaXQtc3RhZ2UiOiAiMXVzIn0sICJ2YWxpZGF0ZSI6IGZhbHNlLCAiZGFzaGJvYXJkIjogeyJzdGF0dXMiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDB9LCAidGFza3MiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDAwMH0sICJ0bHMiOiB7ImNhLWZpbGUiOiBudWxsLCAia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImJva2VoLWFwcGxpY2F0aW9uIjogeyJhbGxvd193ZWJzb2NrZXRfb3JpZ2luIjogWyIqIl0sICJrZWVwX2FsaXZlX21pbGxpc2Vjb25kcyI6IDUwMCwgImNoZWNrX3VudXNlZF9zZXNzaW9uc19taWxsaXNlY29uZHMiOiA1MDB9fSwgImxvY2tzIjogeyJsZWFzZS12YWxpZGF0aW9uLWludGVydmFsIjogIjEwcyIsICJsZWFzZS10aW1lb3V0IjogIjMwcyJ9LCAiaHR0cCI6IHsicm91dGVzIjogWyJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5wcm9tZXRoZXVzIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmluZm8iLCAiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIuanNvbiIsICJkaXN0cmlidXRlZC5odHRwLmhlYWx0aCIsICJkaXN0cmlidXRlZC5odHRwLnByb3h5IiwgImRpc3RyaWJ1dGVkLmh0dHAuc3RhdGljcyJdfSwgImFsbG93ZWQtaW1wb3J0cyI6IFsiZGFzayIsICJkaXN0cmlidXRlZCJdLCAiYWN0aXZlLW1lbW9yeS1tYW5hZ2VyIjogeyJzdGFydCI6IHRydWUsICJpbnRlcnZhbCI6ICIycyIsICJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAicG9saWNpZXMiOiBbeyJjbGFzcyI6ICJkaXN0cmlidXRlZC5hY3RpdmVfbWVtb3J5X21hbmFnZXIuUmVkdWNlUmVwbGljYXMifV19fSwgIndvcmtlciI6IHsiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAibXVsdGlwcm9jZXNzaW5nLW1ldGhvZCI6ICJzcGF3biIsICJ1c2UtZmlsZS1sb2NraW5nIjogdHJ1ZSwgInRyYW5zZmVyIjogeyJtZXNzYWdlLWJ5dGVzLWxpbWl0IjogIjUwTUIifSwgImNvbm5lY3Rpb25zIjogeyJvdXRnb2luZyI6IDUwLCAiaW5jb21pbmciOiAxMH0sICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImRhZW1vbiI6IHRydWUsICJ2YWxpZGF0ZSI6IGZhbHNlLCAicmVzb3VyY2VzIjoge30sICJsaWZldGltZSI6IHsiZHVyYXRpb24iOiBudWxsLCAic3RhZ2dlciI6ICIwIHNlY29uZHMiLCAicmVzdGFydCI6IGZhbHNlfSwgInByb2ZpbGUiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMTBtcyIsICJjeWNsZSI6ICIxMDAwbXMiLCAibG93LWxldmVsIjogZmFsc2V9LCAibWVtb3J5IjogeyJyZWNlbnQtdG8tb2xkLXRpbWUiOiAiMzBzIiwgInJlYmFsYW5jZSI6IHsibWVhc3VyZSI6ICJvcHRpbWlzdGljIiwgInNlbmRlci1taW4iOiAwLjMsICJyZWNpcGllbnQtbWF4IjogMC42LCAic2VuZGVyLXJlY2lwaWVudC1nYXAiOiAwLjF9LCAidHJhbnNmZXIiOiAwLjEsICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NSwgIm1heC1zcGlsbCI6IGZhbHNlLCAic3BpbGwtY29tcHJlc3Npb24iOiAiYXV0byIsICJtb25pdG9yLWludGVydmFsIjogIjEwMG1zIn0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7fSwgInByZS1zcGF3bi1lbnZpcm9uIjogeyJNQUxMT0NfVFJJTV9USFJFU0hPTERfIjogNjU1MzYsICJPTVBfTlVNX1RIUkVBRFMiOiAxLCAiTUtMX05VTV9USFJFQURTIjogMSwgIk9QRU5CTEFTX05VTV9USFJFQURTIjogMX19LCAiY2xpZW50IjogeyJoZWFydGJlYXQiOiAiNXMiLCAic2NoZWR1bGVyLWluZm8taW50ZXJ2YWwiOiAiMnMiLCAic2VjdXJpdHktbG9hZGVyIjogbnVsbCwgInByZWxvYWQiOiBbXSwgInByZWxvYWQtYXJndiI6IFtdfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogZmFsc2UsICJzaGFyZCI6ICI2NE1pQiIsICJvZmZsb2FkIjogIjEwTWlCIiwgImRlZmF1bHQtc2NoZW1lIjogInRjcCIsICJzb2NrZXQtYmFja2xvZyI6IDIwNDgsICJ1Y3giOiB7ImN1ZGEtY29weSI6IG51bGwsICJ0Y3AiOiBudWxsLCAibnZsaW5rIjogbnVsbCwgImluZmluaWJhbmQiOiBudWxsLCAicmRtYWNtIjogbnVsbCwgImNyZWF0ZS1jdWRhLWNvbnRleHQiOiBudWxsLCAiZW52aXJvbm1lbnQiOiB7fX0sICJ6c3RkIjogeyJsZXZlbCI6IDMsICJ0aHJlYWRzIjogMH0sICJ0aW1lb3V0cyI6IHsiY29ubmVjdCI6ICIzMHMiLCAidGNwIjogIjMwcyJ9LCAicmVxdWlyZS1lbmNyeXB0aW9uIjogbnVsbCwgInRscyI6IHsiY2lwaGVycyI6IG51bGwsICJtaW4tdmVyc2lvbiI6IDEuMiwgIm1heC12ZXJzaW9uIjogbnVsbCwgImNhLWZpbGUiOiBudWxsLCAic2NoZWR1bGVyIjogeyJjZXJ0IjogbnVsbCwgImtleSI6IG51bGx9LCAid29ya2VyIjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9LCAiY2xpZW50IjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9fSwgIndlYnNvY2tldHMiOiB7InNoYXJkIjogIjhNaUIifX0sICJkaWFnbm9zdGljcyI6IHsibnZtbCI6IHRydWUsICJjdWRmIjogZmFsc2UsICJjb21wdXRhdGlvbnMiOiB7Im1heC1oaXN0b3J5IjogMTAwLCAibmZyYW1lcyI6IDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiYXN5bmNpbyIsICJmdW5jdG9vbHMiLCAidGhyZWFkaW5nIiwgImRhdGFzaGFkZXIiLCAiZGFzayIsICJkZWJ1Z3B5IiwgImRpc3RyaWJ1dGVkIiwgImNvaWxlZCIsICJjdWRmIiwgImN1bWwiLCAibWF0cGxvdGxpYiIsICJwbHVnZ3kiLCAicHJlZmVjdCIsICJyZWNodW5rZXIiLCAieGFycmF5IiwgInhnYm9vc3QiLCAieGRpc3QiLCAiX19jaGFubmVsZXhlY19fIiwgImV4ZWNuZXQiXSwgImlnbm9yZS1maWxlcyI6IFsicnVucHlcXC5weSIsICJweXRlc3QiLCAicHlcXC50ZXN0IiwgInB5dGVzdC1zY3JpcHRcXC5weSIsICJfcHl0ZXN0IiwgInB5Y2hhcm0iLCAidnNjb2RlX3B5dGVzdCIsICJnZXRfb3V0cHV0X3ZpYV9tYXJrZXJzXFwucHkiXX0sICJlcnJlZC10YXNrcyI6IHsibWF4LWhpc3RvcnkiOiAxMDB9fSwgInAycCI6IHsiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMTAsICJkZWxheSI6IHsibWluIjogIjFzIiwgIm1heCI6ICIzMHMifX19LCAiZGlzayI6IHRydWV9LCAiZGFzaGJvYXJkIjogeyJsaW5rIjogIntzY2hlbWV9Oi8ve2hvc3R9Ontwb3J0fS9zdGF0dXMiLCAiZXhwb3J0LXRvb2wiOiBmYWxzZSwgImdyYXBoLW1heC1pdGVtcyI6IDUwMDAsICJwcm9tZXRoZXVzIjogeyJuYW1lc3BhY2UiOiAiZGFzayJ9fSwgImFkbWluIjogeyJsYXJnZS1ncmFwaC13YXJuaW5nLXRocmVzaG9sZCI6ICIxME1CIiwgInRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MiLCAiY3ljbGUiOiAiMXMifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShhc2N0aW1lKXMgLSAlKG5hbWUpcyAtICUobGV2ZWxuYW1lKXMgLSAlKG1lc3NhZ2UpcyIsICJsb3ctbGV2ZWwtbG9nLWxlbmd0aCI6IDEwMDAsICJwZGItb24tZXJyIjogZmFsc2UsICJzeXN0ZW0tbW9uaXRvciI6IHsiaW50ZXJ2YWwiOiAiNTAwbXMiLCAibG9nLWxlbmd0aCI6IDcyMDAsICJkaXNrIjogdHJ1ZSwgImhvc3QtY3B1IjogZmFsc2UsICJnaWwiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMW1zIn19LCAiZXZlbnQtbG9vcCI6ICJ0b3JuYWRvIn0sICJybW0iOiB7InBvb2wtc2l6ZSI6IG51bGx9fSwgImNsb3VkcHJvdmlkZXIiOiB7ImVjcyI6IHsiZmFyZ2F0ZV9zY2hlZHVsZXIiOiBmYWxzZSwgImZhcmdhdGVfc3BvdCI6IGZhbHNlLCAiZmFyZ2F0ZV93b3JrZXJzIjogZmFsc2UsICJmYXJnYXRlX3VzZV9wcml2YXRlX2lwIjogZmFsc2UsICJzY2hlZHVsZXJfY3B1IjogMTAyNCwgInNjaGVkdWxlcl9tZW0iOiA0MDk2LCAid29ya2VyX2NwdSI6IDQwOTYsICJ3b3JrZXJfbWVtIjogMTYzODQsICJ3b3JrZXJfZ3B1IjogMCwgIm5fd29ya2VycyI6IDAsICJzY2hlZHVsZXJfdGltZW91dCI6ICI1IG1pbnV0ZXMiLCAiaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJjcHVfYXJjaGl0ZWN0dXJlIjogIlg4Nl82NCIsICJncHVfaW1hZ2UiOiAicmFwaWRzYWkvcmFwaWRzYWk6bGF0ZXN0IiwgImNsdXN0ZXJfbmFtZV90ZW1wbGF0ZSI6ICJkYXNrLXt1dWlkfSIsICJjbHVzdGVyX2FybiI6ICIiLCAiZXhlY3V0aW9uX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfcG9saWNpZXMiOiBbXSwgImNsb3Vkd2F0Y2hfbG9nc19ncm91cCI6ICIiLCAiY2xvdWR3YXRjaF9sb2dzX3N0cmVhbV9wcmVmaXgiOiAie2NsdXN0ZXJfbmFtZX0iLCAiY2xvdWR3YXRjaF9sb2dzX2RlZmF1bHRfcmV0ZW50aW9uIjogMzAsICJ2cGMiOiAiZGVmYXVsdCIsICJzdWJuZXRzIjogW10sICJzZWN1cml0eV9ncm91cHMiOiBbXSwgInRhZ3MiOiB7fSwgImVudmlyb25tZW50Ijoge30sICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJhbWkiOiBudWxsLCAiaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJzY2hlZHVsZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJ3b3JrZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2cGMiOiBudWxsLCAic3VibmV0X2lkIjogbnVsbCwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAiZmlsZXN5c3RlbV9zaXplIjogNDAsICJrZXlfbmFtZSI6IG51bGwsICJpYW1faW5zdGFuY2VfcHJvZmlsZSI6IHt9LCAiaW5zdGFuY2VfdGFncyI6IHsiY3JlYXRlZEJ5IjogImRhc2stY2xvdWRwcm92aWRlciJ9LCAidm9sdW1lX3RhZ3MiOiB7ImNyZWF0ZWRCeSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgImVuYWJsZV9kZXRhaWxlZF9tb25pdG9yaW5nIjogZmFsc2UsICJ1c2VfcHJpdmF0ZV9pcCI6IGZhbHNlfSwgImF6dXJlIjogeyJsb2NhdGlvbiI6IG51bGwsICJyZXNvdXJjZV9ncm91cCI6IG51bGwsICJzdWJzY3JpcHRpb25faWQiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzdWJuZXQiOiBudWxsLCAic2VjdXJpdHlfZ3JvdXAiOiBudWxsLCAicHVibGljX2luZ3Jlc3MiOiB0cnVlLCAidm1fc2l6ZSI6ICJTdGFuZGFyZF9EUzFfdjIiLCAiZGlza19zaXplIjogNTAsICJzY2hlZHVsZXJfdm1fc2l6ZSI6IG51bGwsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2bV9pbWFnZSI6IHsicHVibGlzaGVyIjogIkNhbm9uaWNhbCIsICJvZmZlciI6ICJVYnVudHVTZXJ2ZXIiLCAic2t1IjogIjE4LjA0LUxUUyIsICJ2ZXJzaW9uIjogImxhdGVzdCJ9LCAiYm9vdHN0cmFwIjogdHJ1ZSwgImF1dG9fc2h1dGRvd24iOiB0cnVlLCAibWFya2V0cGxhY2VfcGxhbiI6IG51bGwsICJleHRyYV9vcHRpb25zIjoge319fSwgImRpZ2l0YWxvY2VhbiI6IHsidG9rZW4iOiBudWxsLCAicmVnaW9uIjogIm55YzMiLCAic2l6ZSI6ICJzLTF2Y3B1LTFnYiIsICJpbWFnZSI6ICJ1YnVudHUtMjAtMDQteDY0In0sICJnY3AiOiB7InNvdXJjZV9pbWFnZSI6ICJwcm9qZWN0cy91YnVudHUtb3MtY2xvdWQvZ2xvYmFsL2ltYWdlcy91YnVudHUtbWluaW1hbC0xODA0LWJpb25pYy12MjAyMDEwMTQiLCAiem9uZSI6ICJ1cy1lYXN0MS1jIiwgIm5ldHdvcmsiOiAiZGVmYXVsdCIsICJuZXR3b3JrX3Byb2plY3RpZCI6IG51bGwsICJwcm9qZWN0aWQiOiAiIiwgIm9uX2hvc3RfbWFpbnRlbmFuY2UiOiAiVEVSTUlOQVRFIiwgIm1hY2hpbmVfdHlwZSI6ICJuMS1zdGFuZGFyZC0xIiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDUwLCAibmdwdXMiOiAiIiwgImdwdV90eXBlIjogIiIsICJkaXNrX3R5cGUiOiAicGQtc3RhbmRhcmQiLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QiLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJpbnN0YW5jZV9sYWJlbHMiOiB7ImNvbnRhaW5lcl92bSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgInNlcnZpY2VfYWNjb3VudCI6ICJkZWZhdWx0In0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGlvbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9LCAiaWJtIjogeyJhcGlfa2V5IjogbnVsbCwgImltYWdlIjogImdoY3IuaW8vZGFzay9kYXNrOmxhdGVzdCIsICJyZWdpb24iOiAidXMtZWFzdCIsICJwcm9qZWN0X2lkIjogbnVsbCwgInNjaGVkdWxlcl9jcHUiOiAiMS4wIiwgInNjaGVkdWxlcl9tZW0iOiAiNEciLCAic2NoZWR1bGVyX3RpbWVvdXQiOiA2MDAsICJ3b3JrZXJfY3B1IjogIjIuMCIsICJ3b3JrZXJfbWVtIjogIjhHIiwgIndvcmtlcl90aHJlYWRzIjogMX0sICJvcGVuc3RhY2siOiB7InJlZ2lvbiI6ICJSZWdpb25PbmUiLCAic2l6ZSI6IG51bGwsICJhdXRoX3VybCI6IG51bGwsICJhcHBsaWNhdGlvbl9jcmVkZW50aWFsX2lkIjogbnVsbCwgImFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfc2VjcmV0IjogbnVsbCwgImF1dGhfdHlwZSI6ICJ2M2FwcGxpY2F0aW9uY3JlZGVudGlhbCIsICJuZXR3b3JrX2lkIjogbnVsbCwgImltYWdlIjogbnVsbCwgImtleXBhaXJfbmFtZSI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJleHRlcm5hbF9uZXR3b3JrX2lkIjogbnVsbCwgImNyZWF0ZV9mbG9hdGluZ19pcCI6IGZhbHNlLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QifX19\" python -m distributed.cli.dask_spec tcp://18.220.177.101:8786 --spec ''{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"name\": \"dask-d7369884-worker-592c886c\"}}'''\n", + "\n", + " \n", + " # Shutdown when command is done\n", + " - shutdown -h now\n", + " \n", + "\n", + "Cloud init\n", + "==========\n", + "\n", + "\n", + "#cloud-config\n", + "\n", + "\n", + "# Bootstrap\n", + "packages:\n", + " - apt-transport-https\n", + " - ca-certificates\n", + " - curl\n", + " - gnupg-agent\n", + " - software-properties-common\n", + " - ubuntu-drivers-common\n", + "\n", + "# Enable ipv4 forwarding, required on CIS hardened machines\n", + "write_files:\n", + " - path: /etc/sysctl.d/enabled_ipv4_forwarding.conf\n", + " content: |\n", + " net.ipv4.conf.all.forwarding=1\n", + "\n", + "# create the docker group\n", + "groups:\n", + " - docker\n", + "\n", + "# Add default auto created user to docker group\n", + "system_info:\n", + " default_user:\n", + " groups: [docker]\n", + "\n", + "\n", + "runcmd:\n", + " \n", + " # Install Docker\n", + " - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -\n", + " - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"\n", + " - apt-get update -y\n", + " - apt-get install -y docker-ce docker-ce-cli containerd.io\n", + " - systemctl start docker\n", + " - systemctl enable docker\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " # Run container\n", + " - 'docker run --net=host -e EXTRA_CONDA_PACKAGES=\"s3fs==2024.10.0\" daskdev/dask:latest env DASK_INTERNAL_INHERIT_CONFIG=\"eyJ0ZW1wb3JhcnktZGlyZWN0b3J5IjogbnVsbCwgInZpc3VhbGl6YXRpb24iOiB7ImVuZ2luZSI6IG51bGx9LCAidG9rZW5pemUiOiB7ImVuc3VyZS1kZXRlcm1pbmlzdGljIjogZmFsc2V9LCAiZGF0YWZyYW1lIjogeyJiYWNrZW5kIjogInBhbmRhcyIsICJzaHVmZmxlIjogeyJtZXRob2QiOiBudWxsLCAiY29tcHJlc3Npb24iOiBudWxsfSwgInBhcnF1ZXQiOiB7Im1ldGFkYXRhLXRhc2stc2l6ZS1sb2NhbCI6IDUxMiwgIm1ldGFkYXRhLXRhc2stc2l6ZS1yZW1vdGUiOiAxLCAibWluaW11bS1wYXJ0aXRpb24tc2l6ZSI6IDc1MDAwMDAwfSwgImNvbnZlcnQtc3RyaW5nIjogbnVsbCwgInF1ZXJ5LXBsYW5uaW5nIjogbnVsbH0sICJhcnJheSI6IHsiYmFja2VuZCI6ICJudW1weSIsICJjaHVuay1zaXplIjogIjEyOE1pQiIsICJjaHVuay1zaXplLXRvbGVyYW5jZSI6IDEuMjUsICJyZWNodW5rIjogeyJtZXRob2QiOiBudWxsLCAidGhyZXNob2xkIjogNH0sICJzdmciOiB7InNpemUiOiAxMjB9LCAic2xpY2luZyI6IHsic3BsaXQtbGFyZ2UtY2h1bmtzIjogbnVsbH0sICJxdWVyeS1wbGFubmluZyI6IG51bGx9LCAib3B0aW1pemF0aW9uIjogeyJhbm5vdGF0aW9ucyI6IHsiZnVzZSI6IHRydWV9LCAiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJhZG1pbiI6IHsidHJhY2ViYWNrIjogeyJzaG9ydGVuIjogWyJjb25jdXJyZW50W1xcXFxcXC9dZnV0dXJlc1tcXFxcXFwvXSIsICJkYXNrW1xcXFxcXC9dKGJhc2V8Y29yZXxsb2NhbHxtdWx0aXByb2Nlc3Npbmd8b3B0aW1pemF0aW9ufHRocmVhZGVkfHV0aWxzKVxcLnB5IiwgImRhc2tbXFxcXFxcL11hcnJheVtcXFxcXFwvXWNvcmVcXC5weSIsICJkYXNrW1xcXFxcXC9dZGF0YWZyYW1lW1xcXFxcXC9dKGNvcmV8bWV0aG9kcylcXC5weSIsICJkaXN0cmlidXRlZFtcXFxcXFwvXShjbGllbnR8c2NoZWR1bGVyfHV0aWxzfHdvcmtlcilcXC5weSIsICJ0b3JuYWRvW1xcXFxcXC9dZ2VuXFwucHkiLCAicGFuZGFzW1xcXFxcXC9dY29yZVtcXFxcXFwvXSJdfX0sICJkaXN0cmlidXRlZCI6IHsidmVyc2lvbiI6IDIsICJzY2hlZHVsZXIiOiB7ImFsbG93ZWQtZmFpbHVyZXMiOiAzLCAiYmFuZHdpZHRoIjogMTAwMDAwMDAwLCAiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAiY29udGFjdC1hZGRyZXNzIjogbnVsbCwgImRlZmF1bHQtZGF0YS1zaXplIjogIjFraUIiLCAiZXZlbnRzLWNsZWFudXAtZGVsYXkiOiAiMWgiLCAiaWRsZS10aW1lb3V0IjogbnVsbCwgIm5vLXdvcmtlcnMtdGltZW91dCI6IG51bGwsICJ3b3JrLXN0ZWFsaW5nIjogdHJ1ZSwgIndvcmstc3RlYWxpbmctaW50ZXJ2YWwiOiAiMTAwbXMiLCAid29ya2VyLXNhdHVyYXRpb24iOiAxLjEsICJ3b3JrZXItdHRsIjogIjUgbWludXRlcyIsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyIsICJzcGxpdC10YXNrc2h1ZmZsZSI6ICIxdXMiLCAic3BsaXQtc3RhZ2UiOiAiMXVzIn0sICJ2YWxpZGF0ZSI6IGZhbHNlLCAiZGFzaGJvYXJkIjogeyJzdGF0dXMiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDB9LCAidGFza3MiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDAwMH0sICJ0bHMiOiB7ImNhLWZpbGUiOiBudWxsLCAia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImJva2VoLWFwcGxpY2F0aW9uIjogeyJhbGxvd193ZWJzb2NrZXRfb3JpZ2luIjogWyIqIl0sICJrZWVwX2FsaXZlX21pbGxpc2Vjb25kcyI6IDUwMCwgImNoZWNrX3VudXNlZF9zZXNzaW9uc19taWxsaXNlY29uZHMiOiA1MDB9fSwgImxvY2tzIjogeyJsZWFzZS12YWxpZGF0aW9uLWludGVydmFsIjogIjEwcyIsICJsZWFzZS10aW1lb3V0IjogIjMwcyJ9LCAiaHR0cCI6IHsicm91dGVzIjogWyJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5wcm9tZXRoZXVzIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmluZm8iLCAiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIuanNvbiIsICJkaXN0cmlidXRlZC5odHRwLmhlYWx0aCIsICJkaXN0cmlidXRlZC5odHRwLnByb3h5IiwgImRpc3RyaWJ1dGVkLmh0dHAuc3RhdGljcyJdfSwgImFsbG93ZWQtaW1wb3J0cyI6IFsiZGFzayIsICJkaXN0cmlidXRlZCJdLCAiYWN0aXZlLW1lbW9yeS1tYW5hZ2VyIjogeyJzdGFydCI6IHRydWUsICJpbnRlcnZhbCI6ICIycyIsICJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAicG9saWNpZXMiOiBbeyJjbGFzcyI6ICJkaXN0cmlidXRlZC5hY3RpdmVfbWVtb3J5X21hbmFnZXIuUmVkdWNlUmVwbGljYXMifV19fSwgIndvcmtlciI6IHsiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAibXVsdGlwcm9jZXNzaW5nLW1ldGhvZCI6ICJzcGF3biIsICJ1c2UtZmlsZS1sb2NraW5nIjogdHJ1ZSwgInRyYW5zZmVyIjogeyJtZXNzYWdlLWJ5dGVzLWxpbWl0IjogIjUwTUIifSwgImNvbm5lY3Rpb25zIjogeyJvdXRnb2luZyI6IDUwLCAiaW5jb21pbmciOiAxMH0sICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImRhZW1vbiI6IHRydWUsICJ2YWxpZGF0ZSI6IGZhbHNlLCAicmVzb3VyY2VzIjoge30sICJsaWZldGltZSI6IHsiZHVyYXRpb24iOiBudWxsLCAic3RhZ2dlciI6ICIwIHNlY29uZHMiLCAicmVzdGFydCI6IGZhbHNlfSwgInByb2ZpbGUiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMTBtcyIsICJjeWNsZSI6ICIxMDAwbXMiLCAibG93LWxldmVsIjogZmFsc2V9LCAibWVtb3J5IjogeyJyZWNlbnQtdG8tb2xkLXRpbWUiOiAiMzBzIiwgInJlYmFsYW5jZSI6IHsibWVhc3VyZSI6ICJvcHRpbWlzdGljIiwgInNlbmRlci1taW4iOiAwLjMsICJyZWNpcGllbnQtbWF4IjogMC42LCAic2VuZGVyLXJlY2lwaWVudC1nYXAiOiAwLjF9LCAidHJhbnNmZXIiOiAwLjEsICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NSwgIm1heC1zcGlsbCI6IGZhbHNlLCAic3BpbGwtY29tcHJlc3Npb24iOiAiYXV0byIsICJtb25pdG9yLWludGVydmFsIjogIjEwMG1zIn0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7fSwgInByZS1zcGF3bi1lbnZpcm9uIjogeyJNQUxMT0NfVFJJTV9USFJFU0hPTERfIjogNjU1MzYsICJPTVBfTlVNX1RIUkVBRFMiOiAxLCAiTUtMX05VTV9USFJFQURTIjogMSwgIk9QRU5CTEFTX05VTV9USFJFQURTIjogMX19LCAiY2xpZW50IjogeyJoZWFydGJlYXQiOiAiNXMiLCAic2NoZWR1bGVyLWluZm8taW50ZXJ2YWwiOiAiMnMiLCAic2VjdXJpdHktbG9hZGVyIjogbnVsbCwgInByZWxvYWQiOiBbXSwgInByZWxvYWQtYXJndiI6IFtdfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogZmFsc2UsICJzaGFyZCI6ICI2NE1pQiIsICJvZmZsb2FkIjogIjEwTWlCIiwgImRlZmF1bHQtc2NoZW1lIjogInRjcCIsICJzb2NrZXQtYmFja2xvZyI6IDIwNDgsICJ1Y3giOiB7ImN1ZGEtY29weSI6IG51bGwsICJ0Y3AiOiBudWxsLCAibnZsaW5rIjogbnVsbCwgImluZmluaWJhbmQiOiBudWxsLCAicmRtYWNtIjogbnVsbCwgImNyZWF0ZS1jdWRhLWNvbnRleHQiOiBudWxsLCAiZW52aXJvbm1lbnQiOiB7fX0sICJ6c3RkIjogeyJsZXZlbCI6IDMsICJ0aHJlYWRzIjogMH0sICJ0aW1lb3V0cyI6IHsiY29ubmVjdCI6ICIzMHMiLCAidGNwIjogIjMwcyJ9LCAicmVxdWlyZS1lbmNyeXB0aW9uIjogbnVsbCwgInRscyI6IHsiY2lwaGVycyI6IG51bGwsICJtaW4tdmVyc2lvbiI6IDEuMiwgIm1heC12ZXJzaW9uIjogbnVsbCwgImNhLWZpbGUiOiBudWxsLCAic2NoZWR1bGVyIjogeyJjZXJ0IjogbnVsbCwgImtleSI6IG51bGx9LCAid29ya2VyIjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9LCAiY2xpZW50IjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9fSwgIndlYnNvY2tldHMiOiB7InNoYXJkIjogIjhNaUIifX0sICJkaWFnbm9zdGljcyI6IHsibnZtbCI6IHRydWUsICJjdWRmIjogZmFsc2UsICJjb21wdXRhdGlvbnMiOiB7Im1heC1oaXN0b3J5IjogMTAwLCAibmZyYW1lcyI6IDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiYXN5bmNpbyIsICJmdW5jdG9vbHMiLCAidGhyZWFkaW5nIiwgImRhdGFzaGFkZXIiLCAiZGFzayIsICJkZWJ1Z3B5IiwgImRpc3RyaWJ1dGVkIiwgImNvaWxlZCIsICJjdWRmIiwgImN1bWwiLCAibWF0cGxvdGxpYiIsICJwbHVnZ3kiLCAicHJlZmVjdCIsICJyZWNodW5rZXIiLCAieGFycmF5IiwgInhnYm9vc3QiLCAieGRpc3QiLCAiX19jaGFubmVsZXhlY19fIiwgImV4ZWNuZXQiXSwgImlnbm9yZS1maWxlcyI6IFsicnVucHlcXC5weSIsICJweXRlc3QiLCAicHlcXC50ZXN0IiwgInB5dGVzdC1zY3JpcHRcXC5weSIsICJfcHl0ZXN0IiwgInB5Y2hhcm0iLCAidnNjb2RlX3B5dGVzdCIsICJnZXRfb3V0cHV0X3ZpYV9tYXJrZXJzXFwucHkiXX0sICJlcnJlZC10YXNrcyI6IHsibWF4LWhpc3RvcnkiOiAxMDB9fSwgInAycCI6IHsiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMTAsICJkZWxheSI6IHsibWluIjogIjFzIiwgIm1heCI6ICIzMHMifX19LCAiZGlzayI6IHRydWV9LCAiZGFzaGJvYXJkIjogeyJsaW5rIjogIntzY2hlbWV9Oi8ve2hvc3R9Ontwb3J0fS9zdGF0dXMiLCAiZXhwb3J0LXRvb2wiOiBmYWxzZSwgImdyYXBoLW1heC1pdGVtcyI6IDUwMDAsICJwcm9tZXRoZXVzIjogeyJuYW1lc3BhY2UiOiAiZGFzayJ9fSwgImFkbWluIjogeyJsYXJnZS1ncmFwaC13YXJuaW5nLXRocmVzaG9sZCI6ICIxME1CIiwgInRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MiLCAiY3ljbGUiOiAiMXMifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShhc2N0aW1lKXMgLSAlKG5hbWUpcyAtICUobGV2ZWxuYW1lKXMgLSAlKG1lc3NhZ2UpcyIsICJsb3ctbGV2ZWwtbG9nLWxlbmd0aCI6IDEwMDAsICJwZGItb24tZXJyIjogZmFsc2UsICJzeXN0ZW0tbW9uaXRvciI6IHsiaW50ZXJ2YWwiOiAiNTAwbXMiLCAibG9nLWxlbmd0aCI6IDcyMDAsICJkaXNrIjogdHJ1ZSwgImhvc3QtY3B1IjogZmFsc2UsICJnaWwiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMW1zIn19LCAiZXZlbnQtbG9vcCI6ICJ0b3JuYWRvIn0sICJybW0iOiB7InBvb2wtc2l6ZSI6IG51bGx9fSwgImNsb3VkcHJvdmlkZXIiOiB7ImVjcyI6IHsiZmFyZ2F0ZV9zY2hlZHVsZXIiOiBmYWxzZSwgImZhcmdhdGVfc3BvdCI6IGZhbHNlLCAiZmFyZ2F0ZV93b3JrZXJzIjogZmFsc2UsICJmYXJnYXRlX3VzZV9wcml2YXRlX2lwIjogZmFsc2UsICJzY2hlZHVsZXJfY3B1IjogMTAyNCwgInNjaGVkdWxlcl9tZW0iOiA0MDk2LCAid29ya2VyX2NwdSI6IDQwOTYsICJ3b3JrZXJfbWVtIjogMTYzODQsICJ3b3JrZXJfZ3B1IjogMCwgIm5fd29ya2VycyI6IDAsICJzY2hlZHVsZXJfdGltZW91dCI6ICI1IG1pbnV0ZXMiLCAiaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJjcHVfYXJjaGl0ZWN0dXJlIjogIlg4Nl82NCIsICJncHVfaW1hZ2UiOiAicmFwaWRzYWkvcmFwaWRzYWk6bGF0ZXN0IiwgImNsdXN0ZXJfbmFtZV90ZW1wbGF0ZSI6ICJkYXNrLXt1dWlkfSIsICJjbHVzdGVyX2FybiI6ICIiLCAiZXhlY3V0aW9uX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfcG9saWNpZXMiOiBbXSwgImNsb3Vkd2F0Y2hfbG9nc19ncm91cCI6ICIiLCAiY2xvdWR3YXRjaF9sb2dzX3N0cmVhbV9wcmVmaXgiOiAie2NsdXN0ZXJfbmFtZX0iLCAiY2xvdWR3YXRjaF9sb2dzX2RlZmF1bHRfcmV0ZW50aW9uIjogMzAsICJ2cGMiOiAiZGVmYXVsdCIsICJzdWJuZXRzIjogW10sICJzZWN1cml0eV9ncm91cHMiOiBbXSwgInRhZ3MiOiB7fSwgImVudmlyb25tZW50Ijoge30sICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJhbWkiOiBudWxsLCAiaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJzY2hlZHVsZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJ3b3JrZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2cGMiOiBudWxsLCAic3VibmV0X2lkIjogbnVsbCwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAiZmlsZXN5c3RlbV9zaXplIjogNDAsICJrZXlfbmFtZSI6IG51bGwsICJpYW1faW5zdGFuY2VfcHJvZmlsZSI6IHt9LCAiaW5zdGFuY2VfdGFncyI6IHsiY3JlYXRlZEJ5IjogImRhc2stY2xvdWRwcm92aWRlciJ9LCAidm9sdW1lX3RhZ3MiOiB7ImNyZWF0ZWRCeSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgImVuYWJsZV9kZXRhaWxlZF9tb25pdG9yaW5nIjogZmFsc2UsICJ1c2VfcHJpdmF0ZV9pcCI6IGZhbHNlfSwgImF6dXJlIjogeyJsb2NhdGlvbiI6IG51bGwsICJyZXNvdXJjZV9ncm91cCI6IG51bGwsICJzdWJzY3JpcHRpb25faWQiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzdWJuZXQiOiBudWxsLCAic2VjdXJpdHlfZ3JvdXAiOiBudWxsLCAicHVibGljX2luZ3Jlc3MiOiB0cnVlLCAidm1fc2l6ZSI6ICJTdGFuZGFyZF9EUzFfdjIiLCAiZGlza19zaXplIjogNTAsICJzY2hlZHVsZXJfdm1fc2l6ZSI6IG51bGwsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2bV9pbWFnZSI6IHsicHVibGlzaGVyIjogIkNhbm9uaWNhbCIsICJvZmZlciI6ICJVYnVudHVTZXJ2ZXIiLCAic2t1IjogIjE4LjA0LUxUUyIsICJ2ZXJzaW9uIjogImxhdGVzdCJ9LCAiYm9vdHN0cmFwIjogdHJ1ZSwgImF1dG9fc2h1dGRvd24iOiB0cnVlLCAibWFya2V0cGxhY2VfcGxhbiI6IG51bGwsICJleHRyYV9vcHRpb25zIjoge319fSwgImRpZ2l0YWxvY2VhbiI6IHsidG9rZW4iOiBudWxsLCAicmVnaW9uIjogIm55YzMiLCAic2l6ZSI6ICJzLTF2Y3B1LTFnYiIsICJpbWFnZSI6ICJ1YnVudHUtMjAtMDQteDY0In0sICJnY3AiOiB7InNvdXJjZV9pbWFnZSI6ICJwcm9qZWN0cy91YnVudHUtb3MtY2xvdWQvZ2xvYmFsL2ltYWdlcy91YnVudHUtbWluaW1hbC0xODA0LWJpb25pYy12MjAyMDEwMTQiLCAiem9uZSI6ICJ1cy1lYXN0MS1jIiwgIm5ldHdvcmsiOiAiZGVmYXVsdCIsICJuZXR3b3JrX3Byb2plY3RpZCI6IG51bGwsICJwcm9qZWN0aWQiOiAiIiwgIm9uX2hvc3RfbWFpbnRlbmFuY2UiOiAiVEVSTUlOQVRFIiwgIm1hY2hpbmVfdHlwZSI6ICJuMS1zdGFuZGFyZC0xIiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDUwLCAibmdwdXMiOiAiIiwgImdwdV90eXBlIjogIiIsICJkaXNrX3R5cGUiOiAicGQtc3RhbmRhcmQiLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QiLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJpbnN0YW5jZV9sYWJlbHMiOiB7ImNvbnRhaW5lcl92bSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgInNlcnZpY2VfYWNjb3VudCI6ICJkZWZhdWx0In0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGlvbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9LCAiaWJtIjogeyJhcGlfa2V5IjogbnVsbCwgImltYWdlIjogImdoY3IuaW8vZGFzay9kYXNrOmxhdGVzdCIsICJyZWdpb24iOiAidXMtZWFzdCIsICJwcm9qZWN0X2lkIjogbnVsbCwgInNjaGVkdWxlcl9jcHUiOiAiMS4wIiwgInNjaGVkdWxlcl9tZW0iOiAiNEciLCAic2NoZWR1bGVyX3RpbWVvdXQiOiA2MDAsICJ3b3JrZXJfY3B1IjogIjIuMCIsICJ3b3JrZXJfbWVtIjogIjhHIiwgIndvcmtlcl90aHJlYWRzIjogMX0sICJvcGVuc3RhY2siOiB7InJlZ2lvbiI6ICJSZWdpb25PbmUiLCAic2l6ZSI6IG51bGwsICJhdXRoX3VybCI6IG51bGwsICJhcHBsaWNhdGlvbl9jcmVkZW50aWFsX2lkIjogbnVsbCwgImFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfc2VjcmV0IjogbnVsbCwgImF1dGhfdHlwZSI6ICJ2M2FwcGxpY2F0aW9uY3JlZGVudGlhbCIsICJuZXR3b3JrX2lkIjogbnVsbCwgImltYWdlIjogbnVsbCwgImtleXBhaXJfbmFtZSI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJleHRlcm5hbF9uZXR3b3JrX2lkIjogbnVsbCwgImNyZWF0ZV9mbG9hdGluZ19pcCI6IGZhbHNlLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QifX19\" python -m distributed.cli.dask_spec tcp://18.220.177.101:8786 --spec ''{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"name\": \"dask-d7369884-worker-a09fb848\"}}'''\n", + "\n", + " \n", + " # Shutdown when command is done\n", + " - shutdown -h now\n", + " \n", + "\n", + "Cloud init\n", + "==========\n", + "\n", + "\n", + "#cloud-config\n", + "\n", + "\n", + "# Bootstrap\n", + "packages:\n", + " - apt-transport-https\n", + " - ca-certificates\n", + " - curl\n", + " - gnupg-agent\n", + " - software-properties-common\n", + " - ubuntu-drivers-common\n", + "\n", + "# Enable ipv4 forwarding, required on CIS hardened machines\n", + "write_files:\n", + " - path: /etc/sysctl.d/enabled_ipv4_forwarding.conf\n", + " content: |\n", + " net.ipv4.conf.all.forwarding=1\n", + "\n", + "# create the docker group\n", + "groups:\n", + " - docker\n", + "\n", + "# Add default auto created user to docker group\n", + "system_info:\n", + " default_user:\n", + " groups: [docker]\n", + "\n", + "\n", + "runcmd:\n", + " \n", + " # Install Docker\n", + " - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -\n", + " - add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\"\n", + " - apt-get update -y\n", + " - apt-get install -y docker-ce docker-ce-cli containerd.io\n", + " - systemctl start docker\n", + " - systemctl enable docker\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " # Run container\n", + " - 'docker run --net=host -e EXTRA_CONDA_PACKAGES=\"s3fs==2024.10.0\" daskdev/dask:latest env DASK_INTERNAL_INHERIT_CONFIG=\"eyJ0ZW1wb3JhcnktZGlyZWN0b3J5IjogbnVsbCwgInZpc3VhbGl6YXRpb24iOiB7ImVuZ2luZSI6IG51bGx9LCAidG9rZW5pemUiOiB7ImVuc3VyZS1kZXRlcm1pbmlzdGljIjogZmFsc2V9LCAiZGF0YWZyYW1lIjogeyJiYWNrZW5kIjogInBhbmRhcyIsICJzaHVmZmxlIjogeyJtZXRob2QiOiBudWxsLCAiY29tcHJlc3Npb24iOiBudWxsfSwgInBhcnF1ZXQiOiB7Im1ldGFkYXRhLXRhc2stc2l6ZS1sb2NhbCI6IDUxMiwgIm1ldGFkYXRhLXRhc2stc2l6ZS1yZW1vdGUiOiAxLCAibWluaW11bS1wYXJ0aXRpb24tc2l6ZSI6IDc1MDAwMDAwfSwgImNvbnZlcnQtc3RyaW5nIjogbnVsbCwgInF1ZXJ5LXBsYW5uaW5nIjogbnVsbH0sICJhcnJheSI6IHsiYmFja2VuZCI6ICJudW1weSIsICJjaHVuay1zaXplIjogIjEyOE1pQiIsICJjaHVuay1zaXplLXRvbGVyYW5jZSI6IDEuMjUsICJyZWNodW5rIjogeyJtZXRob2QiOiBudWxsLCAidGhyZXNob2xkIjogNH0sICJzdmciOiB7InNpemUiOiAxMjB9LCAic2xpY2luZyI6IHsic3BsaXQtbGFyZ2UtY2h1bmtzIjogbnVsbH0sICJxdWVyeS1wbGFubmluZyI6IG51bGx9LCAib3B0aW1pemF0aW9uIjogeyJhbm5vdGF0aW9ucyI6IHsiZnVzZSI6IHRydWV9LCAiZnVzZSI6IHsiYWN0aXZlIjogbnVsbCwgImF2ZS13aWR0aCI6IDEsICJtYXgtd2lkdGgiOiBudWxsLCAibWF4LWhlaWdodCI6IEluZmluaXR5LCAibWF4LWRlcHRoLW5ldy1lZGdlcyI6IG51bGwsICJzdWJncmFwaHMiOiBudWxsLCAicmVuYW1lLWtleXMiOiB0cnVlfX0sICJhZG1pbiI6IHsidHJhY2ViYWNrIjogeyJzaG9ydGVuIjogWyJjb25jdXJyZW50W1xcXFxcXC9dZnV0dXJlc1tcXFxcXFwvXSIsICJkYXNrW1xcXFxcXC9dKGJhc2V8Y29yZXxsb2NhbHxtdWx0aXByb2Nlc3Npbmd8b3B0aW1pemF0aW9ufHRocmVhZGVkfHV0aWxzKVxcLnB5IiwgImRhc2tbXFxcXFxcL11hcnJheVtcXFxcXFwvXWNvcmVcXC5weSIsICJkYXNrW1xcXFxcXC9dZGF0YWZyYW1lW1xcXFxcXC9dKGNvcmV8bWV0aG9kcylcXC5weSIsICJkaXN0cmlidXRlZFtcXFxcXFwvXShjbGllbnR8c2NoZWR1bGVyfHV0aWxzfHdvcmtlcilcXC5weSIsICJ0b3JuYWRvW1xcXFxcXC9dZ2VuXFwucHkiLCAicGFuZGFzW1xcXFxcXC9dY29yZVtcXFxcXFwvXSJdfX0sICJkaXN0cmlidXRlZCI6IHsidmVyc2lvbiI6IDIsICJzY2hlZHVsZXIiOiB7ImFsbG93ZWQtZmFpbHVyZXMiOiAzLCAiYmFuZHdpZHRoIjogMTAwMDAwMDAwLCAiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAiY29udGFjdC1hZGRyZXNzIjogbnVsbCwgImRlZmF1bHQtZGF0YS1zaXplIjogIjFraUIiLCAiZXZlbnRzLWNsZWFudXAtZGVsYXkiOiAiMWgiLCAiaWRsZS10aW1lb3V0IjogbnVsbCwgIm5vLXdvcmtlcnMtdGltZW91dCI6IG51bGwsICJ3b3JrLXN0ZWFsaW5nIjogdHJ1ZSwgIndvcmstc3RlYWxpbmctaW50ZXJ2YWwiOiAiMTAwbXMiLCAid29ya2VyLXNhdHVyYXRpb24iOiAxLjEsICJ3b3JrZXItdHRsIjogIjUgbWludXRlcyIsICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgInVua25vd24tdGFzay1kdXJhdGlvbiI6ICI1MDBtcyIsICJkZWZhdWx0LXRhc2stZHVyYXRpb25zIjogeyJyZWNodW5rLXNwbGl0IjogIjF1cyIsICJzcGxpdC1zaHVmZmxlIjogIjF1cyIsICJzcGxpdC10YXNrc2h1ZmZsZSI6ICIxdXMiLCAic3BsaXQtc3RhZ2UiOiAiMXVzIn0sICJ2YWxpZGF0ZSI6IGZhbHNlLCAiZGFzaGJvYXJkIjogeyJzdGF0dXMiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDB9LCAidGFza3MiOiB7InRhc2stc3RyZWFtLWxlbmd0aCI6IDEwMDAwMH0sICJ0bHMiOiB7ImNhLWZpbGUiOiBudWxsLCAia2V5IjogbnVsbCwgImNlcnQiOiBudWxsfSwgImJva2VoLWFwcGxpY2F0aW9uIjogeyJhbGxvd193ZWJzb2NrZXRfb3JpZ2luIjogWyIqIl0sICJrZWVwX2FsaXZlX21pbGxpc2Vjb25kcyI6IDUwMCwgImNoZWNrX3VudXNlZF9zZXNzaW9uc19taWxsaXNlY29uZHMiOiA1MDB9fSwgImxvY2tzIjogeyJsZWFzZS12YWxpZGF0aW9uLWludGVydmFsIjogIjEwcyIsICJsZWFzZS10aW1lb3V0IjogIjMwcyJ9LCAiaHR0cCI6IHsicm91dGVzIjogWyJkaXN0cmlidXRlZC5odHRwLnNjaGVkdWxlci5wcm9tZXRoZXVzIiwgImRpc3RyaWJ1dGVkLmh0dHAuc2NoZWR1bGVyLmluZm8iLCAiZGlzdHJpYnV0ZWQuaHR0cC5zY2hlZHVsZXIuanNvbiIsICJkaXN0cmlidXRlZC5odHRwLmhlYWx0aCIsICJkaXN0cmlidXRlZC5odHRwLnByb3h5IiwgImRpc3RyaWJ1dGVkLmh0dHAuc3RhdGljcyJdfSwgImFsbG93ZWQtaW1wb3J0cyI6IFsiZGFzayIsICJkaXN0cmlidXRlZCJdLCAiYWN0aXZlLW1lbW9yeS1tYW5hZ2VyIjogeyJzdGFydCI6IHRydWUsICJpbnRlcnZhbCI6ICIycyIsICJtZWFzdXJlIjogIm9wdGltaXN0aWMiLCAicG9saWNpZXMiOiBbeyJjbGFzcyI6ICJkaXN0cmlidXRlZC5hY3RpdmVfbWVtb3J5X21hbmFnZXIuUmVkdWNlUmVwbGljYXMifV19fSwgIndvcmtlciI6IHsiYmxvY2tlZC1oYW5kbGVycyI6IFtdLCAibXVsdGlwcm9jZXNzaW5nLW1ldGhvZCI6ICJzcGF3biIsICJ1c2UtZmlsZS1sb2NraW5nIjogdHJ1ZSwgInRyYW5zZmVyIjogeyJtZXNzYWdlLWJ5dGVzLWxpbWl0IjogIjUwTUIifSwgImNvbm5lY3Rpb25zIjogeyJvdXRnb2luZyI6IDUwLCAiaW5jb21pbmciOiAxMH0sICJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImRhZW1vbiI6IHRydWUsICJ2YWxpZGF0ZSI6IGZhbHNlLCAicmVzb3VyY2VzIjoge30sICJsaWZldGltZSI6IHsiZHVyYXRpb24iOiBudWxsLCAic3RhZ2dlciI6ICIwIHNlY29uZHMiLCAicmVzdGFydCI6IGZhbHNlfSwgInByb2ZpbGUiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMTBtcyIsICJjeWNsZSI6ICIxMDAwbXMiLCAibG93LWxldmVsIjogZmFsc2V9LCAibWVtb3J5IjogeyJyZWNlbnQtdG8tb2xkLXRpbWUiOiAiMzBzIiwgInJlYmFsYW5jZSI6IHsibWVhc3VyZSI6ICJvcHRpbWlzdGljIiwgInNlbmRlci1taW4iOiAwLjMsICJyZWNpcGllbnQtbWF4IjogMC42LCAic2VuZGVyLXJlY2lwaWVudC1nYXAiOiAwLjF9LCAidHJhbnNmZXIiOiAwLjEsICJ0YXJnZXQiOiAwLjYsICJzcGlsbCI6IDAuNywgInBhdXNlIjogMC44LCAidGVybWluYXRlIjogMC45NSwgIm1heC1zcGlsbCI6IGZhbHNlLCAic3BpbGwtY29tcHJlc3Npb24iOiAiYXV0byIsICJtb25pdG9yLWludGVydmFsIjogIjEwMG1zIn0sICJodHRwIjogeyJyb3V0ZXMiOiBbImRpc3RyaWJ1dGVkLmh0dHAud29ya2VyLnByb21ldGhldXMiLCAiZGlzdHJpYnV0ZWQuaHR0cC5oZWFsdGgiLCAiZGlzdHJpYnV0ZWQuaHR0cC5zdGF0aWNzIl19fSwgIm5hbm55IjogeyJwcmVsb2FkIjogW10sICJwcmVsb2FkLWFyZ3YiOiBbXSwgImVudmlyb24iOiB7fSwgInByZS1zcGF3bi1lbnZpcm9uIjogeyJNQUxMT0NfVFJJTV9USFJFU0hPTERfIjogNjU1MzYsICJPTVBfTlVNX1RIUkVBRFMiOiAxLCAiTUtMX05VTV9USFJFQURTIjogMSwgIk9QRU5CTEFTX05VTV9USFJFQURTIjogMX19LCAiY2xpZW50IjogeyJoZWFydGJlYXQiOiAiNXMiLCAic2NoZWR1bGVyLWluZm8taW50ZXJ2YWwiOiAiMnMiLCAic2VjdXJpdHktbG9hZGVyIjogbnVsbCwgInByZWxvYWQiOiBbXSwgInByZWxvYWQtYXJndiI6IFtdfSwgImRlcGxveSI6IHsibG9zdC13b3JrZXItdGltZW91dCI6ICIxNXMiLCAiY2x1c3Rlci1yZXByLWludGVydmFsIjogIjUwMG1zIn0sICJhZGFwdGl2ZSI6IHsiaW50ZXJ2YWwiOiAiMXMiLCAidGFyZ2V0LWR1cmF0aW9uIjogIjVzIiwgIm1pbmltdW0iOiAwLCAibWF4aW11bSI6IEluZmluaXR5LCAid2FpdC1jb3VudCI6IDN9LCAiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMCwgImRlbGF5IjogeyJtaW4iOiAiMXMiLCAibWF4IjogIjIwcyJ9fSwgImNvbXByZXNzaW9uIjogZmFsc2UsICJzaGFyZCI6ICI2NE1pQiIsICJvZmZsb2FkIjogIjEwTWlCIiwgImRlZmF1bHQtc2NoZW1lIjogInRjcCIsICJzb2NrZXQtYmFja2xvZyI6IDIwNDgsICJ1Y3giOiB7ImN1ZGEtY29weSI6IG51bGwsICJ0Y3AiOiBudWxsLCAibnZsaW5rIjogbnVsbCwgImluZmluaWJhbmQiOiBudWxsLCAicmRtYWNtIjogbnVsbCwgImNyZWF0ZS1jdWRhLWNvbnRleHQiOiBudWxsLCAiZW52aXJvbm1lbnQiOiB7fX0sICJ6c3RkIjogeyJsZXZlbCI6IDMsICJ0aHJlYWRzIjogMH0sICJ0aW1lb3V0cyI6IHsiY29ubmVjdCI6ICIzMHMiLCAidGNwIjogIjMwcyJ9LCAicmVxdWlyZS1lbmNyeXB0aW9uIjogbnVsbCwgInRscyI6IHsiY2lwaGVycyI6IG51bGwsICJtaW4tdmVyc2lvbiI6IDEuMiwgIm1heC12ZXJzaW9uIjogbnVsbCwgImNhLWZpbGUiOiBudWxsLCAic2NoZWR1bGVyIjogeyJjZXJ0IjogbnVsbCwgImtleSI6IG51bGx9LCAid29ya2VyIjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9LCAiY2xpZW50IjogeyJrZXkiOiBudWxsLCAiY2VydCI6IG51bGx9fSwgIndlYnNvY2tldHMiOiB7InNoYXJkIjogIjhNaUIifX0sICJkaWFnbm9zdGljcyI6IHsibnZtbCI6IHRydWUsICJjdWRmIjogZmFsc2UsICJjb21wdXRhdGlvbnMiOiB7Im1heC1oaXN0b3J5IjogMTAwLCAibmZyYW1lcyI6IDAsICJpZ25vcmUtbW9kdWxlcyI6IFsiYXN5bmNpbyIsICJmdW5jdG9vbHMiLCAidGhyZWFkaW5nIiwgImRhdGFzaGFkZXIiLCAiZGFzayIsICJkZWJ1Z3B5IiwgImRpc3RyaWJ1dGVkIiwgImNvaWxlZCIsICJjdWRmIiwgImN1bWwiLCAibWF0cGxvdGxpYiIsICJwbHVnZ3kiLCAicHJlZmVjdCIsICJyZWNodW5rZXIiLCAieGFycmF5IiwgInhnYm9vc3QiLCAieGRpc3QiLCAiX19jaGFubmVsZXhlY19fIiwgImV4ZWNuZXQiXSwgImlnbm9yZS1maWxlcyI6IFsicnVucHlcXC5weSIsICJweXRlc3QiLCAicHlcXC50ZXN0IiwgInB5dGVzdC1zY3JpcHRcXC5weSIsICJfcHl0ZXN0IiwgInB5Y2hhcm0iLCAidnNjb2RlX3B5dGVzdCIsICJnZXRfb3V0cHV0X3ZpYV9tYXJrZXJzXFwucHkiXX0sICJlcnJlZC10YXNrcyI6IHsibWF4LWhpc3RvcnkiOiAxMDB9fSwgInAycCI6IHsiY29tbSI6IHsicmV0cnkiOiB7ImNvdW50IjogMTAsICJkZWxheSI6IHsibWluIjogIjFzIiwgIm1heCI6ICIzMHMifX19LCAiZGlzayI6IHRydWV9LCAiZGFzaGJvYXJkIjogeyJsaW5rIjogIntzY2hlbWV9Oi8ve2hvc3R9Ontwb3J0fS9zdGF0dXMiLCAiZXhwb3J0LXRvb2wiOiBmYWxzZSwgImdyYXBoLW1heC1pdGVtcyI6IDUwMDAsICJwcm9tZXRoZXVzIjogeyJuYW1lc3BhY2UiOiAiZGFzayJ9fSwgImFkbWluIjogeyJsYXJnZS1ncmFwaC13YXJuaW5nLXRocmVzaG9sZCI6ICIxME1CIiwgInRpY2siOiB7ImludGVydmFsIjogIjIwbXMiLCAibGltaXQiOiAiM3MiLCAiY3ljbGUiOiAiMXMifSwgIm1heC1lcnJvci1sZW5ndGgiOiAxMDAwMCwgImxvZy1sZW5ndGgiOiAxMDAwMCwgImxvZy1mb3JtYXQiOiAiJShhc2N0aW1lKXMgLSAlKG5hbWUpcyAtICUobGV2ZWxuYW1lKXMgLSAlKG1lc3NhZ2UpcyIsICJsb3ctbGV2ZWwtbG9nLWxlbmd0aCI6IDEwMDAsICJwZGItb24tZXJyIjogZmFsc2UsICJzeXN0ZW0tbW9uaXRvciI6IHsiaW50ZXJ2YWwiOiAiNTAwbXMiLCAibG9nLWxlbmd0aCI6IDcyMDAsICJkaXNrIjogdHJ1ZSwgImhvc3QtY3B1IjogZmFsc2UsICJnaWwiOiB7ImVuYWJsZWQiOiB0cnVlLCAiaW50ZXJ2YWwiOiAiMW1zIn19LCAiZXZlbnQtbG9vcCI6ICJ0b3JuYWRvIn0sICJybW0iOiB7InBvb2wtc2l6ZSI6IG51bGx9fSwgImNsb3VkcHJvdmlkZXIiOiB7ImVjcyI6IHsiZmFyZ2F0ZV9zY2hlZHVsZXIiOiBmYWxzZSwgImZhcmdhdGVfc3BvdCI6IGZhbHNlLCAiZmFyZ2F0ZV93b3JrZXJzIjogZmFsc2UsICJmYXJnYXRlX3VzZV9wcml2YXRlX2lwIjogZmFsc2UsICJzY2hlZHVsZXJfY3B1IjogMTAyNCwgInNjaGVkdWxlcl9tZW0iOiA0MDk2LCAid29ya2VyX2NwdSI6IDQwOTYsICJ3b3JrZXJfbWVtIjogMTYzODQsICJ3b3JrZXJfZ3B1IjogMCwgIm5fd29ya2VycyI6IDAsICJzY2hlZHVsZXJfdGltZW91dCI6ICI1IG1pbnV0ZXMiLCAiaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJjcHVfYXJjaGl0ZWN0dXJlIjogIlg4Nl82NCIsICJncHVfaW1hZ2UiOiAicmFwaWRzYWkvcmFwaWRzYWk6bGF0ZXN0IiwgImNsdXN0ZXJfbmFtZV90ZW1wbGF0ZSI6ICJkYXNrLXt1dWlkfSIsICJjbHVzdGVyX2FybiI6ICIiLCAiZXhlY3V0aW9uX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfYXJuIjogIiIsICJ0YXNrX3JvbGVfcG9saWNpZXMiOiBbXSwgImNsb3Vkd2F0Y2hfbG9nc19ncm91cCI6ICIiLCAiY2xvdWR3YXRjaF9sb2dzX3N0cmVhbV9wcmVmaXgiOiAie2NsdXN0ZXJfbmFtZX0iLCAiY2xvdWR3YXRjaF9sb2dzX2RlZmF1bHRfcmV0ZW50aW9uIjogMzAsICJ2cGMiOiAiZGVmYXVsdCIsICJzdWJuZXRzIjogW10sICJzZWN1cml0eV9ncm91cHMiOiBbXSwgInRhZ3MiOiB7fSwgImVudmlyb25tZW50Ijoge30sICJza2lwX2NsZWFudXAiOiBmYWxzZX0sICJlYzIiOiB7InJlZ2lvbiI6IG51bGwsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJib290c3RyYXAiOiB0cnVlLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJhbWkiOiBudWxsLCAiaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJzY2hlZHVsZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJ3b3JrZXJfaW5zdGFuY2VfdHlwZSI6ICJ0Mi5taWNybyIsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2cGMiOiBudWxsLCAic3VibmV0X2lkIjogbnVsbCwgInNlY3VyaXR5X2dyb3VwcyI6IFtdLCAiZmlsZXN5c3RlbV9zaXplIjogNDAsICJrZXlfbmFtZSI6IG51bGwsICJpYW1faW5zdGFuY2VfcHJvZmlsZSI6IHt9LCAiaW5zdGFuY2VfdGFncyI6IHsiY3JlYXRlZEJ5IjogImRhc2stY2xvdWRwcm92aWRlciJ9LCAidm9sdW1lX3RhZ3MiOiB7ImNyZWF0ZWRCeSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgImVuYWJsZV9kZXRhaWxlZF9tb25pdG9yaW5nIjogZmFsc2UsICJ1c2VfcHJpdmF0ZV9pcCI6IGZhbHNlfSwgImF6dXJlIjogeyJsb2NhdGlvbiI6IG51bGwsICJyZXNvdXJjZV9ncm91cCI6IG51bGwsICJzdWJzY3JpcHRpb25faWQiOiBudWxsLCAiYXp1cmV2bSI6IHsidm5ldCI6IG51bGwsICJzdWJuZXQiOiBudWxsLCAic2VjdXJpdHlfZ3JvdXAiOiBudWxsLCAicHVibGljX2luZ3Jlc3MiOiB0cnVlLCAidm1fc2l6ZSI6ICJTdGFuZGFyZF9EUzFfdjIiLCAiZGlza19zaXplIjogNTAsICJzY2hlZHVsZXJfdm1fc2l6ZSI6IG51bGwsICJkb2NrZXJfaW1hZ2UiOiAiZGFza2Rldi9kYXNrOmxhdGVzdCIsICJ2bV9pbWFnZSI6IHsicHVibGlzaGVyIjogIkNhbm9uaWNhbCIsICJvZmZlciI6ICJVYnVudHVTZXJ2ZXIiLCAic2t1IjogIjE4LjA0LUxUUyIsICJ2ZXJzaW9uIjogImxhdGVzdCJ9LCAiYm9vdHN0cmFwIjogdHJ1ZSwgImF1dG9fc2h1dGRvd24iOiB0cnVlLCAibWFya2V0cGxhY2VfcGxhbiI6IG51bGwsICJleHRyYV9vcHRpb25zIjoge319fSwgImRpZ2l0YWxvY2VhbiI6IHsidG9rZW4iOiBudWxsLCAicmVnaW9uIjogIm55YzMiLCAic2l6ZSI6ICJzLTF2Y3B1LTFnYiIsICJpbWFnZSI6ICJ1YnVudHUtMjAtMDQteDY0In0sICJnY3AiOiB7InNvdXJjZV9pbWFnZSI6ICJwcm9qZWN0cy91YnVudHUtb3MtY2xvdWQvZ2xvYmFsL2ltYWdlcy91YnVudHUtbWluaW1hbC0xODA0LWJpb25pYy12MjAyMDEwMTQiLCAiem9uZSI6ICJ1cy1lYXN0MS1jIiwgIm5ldHdvcmsiOiAiZGVmYXVsdCIsICJuZXR3b3JrX3Byb2plY3RpZCI6IG51bGwsICJwcm9qZWN0aWQiOiAiIiwgIm9uX2hvc3RfbWFpbnRlbmFuY2UiOiAiVEVSTUlOQVRFIiwgIm1hY2hpbmVfdHlwZSI6ICJuMS1zdGFuZGFyZC0xIiwgImZpbGVzeXN0ZW1fc2l6ZSI6IDUwLCAibmdwdXMiOiAiIiwgImdwdV90eXBlIjogIiIsICJkaXNrX3R5cGUiOiAicGQtc3RhbmRhcmQiLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QiLCAiYXV0b19zaHV0ZG93biI6IHRydWUsICJwdWJsaWNfaW5ncmVzcyI6IHRydWUsICJpbnN0YW5jZV9sYWJlbHMiOiB7ImNvbnRhaW5lcl92bSI6ICJkYXNrLWNsb3VkcHJvdmlkZXIifSwgInNlcnZpY2VfYWNjb3VudCI6ICJkZWZhdWx0In0sICJoZXR6bmVyIjogeyJ0b2tlbiI6IG51bGwsICJsb2NhdGlvbiI6ICJmc24xIiwgInNlcnZlcl90eXBlIjogImN4MTEiLCAiaW1hZ2UiOiAidWJ1bnR1LTIwLjA0IiwgImRvY2tlcl9pbWFnZSI6ICJkYXNrZGV2L2Rhc2s6bGF0ZXN0IiwgImJvb3RzdHJhcCI6IHRydWV9LCAiaWJtIjogeyJhcGlfa2V5IjogbnVsbCwgImltYWdlIjogImdoY3IuaW8vZGFzay9kYXNrOmxhdGVzdCIsICJyZWdpb24iOiAidXMtZWFzdCIsICJwcm9qZWN0X2lkIjogbnVsbCwgInNjaGVkdWxlcl9jcHUiOiAiMS4wIiwgInNjaGVkdWxlcl9tZW0iOiAiNEciLCAic2NoZWR1bGVyX3RpbWVvdXQiOiA2MDAsICJ3b3JrZXJfY3B1IjogIjIuMCIsICJ3b3JrZXJfbWVtIjogIjhHIiwgIndvcmtlcl90aHJlYWRzIjogMX0sICJvcGVuc3RhY2siOiB7InJlZ2lvbiI6ICJSZWdpb25PbmUiLCAic2l6ZSI6IG51bGwsICJhdXRoX3VybCI6IG51bGwsICJhcHBsaWNhdGlvbl9jcmVkZW50aWFsX2lkIjogbnVsbCwgImFwcGxpY2F0aW9uX2NyZWRlbnRpYWxfc2VjcmV0IjogbnVsbCwgImF1dGhfdHlwZSI6ICJ2M2FwcGxpY2F0aW9uY3JlZGVudGlhbCIsICJuZXR3b3JrX2lkIjogbnVsbCwgImltYWdlIjogbnVsbCwgImtleXBhaXJfbmFtZSI6IG51bGwsICJzZWN1cml0eV9ncm91cCI6IG51bGwsICJleHRlcm5hbF9uZXR3b3JrX2lkIjogbnVsbCwgImNyZWF0ZV9mbG9hdGluZ19pcCI6IGZhbHNlLCAiZG9ja2VyX2ltYWdlIjogImRhc2tkZXYvZGFzazpsYXRlc3QifX19\" python -m distributed.cli.dask_spec tcp://18.220.177.101:8786 --spec ''{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"name\": \"dask-d7369884-worker-f9596efa\"}}'''\n", + "\n", + " \n", + " # Shutdown when command is done\n", + " - shutdown -h now\n", + " \n", + "Created instance i-0e2f3a60157449305 as dask-d7369884-worker-87177d09\n", + "Created instance i-08464c37bec5899d5 as dask-d7369884-worker-f9596efa\n", + "Created instance i-03c7260fe1621977b as dask-d7369884-worker-592c886c\n", + "Created instance i-01836314d011735d9 as dask-d7369884-worker-a09fb848\n" + ] + } + ], "source": [ "env_vars = {\"EXTRA_CONDA_PACKAGES\": \"s3fs==2024.10.0\"}\n", "cluster = EC2Cluster(\n", @@ -42,7 +350,8 @@ " # https://github.com/dask/dask-cloudprovider/issues/249\n", " security=False,\n", " n_workers=4,\n", - " instance_type=\"r6i.16xlarge\",\n", + " scheduler_instance_type=\"c6i.xlarge\",\n", + " worker_instance_type=\"r6i.16xlarge\",\n", " # Region for accessing bodo-example-data\n", " region=\"us-east-2\",\n", " env_vars=env_vars,\n", @@ -52,9 +361,31 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 35, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "http://18.220.177.101:8787/status\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/scottroutledge/miniforge3/envs/benchmark_dask/lib/python3.10/site-packages/distributed/client.py:1606: VersionMismatchWarning: Mismatched versions found\n", + "\n", + "+---------+-----------------+-----------------+---------+\n", + "| Package | Client | Scheduler | Workers |\n", + "+---------+-----------------+-----------------+---------+\n", + "| python | 3.10.15.final.0 | 3.10.12.final.0 | None |\n", + "+---------+-----------------+-----------------+---------+\n", + " warnings.warn(version_module.VersionMismatchWarning(msg[0][\"warning\"]))\n" + ] + } + ], "source": [ "client = Client(cluster)\n", "print(client.dashboard_link)" @@ -62,20 +393,20 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "# first parquet file\n", "dataset = [\n", " f\"s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/fhvhv_tripdata_2019-{i:02}.parquet\"\n", - " for i in range(2, 3)\n", + " for i in range(2, 8)\n", "]" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 36, "metadata": {}, "outputs": [], "source": [ @@ -85,7 +416,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 37, "metadata": {}, "outputs": [], "source": [ @@ -175,16 +506,16 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 39, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Total time for IO and compute: 908.5563251972198\n", - "Total time for IO and compute: 907.8074090480804\n", - "Total time for IO and compute: 929.6480667591095\n" + "Total time for IO and compute: 932.2804353237152\n", + "Total time for IO and compute: 888.8601548671722\n", + "Total time for IO and compute: 885.689935207367\n" ] } ], @@ -198,9 +529,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 40, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Terminated dask-d7369884-worker-87177d09 (i-0e2f3a60157449305)\n", + "Terminated dask-d7369884-worker-592c886c (i-03c7260fe1621977b)\n", + "Terminated dask-d7369884-worker-f9596efa (i-08464c37bec5899d5)\n", + "Terminated dask-d7369884-worker-a09fb848 (i-01836314d011735d9)\n", + "Terminated dask-d7369884-scheduler (i-0311fa1be461bd2a9)\n" + ] + } + ], "source": [ "client.close()\n", "cluster.close()" diff --git a/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.py b/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.py index d5e724311a..b8744f374d 100644 --- a/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.py +++ b/benchmarks/nyc_taxi/dask/nyc_taxi_precipitation.py @@ -13,24 +13,18 @@ import dask.dataframe as dd from dask.distributed import Client -from dask_cloudprovider.aws import EC2Cluster -def get_monthly_travels_weather(): +def get_monthly_travels_weather(weather_dataset, hvfhv_dataset, storage_options=None): start = time.time() central_park_weather_observations = dd.read_csv( - "s3://bodo-example-data/nyc-taxi/central_park_weather.csv", - parse_dates=["DATE"], - storage_options={"anon": True}, + weather_dataset, parse_dates=["DATE"], storage_options=storage_options ) central_park_weather_observations = central_park_weather_observations.rename( columns={"DATE": "date", "PRCP": "precipitation"} ) - fhvhv_tripdata = dd.read_parquet( - "s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/", - storage_options={"anon": True}, - ) + fhvhv_tripdata = dd.read_parquet(hvfhv_dataset, storage_options=storage_options) central_park_weather_observations["date"] = central_park_weather_observations[ "date" @@ -103,25 +97,48 @@ def get_time_bucket(t): return end - start -def main(): +def local_get_monthly_travels_weather(weather_dataset, hvfhv_dataset): + """Run Dask on local cluster.""" + with Client(): + total_time = get_monthly_travels_weather(weather_dataset, hvfhv_dataset) + print("Total time for IO and compute:", total_time) + + +def ec2_get_monthly_travels_weather(weather_dataset, hvfhv_dataset): + """Run Dask on EC2 cluster.""" + from dask_cloudprovider.aws import EC2Cluster + + # for reading from S3 env_vars = {"EXTRA_CONDA_PACKAGES": "s3fs==2024.10.0"} + + # use an anoymous session to avoid passing credentials to cluster + s3_options = {"anon": True} + with EC2Cluster( # NOTE: Setting security = False to avoid large config size # https://github.com/dask/dask-cloudprovider/issues/249 security=False, n_workers=4, - instance_type="r6i.16xlarge", + scheduler_instance_type="c6i.xlarge", + worker_instance_type="r6i.16xlarge", # Region for accessing bodo-example-data region="us-east-2", env_vars=env_vars, ) as cluster: with Client(cluster) as client: for _ in range(3): - future = client.submit(get_monthly_travels_weather) + future = client.submit( + get_monthly_travels_weather, + weather_dataset, + hvfhv_dataset, + storage_options=s3_options, + ) total_time = future.result() client.restart() print("Total time for IO and compute:", total_time) if __name__ == "__main__": - main() + hvfhv_dataset = "s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/" + weather_dataset = "s3://bodo-example-data/nyc-taxi/central_park_weather.csv" + ec2_get_monthly_travels_weather(weather_dataset, hvfhv_dataset) diff --git a/benchmarks/nyc_taxi/local_versions.py b/benchmarks/nyc_taxi/local_versions.py new file mode 100755 index 0000000000..496bdcf93a --- /dev/null +++ b/benchmarks/nyc_taxi/local_versions.py @@ -0,0 +1,92 @@ +"""Run local version of Bodo, Dask, Modin, or PySpark benchmarks. +Accepts a parquet file to use as the High Volume For Hire Vehicle dataset +(defaults to first 5 million rows of the dataset). For an exhaustive list of +files, see s3://bodo-example-data/nyc-taxi/fhvhv. You can also optionally +specify which system to run the benchmark on. The default is Bodo. To run +all systems at once, use run_local.sh. + +usage: + python run_local.py --dataset FILE --system SYSTEM +""" + +import argparse +import os + +import boto3 +from botocore import UNSIGNED +from botocore.config import Config +from nyc_taxi.bodo.nyc_taxi_precipitation import ( + get_monthly_travels_weather as bodo_get_monthly_travels_weather, +) +from nyc_taxi.dask.nyc_taxi_precipitation import ( + local_get_monthly_travels_weather as dask_get_monthly_travels_weather, +) +from nyc_taxi.modin_ray.nyc_taxi_precipitation import ( + local_get_monthly_travels_weather as modin_get_monthly_travels_weather, +) +from nyc_taxi.spark.spark_nyc_taxi_precipitation import ( + get_monthly_travels_weather as spark_get_monthly_travels_weather, +) + +SMALL_DATASET_PATH_S3 = "nyc-taxi/fhvhv_5M_rows.pq" +WEATHER_DATASET_PATH_S3 = "nyc-taxi/central_park_weather.csv" +BUCKET_NAME = "bodo-example-data" + + +def download_data_s3(path_to_s3: str, local_data_dir: str = "data") -> str: + """Download the dataset from S3 if already exists, skip download.""" + file_name = path_to_s3.split("/", -1)[1] + local_path = os.path.join(local_data_dir, file_name) + + if os.path.exists(local_path): + return local_path + + print("Downloading dataset from S3...") + + s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED)) + + if not os.path.exists(local_data_dir): + os.mkdir(local_data_dir) + + s3.download_file(BUCKET_NAME, path_to_s3, local_path) + return local_path + + +def main(hvfhv_path_s3: str, system: str): + weather_path = download_data_s3(WEATHER_DATASET_PATH_S3) + hvfhv_path = download_data_s3(hvfhv_path_s3) + + get_monthly_travels_weather_impls = { + "bodo": bodo_get_monthly_travels_weather, + "dask": dask_get_monthly_travels_weather, + "modin": modin_get_monthly_travels_weather, + "spark": spark_get_monthly_travels_weather, + } + + get_monthly_travels_weather = get_monthly_travels_weather_impls[system] + + print(f"Running {system.capitalize()}...") + get_monthly_travels_weather(weather_path, hvfhv_path) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--dataset", + "-d", + required=False, + default=SMALL_DATASET_PATH_S3, + help="Path to parquet file to use for local benchmark. Include everything after s3://bodo-example-data.", + ) + parser.add_argument( + "--system", + "-s", + required=False, + default="bodo", + help="System to run benchmark on. Options are bodo, dask, modin or spark.", + ) + args = parser.parse_args() + hvfhv_path_s3 = args.dataset + system = args.system + + main(hvfhv_path_s3, system) diff --git a/benchmarks/nyc_taxi/modin-ray/README.md b/benchmarks/nyc_taxi/modin_ray/README.md similarity index 100% rename from benchmarks/nyc_taxi/modin-ray/README.md rename to benchmarks/nyc_taxi/modin_ray/README.md diff --git a/benchmarks/nyc_taxi/modin-ray/env.yml b/benchmarks/nyc_taxi/modin_ray/env.yml similarity index 100% rename from benchmarks/nyc_taxi/modin-ray/env.yml rename to benchmarks/nyc_taxi/modin_ray/env.yml diff --git a/benchmarks/nyc_taxi/modin-ray/modin-cluster.yaml b/benchmarks/nyc_taxi/modin_ray/modin-cluster.yaml similarity index 99% rename from benchmarks/nyc_taxi/modin-ray/modin-cluster.yaml rename to benchmarks/nyc_taxi/modin_ray/modin-cluster.yaml index 2cfba616b3..a4669a12a0 100644 --- a/benchmarks/nyc_taxi/modin-ray/modin-cluster.yaml +++ b/benchmarks/nyc_taxi/modin_ray/modin-cluster.yaml @@ -173,6 +173,7 @@ setup_commands: - conda activate modin && pip install -U fsspec>=2022.11.0 s3fs boto3 pyopenssl - echo "conda activate modin" >> ~/.bashrc - echo 'export MODIN_RAY_CLUSTER=True' >> ~/.bashrc + - echo 'conda list' # Custom commands that will be run on the head node after common setup. head_setup_commands: diff --git a/benchmarks/nyc_taxi/modin-ray/nyc_taxi_precipitation.py b/benchmarks/nyc_taxi/modin_ray/nyc_taxi_precipitation.py similarity index 74% rename from benchmarks/nyc_taxi/modin-ray/nyc_taxi_precipitation.py rename to benchmarks/nyc_taxi/modin_ray/nyc_taxi_precipitation.py index 8ad6cb48b9..1d38ec19d8 100644 --- a/benchmarks/nyc_taxi/modin-ray/nyc_taxi_precipitation.py +++ b/benchmarks/nyc_taxi/modin_ray/nyc_taxi_precipitation.py @@ -1,33 +1,27 @@ import time -import ray - -ray.init(address="auto") -cpu_count = ray.cluster_resources()["CPU"] -print("RAY CPU COUNT: ", cpu_count) import modin.pandas as pd +import ray from modin.pandas.io import to_ray -def run_modin(): - start = time.time() +def get_monthly_travels_weather(weather_dataset, hvfhv_dataset): + start_read = time.time() central_park_weather_observations = pd.read_csv( - "s3://bodo-example-data/nyc-taxi/central_park_weather.csv", + weather_dataset, parse_dates=["DATE"], - storage_options={"anon": True}, ) central_park_weather_observations = central_park_weather_observations.rename( columns={"DATE": "date", "PRCP": "precipitation"}, copy=False, ) fhvhv_tripdata = pd.read_parquet( - "s3://bodo-example-data/nyc-taxi/fhvhv_tripdata/", - storage_options={"anon": True}, + hvfhv_dataset, ) end = time.time() - print("Reading Time: ", (end - start)) + print("Reading Time: ", (end - start_read)) - start = time.time() + start_compute = time.time() central_park_weather_observations["date"] = central_park_weather_observations[ "date" @@ -90,16 +84,29 @@ def get_time_bucket(t): copy=False, ) end = time.time() - print("Monthly Taxi Travel Times Computation Time: ", end - start) - print(monthly_trips_weather.head()) + print("Monthly Taxi Travel Times Computation Time: ", end - start_compute) - start = time.time() + start_write = time.time() monthly_trips_weather_ray = to_ray(monthly_trips_weather) monthly_trips_weather_ray.write_parquet("local:///tmp/data/modin_result.pq") end = time.time() - print("Writing time:", (end - start)) + print("Writing time:", (end - start_write)) + print("Total E2E time:", (end - start_read)) return monthly_trips_weather +def local_get_monthly_travels_weather(weather_dataset, hvfhv_dataset): + """Run Modin on local Ray cluster""" + ray.init() + get_monthly_travels_weather(weather_dataset, hvfhv_dataset) + ray.shutdown() + + if __name__ == "__main__": - result = run_modin() + ray.init(address="auto") + cpu_count = ray.cluster_resources()["CPU"] + print("RAY CPU COUNT: ", cpu_count) + + weather_dataset = "s3://bodo-example-data/nyc-taxi/central_park_weather.csv" + hvfhv_dataset = "s3://bodo-example-data/nyc-taxi/fhvhv/" + get_monthly_travels_weather(weather_dataset, hvfhv_dataset) diff --git a/benchmarks/nyc_taxi/modin-ray/run_modin.sh b/benchmarks/nyc_taxi/modin_ray/run_modin.sh similarity index 100% rename from benchmarks/nyc_taxi/modin-ray/run_modin.sh rename to benchmarks/nyc_taxi/modin_ray/run_modin.sh diff --git a/benchmarks/nyc_taxi/modin-ray/scale_cluster.py b/benchmarks/nyc_taxi/modin_ray/scale_cluster.py similarity index 100% rename from benchmarks/nyc_taxi/modin-ray/scale_cluster.py rename to benchmarks/nyc_taxi/modin_ray/scale_cluster.py diff --git a/benchmarks/nyc_taxi/run_local.sh b/benchmarks/nyc_taxi/run_local.sh new file mode 100755 index 0000000000..cac584ac92 --- /dev/null +++ b/benchmarks/nyc_taxi/run_local.sh @@ -0,0 +1,16 @@ +#!/bin/bash + +# Run local benchmark on all systems. +# Note: uses sleep to ensure resources are reset before running the next system. + +python -m nyc_taxi.local_versions -s dask +sleep 4 + +python -m nyc_taxi.local_versions -s bodo +sleep 4 + +python -m nyc_taxi.local_versions -s modin +sleep 4 + +python -m nyc_taxi.local_versions -s spark + diff --git a/benchmarks/nyc_taxi/spark/spark_nyc_taxi_precipitation.py b/benchmarks/nyc_taxi/spark/spark_nyc_taxi_precipitation.py index bbe271af24..1c721e1903 100644 --- a/benchmarks/nyc_taxi/spark/spark_nyc_taxi_precipitation.py +++ b/benchmarks/nyc_taxi/spark/spark_nyc_taxi_precipitation.py @@ -11,7 +11,7 @@ ) -def get_monthly_travels_weather(): +def get_monthly_travels_weather(weather_dataset, hvfhv_dataset): spark = ( SparkSession.builder.appName("MonthlyTravelsWeather") .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.2") @@ -27,7 +27,7 @@ def get_monthly_travels_weather(): # Read in weather data using pandas-on-Spark central_park_weather_observations = ps.read_csv( - "s3a://bodo-example-data/nyc-taxi/central_park_weather.csv", + weather_dataset, ).rename(columns={"DATE": "date", "PRCP": "precipitation"}) central_park_weather_observations["date"] = ps.to_datetime( @@ -36,9 +36,7 @@ def get_monthly_travels_weather(): # Read in trip data using spark, this reads a re-written dataset because spark doesn't support reading the original dataset # due to schema unification issues - fhvhv_tripdata = spark.read.parquet( - "s3a://bodo-example-data/nyc-taxi/fhvhv_tripdata_rewrite/" - ).drop("__index_level_0__") + fhvhv_tripdata = spark.read.parquet(hvfhv_dataset).drop("__index_level_0__") # Convert datetime columns and create necessary features fhvhv_tripdata = ( @@ -113,4 +111,7 @@ def get_time_bucket(t): print("Execution time:", time.time() - start) -get_monthly_travels_weather() +if __name__ == "__main__": + weather_dataset = "s3a://bodo-example-data/nyc-taxi/central_park_weather.csv" + hvfhv_dataset = "s3a://bodo-example-data/nyc-taxi/fhvhv_tripdata_rewrite/" + get_monthly_travels_weather(weather_dataset, hvfhv_dataset)