-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test CometDriverPlugin in Kube/Yarn #826
Comments
Related to #605 |
I tested this on YARN and I can confirm that I launch a spark shell with two containers on YARN with no Comet. YARN reports 7168MB allocated, which corresponds to (2048 MB memory + 1024 MB memory overhead) + 1024 MB YARN AM container, the cluster's default settings.
Then I launch a spark shell with Comet enabled:
Spark session doesn't report anything unusual:
but YARN reports 9216 MB in use, an increase of 1024 MB per container. Spark History Server reports that the new memory overhead is 1433M, which is just 409MB per container. This matches the expected increase, since it's equal to 2048*0.2, the default value of The difference is explained by the minimum allocation setting of YARN, it allocates memory in chunks of 1GB, which I have proven by running:
and getting 9216 MB allocated in YARN. Changing Changing |
Thanks @orthoxerox for running the experiment. Please correct me if I'm wrong so both Comet memory params increases the YARN container memory usage, at the same time |
@comphead That's right, but this applies only to the introspection from inside the Spark application. If you open Spark history server/application master, the Executor tab shows the amended value of |
Historical server has slightly other implementation. Great, thanks for the help @orthoxerox |
In Comet there is a Spark driver plugin that overrides
spark.executor.memoryOverhead
value to respect native part needed for the Comet.Resource managers like YARN/Kube considers
spark.executor.memoryOverhead
param when containers get started.Its needed to test the plugin and see if the container created with extra native memory amount set with
Originally posted by @comphead in #684 (comment)
The text was updated successfully, but these errors were encountered: