The Spark operator installs a vector agent as a side-car container in every application Pod except the job Pod that runs spark-submit. It also configures the logging framework to output logs in XML format. This is the same format used across all Stackable products and it enables the vector aggregator to collect logs across the entire platform.
It is the user’s responsibility to install and configure the vector aggregator, but the agents can discover the aggregator automatically using a discovery ConfigMap as described in the logging concepts.
|
Note
|
Only logs produced by the application’s driver and executors are collected. Logs produced by spark-submit are discarded.
|