Skip to content

[SPARK-11105][yarn] Distribute log4j.properties to executors #9118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vundela wants to merge 11 commits into from
Closed

[SPARK-11105][yarn] Distribute log4j.properties to executors #9118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
cc09f86
[SPARK-11105] Distribute log4j.properties to executors
Oct 14, 2015
2772c1d
Changing vocabulary
Oct 14, 2015
4ffcb3a
Fixing the possible NullPointerIssue
Oct 14, 2015
f007acf
-a
Oct 14, 2015
85aecef
-a
Oct 14, 2015
9669a32
Making changes as suggested by Marcelo
Oct 14, 2015
2e34edc
Uploading log4j conf through archived files
Oct 19, 2015
187eb99
Indentation
Oct 19, 2015
39c75a2
Uploading documentation to reflect the new behaviour
Oct 19, 2015
e1e474a
Making changes as per suggestion
Oct 19, 2015
b97a2b5
Style change
Oct 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,17 @@ all environment variables used for launching each container. This process is use
classpath problems in particular. (Note that enabling this requires admin privileges on cluster
settings and a restart of all node managers. Thus, this is not applicable to hosted clusters).

To use a custom log4j configuration for the application master or executors, there are two options:
To use a custom log4j configuration for the application master or executors, here are the options:

- upload a custom `log4j.properties` using `spark-submit`, by adding it to the `--files` list of files
to be uploaded with the application.
- add `-Dlog4j.configuration=<location of configuration file>` to `spark.driver.extraJavaOptions`
(for the driver) or `spark.executor.extraJavaOptions` (for executors). Note that if using a file,
the `file:` protocol should be explicitly provided, and the file needs to exist locally on all
the nodes.
- update the `$SPARK_CONF_DIR/log4j.properties` file and it will be automatically uploaded along
with the other configurations. Note that other 2 options has higher priority than this option if
multiple options are specified.

Note that for the first option, both executors and the application master will share the same
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
Expand Down
13 changes: 13 additions & 0 deletions yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,19 @@ private[spark] class Client(
*/
private def createConfArchive(): File = {
val hadoopConfFiles = new HashMap[String, File]()

// Uploading $SPARK_CONF_DIR/log4j.properties file to the distributed cache to make sure that
// the executors will use the latest configurations instead of the default values. This is
// required when user changes log4j.properties directly to set the log configurations. If
// configuration file is provided through --files then executors will be taking configurations
// from --files instead of $SPARK_CONF_DIR/log4j.properties.
Copy link
Contributor

@tgravescs tgravescs Oct 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does --files work? Is it just because it gets put on classpath first?

Copy link
Author

@vundela vundela Oct 20, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thats correct and also in this case user knows what he is doing and his intension is to overwrite the default behavior

val log4jFileName = "log4j.properties"
Option(Utils.getContextOrSparkClassLoader.getResource(log4jFileName)).foreach { url =>
if (url.getProtocol == "file") {
hadoopConfFiles(log4jFileName) = new File(url.getPath)
}
}

Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR").foreach { envKey =>
sys.env.get(envKey).foreach { path =>
val dir = new File(path)
Expand Down