Skip to content

Commit cea5cbc

Browse files
HeartSaVioRHyukjinKwon
authored andcommitted
[SPARK-30827][DOCS] Document direct relationship among configurations in "spark.history.*" namespace
### What changes were proposed in this pull request? This patch adds direct relationship among configurations under "spark.history" namespace. ### Why are the changes needed? Refer the discussion thread: https://lists.apache.org/thread.html/r43c4e57cace116aca1f0f099e8a577cf202859e3671a04077867b84a%40%3Cdev.spark.apache.org%3E ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Locally ran jekyll and confirmed. Screenshots for the modified spots: <img width="1159" alt="Screen Shot 2020-02-15 at 8 20 14 PM" src="https://user-images.githubusercontent.com/1317309/74587003-d5922b00-5030-11ea-954b-ee37fc08470a.png"> <img width="1158" alt="Screen Shot 2020-02-15 at 8 20 44 PM" src="https://user-images.githubusercontent.com/1317309/74587005-d62ac180-5030-11ea-98fc-98b1c9d83ff4.png"> <img width="1149" alt="Screen Shot 2020-02-15 at 8 19 56 PM" src="https://user-images.githubusercontent.com/1317309/74587002-d1660d80-5030-11ea-84b5-dec3d7f5c97c.png"> Closes #27575 from HeartSaVioR/SPARK-30827. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan.opensource@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 5445fe9) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent 7c09b57 commit cea5cbc

File tree

2 files changed

+9
-11
lines changed

2 files changed

+9
-11
lines changed

core/src/main/scala/org/apache/spark/internal/config/History.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ private[spark] object History {
162162
val APPLY_CUSTOM_EXECUTOR_LOG_URL_TO_INCOMPLETE_APP =
163163
ConfigBuilder("spark.history.custom.executor.log.url.applyIncompleteApplication")
164164
.doc("Whether to apply custom executor log url, as specified by " +
165-
"`spark.history.custom.executor.log.url`, to incomplete application as well. " +
165+
s"${CUSTOM_EXECUTOR_LOG_URL.key}, to incomplete application as well. " +
166166
"Even if this is true, this still only affects the behavior of the history server, " +
167167
"not running spark applications.")
168168
.booleanConf

docs/monitoring.md

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -159,23 +159,21 @@ Security options for the Spark History Server are covered more detail in the
159159
<td>false</td>
160160
<td>
161161
Indicates whether the history server should use kerberos to login. This is required
162-
if the history server is accessing HDFS files on a secure Hadoop cluster. If this is
163-
true, it uses the configs <code>spark.history.kerberos.principal</code> and
164-
<code>spark.history.kerberos.keytab</code>.
162+
if the history server is accessing HDFS files on a secure Hadoop cluster.
165163
</td>
166164
</tr>
167165
<tr>
168166
<td>spark.history.kerberos.principal</td>
169167
<td>(none)</td>
170168
<td>
171-
Kerberos principal name for the History Server.
169+
When <code>spark.history.kerberos.enabled=true</code>, specifies kerberos principal name for the History Server.
172170
</td>
173171
</tr>
174172
<tr>
175173
<td>spark.history.kerberos.keytab</td>
176174
<td>(none)</td>
177175
<td>
178-
Location of the kerberos keytab file for the History Server.
176+
When <code>spark.history.kerberos.enabled=true</code>, specifies location of the kerberos keytab file for the History Server.
179177
</td>
180178
</tr>
181179
<tr>
@@ -189,7 +187,7 @@ Security options for the Spark History Server are covered more detail in the
189187
<td>spark.history.fs.cleaner.interval</td>
190188
<td>1d</td>
191189
<td>
192-
How often the filesystem job history cleaner checks for files to delete.
190+
When <code>spark.history.fs.cleaner.enabled=true</code>, specifies how often the filesystem job history cleaner checks for files to delete.
193191
Files are deleted if at least one of two conditions holds.
194192
First, they're deleted if they're older than <code>spark.history.fs.cleaner.maxAge</code>.
195193
They are also deleted if the number of files is more than
@@ -201,14 +199,14 @@ Security options for the Spark History Server are covered more detail in the
201199
<td>spark.history.fs.cleaner.maxAge</td>
202200
<td>7d</td>
203201
<td>
204-
Job history files older than this will be deleted when the filesystem history cleaner runs.
202+
When <code>spark.history.fs.cleaner.enabled=true</code>, job history files older than this will be deleted when the filesystem history cleaner runs.
205203
</td>
206204
</tr>
207205
<tr>
208206
<td>spark.history.fs.cleaner.maxNum</td>
209207
<td>Int.MaxValue</td>
210208
<td>
211-
The maximum number of files in the event log directory.
209+
When <code>spark.history.fs.cleaner.enabled=true</code>, specifies the maximum number of files in the event log directory.
212210
Spark tries to clean up the completed attempt logs to maintain the log directory under this limit.
213211
This should be smaller than the underlying file system limit like
214212
`dfs.namenode.fs-limits.max-directory-items` in HDFS.
@@ -242,15 +240,15 @@ Security options for the Spark History Server are covered more detail in the
242240
<td>spark.history.fs.driverlog.cleaner.interval</td>
243241
<td><code>spark.history.fs.cleaner.interval</code></td>
244242
<td>
245-
How often the filesystem driver log cleaner checks for files to delete.
243+
When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, specifies how often the filesystem driver log cleaner checks for files to delete.
246244
Files are only deleted if they are older than <code>spark.history.fs.driverlog.cleaner.maxAge</code>
247245
</td>
248246
</tr>
249247
<tr>
250248
<td>spark.history.fs.driverlog.cleaner.maxAge</td>
251249
<td><code>spark.history.fs.cleaner.maxAge</code></td>
252250
<td>
253-
Driver log files older than this will be deleted when the driver log cleaner runs.
251+
When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, driver log files older than this will be deleted when the driver log cleaner runs.
254252
</td>
255253
</tr>
256254
<tr>

0 commit comments

Comments
 (0)