[SPARK-3562]Periodic cleanup event logs

[viper-kun]

No description provided.

[SparkQA]

Can one of the admins verify this patch?

[srowen]

Can you explain this patch? What problem does it solve and why? There is no JIRA here either.

[viper-kun]

@srowen If we run spark application frequently, it will write many spark event log into spark.eventLog.dir. After a long time later, there will be many spark event log that we do not concern in the spark.eventLog.dir. So we need delete some logs that exceeding the time limit.

[andrewor14]

Hey @viper-kun can you file a JIRA and put it in the title of this PR? See how other Spark PRs are formatted.

[andrewor14]

ok to test

[SparkQA]

QA tests have started for PR 2391 at commit 79f4928.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2391 at commit 79f4928.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Pretty sure you don't want to clean logs on initialization. Otherwise your history server will always display nothing!

[viper-kun]

yes，you are right. it is no need to clean it on initialization. And i can not change it, because of unknown repository. And i will submit other commit.

[andrewor14]

@viper-kun I'm not sure if it's the HistoryServer's responsibility to delete the application logs. The HistoryServer is intended to be a serving daemon; it should not modify application output, since it did not create any output in the first place. If I'm not wrong, Hadoop's JobHistoryServer has the same semantics (i.e. it doesn't randomly delete your logs). Perhaps @srowen can confirm this.

[viper-kun]

@andrewor14 i have checked Hadoop's JobHistoryServer. it is JobHistoryServer's responsibility to delete the application logs.

[vanzin]

nit: cleans, logs

[vanzin]

Looks ok to me, mostly a few nits.

Could you also update docs/monitoring.md and add the new conf parameters?

[viper-kun]

@vanzin , @andrewor14 .Thanks for your opinions. Because the source branch had been deleted by me, i can change it in this commit. i submit another commit[#2471] and change the source according your opinions.

[andrewor14]

@viper-kun Since there is now a new PR, would you mind closing this one? It's confusing to have two PRs that do the same thing.