Skip to content

Commit 06ee53d

Browse files
committed
Reflect review comments
1 parent 1be03b0 commit 06ee53d

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

docs/structured-streaming-programming-guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -547,7 +547,7 @@ Here are the details of all the sources in Spark.
547547
<br/>
548548
<code>maxFileAge</code>: Maximum age of a file that can be found in this directory, before it is ignored. For the first batch all files will be considered valid. If <code>latestFirst</code> is set to `true` and <code>maxFilesPerTrigger</code> is set, then this parameter will be ignored, because old files that are valid, and should be processed, may be ignored. The max age is specified with respect to the timestamp of the latest file, and not the timestamp of the current system.(default: 1 week)
549549
<br/>
550-
<code>inputRetention</code>: Maximum age of a file that can be found in this directory, before it is ignored.<br/>
550+
<code>inputRetention</code>: Maximum age of a file that can be found in this directory, before it is ignored. (e.g. 14d, default: None)<br/>
551551
This is the "hard" limit of input data retention - input files older than the max age will be ignored regardless of source options (while `maxFileAgeMs` depends on the condition), as well as entries in checkpoint metadata will be purged based on this.<br/>
552552
Unlike `maxFileAgeMs`, the max age is specified with respect to the timestamp of the current system, to provide consistent behavior regardless of metadata entries.<br/>
553553
NOTE 1: Please be careful to set the value if the query replays from the old input files.<br/>

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ class FileStreamSource(
7272
}}
7373

7474
/** exposed for testing */
75-
private[sql] var clockForRetention: Clock = new SystemClock
75+
var clockForRetention: Clock = new SystemClock
7676
private val inputRetentionMs = sourceOptions.inputRetentionMs.getOrElse(Long.MaxValue)
7777

7878
private val metadataLog =

0 commit comments

Comments
 (0)