[SPARK-14078] Streaming Parquet Based FileSink #11897
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val startOffset: Option[Offset] = None, | ||
| val endOffset: Option[Offset] = None | ||
| ) extends Exception(message, cause) { | ||
| ) extends Exception(message, cause) with Serializable { |
Member
There was a problem hiding this comment.
nit: Exception has already extended Serializable. Not need to add it again.
|
Test build #53807 has finished for PR 11897 at commit
|
| path: String, | ||
| fileFormat: FileFormat) extends Sink with Logging { | ||
|
|
||
| val basePath = new Path(path) |
| .startStream(outputDir) | ||
|
|
||
| inputData.addData(1, 2, 3) | ||
| failAfter(streamingTimeout) { query.processAllAvailable() } |
Member
There was a problem hiding this comment.
There is a race condition here: noNewData may become true before processAllAvailable.
Member
There was a problem hiding this comment.
Never mind. I just realized it will continue to set noNewData to true
Member
|
Looks pretty good. Just some nits. |
|
Test build #53841 has finished for PR 11897 at commit
|
|
Test build #53847 has finished for PR 11897 at commit
|
|
|
||
| override def getOffset: Option[Offset] = Some(fetchMaxOffset()).filterNot(_.offset == -1) | ||
|
|
||
| override def toString: String = s"FileSink[$path]" |
Contributor
Author
There was a problem hiding this comment.
Haha, yes. I'll fix this in a follow up.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new
Sinkimplementation that writes out Parquet files. In order to correctly handle partial failures while maintaining exactly once semantics, the files for each batch are written out to a unique directory and then atomically appended to a metadata log. When a parquet basedDataSourceis initialized for reading, we first check for this log directory and use it instead of file listing when present.Unit tests are added, as well as a stress test that checks the answer after non-deterministic injected failures.