Skip to content

Conversation

@HeartSaVioR
Copy link
Contributor

@HeartSaVioR HeartSaVioR commented Oct 29, 2019

What changes were proposed in this pull request?

This patch fixes the bug that ContinuousMemoryStream[String] throws error regarding ClassCastException - cast String to UTFString. This is because ContinuousMemoryStream and ContinuousRecordEndpoint uses origin input as it is for underlying data structure of Row, and encoding is missing here.

To force encoding, this patch changes the element type of underlying array to UnsafeRow instead of Any for ContinuousRecordEndpoint - ContinuousMemoryStream and TextSocketContinuousStream are modified to reflect the change.

Why are the changes needed?

Above section describes the bug.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add new UT to check for availability on couple of types.

@HeartSaVioR
Copy link
Contributor Author

@SparkQA
Copy link

SparkQA commented Oct 30, 2019

Test build #112868 has finished for PR 26300 at commit f457f40.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class ContinuousRecordEndpoint(buckets: Seq[Seq[UnsafeRow]], lock: Object)

@SparkQA
Copy link

SparkQA commented Oct 30, 2019

Test build #112876 has finished for PR 26300 at commit 8cf2eba.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Nov 5, 2019

Looks good as far as I can tell.

@vanzin
Copy link
Contributor

vanzin commented Nov 5, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Nov 6, 2019

Test build #113289 has finished for PR 26300 at commit 8cf2eba.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Nov 6, 2019

Merging to master.

@vanzin vanzin closed this in 782992c Nov 6, 2019
@HeartSaVioR
Copy link
Contributor Author

Thanks for reviewing and merging!

@HeartSaVioR HeartSaVioR deleted the SPARK-29642 branch November 6, 2019 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants