Skip to content

[SPARK-19859][SS]The new watermark should override the old one #17199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zsxwing wants to merge 1 commit into from
Closed

[SPARK-19859][SS]The new watermark should override the old one #17199

zsxwing wants to merge 1 commit into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented Mar 7, 2017

What changes were proposed in this pull request?

The new watermark should override the old one. Otherwise, we just pick up the first column which has a watermark, it may be unexpected.

How was this patch tested?

The new test.

@zsxwing
Copy link
Member Author

zsxwing commented Mar 7, 2017

cc @marmbrus

@marmbrus
Copy link
Contributor

marmbrus commented Mar 8, 2017

LGTM

@SparkQA
Copy link

SparkQA commented Mar 8, 2017

Test build #74147 has finished for PR 17199 at commit 6527cb1.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Mar 8, 2017

retest this please

@SparkQA
Copy link

SparkQA commented Mar 8, 2017

Test build #74166 has finished for PR 17199 at commit 6527cb1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Mar 8, 2017 •
edited
Loading

Thanks! Merging to master and 2.1.

asfgit pushed a commit that referenced this pull request Mar 8, 2017
@zsxwing
[SPARK-19859][SS] The new watermark should override the old one
3b648a6 
## What changes were proposed in this pull request?

The new watermark should override the old one. Otherwise, we just pick up the first column which has a watermark, it may be unexpected.

## How was this patch tested?

The new test.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #17199 from zsxwing/SPARK-19859.

(cherry picked from commit d8830c5)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>
@asfgit asfgit closed this in d8830c5 Mar 8, 2017
@zsxwing zsxwing deleted the SPARK-19859 branch March 8, 2017 04:38
@zsxwing zsxwing mentioned this pull request Mar 9, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zsxwing @marmbrus @SparkQA