Skip to content

Conversation

@yucai
Copy link
Contributor

@yucai yucai commented Jul 17, 2018

What changes were proposed in this pull request?

Currently, ColumnarBatch's bytesRead needs to be updated every 4096 * 1000 rows, which makes the metrics out of date. This PR makes it update at each batch.

How was this patch tested?

Existing UTs.

@kiszk
Copy link
Member

kiszk commented Jul 17, 2018

Is it better to add a test to InputOutputMetricsSuite?

@SparkQA
Copy link

SparkQA commented Jul 17, 2018

Test build #93168 has finished for PR 21791 at commit b9320c8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yucai yucai changed the title [SPARK-24832][SQL] Improve inputMetrics's bytesRead update for ColumnarBatch [SPARK-24925][SQL] input bytesRead metrics fluctuate from time to time Jul 26, 2018
@kiszk
Copy link
Member

kiszk commented Sep 8, 2018

gentle ping @yucai

@yucai
Copy link
Contributor Author

yucai commented Sep 8, 2018

@kiszk I see #22324 has been solved, which is one of my PR's dependency actually (see https://issues.apache.org/jira/browse/SPARK-24925?focusedCommentId=16556818&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16556818) , I am stuck on that for a long time, I will update this one recently. Thanks!

@kiszk
Copy link
Member

kiszk commented Sep 8, 2018

I see. thank you for sharing the dependency and status.

@SparkQA
Copy link

SparkQA commented Dec 28, 2018

Test build #100488 has finished for PR 21791 at commit b9320c8.

  • This patch fails due to an unknown error code, -9.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 12, 2019

Test build #107615 has finished for PR 21791 at commit b9320c8.

  • This patch fails R style tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@github-actions
Copy link

github-actions bot commented Jan 9, 2020

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

@github-actions github-actions bot added the Stale label Jan 9, 2020
@github-actions github-actions bot closed this Jan 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants