Skip to content

Conversation

@liyuance
Copy link

when I use transformation in SparkSQL, may throw java.lang.ArrayIndexOutOfBoundsException, as some output lines of the transformation script end with sequential TOK_TABLEROWFORMATFIELD.
like:
A\tB\tC\t\t

@HyukjinKwon
Copy link
Member

Maybe I think the title is incomplete. It would be nicer if the title includes where (in.. where).

@yhuai
Copy link
Contributor

yhuai commented May 20, 2016

ok to test

@SparkQA
Copy link

SparkQA commented May 20, 2016

Test build #58932 has finished for PR 12772 at commit 7a5c3bd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented May 20, 2016

can you add a regression test?

@yhuai
Copy link
Contributor

yhuai commented May 20, 2016

At here, A\tB\tC\t\t represents 5 fields and A\tB\tC\t represents 4 fields, right?

@rxin
Copy link
Contributor

rxin commented May 24, 2016

cc @gatorsmile would you be able to take this over and add a test case?

@gatorsmile
Copy link
Member

Sure, will do it. Thanks!

@gatorsmile
Copy link
Member

Sorry, I am unable to reproduce it. Without the fix, the following test case works well.

    val data = Seq(("A\tB\tC\tD\t\t"), ("E\tF\tD\tE\t"))
    data.toDF("d1").createOrReplaceTempView("script_trans")
    checkAnswer(
    sql("SELECT TRANSFORM (d1) USING 'cat' AS (a, b, c, d) FROM script_trans"),
      Row("A", "B", "C", "D") :: Row("E", "F", "D", "E") :: Nil)

@rxin
Copy link
Contributor

rxin commented May 27, 2016

@liyuance can you comment on this?

@JoshRosen
Copy link
Contributor

@liyuance, if you don't have a reproduction of this issue could you close this PR? You can always re-open it later once it's ready for review. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants