Raise error when dataType was not provided #714

rajagurunath · 2021-07-09T18:07:24Z

throw Error explicitly, if dataType was not provided for DeltaColumnBuilder :

issue Raised and Discussed here: #698

…rovided

zsxwing

Thanks for the contribution. Left some style comments. Could you merge master into your branch so that it will have the fix for CI?

zsxwing · 2021-07-09T18:45:27Z

core/src/main/scala/io/delta/tables/DeltaColumnBuilder.scala

@@ -126,6 +127,10 @@ class DeltaColumnBuilder private[tables](
      metadataBuilder.putString("comment", comment.get)
    }
    val fieldMetadata = metadataBuilder.build()
+    if (dataType == null) {
+      throw DeltaErrors.analysisException("dataType should not be null for a column," +


nit: s"The data type of column $colName is not provided"

Thanks, this new error message has the right amount of information 👍

zsxwing · 2021-07-09T18:46:15Z

core/src/test/scala/io/delta/tables/DeltaTableBuilderSuite.scala

@@ -228,6 +228,18 @@ class DeltaTableBuilderSuite extends QueryTest with SharedSparkSession with Delt
    }
  }

+  test("test addColumn using columnBuilder, without dataType") {
+    val e = intercept[AnalysisException] {
+      io.delta.tables.DeltaTable.create().addColumn(


we can just run

DeltaTable.columnBuilder("value") .generatedAlwaysAs("true") .nullable(true) .build()

This would show that we will catch the error in build().

got it , makes sense !

rajagurunath · 2021-07-10T05:14:48Z

Thanks for reviewing the code and giving the helpful comments !!

jaceklaskowski · 2021-07-10T20:45:25Z

core/src/test/scala/io/delta/tables/DeltaTableBuilderSuite.scala

+    val e = intercept[AnalysisException] {
+      DeltaTable.columnBuilder("value")
+        .generatedAlwaysAs("true")
+        . nullable(true)


nit: remove the space after the dot

got it, Thanks 👍

jaceklaskowski · 2021-07-10T20:46:41Z

core/src/main/scala/io/delta/tables/DeltaColumnBuilder.scala

@@ -126,6 +127,9 @@ class DeltaColumnBuilder private[tables](
      metadataBuilder.putString("comment", comment.get)
    }
    val fieldMetadata = metadataBuilder.build()
+    if (dataType == null) {
+      throw DeltaErrors.analysisException(s"The data type of column $colName is not provided")


nit: of the column

Thanks 👍

jaceklaskowski · 2021-07-10T20:47:30Z

core/src/test/scala/io/delta/tables/DeltaTableBuilderSuite.scala

+        . nullable(true)
+        .build()
+    }
+    assert(e.getMessage.equals(s"The data type of column value is not provided"))


nit: Use == for equals

replaced equals with == and removed the brackets '()' is this okay ?

rajagurunath · 2021-07-11T07:21:25Z

Hi @jaceklaskowski
Thanks a lot for taking the time to review the code and giving comments.

(btw your books are awesome ❤️ )

zsxwing

LGTM. Thanks!

throw Error explicitly, if dataType was not provided for DeltaColumnBuilder : issue Raised and Discussed here: delta-io#698 Closes delta-io#714 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: gurunath <gurunathrajagopal@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #24669 is resolved by zsxwing/cuj6lcz0. GitOrigin-RevId: a9d92bc195693bb1ddca515a8069c38ce99d0497

throw Error explicitly, if dataType was not provided for DeltaColumnBuilder : issue Raised and Discussed here: delta-io#698 Closes delta-io#714 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: gurunath <gurunathrajagopal@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #24669 is resolved by zsxwing/cuj6lcz0. GitOrigin-RevId: a9d92bc195693bb1ddca515a8069c38ce99d0497 (cherry picked from commit 104e2a4)

* [SC-78072][Delta] Fix partitionedBy in DeltaTableBuilder Python API Fix partitionedBy in DeltaTableBuilder Python API Added more usages in the unit tests. Author: Yijia Cui <yijia.cui@databricks.com> GitOrigin-RevId: 99e2e89242c8fa41c960df25dcc382faae439ab1 (cherry picked from commit 59aa330) * [DELTA-OSS-EXTERNAL] Raise error when dataType was not provided throw Error explicitly, if dataType was not provided for DeltaColumnBuilder : issue Raised and Discussed here: delta-io#698 Closes delta-io#714 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: gurunath <gurunathrajagopal@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #24669 is resolved by zsxwing/cuj6lcz0. GitOrigin-RevId: a9d92bc195693bb1ddca515a8069c38ce99d0497 (cherry picked from commit 104e2a4) * [DELTA-OSS-EXTERNAL] Move GCS write to a separate thread to workaround a GCS corruption issue We found a potential GCS corruption due to an issue in GCS Hadoop connector: When writing a file on GCS, if the current thread is interrupted, GCS may close the partial write and upload the incomplete file. Here is a stack trace showing this behavior: ``` at java.io.PipedOutputStream.close(PipedOutputStream.java:175) at java.nio.channels.Channels$WritableByteChannelImpl.implCloseChannel(Channels.java:469) at java.nio.channels.spi.AbstractInterruptibleChannel$1.interrupt(AbstractInterruptibleChannel.java:165) - locked <0xc2f> (a java.lang.Object) at java.nio.channels.spi.AbstractInterruptibleChannel.begin(AbstractInterruptibleChannel.java:173) at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:457) - locked <0xc3b> (a java.lang.Object) at com.google.cloud.hadoop.util.BaseAbstractGoogleAsyncWriteChannel.write(BaseAbstractGoogleAsyncWriteChannel.java:136) - locked <0xc3c> (a com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$2) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:101) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) - locked <0xc3d> (a java.nio.channels.Channels$1) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) - locked <0xc3e> (a java.io.BufferedOutputStream) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.write(GoogleHadoopOutputStream.java:118) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) at java.io.DataOutputStream.write(DataOutputStream.java:107) - locked <0xc3f> (a org.apache.hadoop.fs.FSDataOutputStream) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) ``` This PR moves GCS write in GCSLogStore to a separate new thread to workaround the issue. Closes delta-io#782 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #28761 is resolved by zsxwing/mm7ity34. GitOrigin-RevId: 5bfa936db1983a151da7d6e4f239a200f634690a (cherry picked from commit 7a3f1e8) * [DELTA-OSS-EXTERNAL] Set the correct Hadoop configuration for checkpoint write Checkpoint write doesn't pass the correct Hadoop configuration. It may fail when users specify their storage credentials using Spark session configs. This PR fixes it and adds a test for it. Closes delta-io#784 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #28914 is resolved by zsxwing/66uz7rxl. GitOrigin-RevId: cc42b868f2a3ecffca88424fb7465792dc22c7e6 (cherry picked from commit 43d1422) * [DELTA-OSS-EXTERNAL][BREAKING]Python: fix convertToDelta to return a delta.tables.DeltaTable Fix the return type of DeltaTable.convertToDelta in Python. GitOrigin-RevId: 4cd9bd00708a5cdd3c442bf9715ce9047d2b4893 (cherry picked from commit c586f9a) * [SC-86940][DELTA]Fix potential checkpoint corruption issue for GCS in OSS Delta This PR moves checkpoint write to a separate thread for GCS to workaround the potential checkpoint corruption issue when a thread is interrupted. GitOrigin-RevId: 418df2441923c85c6415f9baec157c923c4c3dca (cherry picked from commit 95e9076) * [DELTA-OSS-EXTERNAL] Convert build to GitHub Actions This PR solves delta-io#792 Closes delta-io#798 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: gurunath <gurunathrajagopal@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> GitOrigin-RevId: 1b35ce70185ddfcdfca9992d695ffbe1de397e40 (cherry picked from commit ee5d31f) * [DELTA-OSS-EXTERNAL] Fix an issue in Merge/Update/Delete when the table path contains special chars getTouchedFiles incorrectly calls toUri which will escape the table path when it contains special chars. This causes Merge/Update/Delete not able to find matches files in this case. This PR removes toUri to fix the issue and adds a test to confirm the fix. Fix delta-io#724 Closes delta-io#741 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: Jarred Parrett <jarredparrett.dev@gmail.com> GitOrigin-RevId: c8e6563fc48e82b86a00b58910eba8983fc5f933 (cherry picked from commit 4359484) * [ES-113602] Minor change in sbt install script Minor change in sbt install script. Author: yaohua <yaohua.zhao@databricks.com> GitOrigin-RevId: 26b0fe9df735739332a482ced59bdd90bd7534ec (cherry picked from commit 4bee7ae) * [DELTA-OSS-EXTERNAL] Fix the sbt launch download url The bintray url is not working now. Use the `repo.typesafe.com` link instead. Closes delta-io#711 Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> Author: Shixiong Zhu <zsxwing@gmail.com> #24449 is resolved by zsxwing/den3b8hd. GitOrigin-RevId: 1f8fdb3bba694ff53001d13ebca9f84dfae0748e (cherry picked from commit 86bbe99) * Setting version to 1.0.1 Co-authored-by: Yijia Cui <yijia.cui@databricks.com> Co-authored-by: gurunath <gurunathrajagopal@gmail.com> Co-authored-by: Shixiong Zhu <zsxwing@gmail.com> Co-authored-by: Tom Lynch <lync0143@gmail.com> Co-authored-by: Jarred Parrett <jarredparrett.dev@gmail.com> Co-authored-by: yaohua <yaohua.zhao@databricks.com> Co-authored-by: Allison Portis <allison.portis@databricks.com>

Improved Error message for DeltaColumnBuilder - if dataType was not p…

714e48d

…rovided

zsxwing requested changes Jul 9, 2021

View reviewed changes

rajagurunath added 3 commits July 10, 2021 09:54

Merge branch 'delta-io:master' into bugfix/dataTypeErrorMsg

9a1b542

PR changes: changed error message and test case

cd58fc3

PR changes: removed commented code

4a1b2bf

jaceklaskowski reviewed Jul 10, 2021

View reviewed changes

PR changes: fixing typos

34d1b23

zsxwing approved these changes Jul 12, 2021

View reviewed changes

pranavanand closed this in 104e2a4 Aug 3, 2021

vkorukanti mentioned this pull request Oct 7, 2021

NullPointerException when DeltaTable.create with Column with no DataType specified #698

Closed

tdas added this to the 1.1.0 milestone Nov 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raise error when dataType was not provided #714

Raise error when dataType was not provided #714

rajagurunath commented Jul 9, 2021

zsxwing left a comment

zsxwing Jul 9, 2021 •

edited

Loading

rajagurunath Jul 10, 2021

zsxwing Jul 9, 2021

rajagurunath Jul 10, 2021

rajagurunath commented Jul 10, 2021

jaceklaskowski Jul 10, 2021

rajagurunath Jul 11, 2021

jaceklaskowski Jul 10, 2021

rajagurunath Jul 11, 2021

jaceklaskowski Jul 10, 2021

rajagurunath Jul 11, 2021

rajagurunath commented Jul 11, 2021

zsxwing left a comment

Raise error when dataType was not provided #714

Raise error when dataType was not provided #714

Conversation

rajagurunath commented Jul 9, 2021

zsxwing left a comment

Choose a reason for hiding this comment

zsxwing Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajagurunath commented Jul 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajagurunath commented Jul 11, 2021

zsxwing left a comment

Choose a reason for hiding this comment

zsxwing Jul 9, 2021 •

edited

Loading