Add REST Catalog tests to Spark 3.5 integration test #11093

haizhou-zhao · 2024-09-06T21:17:59Z

For issue: #11079

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java

nastra · 2024-10-08T08:49:05Z

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java

@@ -521,7 +524,7 @@ public void testFilesTableTimeTravelWithSchemaEvolution() throws Exception {
                optional(3, "category", Types.StringType.get())));

    spark.createDataFrame(newRecords, newSparkSchema).coalesce(1).writeTo(tableName).append();
-
+    table.refresh();


is this refresh only needed for REST?

Yes, RESTTableOperations and other TableOperations has different mechanisms of refreshing metadata.

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

nastra · 2024-10-08T08:58:42Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

+      // JdbcCatalog, then different jdbc connections could provide different views of table
+      // status even belonging to the same catalog. Reference:
+      // https://www.sqlite.org/inmemorydb.html
+      System.setProperty(CatalogProperties.CLIENT_POOL_SIZE, "1");


is this strictly needed to make tests pass? I don't think we set this in any other tests for that specific purpose

Yes, this is needed for test to pass. I attempted to run test without this line and here're the result: https://github.com/apache/iceberg/actions/runs/11297060489/job/31423193928?pr=11093

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

nastra · 2024-10-11T07:23:59Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

+      System.setProperty(CatalogProperties.CLIENT_POOL_SIZE, "1");
+      restServer.start(false);
+      restCatalog = RCKUtils.initCatalogClient();
+      System.clearProperty("rest.port");


maybe we should pass a Map<String, String> to the REST server rather than having to set system properties (which then also need to be cleared again). @danielcweeks thoughts on this?

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

nastra · 2024-10-11T07:32:07Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

  @BeforeEach
  public void before() {
-    this.validationCatalog =


what happens if we don't do any changes to how the validation catalog is configured?

something like you created a table on REST catalog, while validate against Hive catalog.

my point is that I think all tests should be passing when we don't do any changes to the validation catalog

If I understand correctly, the original validationCatalog is set to a HadoopCatalog if catalogName of the catalog being tested is named testhadoop; otherwise, make the validationCatalog the same as catalog (which is strictly a HiveCatalog, as defined by TestBase class).

That will work in the old days, as we only have 2 types catalogs, either Hadoop or Hive, being tested - if the test subject is not a Hadoop Catalog, then setting the validation catalog to Hive Catalog will suffice the validation purpose. However, with the introduction of a 3rd type, REST catalog: when the test subject is a REST catalog, without changing how validationCatalog is initialized, the validationCatalog will be set to a Hive catalog. In this case, conducting test behaviors on REST catalog while validating the status post-change on Hive catalog won't work.

Unless, you are suggesting that we should make changes to TestBase class where the catalog being tested does not strictly need to be a HiveCatalog, and can be any type of catalog.

nastra · 2024-10-11T07:33:29Z

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java

-    Map<String, String> catalogProperties = RCKUtils.environmentCatalogConfig();
+    Map<String, String> catalogProperties = Maps.newHashMap();
+    catalogProperties.putAll(RCKUtils.environmentCatalogConfig());
+    catalogProperties.putAll(Maps.fromProperties(System.getProperties()));


as mentioned in a comment further above, maybe we should consider passing a Map<String, String> to RESTCatalogServer rather than relying on system properties (which have to be cleared after they were configured)

that means we can revert this line here: catalogProperties.putAll(Maps.fromProperties(System.getProperties()));

haizhou-zhao · 2024-10-11T21:31:03Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

+    // JdbcCatalog, then different jdbc connections could provide different views of table
+    // status even belonging to the same catalog. Reference:
+    // https://www.sqlite.org/inmemorydb.html
+    System.setProperty(CatalogProperties.CLIENT_POOL_SIZE, "1");


There was this conversation getting lost (not sure why): #11093 (comment)

haizhou-zhao · 2024-10-11T21:36:19Z

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

+    System.setProperty(CatalogProperties.CLIENT_POOL_SIZE, "1");
+    restServer.start(false);
+    restCatalog = RCKUtils.initCatalogClient();
+    System.clearProperty("rest.port");


This one was also lost: #11093 (comment)

As I mentioned in #11093 (comment) I think we should do this differently rather than passing/re-setting system properties

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java

…alogServer.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

…seWithCatalog.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

…seWithCatalog.java

c-thiel · 2024-11-21T10:56:43Z

@nastra, @danielcweeks is there something stopping us from merging this?
This is currently blocking us on #11317 which is a requirement for REST Catalogs to offer undrop for storages that are not S3.

danielcweeks

Thanks @haizhou-zhao for getting this done!

* Add REST Catalog tests to Spark 3.5 integration test Add REST Catalog tests to Spark 3.4 integration test tmp save Fix integ tests Revert "Add REST Catalog tests to Spark 3.4 integration test" This reverts commit d052416. unneeded changes fix test retrigger checks Fix integ test Fix port already in use Fix unmatched validation catalog spotless Fix sqlite related test failures * Rebase & spotless * code format * unneeded change * unneeded change * Revert "unneeded change" This reverts commit ae29c41. * code format * Use in-mem config to configure RCK * Update open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java * Use RESTServerExtension * check style and test failure * test failure * fix test * fix test * spotless * Update open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> * Update open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> * Update spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> * Spotless and fix test * Apply suggestions from code review * Apply suggestions from code review * Apply suggestions from code review * Update spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java * Package protected RCKUtils * spotless * unintentional change * remove warehouse specification from rest * spotless * move find free port to rest server extension * fix typo * checkstyle * fix unit test --------- Co-authored-by: Haizhou Zhao <haizhouzhao@Haizhous-MacBook-Pro.local> Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

github-actions bot added spark build OPENAPI labels Sep 6, 2024

haizhou-zhao force-pushed the spark-rest-integ-test branch 2 times, most recently from 9481bd4 to 8bcd853 Compare September 18, 2024 23:18

nastra reviewed Oct 8, 2024

View reviewed changes

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java Outdated Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java Outdated Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 8, 2024

View reviewed changes

nastra reviewed Oct 11, 2024

View reviewed changes

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

...5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 11, 2024

View reviewed changes

haizhou-zhao commented Oct 11, 2024

View reviewed changes

nastra reviewed Oct 23, 2024

View reviewed changes

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java Outdated Show resolved Hide resolved

nastra reviewed Oct 23, 2024

View reviewed changes

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java Outdated Show resolved Hide resolved

nastra requested a review from danielcweeks October 23, 2024 06:39

nastra mentioned this pull request Oct 24, 2024

Spark: Add view support to SparkSessionCatalog #11388

Merged

danielcweeks reviewed Oct 24, 2024

View reviewed changes

spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBaseWithCatalog.java Outdated Show resolved Hide resolved

nastra reviewed Oct 30, 2024

View reviewed changes

Haizhou Zhao and others added 23 commits November 14, 2024 16:31

Use RESTServerExtension

0a613f6

check style and test failure

c914475

test failure

d18c3fa

fix test

82cca8b

fix test

4672381

spotless

c1db1f9

Update open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCat…

de4be14

…alogServer.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

Update open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCat…

4e2dd03

…alogServer.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

Update spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBa…

c549eb3

…seWithCatalog.java Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

Spotless and fix test

043cd83

Apply suggestions from code review

b854759

Apply suggestions from code review

bf0dd18

Apply suggestions from code review

b729f4f

Update spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBa…

45b76f4

…seWithCatalog.java

Package protected RCKUtils

c9f06b9

spotless

2f1ad6c

unintentional change

a23bc7f

remove warehouse specification from rest

eb1a345

spotless

048f037

move find free port to rest server extension

bac5d17

fix typo

85655d8

checkstyle

0ef2c2b

fix unit test

2d31dc0

haizhou-zhao force-pushed the spark-rest-integ-test branch from 09f66ed to 2d31dc0 Compare November 15, 2024 01:17

danielcweeks approved these changes Nov 21, 2024

View reviewed changes

danielcweeks merged commit a52afdc into apache:main Nov 21, 2024
32 checks passed

nastra mentioned this pull request Dec 4, 2024

Spark 3.4: Add REST catalog to Spark integration tests #11698

Merged

nastra mentioned this pull request Feb 28, 2025

Adding RESTCatalog based Spark Integ Test #11079

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add REST Catalog tests to Spark 3.5 integration test #11093

Add REST Catalog tests to Spark 3.5 integration test #11093

haizhou-zhao commented Sep 6, 2024

nastra Oct 8, 2024

haizhou-zhao Oct 10, 2024

nastra Oct 8, 2024

haizhou-zhao Oct 11, 2024

nastra Oct 11, 2024 •

edited

Loading

nastra Oct 11, 2024

haizhou-zhao Oct 11, 2024

nastra Oct 14, 2024

haizhou-zhao Oct 14, 2024

nastra Oct 11, 2024

haizhou-zhao Oct 14, 2024

nastra Oct 23, 2024

haizhou-zhao Oct 29, 2024

haizhou-zhao Oct 11, 2024

haizhou-zhao Oct 11, 2024

nastra Oct 14, 2024

haizhou-zhao Oct 14, 2024

c-thiel commented Nov 21, 2024

danielcweeks left a comment

Add REST Catalog tests to Spark 3.5 integration test #11093

Add REST Catalog tests to Spark 3.5 integration test #11093

Conversation

haizhou-zhao commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nastra Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c-thiel commented Nov 21, 2024

danielcweeks left a comment

Choose a reason for hiding this comment

nastra Oct 11, 2024 •

edited

Loading