Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug report] The API for creating an Iceberg catalog has a deprecated parameter "warehouse" that is required in the HTTP payload #5756

Closed
liuchunhao opened this issue Dec 4, 2024 · 4 comments · Fixed by #5923
Assignees
Labels
0.8.0 Release v0.8.0 bug Something isn't working

Comments

@liuchunhao
Copy link
Contributor

Version

main branch

Describe what's wrong

The property key warehouse was removed with rest assigned as a catalog-backend on Web UI, but REST API call for creating a catalog still requires warehouse in the request body and missing warehouse in the HTTP request body will cause the exception below.

No option warehouse provided with restcatalog-backend on Web UI
image

POST http://localhost:8090/api/metalakes/test/catalogs

{
  "name": "1",       
  "type": "relational", 
  "provider": "lakehouse-iceberg",
  "comment": "",      
  "properties": {
    "uri": "1",
    // "warehouse": "",   // Why is this a required property ?
    "catalog-backend": "rest",
    "authType": "simple"
  }
}

API call without “warehouse” causes the exception below:

{
    "code": 1001,
    "type": "IllegalArgumentException",
    "message": "Failed to operate catalog(s) [100] operation [CREATE] under metalake [test], reason [Properties are required and must be set: [warehouse]]",
    "stack": [
        ...
     ]
}

Error message and/or stacktrace

{
    "code": 1001,
    "type": "IllegalArgumentException",
    "message": "Failed to operate catalog(s) [100] operation [CREATE] under metalake [test], reason [Properties are required and must be set: [warehouse]]",
    "stack": [
        ...
     ]
}

How to reproduce

Create a test metalake and then create a catalog by sending the request without warehouse to the REST API as follows :

POST http://localhost:8090/api/metalakes/test/catalogs

{
  "name": "1",       
  "type": "relational", 
  "provider": "lakehouse-iceberg",
  "comment": "",      
  "properties": {
    "uri": "1",
    "catalog-backend": "rest",
    "authType": "simple"
  }
}

Additional context

No response

@liuchunhao liuchunhao added the bug Something isn't working label Dec 4, 2024
@FANNG1
Copy link
Contributor

FANNG1 commented Dec 9, 2024

warehouse is not required for Iceberg REST catalog backend. we should not check the exists of warehouse for REST catalog backend in https://github.com/apache/gravitino/blob/main/iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java#L89 , @liuchunhao would you like to fix it?

@liuchunhao
Copy link
Contributor Author

Sure, I'd be happy to handle it.

jerryshao pushed a commit that referenced this issue Dec 23, 2024
### What changes were proposed in this pull request?

Removed the systematic validations in Iceberg Config and modified the
instantiation of the warehouse attribute and uri attribute in the
IcebergCatalogWrapper to conform with the new possibility (REST).

### Why are the changes needed?

The bug was blocking the creation of a rest catalog.

Fix: [#5756](#5756)

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

./gradlew build && compileDistribution && assembleDistribution +
creation of the docker image there :
https://hub.docker.com/r/fsalhi2/gravitino

Started gravitino with such configs : 

```
### Gravitino General Settings

gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf

### HTTP Server

gravitino.iceberg-rest.host = 0.0.0.0
gravitino.iceberg-rest.httpPort = 9001

### Storage

gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.s3-access-key-id = XXXXX
gravitino.iceberg-rest.s3-secret-access-key = XXXXXX
gravitino.iceberg-rest.s3-path-style-access = true
gravitino.iceberg-rest.s3-endpoint = http://minio:9000/
gravitino.iceberg-rest.s3-region = us-east-1

### JDBC

gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.uri = jdbc:mysql://mysql:3306/
gravitino.iceberg-rest.warehouse = s3://lake/catalog
gravitino.iceberg-rest.jdbc.user = root
gravitino.iceberg-rest.jdbc.password = XXXXXX
gravitino.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
```

Was able to create a catalog through Web UI and start working on the
scheme.
@jerryshao jerryshao added the 0.8.0 Release v0.8.0 label Dec 23, 2024
@jerryshao
Copy link
Contributor

@fsalhi2 would you please leave an comment here, so I can assign the issue to you, thanks.

@fsalhi2
Copy link
Contributor

fsalhi2 commented Dec 23, 2024

Hi @jerryshao , here am i :)

Abyss-lord pushed a commit to Abyss-lord/gravitino that referenced this issue Dec 23, 2024
…pache#5923)

### What changes were proposed in this pull request?

Removed the systematic validations in Iceberg Config and modified the
instantiation of the warehouse attribute and uri attribute in the
IcebergCatalogWrapper to conform with the new possibility (REST).

### Why are the changes needed?

The bug was blocking the creation of a rest catalog.

Fix: [apache#5756](apache#5756)

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

./gradlew build && compileDistribution && assembleDistribution +
creation of the docker image there :
https://hub.docker.com/r/fsalhi2/gravitino

Started gravitino with such configs : 

```
### Gravitino General Settings

gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf

### HTTP Server

gravitino.iceberg-rest.host = 0.0.0.0
gravitino.iceberg-rest.httpPort = 9001

### Storage

gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.s3-access-key-id = XXXXX
gravitino.iceberg-rest.s3-secret-access-key = XXXXXX
gravitino.iceberg-rest.s3-path-style-access = true
gravitino.iceberg-rest.s3-endpoint = http://minio:9000/
gravitino.iceberg-rest.s3-region = us-east-1

### JDBC

gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.uri = jdbc:mysql://mysql:3306/
gravitino.iceberg-rest.warehouse = s3://lake/catalog
gravitino.iceberg-rest.jdbc.user = root
gravitino.iceberg-rest.jdbc.password = XXXXXX
gravitino.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
```

Was able to create a catalog through Web UI and start working on the
scheme.
Abyss-lord pushed a commit to Abyss-lord/gravitino that referenced this issue Dec 29, 2024
…pache#5923)

### What changes were proposed in this pull request?

Removed the systematic validations in Iceberg Config and modified the
instantiation of the warehouse attribute and uri attribute in the
IcebergCatalogWrapper to conform with the new possibility (REST).

### Why are the changes needed?

The bug was blocking the creation of a rest catalog.

Fix: [apache#5756](apache#5756)

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

./gradlew build && compileDistribution && assembleDistribution +
creation of the docker image there :
https://hub.docker.com/r/fsalhi2/gravitino

Started gravitino with such configs : 

```
### Gravitino General Settings

gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf

### HTTP Server

gravitino.iceberg-rest.host = 0.0.0.0
gravitino.iceberg-rest.httpPort = 9001

### Storage

gravitino.iceberg-rest.io-impl = org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.s3-access-key-id = XXXXX
gravitino.iceberg-rest.s3-secret-access-key = XXXXXX
gravitino.iceberg-rest.s3-path-style-access = true
gravitino.iceberg-rest.s3-endpoint = http://minio:9000/
gravitino.iceberg-rest.s3-region = us-east-1

### JDBC

gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.uri = jdbc:mysql://mysql:3306/
gravitino.iceberg-rest.warehouse = s3://lake/catalog
gravitino.iceberg-rest.jdbc.user = root
gravitino.iceberg-rest.jdbc.password = XXXXXX
gravitino.iceberg-rest.jdbc-driver = com.mysql.cj.jdbc.Driver
```

Was able to create a catalog through Web UI and start working on the
scheme.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.8.0 Release v0.8.0 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants