diff --git a/docs/assets/trino/add-config.jpg b/docs/assets/trino/add-config.jpg new file mode 100644 index 00000000000..a94b17128d6 Binary files /dev/null and b/docs/assets/trino/add-config.jpg differ diff --git a/docs/assets/trino/add-link.jpg b/docs/assets/trino/add-link.jpg new file mode 100644 index 00000000000..a64f22ec2bf Binary files /dev/null and b/docs/assets/trino/add-link.jpg differ diff --git a/docs/assets/trino/config.properties b/docs/assets/trino/config.properties new file mode 100644 index 00000000000..00ce823323d --- /dev/null +++ b/docs/assets/trino/config.properties @@ -0,0 +1,45 @@ +# +# Copyright 2024 Datastrato Pvt Ltd. +# This software is licensed under the Apache License version 2. +# + +# +# WARNING +# ^^^^^^^ +# This configuration file is for development only and should NOT be used +# in production. For example configuration, see the Trino documentation. +# sample nodeId to provide consistency across test runs +node.id=ffffffff-ffff-ffff-ffff-ffffffffffff +node.environment=test +node.internal-address=localhost +experimental.concurrent-startup=true + +# Default port is 8080, We change it to 8180 +http-server.http.port=8180 + +discovery.uri=http://localhost:8180 + +exchange.http-client.max-connections=1000 +exchange.http-client.max-connections-per-server=1000 +exchange.http-client.connect-timeout=1m +exchange.http-client.idle-timeout=1m + +scheduler.http-client.max-connections=1000 +scheduler.http-client.max-connections-per-server=1000 +scheduler.http-client.connect-timeout=1m +scheduler.http-client.idle-timeout=1m + +query.client.timeout=5m +query.min-expire-age=30m + +# We removed several catalogs that won't be used in Gravitino +plugin.bundles=\ + ../../plugin/trino-iceberg/pom.xml,\ + ../../plugin/trino-hive/pom.xml,\ + ../../plugin/trino-local-file/pom.xml, \ + ../../plugin/trino-mysql/pom.xml,\ + ../../plugin/trino-postgresql/pom.xml, \ + ../../plugin/trino-exchange-filesystem/pom.xml, \ + ../../plugin/trino-gravitino/pom.xml + +node-scheduler.include-coordinator=true diff --git a/docs/assets/trino/create-gravitino-connector.jpg b/docs/assets/trino/create-gravitino-connector.jpg new file mode 100644 index 00000000000..7c10eda1b65 Binary files /dev/null and b/docs/assets/trino/create-gravitino-connector.jpg differ diff --git a/docs/assets/trino/gravitino.properties b/docs/assets/trino/gravitino.properties new file mode 100644 index 00000000000..b0d8cca3c78 --- /dev/null +++ b/docs/assets/trino/gravitino.properties @@ -0,0 +1,13 @@ +# +# Copyright 2024 Datastrato Pvt Ltd. +# This software is licensed under the Apache License version 2. +# + +# the connector name is always 'gravitino' +connector.name=gravitino + +# uri of the gravitino server, you need to change it according to your environment +gravitino.uri=http://localhost:8090 + +# The name of the metalake to which the connector is connected, you need to change it according to your environment +gravitino.metalake=test diff --git a/docs/assets/trino/pom.xml b/docs/assets/trino/pom.xml new file mode 100644 index 00000000000..77536f2f138 --- /dev/null +++ b/docs/assets/trino/pom.xml @@ -0,0 +1,168 @@ + + + + + 4.0.0 + + io.trino + trino-root + 426 + ../../pom.xml + + + trino-gravitino + trino-plugin + Trino - Graviton Connector + + + ${project.parent.basedir} + + + + + com.google.guava + guava + + + + com.google.inject + guice + + + + io.airlift + bootstrap + + + log4j-to-slf4j + org.apache.logging.log4j + + + + + + io.airlift + configuration + + + + io.airlift + json + + + + io.trino + trino-plugin-toolkit + + + + jakarta.validation + jakarta.validation-api + + + + com.fasterxml.jackson.core + jackson-annotations + provided + + + + io.airlift + slice + provided + + + + io.opentelemetry + opentelemetry-api + provided + + + + io.opentelemetry + opentelemetry-context + provided + + + + io.trino + trino-spi + provided + + + + org.openjdk.jol + jol-core + provided + + + + com.fasterxml.jackson.core + jackson-databind + + + + io.airlift + node + runtime + + + + org.apache.httpcomponents.client5 + httpclient5 + 5.2.1 + + + + org.apache.commons + commons-lang3 + + + + org.antlr + antlr4-runtime + 4.9.2 + + + + org.testng + testng + test + + + io.trino + trino-memory + test + + + io.trino + trino-testing + test + + + + + com.datastrato.gravitino + client-java-runtime + 0.5.0-SNAPSHOT + + + + com.datastrato.gravitino + bundled-catalog + 0.5.0-SNAPSHOT + + + + org.apache.commons + commons-collections4 + 4.4 + + + diff --git a/docs/assets/trino/show-catalogs.jpg b/docs/assets/trino/show-catalogs.jpg new file mode 100644 index 00000000000..9ff05eee081 Binary files /dev/null and b/docs/assets/trino/show-catalogs.jpg differ diff --git a/docs/assets/trino/start-trino.jpg b/docs/assets/trino/start-trino.jpg new file mode 100644 index 00000000000..c8dcd3f78f5 Binary files /dev/null and b/docs/assets/trino/start-trino.jpg differ diff --git a/docs/trino-connector/development.md b/docs/trino-connector/development.md new file mode 100644 index 00000000000..c8af1a181e1 --- /dev/null +++ b/docs/trino-connector/development.md @@ -0,0 +1,79 @@ +--- +title: "Gravitino connector development" +slug: /trino-connector/development +keyword: gravitino connector development +license: "Copyright 2024 Datastrato Pvt Ltd. +This software is licensed under the Apache License version 2." +--- + +This document is to guide users through the development of the Gravitino connector for Trino locally. + +## Prerequisites + +Before you start developing the Gravitino trino connector, you need to have the following prerequisites: + +1. You need to start the Gravitino server locally, for more information, please refer to the [start Gravitino server](../how-to-install.md) +2. Create a catalog in the Gravitino server, for more information, please refer to the [Gravitino metadata management](../manage-metadata-using-gravitino.md). Assuming we have just created a MySQL catalog using the following command: + +```curl +curl -X POST -H "Content-Type: application/json" -d '{"name":"test","comment":"comment","properties":{}}' http://localhost:8090/api/metalakes + +curl -X POST -H "Content-Type: application/json" -d '{"name":"mysql_catalog3","type":"RELATIONAL","comment":"comment","provider":"jdbc-mysql", "properties":{ + "jdbc-url": "jdbc:mysql://127.0.0.1:3306?useSSL=false&allowPublicKeyRetrieval=true", + "jdbc-user": "root", + "jdbc-password": "123456", + "jdbc-driver": "com.mysql.cj.jdbc.Driver" +}}' http://localhost:8090/api/metalakes/test/catalogs +``` + +:::note +Please change the above `localhost`, `port` and the names of metalake and catalogs accordingly. +::: + + +## Development environment + +To develop the Gravitino connector locally, you need to do the following steps: + +### IDEA + +1. Clone the Trino repository from the [GitHub](https://github.com/trinodb/trino) repository. We advise you to use the release version 426 or 435. +2. Open the Trino project in your IDEA. +3. Create a new module for the Gravitino connector in the Trino project as the following picture (we will use the name `trino-gravitino` as the module name in the following steps). ![trino-gravitino](../assets/trino/create-gravitino-connector.jpg) +4. Add a soft link to the Gravitino trino connector module in the Trino project. Assuming the src java main directory of the Gravitino trino connector in project Gravitino is `gravitino/path/to/gravitino-trino-connector/src/main/java`, +and the src java main directory of trino-gravitino in the Trino project is `trino/path/to/trino-gravitino/src/main/java`, you can use the following command to create a soft link: + +```shell +ln -s gravitino/path/to/trino-connector/src/main/java trino/path/to/trino-gravitino/src/main/java +``` +then you can see the `gravitino-trino-connecor` source files and directories in the `trino-gravitino` module as follows: + +![trino-gravitino-structure](../assets/trino/add-link.jpg) + +5. Change the `pom.xml` file in the `trino-gravitino` module accordingly. This is a [example](../assets/trino/pom.xml) of the `pom.xml` file in the `trino-gravitino` module. +6. Try to compile module `trino-gravitino` to see if there are any errors. +```shell +# build the whole trino project +./mvnw -pl '!core/trino-server-rpm' package -DskipTests -Dair.check.skip-all=true + + +# build the trino-gravitino module if we change the code in the trino-gravitino module +./mvnw clean -pl 'plugin/trino-gravitino' package -DskipTests -Dcheckstyle.skip -Dair.check.skip-checkstyle=true -DskipTests -Dair.check.skip-all=true +``` +7. Set up the configuration for the Gravitino connector in the Trino project. You can do as the following picture shows: +![](../assets/trino/add-config.jpg) + +The corresponding configuration files are here: [gravitino.properties](../assets/trino/gravitino.properties) and [config.properties](../assets/trino/config.properties). + +8. Start the Trino server and connect to the Gravitino server. +![](../assets/trino/start-trino.jpg) +9. If `DevelopmentServer` has started successfully, you can connect to the Trino server using the `trino-cli` and run the following command to see if the Gravitino connector is available: +```shell +java -jar trino-cli-429-executable.jar --server localhost:8180 +``` +:::note +The `trino-cli-429-executable.jar` is the Trino CLI jar file, you can download it from the [Trino release page](https://trino.io/docs/current/client/cli.html). **Users can use the version of the Trino CLI jar file according to the version of the Trino server.** +::: + +10. If nothing goes wrong, you can start developing the Gravitino connector in the Gravitino project and debug it in the Trino project. +![](../assets/trino/show-catalogs.jpg) diff --git a/docs/trino-connector/index.md b/docs/trino-connector/index.md index 4c1d2e9f5be..25c8fb766b9 100644 --- a/docs/trino-connector/index.md +++ b/docs/trino-connector/index.md @@ -18,3 +18,4 @@ Gravitino connector index: - [MySQL](catalog-mysql.md) - [PostgreSQL](catalog-postgresql.md) - [Supported SQL](sql-support.md) + - [Development](development.md)