From 2411dbe0e6d9e78931535472d8ff0d8ab0af5b8a Mon Sep 17 00:00:00 2001 From: slothever <18522955+wsjz@users.noreply.github.com> Date: Mon, 25 Dec 2023 23:05:44 +0800 Subject: [PATCH] [doc](multi-catalog)add krb and some ertificates FAQs (#28858) add some security docs --- docs/en/docs/lakehouse/faq.md | 18 ++++++++++++++++++ docs/en/docs/lakehouse/multi-catalog/hive.md | 2 ++ .../docs/lakehouse/multi-catalog/iceberg.md | 2 ++ docs/zh-CN/docs/lakehouse/faq.md | 19 +++++++++++++++++++ .../docs/lakehouse/multi-catalog/hive.md | 2 ++ .../docs/lakehouse/multi-catalog/iceberg.md | 2 ++ 6 files changed, 45 insertions(+) diff --git a/docs/en/docs/lakehouse/faq.md b/docs/en/docs/lakehouse/faq.md index ee441961c963cd..3b20fa878c2005 100644 --- a/docs/en/docs/lakehouse/faq.md +++ b/docs/en/docs/lakehouse/faq.md @@ -27,6 +27,19 @@ under the License. # FAQ +## Certificates + +1. If an error is reported: `curl 77: Problem with the SSL CA cert.`, need update your certificate. + - Download the latest certificate from `https://curl.haxx.se/docs/caextract.html`. + - Place the downloaded cacert-xxx.pem in the `/etc/ssl/certs/` directory. For example: `sudo cp cacert-xxx.pem /etc/ssl/certs/ca-certificates.crt`. + +2. If an error is reported: `ERROR 1105 (HY000): errCode = 2, detailMessage = (x.x.x.x)[CANCELLED][INTERNAL_ERROR]error setting certificate verify locations: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none`. + +``` +yum install -y ca-certificates +ln -s /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt /etc/ssl/certs/ca-certificates.crt +``` + ## Kerberos @@ -58,6 +71,11 @@ under the License. - The principal used must exist in the klist, use `klist -kt your.keytab` to check. - Ensure the catalog configuration correct, such as missing the `yarn.resourcemanager.principal`. - If the preceding checks are correct, the JDK version installed by yum or other package-management utility in the current system maybe have an unsupported encryption algorithm. It is recommended to install JDK by yourself and set `JAVA_HOME` environment variable. + - Kerberos uses AES-256 by default for encryption. If you use Oracle JDK, you must install JCE. In the case of OpenJDK, some distributions of OpenJDK automatically provide the JCE Unlimited Strength Jurisdiction Policy Files, so it's not need to install JCE. + - The JCE version corresponds to the JDK version. You need to select the JCE according to the JDK version. Download the JCE zip package and decompress it into `$JAVA_HOME/jre/lib/security`: + - JDK6：[JCE6](http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html) + - JDK7：[JCE7](http://www.oracle.com/technetwork/java/embedded/embedded-se/downloads/jce-7-download-432124.html) + - JDK8：[JCE8](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html) 5. An error is reported when using KMS to access HDFS: `java.security.InvalidKeyException: Illegal key size` diff --git a/docs/en/docs/lakehouse/multi-catalog/hive.md b/docs/en/docs/lakehouse/multi-catalog/hive.md index 754c3954f5268e..25fddea125064d 100644 --- a/docs/en/docs/lakehouse/multi-catalog/hive.md +++ b/docs/en/docs/lakehouse/multi-catalog/hive.md @@ -168,6 +168,8 @@ CREATE CATALOG hive PROPERTIES ( ### Hive With Glue +> When connecting Glue, if it's not on the EC2 environment, need copy the `~/.aws` from the EC2 environment to the current environment. And can also download and configure the [AWS Cli tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), which also creates the `.aws` directory under the current user directory. + ```sql CREATE CATALOG hive PROPERTIES ( "type"="hms", diff --git a/docs/en/docs/lakehouse/multi-catalog/iceberg.md b/docs/en/docs/lakehouse/multi-catalog/iceberg.md index 2baa05770ff135..07325114ca1d56 100644 --- a/docs/en/docs/lakehouse/multi-catalog/iceberg.md +++ b/docs/en/docs/lakehouse/multi-catalog/iceberg.md @@ -96,6 +96,8 @@ CREATE CATALOG iceberg PROPERTIES ( #### AWS Glue +> When connecting Glue, if it's not on the EC2 environment, need copy the `~/.aws` from the EC2 environment to the current environment. And can also download and configure the [AWS Cli tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), which also creates the `.aws` directory under the current user directory. + ```sql CREATE CATALOG glue PROPERTIES ( "type"="iceberg", diff --git a/docs/zh-CN/docs/lakehouse/faq.md b/docs/zh-CN/docs/lakehouse/faq.md index 8651784e2e81b0..7c608eb61ae234 100644 --- a/docs/zh-CN/docs/lakehouse/faq.md +++ b/docs/zh-CN/docs/lakehouse/faq.md @@ -27,6 +27,19 @@ under the License. # 常见问题 +## 证书问题 + +1. 查询时报错 `curl 77: Problem with the SSL CA cert.`。说明当前系统证书过旧，需要更新本地证书。 + - 可以从 `https://curl.haxx.se/docs/caextract.html` 下载最新的CA证书。 + - 将下载后的cacert-xxx.pem放到`/etc/ssl/certs/`目录，例如：`sudo cp cacert-xxx.pem /etc/ssl/certs/ca-certificates.crt`。 + +2. 查询时报错：`ERROR 1105 (HY000): errCode = 2, detailMessage = (x.x.x.x)[CANCELLED][INTERNAL_ERROR]error setting certificate verify locations: CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none`. + +``` +yum install -y ca-certificates +ln -s /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt /etc/ssl/certs/ca-certificates.crt +``` + ## Kerberos 1. 连接 Kerberos 认证的 Hive Metastore 报错：`GSS initiate failed` @@ -58,6 +71,12 @@ under the License. - 用到的principal必须在klist中存在，使用`klist -kt your.keytab`检查。 - 检查catalog配置是否正确，比如漏配`yarn.resourcemanager.principal`。 - 若上述检查没问题，则当前系统yum或者其他包管理软件安装的JDK版本存在不支持的加密算法，建议自行安装JDK并设置`JAVA_HOME`环境变量。 + - Kerberos默认使用AES-256来进行加密。如果使用Oracle JDK，则必须安装JCE。如果是OpenJDK，OpenJDK的某些发行版会自动提供无限强度的JCE，因此不需要安装JCE。 + - JCE与JDK版本是对应的，需要根据JDK的版本来选择JCE版本，下载JCE的zip包并解压到`$JAVA_HOME/jre/lib/security`目录下： + - JDK6：[JCE6](http://www.oracle.com/technetwork/java/javase/downloads/jce-6-download-429243.html) + - JDK7：[JCE7](http://www.oracle.com/technetwork/java/embedded/embedded-se/downloads/jce-7-download-432124.html) + - JDK8：[JCE8](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html) + 5. 使用 KMS 访问 HDFS 时报错：`java.security.InvalidKeyException: Illegal key size` diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md b/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md index 16f4565522a300..e75977c25f4766 100644 --- a/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md +++ b/docs/zh-CN/docs/lakehouse/multi-catalog/hive.md @@ -168,6 +168,8 @@ CREATE CATALOG hive PROPERTIES ( ### Hive With Glue +> 连接Glue时，如果是在非EC2环境，需要将EC2环境里的 `~/.aws` 目录拷贝到当前环境里。也可以下载[AWS Cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)工具进行配置，这种方式也会在当前用户目录下创建`.aws`目录。 + ```sql CREATE CATALOG hive PROPERTIES ( "type"="hms", diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md b/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md index 3e6a4826d02049..9bf61f6d9f7646 100644 --- a/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md +++ b/docs/zh-CN/docs/lakehouse/multi-catalog/iceberg.md @@ -96,6 +96,8 @@ CREATE CATALOG iceberg PROPERTIES ( #### AWS Glue +> 连接Glue时，如果是在非EC2环境，需要将EC2环境里的 `~/.aws` 目录拷贝到当前环境里。也可以下载[AWS Cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)工具进行配置，这种方式也会在当前用户目录下创建`.aws`目录。 + ```sql CREATE CATALOG glue PROPERTIES ( "type"="iceberg",