File tree Expand file tree Collapse file tree 3 files changed +18
-9
lines changed
standalone-metastore/packaging/src/docker Expand file tree Collapse file tree 3 files changed +18
-9
lines changed Original file line number Diff line number Diff line change @@ -133,10 +133,16 @@ change the `POSTGRES_LOCAL_PATH` to the path of the downloaded jar.
133133
134134- Metastore with S3 support
135135
136- Download aws-java-sdk-bundle-xxx.jar and place it under the jars directory:
136+ Download AWS SDK bundle and place it under the jars directory:
137+
138+ Note: Hadoop 3.4.1 requires both AWS SDK V1 and V2 for its S3A connector:
137139wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.770/aws-java-sdk-bundle-1.12.770.jar -P jars/
140+ wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.26.19/bundle-2.26.19.jar -P jars/
138141
139- Add the ` fs.s3a.access.key ` and ` fs.s3a.secret.key ` properties in ` metastore-site.xml ` under the conf directory.
142+ Then add the following configs to the ` metastore-site.xml ` under the conf directory:
143+ - fs.s3a.endpoint
144+ - fs.s3a.access.key
145+ - fs.s3a.secret.key
140146
141147Then,
142148``` shell
Original file line number Diff line number Diff line change 3434 </property >
3535
3636 <!--
37+ <property>
38+ <name>fs.s3a.endpoint</name>
39+ <value>s3.us-west-2.amazonaws.com</value>
40+ </property>
3741 <property>
3842 <name>fs.s3a.access.key</name>
3943 <value>YOUR_AWS_ACCESS_KEY</value>
Original file line number Diff line number Diff line change @@ -25,16 +25,15 @@ set -x
2525STAGING_DIR=" /tmp/ext-jars"
2626
2727TOOLS_LIB=" ${HADOOP_HOME} /share/hadoop/tools/lib"
28- COMMON_LIB =" ${HADOOP_HOME} /share/hadoop/common/ lib"
28+ HIVE_LIB =" ${HIVE_HOME} / lib/ "
2929
3030# Checks if /tmp/ext-jars is mounted (via Docker volume).
3131if [ -d " $STAGING_DIR " ]; then
32- # Check for aws-java-sdk-bundle (Wildcard handles versions)
33- if ls " $STAGING_DIR " /aws-java-sdk-bundle-* .jar 1> /dev/null 2>&1 ; then
34- echo " --> Installing AWS SDK Bundle..."
35- cp " $STAGING_DIR " /aws-java-sdk-bundle-* .jar " $COMMON_LIB /"
36- echo " --> activating hadoop-aws from tools..."
37- cp " $TOOLS_LIB " /hadoop-aws-* .jar " $COMMON_LIB /"
32+ if ls " $STAGING_DIR " /* .jar 1> /dev/null 2>&1 ; then
33+ echo " --> Copying custom jars from volume to Hive..."
34+ cp -vf " $STAGING_DIR " /* .jar " $HIVE_LIB /"
35+ else
36+ echo " --> Volume mounted at $STAGING_DIR , but no jars found."
3837 fi
3938fi
4039
You can’t perform that action at this time.
0 commit comments