Skip to content

Commit db51c7a

Browse files
zhengchenyuzhengchenyu
authored
Merge branch 'trunk' into YARN-11153 
2 parents ada446b + b5b9a81 commit db51c7a

File tree

2,438 files changed

+452152
-81731
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,438 files changed

+452152
-81731
lines changed

‎.asf.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@
1414
# limitations under the License.
1515

1616
github:
17+
ghp_path: /
18+
ghp_branch: gh-pages
1719
enabled_merge_buttons:
1820
squash: true
1921
merge: false

‎.github/labeler.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
# Pull Request Labeler Github Action Configuration: https://github.com/marketplace/actions/labeler
21+
22+
trunk:
23+
- '**'
24+
INFRA:
25+
- .asf.yaml
26+
- .gitattributes
27+
- .gitignore
28+
- .github/**
29+
- dev-support/**
30+
- start-build-env.sh
31+
BUILD:
32+
- '**/pom.xml'
33+
COMMON:
34+
- hadoop-common-project/**
35+
HDFS:
36+
- hadoop-hdfs-project/**
37+
RBF:
38+
- hadoop-hdfs-project/hadoop-hdfs-rbf/**
39+
NATIVE:
40+
- hadoop-hdfs-project/hadoop-hdfs-native-client/**
41+
- hadoop-common-project/hadoop-common/src/main/native/**
42+
YARN:
43+
- hadoop-yarn-project/**
44+
MAPREDUCE:
45+
- hadoop-mapreduce-project/**
46+
DISTCP:
47+
- hadoop-tools/hadoop-distcp/**
48+
TOOLS:
49+
- hadoop-tools/**
50+
AWS:
51+
- hadoop-tools/hadoop-aws/**
52+
ABFS:
53+
- hadoop-tools/hadoop-azure/**
54+
DYNAMOMETER:
55+
- hadoop-tools/hadoop-dynamometer/**
56+
MAVEN-PLUGINS:
57+
- hadoop-maven-plugins/**

‎.github/workflows/labeler.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Pull Request Labeler"
21+
on: pull_request_target
22+
23+
permissions:
24+
contents: read
25+
pull-requests: write
26+
27+
jobs:
28+
triage:
29+
runs-on: ubuntu-latest
30+
steps:
31+
- uses: actions/checkout@v3
32+
with:
33+
sparse-checkout: |
34+
.github
35+
- uses: actions/labeler@v4.3.0
36+
with:
37+
repo-token: ${{ secrets.GITHUB_TOKEN }}
38+
sync-labels: true
39+
configuration-path: .github/labeler.yml
40+
dot: true

‎.github/workflows/website.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
17+
name: website
18+
19+
# Controls when the action will run.
20+
on:
21+
push:
22+
branches: [ trunk ]
23+
24+
jobs:
25+
build:
26+
runs-on: ubuntu-latest
27+
steps:
28+
- name: Checkout Hadoop trunk
29+
uses: actions/checkout@v3
30+
with:
31+
repository: apache/hadoop
32+
- name: Set up JDK 8
33+
uses: actions/setup-java@v3
34+
with:
35+
java-version: '8'
36+
distribution: 'temurin'
37+
- name: Cache local Maven repository
38+
uses: actions/cache@v3
39+
with:
40+
path: ~/.m2/repository
41+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
42+
restore-keys: |
43+
${{ runner.os }}-maven-
44+
- name: Build Hadoop maven plugins
45+
run: cd hadoop-maven-plugins && mvn --batch-mode install
46+
- name: Build Hadoop
47+
run: mvn clean install -DskipTests -DskipShade
48+
- name: Build document
49+
run: mvn clean site
50+
- name: Stage document
51+
run: mvn site:stage -DstagingDirectory=${GITHUB_WORKSPACE}/staging/
52+
- name: Deploy to GitHub Pages
53+
uses: peaceiris/actions-gh-pages@v3
54+
with:
55+
github_token: ${{ secrets.GITHUB_TOKEN }}
56+
publish_dir: ./staging/hadoop-project
57+
user_name: 'github-actions[bot]'
58+
user_email: 'github-actions[bot]@users.noreply.github.com'
59+

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
*.patch
1212
*.diff
1313
.idea
14+
.vscode
1415
.svn
1516
.classpath
1617
.project

‎.yetus/excludes.txt‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
dev-support/docker/Dockerfile_windows_10

‎BUILDING.txt‎

Lines changed: 84 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -492,39 +492,66 @@ Building on CentOS 8
492492

493493
----------------------------------------------------------------------------------
494494

495-
Building on Windows
495+
Building on Windows 10
496496

497497
----------------------------------------------------------------------------------
498498
Requirements:
499499

500-
* Windows System
500+
* Windows 10
501501
* JDK 1.8
502-
* Maven 3.0 or later
503-
* Boost 1.72
504-
* Protocol Buffers 3.7.1
505-
* CMake 3.19 or newer
506-
* Visual Studio 2010 Professional or Higher
507-
* Windows SDK 8.1 (if building CPU rate control for the container executor)
508-
* zlib headers (if building native code bindings for zlib)
502+
* Maven 3.0 or later (maven.apache.org)
503+
* Boost 1.72 (boost.org)
504+
* Protocol Buffers 3.7.1 (https://github.com/protocolbuffers/protobuf/releases)
505+
* CMake 3.19 or newer (cmake.org)
506+
* Visual Studio 2019 (visualstudio.com)
507+
* Windows SDK 8.1 (optional, if building CPU rate control for the container executor. Get this from
508+
http://msdn.microsoft.com/en-us/windows/bg162891.aspx)
509+
* Zlib (zlib.net, if building native code bindings for zlib)
510+
* Git (preferably, get this from https://git-scm.com/download/win since the package also contains
511+
Unix command-line tools that are needed during packaging).
512+
* Python (python.org, for generation of docs using 'mvn site')
509513
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
510-
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
511-
tools must be present on your PATH.
512-
* Python ( for generation of docs using 'mvn site')
513514

514-
Unix command-line tools are also included with the Windows Git package which
515-
can be downloaded from http://git-scm.com/downloads
515+
----------------------------------------------------------------------------------
516516

517-
If using Visual Studio, it must be Professional level or higher.
518-
Do not use Visual Studio Express. It does not support compiling for 64-bit,
519-
which is problematic if running a 64-bit system.
517+
Building guidelines:
520518

521-
The Windows SDK 8.1 is available to download at:
519+
Hadoop repository provides the Dockerfile for building Hadoop on Windows 10, located at
520+
dev-support/docker/Dockerfile_windows_10. It is highly recommended to use this and create the
521+
Docker image for building Hadoop on Windows 10, since you don't have to install anything else
522+
other than Docker and no additional steps are required in terms of aligning the environment with
523+
the necessary paths etc.
522524

523-
http://msdn.microsoft.com/en-us/windows/bg162891.aspx
525+
However, if you still prefer taking the route of not using Docker, this Dockerfile_windows_10 will
526+
still be immensely useful as a raw guide for all the steps involved in creating the environment
527+
needed to build Hadoop on Windows 10.
524528

525-
Cygwin is not required.
529+
Building using the Docker:
530+
We first need to build the Docker image for building Hadoop on Windows 10. Run this command from
531+
the root of the Hadoop repository.
532+
> docker build -t hadoop-windows-10-builder -f .\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
533+
534+
Start the container with the image that we just built.
535+
> docker run --rm -it hadoop-windows-10-builder
536+
537+
You can now clone the Hadoop repo inside this container and proceed with the build.
538+
539+
NOTE:
540+
While one may perceive the idea of mounting the locally cloned (on the host filesystem) Hadoop
541+
repository into the container (using the -v option), we have seen the build to fail owing to some
542+
files not being able to be located by Maven. Thus, we suggest cloning the Hadoop repository to a
543+
non-mounted folder inside the container and proceed with the build. When the build is completed,
544+
you may use the "docker cp" command to copy the built Hadoop tar.gz file from the docker container
545+
to the host filesystem. If you still would like to mount the Hadoop codebase, a workaround would
546+
be to copy the mounted Hadoop codebase into another folder (which doesn't point to a mount) in the
547+
container's filesystem and use this for building.
548+
549+
However, we noticed no build issues when the Maven repository from the host filesystem was mounted
550+
into the container. One may use this to greatly reduce the build time. Assuming that the Maven
551+
repository is located at D:\Maven\Repository in the host filesystem, one can use the following
552+
command to mount the same onto the default Maven repository location while launching the container.
553+
> docker run --rm -v D:\Maven\Repository:C:\Users\ContainerAdministrator\.m2\repository -it hadoop-windows-10-builder
526554

527-
----------------------------------------------------------------------------------
528555
Building:
529556

530557
Keep the source code tree in a short path to avoid running into problems related
@@ -540,6 +567,24 @@ configure the bit-ness of the build, and set several optional components.
540567
Several tests require that the user must have the Create Symbolic Links
541568
privilege.
542569

570+
To simplify the installation of Boost, Protocol buffers, OpenSSL and Zlib dependencies we can use
571+
vcpkg (https://github.com/Microsoft/vcpkg.git). Upon cloning the vcpkg repo, checkout the commit
572+
7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d to get the required versions of the dependencies
573+
mentioned above.
574+
> git clone https://github.com/Microsoft/vcpkg.git
575+
> cd vcpkg
576+
> git checkout 7ffa425e1db8b0c3edf9c50f2f3a0f25a324541d
577+
> .\bootstrap-vcpkg.bat
578+
> .\vcpkg.exe install boost:x64-windows
579+
> .\vcpkg.exe install protobuf:x64-windows
580+
> .\vcpkg.exe install openssl:x64-windows
581+
> .\vcpkg.exe install zlib:x64-windows
582+
583+
Set the following environment variables -
584+
(Assuming that vcpkg was checked out at C:\vcpkg)
585+
> set PROTOBUF_HOME=C:\vcpkg\installed\x64-windows
586+
> set MAVEN_OPTS=-Xmx2048M -Xss128M
587+
543588
All Maven goals are the same as described above with the exception that
544589
native code is built by enabling the 'native-win' Maven profile. -Pnative-win
545590
is enabled by default when building on Windows since the native components
@@ -557,6 +602,24 @@ the zlib 1.2.7 source tree.
557602

558603
http://www.zlib.net/
559604

605+
606+
Build command:
607+
The following command builds all the modules in the Hadoop project and generates the tar.gz file in
608+
hadoop-dist/target upon successful build. Run these commands from an
609+
"x64 Native Tools Command Prompt for VS 2019" which can be found under "Visual Studio 2019" in the
610+
Windows start menu. If you're using the Docker image from Dockerfile_windows_10, you'll be
611+
logged into "x64 Native Tools Command Prompt for VS 2019" automatically when you start the
612+
container.
613+
614+
> set classpath=
615+
> set PROTOBUF_HOME=C:\vcpkg\installed\x64-windows
616+
> mvn clean package -Dhttps.protocols=TLSv1.2 -DskipTests -DskipDocs -Pnative-win,dist^
617+
-Drequire.openssl -Drequire.test.libhadoop -Pyarn-ui -Dshell-executable=C:\Git\bin\bash.exe^
618+
-Dtar -Dopenssl.prefix=C:\vcpkg\installed\x64-windows^
619+
-Dcmake.prefix.path=C:\vcpkg\installed\x64-windows^
620+
-Dwindows.cmake.toolchain.file=C:\vcpkg\scripts\buildsystems\vcpkg.cmake -Dwindows.cmake.build.type=RelWithDebInfo^
621+
-Dwindows.build.hdfspp.dll=off -Dwindows.no.sasl=on -Duse.platformToolsetVersion=v142
622+
560623
----------------------------------------------------------------------------------
561624
Building distributions:
562625

0 commit comments

Comments
 (0)