From d28a2c6b4ad0774f9ab2a10cf256a3c55a565a6c Mon Sep 17 00:00:00 2001 From: Jialiang Date: Fri, 21 Mar 2025 13:44:20 +0800 Subject: [PATCH] AMBARI-26340: Update build Docs for Ambari 3.0 --- docs/ambari-dev/bigtop-guide.md | 154 ++++++++++++++ docs/ambari-dev/building-from-source.md | 126 +++++++++++ docs/ambari-dev/how-to-contribute.md | 22 +- docs/ambari-dev/running-tests.md | 109 ++++++++++ docs/quick-start/installation-guide.md | 48 ++++- docs/release-notes.md | 149 +++++++------ sidebars.js | 3 + src/pages/team/languages.json | 80 ++++--- .../version-3.0.0/ambari-dev/bigtop-guide.md | 154 ++++++++++++++ .../ambari-dev/building-from-source.md | 125 +++++++++++ .../ambari-dev/how-to-contribute.md | 17 +- .../version-3.0.0/ambari-dev/running-tests.md | 109 ++++++++++ .../quick-start/installation-guide.md | 48 ++++- versioned_docs/version-3.0.0/release-notes.md | 199 +++++++++++++----- .../version-3.0.0-sidebars.json | 5 + 15 files changed, 1177 insertions(+), 171 deletions(-) create mode 100644 docs/ambari-dev/bigtop-guide.md create mode 100644 docs/ambari-dev/building-from-source.md create mode 100644 docs/ambari-dev/running-tests.md create mode 100644 versioned_docs/version-3.0.0/ambari-dev/bigtop-guide.md create mode 100644 versioned_docs/version-3.0.0/ambari-dev/building-from-source.md create mode 100644 versioned_docs/version-3.0.0/ambari-dev/running-tests.md diff --git a/docs/ambari-dev/bigtop-guide.md b/docs/ambari-dev/bigtop-guide.md new file mode 100644 index 00000000..875468a2 --- /dev/null +++ b/docs/ambari-dev/bigtop-guide.md @@ -0,0 +1,154 @@ +--- +title: Compiling Components for Ambari Bigtop Stack +--- + +# Compiling Components for Ambari Bigtop Stack + +## Introduction + +Apache Bigtop is designed for infrastructure engineers and data scientists seeking comprehensive packaging, testing, and configuration of leading open-source big data components. Bigtop supports a wide range of components/projects, including but not limited to Hadoop, HBase, and Spark. This guide specifically focuses on how to compile components for the **Ambari Bigtop Stack**. + +## Use Cases for Apache Bigtop + +1. **Simplified Package Building**: Bigtop significantly simplifies the process of compiling RPM or DEB packages for big data components across different operating systems through pre-configured Docker images, making it quick and efficient. + +2. **Dependency Management**: Bigtop integrates complex dependencies required during the compilation process, effectively resolving common compilation errors and ensuring a smooth compilation experience through patches in the code. This means users no longer need to worry about official packages failing to compile or setting up complex compilation environments. + +3. **Apache Ambari Support**: Bigtop provides support for Apache Ambari, allowing users to easily package big data software that is compatible with Ambari and meets installation requirements. + +## Getting Started with Bigtop + +This guide uses the official Bigtop 3.3.0 as an example, with CentOS 7 as the compilation operating system. The same operations apply to other systems and versions. + +### Prerequisites + +- Linux environment +- Docker installed on your system +- Git + +### Step-by-Step Guide + +#### 1. Create a Development Directory + +```bash +mkdir ~/dev/ +``` + +#### 2. Clone Bigtop Repository + +```bash +cd ~/dev/ +git clone https://github.com/apache/bigtop.git +``` + +#### 3. Switch to Version 3.3.0 + +```bash +cd bigtop +git checkout release-3.3.0 +``` + +#### 4. Pull the Bigtop CentOS 7 Compilation Environment Image + +```bash +# If you need to compile for other operating systems or architectures (e.g., ARM), +# you can search for the corresponding Bigtop version in the image repository +# https://hub.docker.com/r/bigtop/slaves/tags +docker pull bigtop/slaves:3.3.0-centos-7 +``` + +#### 5. Launch the Container + +**Scenario 1**: If you've previously compiled big data components locally and have a Maven repository cache, it's best to map this directory to the container's default Maven download directory to avoid downloading packages again. + +For example, if your local Maven repository directory is `/data/repository`: + +```bash +cd ~/dev/bigtop +docker run -d -it --network host -v `pwd`:/ws -v /data/repository:/root/.m2/repository --workdir /ws --name bigtopr bigtop/slaves:3.3.0-centos-7 +``` + +**Scenario 2**: If you don't have a Maven cache locally or are unfamiliar with this, you should still map a directory to the Bigtop container to facilitate repeated compilations using downloaded Maven cache. Otherwise, when the container is deleted, your Maven cache will be lost, and dependency downloading is the most time-consuming stage of recompilation. + +```bash +mkdir -p ~/m2/repository +cd ~/dev/bigtop +docker run -d -it --network host -v `pwd`:/ws -v ~/m2/repository:/root/.m2/repository --workdir /ws --name bigtopr bigtop/slaves:3.3.0-centos-7 +``` + +#### 6. Modify Maven Repository Settings (Optional) + +You can configure Maven to use mirrors that are faster for your location. This step is optional but can significantly improve download speeds. + +1. Enter the container: +```bash +docker exec -it bigtopr /bin/bash +``` + +2. Edit the Maven settings file: +```bash +vi /usr/local/maven/conf/settings.xml +``` + +3. Add appropriate mirror repositories based on your location. For example: + +```xml + + + central-mirror + central + Central Repository Mirror + https://repo1.maven.org/maven2/ + + + +``` + +#### 7. Compile Big Data Components + +Enter your running container: + +```bash +docker exec -it bigtopr /bin/bash +``` + +Compile components: + +```bash +. /etc/profile.d/bigtop.sh +./gradlew flink-clean flink-pkg -PparentDir=/usr/bigtop -PpkgSuffix -PbuildThreads=2C repo +``` + +**Explanation of compilation parameters**: + +- `-PparentDir=/usr/bigtop`: Changes the default installation path of the package, making Bigtop-built packages conform to Ambari installation specifications. +- `-PpkgSuffix`: Makes the output package include the Bigtop version number (e.g., hadoop_3_3_0), conforming to Ambari Bigtop service specifications. +- `-PbuildThreads=2C`: Sets the number of threads for compilation (2 times the number of CPU cores). + +## Parallel Compilation for Improved Performance + +A pull request for parallel compilation to speed up the build process has been submitted to the community and is currently under review. Once merged, all Java components in Bigtop will be able to compile in parallel, expected to be available in versions after Bigtop 3.3.1. + +Performance comparison for parallel compilation (after all dependencies are downloaded): + +| Component | Time Before | Time After | +|------------|-------------|------------| +| Alluxio | 21min | 07:43min | +| Hive | 05:33min | 03:04min | +| HBase | 06:18min | 02:55min | +| Zookeeper | 01:25min | 35s | +| Livy | 03:29min | 03:12min | +| Phoenix | 11:23min | 05:32min | +| Zeppelin | 14:15min | 13:19min | +| Flink | 36:27min | 14:16min | +| Hadoop | 50min | 16min | + +Example of parallel compilation command: + +```bash +docker run -d -it --network host -v `pwd`:/ws -v /data/repository:/data/repository --workdir /ws --name bigtop bigtop/slaves:trunk-centos-7 --cpus 16 +source /etc/profile.d/bigtop.sh +./gradlew alluxio-clean alluxio-pkg -PcompileThreads=2C +``` + +This approach shows a 2-3x improvement in compilation speed, with even more significant effects during initial compilation (e.g., Hadoop initial compilation time reduced from 3 hours to 1 hour). diff --git a/docs/ambari-dev/building-from-source.md b/docs/ambari-dev/building-from-source.md new file mode 100644 index 00000000..d8f7b611 --- /dev/null +++ b/docs/ambari-dev/building-from-source.md @@ -0,0 +1,126 @@ +--- +title: Building from Source +--- + +# Building Apache Ambari from Source + +This guide explains how to build Apache Ambari 3.0 and its related subprojects from source code. + +## Prerequisites + +Before you begin, ensure you have the following requirements installed: + +### System Requirements +- Operating System: Rocky Linux 8 or 9 (recommended) +- Python 3 Development Tools (`python3-devel`) + +### Java Requirements +- Ambari Main Project: JDK 17 +- Ambari Metrics: JDK 8 +- Ambari Infra: JDK 8 + +## Building Ambari Main Project + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari.git +cd ambari +``` + +### 2. Build Options + +#### Build Without RPM +To build Ambari without creating RPM packages: +```bash +mvn -B -T 2C clean install package \ + -Drat.skip=true \ + -DskipTests \ + -Dmaven.test.skip=true \ + -Dfindbugs.skip=true \ + -Dcheckstyle.skip=true +``` + +#### Build With RPM +To build Ambari and create RPM packages: +```bash +mvn -B -T 2C clean install package rpm:rpm \ + -Drat.skip=true \ + -DskipTests \ + -Dmaven.test.skip=true \ + -Dfindbugs.skip=true \ + -Dcheckstyle.skip=true +``` + +The RPM packages will be generated at: +- Ambari Agent: `ambari/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-3.0.0.0-SNAPSHOT.x86_64.rpm` +- Ambari Server: `ambari/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-3.0.0.0-SNAPSHOT.x86_64.rpm` + +## Building Ambari Metrics + +:::tip Performance Optimization +To significantly improve build performance, download binary dependencies locally before building: + +1. Create a local directory for dependencies: +```bash +mkdir -p /ws/dl/ +``` + +2. Download the required binary files: +```bash +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/hbase-2.4.13-bin.tar.gz +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/hadoop-3.3.4.tar.gz +wget -P /ws/dl/ https://dl.grafana.com/oss/release/grafana-11.1.4.linux-amd64.tar.gz +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/phoenix-hbase-2.4-5.1.2-bin.tar.gz +``` + +3. Modify the `pom.xml` in ambari-metrics project to use local files: +```xml + + + file:///ws/dl/hbase-2.4.13-bin.tar.gz + file:///ws/dl/hadoop-3.3.4.tar.gz + file:///ws/dl/grafana-11.1.4.linux-amd64.tar.gz + file:///ws/dl/phoenix-hbase-2.4-5.1.2-bin.tar.gz + +``` + +This optimization will save significant time during repeated builds by avoiding large downloads. +::: + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari-metrics.git +cd ambari-metrics +``` + +### 2. Build Options + +#### Build Without RPM +To build Ambari Metrics without creating RPM packages: +```bash +mvn -T 2C clean install -DskipTests +``` + +#### Build With RPM +To build Ambari Metrics and create RPM packages: +```bash +mvn -T 2C clean install -DskipTests -Dbuild-rpm +``` + +To locate the generated RPM packages: +```bash +find ./ -name "*.rpm" +``` + +## Building Ambari Infra + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari-infra.git +cd ambari-infra +``` + +### 2. Build RPM Package +```bash +make rpm +``` diff --git a/docs/ambari-dev/how-to-contribute.md b/docs/ambari-dev/how-to-contribute.md index 1e44e8c5..23897ec2 100644 --- a/docs/ambari-dev/how-to-contribute.md +++ b/docs/ambari-dev/how-to-contribute.md @@ -36,11 +36,21 @@ Repeat these steps for all the branches that needs to be synced with the remote. Apache Ambari uses JIRA to track issues including bugs and improvements, and uses Github pull requests to manage code reviews and code merges. Major design changes are discussed in JIRA and implementation changes are discussed in pull requests after a pull request is created. +:::note Important Changes to JIRA Registration +* JIRA registration is currently closed to the public +* To get a JIRA account: + 1. Register on [Apache JIRA](https://issues.apache.org/jira) + 2. Contact a PMC member to approve your registration +* Alternatively, you can: + 1. Submit your Pull Request first + 2. Community members will help create the corresponding JIRA ticket for you +::: + * Find an existing Apache JIRA that the change pertains to * Do not create a new JIRA if the change is minor and relates to an existing JIRA; add to the existing discussion and work instead * Look for existing pull requests that are linked from the JIRA, to understand if someone is already working on the JIRA -* If the change is new, then create a new JIRA: +* If the change is new and you have JIRA access, then create a new JIRA: * Provide a descriptive Title * Write a detailed Description. For bug reports, this should ideally include a short reproduction of the problem. For new features, it may include a design document. * Fill the required fields: @@ -49,11 +59,11 @@ Apache Ambari uses JIRA to track issues including bugs and improvements, and use * Blocker: pointless to release without this change as the release would be unusable to a large minority of users * Critical: a large minority of users are missing important functionality without this, and/or a workaround is difficult * Major: a small minority of users are missing important functionality without this, and there is a workaround - * Minor: a niche use case is missing some support, but it does not affect usage or is easily worked around - * Trivial: a nice-to-have change but unlikely to be any problem in practice otherwise - * Component. Choose the components that are affected by this change. Choose from Ambari Components - * Affects Version. For Bugs, assign at least one version that is known to exhibit the problem or need the change - * Do not include a patch file; pull requests are used to propose the actual change. + +* If you don't have JIRA access: + * Submit your Pull Request first + * In the PR description, clearly describe the issue or improvement + * A community member will create a JIRA ticket and link it to your PR ### Pull Request diff --git a/docs/ambari-dev/running-tests.md b/docs/ambari-dev/running-tests.md new file mode 100644 index 00000000..310988a7 --- /dev/null +++ b/docs/ambari-dev/running-tests.md @@ -0,0 +1,109 @@ +--- +title: Running Tests +--- + +# Running Tests in Apache Ambari + +This guide explains how to run different types of tests in Apache Ambari. For the official test configuration, you can refer to the [Ambari Jenkinsfile](https://github.com/apache/ambari/blob/trunk/Jenkinsfile). + +## Java Tests + +### Running All Java Tests +To run all Java tests for the Ambari Server: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true +``` + +### Running Specific Java Tests +To run a specific Java test class: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true \ + -Dtest=AmbariServerTest +``` + +To run a specific test method within a test class: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true \ + -Dtest=AmbariServerTest#testMethodName +``` + +### Test Parameters Explained +- `-am`: Also build dependencies +- `-pl ambari-server`: Only build the ambari-server module +- `-DskipPythonTests`: Skip Python tests +- `-Dmaven.test.failure.ignore`: Continue the build even if tests fail +- `-Dmaven.artifact.threads=10`: Use 10 threads for parallel artifact resolution +- `-Drat.skip`: Skip Apache RAT (Release Audit Tool) checks +- `-DskipAdminWebTests`: Skip admin web interface tests +- `-Dtest`: Specify which test class or method to run + +## Python Tests + +### Running All Python Tests +To run all Python tests: +```bash +mvn test -pl ambari-server \ + -DskipJavaTests \ + -Dpython.test.mask="*_test.py" \ + -Dpython.test.skip.pattern="agent_perf.py" +``` + +### Running Specific Python Tests +To run a specific Python test file: +```bash +mvn test -pl ambari-server \ + -DskipJavaTests \ + -Dpython.test.mask="test_file_name.py" +``` + +### Python Test Parameters Explained +- `-DskipJavaTests`: Skip Java tests +- `-Dpython.test.mask`: Pattern to match test files to run +- `-Dpython.test.skip.pattern`: Pattern to match test files to skip + +## Integration Tests + +### Running Integration Tests +To run integration tests: +```bash +mvn verify -pl ambari-server \ + -P integration-tests \ + -DskipPythonTests \ + -DskipJavaTests +``` + +### Integration Test Parameters +- `-P integration-tests`: Activate the integration-tests profile +- `-DskipPythonTests`: Skip Python tests +- `-DskipJavaTests`: Skip Java tests + +## Test Reports + +Test reports can be found in the following locations after test execution: + +### Java Test Reports +- Unit Tests: `ambari-server/target/surefire-reports/` +- Integration Tests: `ambari-server/target/failsafe-reports/` + +### Python Test Reports +- Test Results: `ambari-server/target/python-test-results/` +- Coverage Reports: `ambari-server/target/python-coverage/` + +:::tip +When debugging test failures, check these report directories for detailed test execution logs and stack traces. +::: diff --git a/docs/quick-start/installation-guide.md b/docs/quick-start/installation-guide.md index db52d1d8..485a50eb 100644 --- a/docs/quick-start/installation-guide.md +++ b/docs/quick-start/installation-guide.md @@ -2,6 +2,42 @@ This guide covers the installation and setup of Apache Ambari on bare metal, KVM, Docker, or Vagrant environments. +## Important: Firewall Configuration + +:::tip Before You Begin +Firewall settings can significantly impact cluster functionality by blocking necessary communication ports between components. Please review the following guidelines carefully. +::: + +### Development & Testing Environments +For development or testing environments, consider disabling the firewall: +```bash +systemctl stop firewalld +systemctl disable firewalld +``` + +### Production Environments + +#### For System Administrators +Configure the firewall to allow required Hadoop ecosystem ports based on your specific component deployment: + +| Component | Ports | Purpose | +|-----------|-------|---------| +| Ambari Server | 8080, 8440, 8441 | Web UI, Agent communication | +| Core Hadoop | 8020, 9000, 50070, 50075 | HDFS NameNode, DataNode HTTP | +| YARN | 8032, 8088, 19888 | ResourceManager, UI, JobHistory | +| Hive | 9083, 10000 | Metastore, HiveServer2 | + +:::note NodeManager Ports +YARN NodeManagers allocate containers on dynamically assigned ports (default range: 32768-65535). +This range can be restricted via `yarn.nodemanager.resource.ports` in `yarn-site.xml` +::: + +:::info For General Users +If you're not familiar with advanced network configuration, we recommend: +- Disabling the firewall during initial setup and cluster testing +- Consulting with a network security specialist for production deployments +::: + ## Prerequisites Ensure you have a working environment (bare metal, KVM, Docker, or [Vagrant setup](./environment-setup/vagrant-environment-setup.md)) before proceeding. @@ -204,16 +240,10 @@ Default credentials: ## Troubleshooting -1. If you encounter firewall issues, disable it (for development environments only): -```bash -systemctl stop firewalld -systemctl disable firewalld -``` - -2. Ensure proper hostname resolution by configuring `/etc/hosts` on all nodes. +1. Ensure proper hostname resolution by configuring `/etc/hosts` on all nodes. -3. For MySQL 8 connection issues, verify the JDBC URL includes the correct SSL parameters in `ambari.properties`. +2. For MySQL 8 connection issues, verify the JDBC URL includes the correct SSL parameters in `ambari.properties`. -4. Check service logs: +3. Check service logs: - Ambari Server: `/var/log/ambari-server/ambari-server.log` - Ambari Agent: `/var/log/ambari-agent/ambari-agent.log` diff --git a/docs/release-notes.md b/docs/release-notes.md index 25f84e96..fdc33900 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -6,10 +6,22 @@ sidebar_position: 2 ## Overview +:::tip Major Release Apache Ambari 3.0.0 represents a significant milestone in the project's development, bringing major improvements to cluster management capabilities, user experience, and platform support. This release focuses on modernizing the technology stack, enhancing security features, and improving overall stability and performance. +::: ## Release Highlights +### Major Improvements + +- **Apache Bigtop Integration**: + - Ambari 3.0.0 now uses Apache Bigtop as the default packaging system for component management. + - Bigtop Stack serves as the default project stack, providing a more sustainable and community-driven approach to package management. + +:::tip Bigtop Component Compilation +For developers interested in compiling components for the Ambari Bigtop Stack, we have prepared a **[detailed guide](ambari-dev/bigtop-guide.md)** that walks through the process step by step. +::: + ### Platform Support - Added support for Rocky Linux 8 and 9 ([AMBARI-26133](https://issues.apache.org/jira/browse/AMBARI-26133)) @@ -147,6 +159,7 @@ Apache Ambari 3.0.0 represents a significant milestone in the project's developm - Fixed issue with 'supported-refresh-commands' element ([AMBARI-25863](https://issues.apache.org/jira/browse/AMBARI-25863)) - Reduced excess Zookeeper and Hadoop logging ([AMBARI-24140](https://issues.apache.org/jira/browse/AMBARI-24140)) + - **Bug Fixes**: - Fixed OceanBase support in Ambari MySQL DDL ([AMBARI-26273](https://issues.apache.org/jira/browse/AMBARI-26273)) - Fixed regex pattern flag position in ambari_jinja2 filters ([AMBARI-26269](https://issues.apache.org/jira/browse/AMBARI-26269)) @@ -166,6 +179,77 @@ Apache Ambari 3.0.0 represents a significant milestone in the project's developm - Fixed alter dispatcher ([AMBARI-26240](https://issues.apache.org/jira/browse/AMBARI-26240)) - Corrected spelling mistakes in documentation ([AMBARI-26104](https://issues.apache.org/jira/browse/AMBARI-26104)) +## Acknowledgments + +:::tip Our Amazing Contributors 🌟 +Apache Ambari 3.0.0 represents the collective effort of our incredible community. We are deeply honored to recognize the individuals who have shaped this milestone release: + +
+ +### Core Contributors 💫 +These dedicated individuals have made substantial contributions to this release: + +- **jialiang** 🏆 +- **zrain** 🏆 +- **Peng Lu** 🏆 +- **Mohammad Arshad** 🏆 +- **Sandeep Kumar** 🏆 +- **coldless177** 🏆 +- **Vishal Suvagia** 🏆 +- **zhenye zhang** 🏆 +- **yaolei** 🏆 +- **xjmu** 🏆 +- **Himanshu Maurya** 🏆 +- **timyuer** 🏆 +- **yaruyng** 🏆 +- **tongxiaojun** 🏆 + +### Community Champions 🌠 +Their valuable contributions have helped drive the project forward: + +- **Ananya Singh** +- **rzuo** +- **Viraj Jasani** +- **William Horn** +- **vanshuhassija** +- **Yu Hou** +- **basapuram-kumar** +- **Prabhjyot Singh** +- **Brahma Reddy Battula** +- **Basapuram Kumar** +- **Shreeya Sand** +- **Bhavik Patel** + +### Technical Innovators ⚡ +Their technical expertise has been instrumental: + +- **lupeng** +- **piaolingzxh** +- **Murali Krishna** +- **LiJie20190102** +- **HARSHITH GANDHE** +- **userhimanshuverma** +- **Arnout Engelen** +- **wangda** +- **Dmytro Sen** +- **Rich Bowen** +- **Shubham Sharma** +- **Weijian Wen** +- **Will Guo** +- **guluo** +- **Peng Lee** +- **liqinwyyx** + +
+ +:::note Special Recognition 🎖 +A special note of appreciation goes to our Release Manager(s), whose tireless coordination and attention to detail have been crucial in bringing this release to fruition. +::: + +:::tip Join Our Community! 🤝 +Every contribution, no matter how small, helps make Ambari better. We welcome new contributors to join our vibrant community! +::: + ## Known Issues - Timeline Service Reader may fail to start if HBase is not installed ([AMBARI-26248](https://issues.apache.org/jira/browse/AMBARI-26248)) @@ -230,68 +314,5 @@ sha256sum --check apache-ambari-3.0.0.tar.gz.sha256 # Verify the PGP signature gpg --verify apache-ambari-3.0.0.tar.gz.asc apache-ambari-3.0.0.tar.gz -``` - -The PGP signatures can be verified using the public keys of the Apache Ambari Release Managers, which can be found on the [Apache Ambari Download Page](https://ambari.apache.org/download.html). - -## Acknowledgments -Apache Ambari 3.0.0 is the result of significant contributions from the Apache Ambari community. We would like to thank all the contributors who have helped make this release possible: - -- jialiang -- zrain -- Peng Lu -- Mohammad Arshad -- yaruyng -- tongxiaojun -- Sandeep Kumar -- coldless177 -- Vishal Suvagia -- zhenye zhang -- yaolei -- xjmu -- Himanshu Maurya -- timyuer -- Ananya Singh -- rzuo -- Viraj Jasani -- William Horn -- vanshuhassija -- Yu Hou -- basapuram-kumar -- Prabhjyot Singh -- Brahma Reddy Battula -- Basapuram Kumar -- Shreeya Sand -- Bhavik Patel -- lupeng -- piaolingzxh -- Murali Krishna -- LiJie20190102 -- HARSHITH GANDHE -- userhimanshuverma -- Arnout Engelen -- wangda -- Dmytro Sen -- Rich Bowen -- Shubham Sharma -- Weijian Wen -- Will Guo -- guluo -- Peng Lee -- liqinwyyx -- and many others - -Special thanks to our Release Manager(s) for their dedication and hard work in preparing this release. - -## Community - -We welcome your feedback and contributions to Apache Ambari: - -- [Mailing Lists](https://ambari.apache.org/mail-lists.html) -- [Issue Tracker](https://issues.apache.org/jira/projects/AMBARI) -- [GitHub Repository](https://github.com/apache/ambari) - -## License - -Apache Ambari is released under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). +``` \ No newline at end of file diff --git a/sidebars.js b/sidebars.js index c571cbac..7a9c9b2b 100644 --- a/sidebars.js +++ b/sidebars.js @@ -130,12 +130,15 @@ const sidebars = { collapsed: false, link: {type: 'doc', id: 'ambari-dev/index'}, items: [ + "ambari-dev/building-from-source", + "ambari-dev/running-tests", "ambari-dev/development-process-for-new-major-features", "ambari-dev/ambari-code-layout", "ambari-dev/apache-ambari-jira", "ambari-dev/coding-guidelines-for-ambari", "ambari-dev/how-to-commit", "ambari-dev/how-to-contribute", + "ambari-dev/bigtop-guide", "ambari-dev/code-review-guidelines", "ambari-dev/releasing-ambari", "ambari-dev/admin-view-ambari-admin-development", diff --git a/src/pages/team/languages.json b/src/pages/team/languages.json index 0858890f..1c0eef9e 100644 --- a/src/pages/team/languages.json +++ b/src/pages/team/languages.json @@ -5,13 +5,20 @@ "tip": "(In no particular order)" }, "pmc": [ - + { + "apacheId": "arshad", + "avatarUrl": "https://avatars.githubusercontent.com/u/3433835?v=4", + "email": "arshad@apache.org", + "gitUrl": "https://github.com/arshadmohammad", + "githubId": "arshadmohammad", + "name": "Mohammad Arshad" + }, { "apacheId": "atri", "avatarUrl": "https://avatars.githubusercontent.com/u/1724131?v=4", "email": "atri@apache.org", "gitUrl": "https://github.com/atris", - "githubId": "atri", + "githubId": "atris", "name": "Atri Sharma" }, { @@ -27,7 +34,7 @@ "avatarUrl": "https://avatars.githubusercontent.com/u/938524?v=4", "email": "ddas@hortonworks.com", "gitUrl": "https://github.com/ddraj", - "githubId": "ddas", + "githubId": "ddraj", "name": "Devaraj Das" }, { @@ -53,8 +60,7 @@ "gitUrl": "https://github.com/iwasakims", "githubId": "iwasakims", "name": "Masatake Iwasaki" - } - , + }, { "apacheId": "junhe", "avatarUrl": "https://avatars.githubusercontent.com/u/17865359?v=4", @@ -62,19 +68,15 @@ "gitUrl": "https://github.com/JunHe77", "githubId": "JunHe77", "name": "Jun He" - } - - , + }, { "apacheId": "masatana", "avatarUrl": "https://avatars.githubusercontent.com/u/240015?v=4", "email": "masatana@apache.org", - "gitUrl": "https://github.com/masahirotanaka", - "githubId": "masahirotanaka", + "gitUrl": "https://github.com/masatana", + "githubId": "masatana", "name": "Masahiro Tanaka" - } - - , + }, { "apacheId": "mithmatt", "avatarUrl": "https://avatars.githubusercontent.com/u/5906984?v=4", @@ -82,9 +84,7 @@ "gitUrl": "https://github.com/mithmatt", "githubId": "mithmatt", "name": "Matt" - } - - , + }, { "apacheId": "rvs", "avatarUrl": "https://avatars.githubusercontent.com/u/34680?v=4", @@ -100,8 +100,7 @@ "gitUrl": "https://github.com/sekikn", "githubId": "sekikn", "name": "Kengo Seki" - } - , + }, { "apacheId": "umamahesh", "avatarUrl": "https://avatars.githubusercontent.com/u/1719507?v=4", @@ -109,8 +108,7 @@ "gitUrl": "https://github.com/umamaheswararao", "githubId": "umamaheswararao", "name": "Uma Maheswara Rao G" - } - , + }, { "apacheId": "vgogate", "avatarUrl": "https://avatars.githubusercontent.com/u/913821?v=4", @@ -118,8 +116,7 @@ "gitUrl": "https://github.com/vgogate", "githubId": "vgogate", "name": "Suhas" - } - , + }, { "apacheId": "vishalsuvagia", "avatarUrl": "https://avatars.githubusercontent.com/u/7812238?v=4", @@ -127,8 +124,7 @@ "gitUrl": "https://github.com/vishalsuvagia", "githubId": "vishalsuvagia", "name": "Vishal Suvagia" - } - , + }, { "apacheId": "weichiu", "avatarUrl": "https://avatars.githubusercontent.com/u/2691807?v=4", @@ -144,9 +140,33 @@ "gitUrl": "https://github.com/kevinw66", "githubId": "kevinw66", "name": "Zhiguo Wu" + }, + { + "apacheId": "vjasani", + "avatarUrl": "https://avatars.githubusercontent.com/u/34790606?v=4", + "email": "vjasani@apache.org", + "gitUrl": "https://github.com/virajjasani", + "githubId": "virajjasani", + "name": "Viraj Jasani" } ], "committer": [ + { + "apacheId": "jialiang", + "avatarUrl": "https://avatars.githubusercontent.com/u/18082602?v=4", + "email": "jialiang@apache.org", + "gitUrl": "https://github.com/Jialiangc", + "githubId": "Jialiangc", + "name": "Jialiang Cai" + }, + { + "apacheId": "zhangyu", + "avatarUrl": "https://avatars.githubusercontent.com/u/16508606?v=4", + "email": "zhangyu@apache.org", + "gitUrl": "https://github.com/zRains", + "githubId": "zRains", + "name": "Yu Zhang" + }, { "apacheId": "yaolei", "avatarUrl": "https://avatars.githubusercontent.com/u/13212217?v=4", @@ -156,12 +176,12 @@ "name": "Lei Yao" }, { - "apacheId": "vjasani", - "avatarUrl": "https://avatars.githubusercontent.com/u/34790606?v=4", - "email": "vjasani@apache.org", - "gitUrl": "https://github.com/virajjasani", - "githubId": "virajjasani", - "name": "Viraj Jasani" + "apacheId": "houyu", + "avatarUrl": "https://avatars.githubusercontent.com/u/16263438?v=4 ", + "email": "houyu@apache.org", + "gitUrl": "https://github.com/timyuer", + "githubId": "timyuer", + "name": "Yu Hou" } ] } diff --git a/versioned_docs/version-3.0.0/ambari-dev/bigtop-guide.md b/versioned_docs/version-3.0.0/ambari-dev/bigtop-guide.md new file mode 100644 index 00000000..875468a2 --- /dev/null +++ b/versioned_docs/version-3.0.0/ambari-dev/bigtop-guide.md @@ -0,0 +1,154 @@ +--- +title: Compiling Components for Ambari Bigtop Stack +--- + +# Compiling Components for Ambari Bigtop Stack + +## Introduction + +Apache Bigtop is designed for infrastructure engineers and data scientists seeking comprehensive packaging, testing, and configuration of leading open-source big data components. Bigtop supports a wide range of components/projects, including but not limited to Hadoop, HBase, and Spark. This guide specifically focuses on how to compile components for the **Ambari Bigtop Stack**. + +## Use Cases for Apache Bigtop + +1. **Simplified Package Building**: Bigtop significantly simplifies the process of compiling RPM or DEB packages for big data components across different operating systems through pre-configured Docker images, making it quick and efficient. + +2. **Dependency Management**: Bigtop integrates complex dependencies required during the compilation process, effectively resolving common compilation errors and ensuring a smooth compilation experience through patches in the code. This means users no longer need to worry about official packages failing to compile or setting up complex compilation environments. + +3. **Apache Ambari Support**: Bigtop provides support for Apache Ambari, allowing users to easily package big data software that is compatible with Ambari and meets installation requirements. + +## Getting Started with Bigtop + +This guide uses the official Bigtop 3.3.0 as an example, with CentOS 7 as the compilation operating system. The same operations apply to other systems and versions. + +### Prerequisites + +- Linux environment +- Docker installed on your system +- Git + +### Step-by-Step Guide + +#### 1. Create a Development Directory + +```bash +mkdir ~/dev/ +``` + +#### 2. Clone Bigtop Repository + +```bash +cd ~/dev/ +git clone https://github.com/apache/bigtop.git +``` + +#### 3. Switch to Version 3.3.0 + +```bash +cd bigtop +git checkout release-3.3.0 +``` + +#### 4. Pull the Bigtop CentOS 7 Compilation Environment Image + +```bash +# If you need to compile for other operating systems or architectures (e.g., ARM), +# you can search for the corresponding Bigtop version in the image repository +# https://hub.docker.com/r/bigtop/slaves/tags +docker pull bigtop/slaves:3.3.0-centos-7 +``` + +#### 5. Launch the Container + +**Scenario 1**: If you've previously compiled big data components locally and have a Maven repository cache, it's best to map this directory to the container's default Maven download directory to avoid downloading packages again. + +For example, if your local Maven repository directory is `/data/repository`: + +```bash +cd ~/dev/bigtop +docker run -d -it --network host -v `pwd`:/ws -v /data/repository:/root/.m2/repository --workdir /ws --name bigtopr bigtop/slaves:3.3.0-centos-7 +``` + +**Scenario 2**: If you don't have a Maven cache locally or are unfamiliar with this, you should still map a directory to the Bigtop container to facilitate repeated compilations using downloaded Maven cache. Otherwise, when the container is deleted, your Maven cache will be lost, and dependency downloading is the most time-consuming stage of recompilation. + +```bash +mkdir -p ~/m2/repository +cd ~/dev/bigtop +docker run -d -it --network host -v `pwd`:/ws -v ~/m2/repository:/root/.m2/repository --workdir /ws --name bigtopr bigtop/slaves:3.3.0-centos-7 +``` + +#### 6. Modify Maven Repository Settings (Optional) + +You can configure Maven to use mirrors that are faster for your location. This step is optional but can significantly improve download speeds. + +1. Enter the container: +```bash +docker exec -it bigtopr /bin/bash +``` + +2. Edit the Maven settings file: +```bash +vi /usr/local/maven/conf/settings.xml +``` + +3. Add appropriate mirror repositories based on your location. For example: + +```xml + + + central-mirror + central + Central Repository Mirror + https://repo1.maven.org/maven2/ + + + +``` + +#### 7. Compile Big Data Components + +Enter your running container: + +```bash +docker exec -it bigtopr /bin/bash +``` + +Compile components: + +```bash +. /etc/profile.d/bigtop.sh +./gradlew flink-clean flink-pkg -PparentDir=/usr/bigtop -PpkgSuffix -PbuildThreads=2C repo +``` + +**Explanation of compilation parameters**: + +- `-PparentDir=/usr/bigtop`: Changes the default installation path of the package, making Bigtop-built packages conform to Ambari installation specifications. +- `-PpkgSuffix`: Makes the output package include the Bigtop version number (e.g., hadoop_3_3_0), conforming to Ambari Bigtop service specifications. +- `-PbuildThreads=2C`: Sets the number of threads for compilation (2 times the number of CPU cores). + +## Parallel Compilation for Improved Performance + +A pull request for parallel compilation to speed up the build process has been submitted to the community and is currently under review. Once merged, all Java components in Bigtop will be able to compile in parallel, expected to be available in versions after Bigtop 3.3.1. + +Performance comparison for parallel compilation (after all dependencies are downloaded): + +| Component | Time Before | Time After | +|------------|-------------|------------| +| Alluxio | 21min | 07:43min | +| Hive | 05:33min | 03:04min | +| HBase | 06:18min | 02:55min | +| Zookeeper | 01:25min | 35s | +| Livy | 03:29min | 03:12min | +| Phoenix | 11:23min | 05:32min | +| Zeppelin | 14:15min | 13:19min | +| Flink | 36:27min | 14:16min | +| Hadoop | 50min | 16min | + +Example of parallel compilation command: + +```bash +docker run -d -it --network host -v `pwd`:/ws -v /data/repository:/data/repository --workdir /ws --name bigtop bigtop/slaves:trunk-centos-7 --cpus 16 +source /etc/profile.d/bigtop.sh +./gradlew alluxio-clean alluxio-pkg -PcompileThreads=2C +``` + +This approach shows a 2-3x improvement in compilation speed, with even more significant effects during initial compilation (e.g., Hadoop initial compilation time reduced from 3 hours to 1 hour). diff --git a/versioned_docs/version-3.0.0/ambari-dev/building-from-source.md b/versioned_docs/version-3.0.0/ambari-dev/building-from-source.md new file mode 100644 index 00000000..514760b6 --- /dev/null +++ b/versioned_docs/version-3.0.0/ambari-dev/building-from-source.md @@ -0,0 +1,125 @@ +--- +title: Building from Source +--- + +# Building Apache Ambari from Source + +This guide explains how to build Apache Ambari 3.0 and its related subprojects from source code. + +## Prerequisites + +Before you begin, ensure you have the following requirements installed: + +### System Requirements +- Operating System: Rocky Linux 8 or 9 (recommended) +- Python 3 Development Tools (`python3-devel`) + +### Java Requirements +- Ambari Main Project: JDK 17 +- Ambari Metrics: JDK 8 +- Ambari Infra: JDK 8 + +## Building Ambari Main Project + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari.git +cd ambari +``` + +### 2. Build Options + +#### Build Without RPM +To build Ambari without creating RPM packages: +```bash +mvn -B -T 2C clean install package \ + -Drat.skip=true \ + -DskipTests \ + -Dmaven.test.skip=true \ + -Dfindbugs.skip=true \ + -Dcheckstyle.skip=true +``` + +#### Build With RPM +To build Ambari and create RPM packages: +```bash +mvn -B -T 2C clean install package rpm:rpm \ + -Drat.skip=true \ + -DskipTests \ + -Dmaven.test.skip=true \ + -Dfindbugs.skip=true \ + -Dcheckstyle.skip=true +``` + +The RPM packages will be generated at: +- Ambari Agent: `ambari/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-3.0.0.0-SNAPSHOT.x86_64.rpm` +- Ambari Server: `ambari/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-3.0.0.0-SNAPSHOT.x86_64.rpm` + +## Building Ambari Metrics + +:::tip Performance Optimization +To significantly improve build performance, download binary dependencies locally before building: + +1. Create a local directory for dependencies: +```bash +mkdir -p /ws/dl/ +``` + +2. Download the required binary files: +```bash +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/hbase-2.4.13-bin.tar.gz +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/hadoop-3.3.4.tar.gz +wget -P /ws/dl/ https://dl.grafana.com/oss/release/grafana-11.1.4.linux-amd64.tar.gz +wget -P /ws/dl/ http://repo.bigtop.apache.org.s3.amazonaws.com/bigtop-stack-binary/3.2.0/centos-7/x86_64/phoenix-hbase-2.4-5.1.2-bin.tar.gz +``` + +3. Modify the `pom.xml` in ambari-metrics project to use local files: +```xml + + + file:///ws/dl/hbase-2.4.13-bin.tar.gz + file:///ws/dl/hadoop-3.3.4.tar.gz + file:///ws/dl/grafana-11.1.4.linux-amd64.tar.gz + file:///ws/dl/phoenix-hbase-2.4-5.1.2-bin.tar.gz + +``` + +This optimization will save significant time during repeated builds by avoiding large downloads. +::: + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari-metrics.git +cd ambari-metrics +``` + +### 2. Build Options + +#### Build Without RPM +To build Ambari Metrics without creating RPM packages: +```bash +mvn -T 2C clean install -DskipTests +``` + +#### Build With RPM +To build Ambari Metrics and create RPM packages: +```bash +mvn -T 2C clean install -DskipTests -Dbuild-rpm +``` + +To locate the generated RPM packages: +```bash +find ./ -name "*.rpm" +``` + +## Building Ambari Infra + +### 1. Clone the Repository +```bash +git clone git@github.com:apache/ambari-infra.git +cd ambari-infra +``` + +### 2. Build RPM Package +```bash +make rpm diff --git a/versioned_docs/version-3.0.0/ambari-dev/how-to-contribute.md b/versioned_docs/version-3.0.0/ambari-dev/how-to-contribute.md index 1e44e8c5..216d0bf1 100644 --- a/versioned_docs/version-3.0.0/ambari-dev/how-to-contribute.md +++ b/versioned_docs/version-3.0.0/ambari-dev/how-to-contribute.md @@ -36,11 +36,21 @@ Repeat these steps for all the branches that needs to be synced with the remote. Apache Ambari uses JIRA to track issues including bugs and improvements, and uses Github pull requests to manage code reviews and code merges. Major design changes are discussed in JIRA and implementation changes are discussed in pull requests after a pull request is created. +:::note Important Changes to JIRA Registration +* JIRA registration is currently closed to the public +* To get a JIRA account: + 1. Register on [Apache JIRA](https://issues.apache.org/jira) + 2. Contact a PMC member to approve your registration +* Alternatively, you can: + 1. Submit your Pull Request first + 2. Community members will help create the corresponding JIRA ticket for you +::: + * Find an existing Apache JIRA that the change pertains to * Do not create a new JIRA if the change is minor and relates to an existing JIRA; add to the existing discussion and work instead * Look for existing pull requests that are linked from the JIRA, to understand if someone is already working on the JIRA -* If the change is new, then create a new JIRA: +* If the change is new and you have JIRA access, then create a new JIRA: * Provide a descriptive Title * Write a detailed Description. For bug reports, this should ideally include a short reproduction of the problem. For new features, it may include a design document. * Fill the required fields: @@ -55,6 +65,11 @@ Apache Ambari uses JIRA to track issues including bugs and improvements, and use * Affects Version. For Bugs, assign at least one version that is known to exhibit the problem or need the change * Do not include a patch file; pull requests are used to propose the actual change. +* If you don't have JIRA access: + * Submit your Pull Request first + * In the PR description, clearly describe the issue or improvement + * A community member will create a JIRA ticket and link it to your PR + ### Pull Request Apache Ambari uses [Github pull requests](https://github.com/apache/ambari/pulls) to review and merge changes to the source code. Before creating a pull request, one must have a fork of apache/ambari checked out. Follow instructions in step 1 to create a fork if you haven't already. diff --git a/versioned_docs/version-3.0.0/ambari-dev/running-tests.md b/versioned_docs/version-3.0.0/ambari-dev/running-tests.md new file mode 100644 index 00000000..310988a7 --- /dev/null +++ b/versioned_docs/version-3.0.0/ambari-dev/running-tests.md @@ -0,0 +1,109 @@ +--- +title: Running Tests +--- + +# Running Tests in Apache Ambari + +This guide explains how to run different types of tests in Apache Ambari. For the official test configuration, you can refer to the [Ambari Jenkinsfile](https://github.com/apache/ambari/blob/trunk/Jenkinsfile). + +## Java Tests + +### Running All Java Tests +To run all Java tests for the Ambari Server: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true +``` + +### Running Specific Java Tests +To run a specific Java test class: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true \ + -Dtest=AmbariServerTest +``` + +To run a specific test method within a test class: +```bash +mvn -am test -pl ambari-server \ + -DskipPythonTests \ + -Dmaven.test.failure.ignore \ + -Dmaven.artifact.threads=10 \ + -Drat.skip \ + -DskipAdminWebTests=true \ + -Dtest=AmbariServerTest#testMethodName +``` + +### Test Parameters Explained +- `-am`: Also build dependencies +- `-pl ambari-server`: Only build the ambari-server module +- `-DskipPythonTests`: Skip Python tests +- `-Dmaven.test.failure.ignore`: Continue the build even if tests fail +- `-Dmaven.artifact.threads=10`: Use 10 threads for parallel artifact resolution +- `-Drat.skip`: Skip Apache RAT (Release Audit Tool) checks +- `-DskipAdminWebTests`: Skip admin web interface tests +- `-Dtest`: Specify which test class or method to run + +## Python Tests + +### Running All Python Tests +To run all Python tests: +```bash +mvn test -pl ambari-server \ + -DskipJavaTests \ + -Dpython.test.mask="*_test.py" \ + -Dpython.test.skip.pattern="agent_perf.py" +``` + +### Running Specific Python Tests +To run a specific Python test file: +```bash +mvn test -pl ambari-server \ + -DskipJavaTests \ + -Dpython.test.mask="test_file_name.py" +``` + +### Python Test Parameters Explained +- `-DskipJavaTests`: Skip Java tests +- `-Dpython.test.mask`: Pattern to match test files to run +- `-Dpython.test.skip.pattern`: Pattern to match test files to skip + +## Integration Tests + +### Running Integration Tests +To run integration tests: +```bash +mvn verify -pl ambari-server \ + -P integration-tests \ + -DskipPythonTests \ + -DskipJavaTests +``` + +### Integration Test Parameters +- `-P integration-tests`: Activate the integration-tests profile +- `-DskipPythonTests`: Skip Python tests +- `-DskipJavaTests`: Skip Java tests + +## Test Reports + +Test reports can be found in the following locations after test execution: + +### Java Test Reports +- Unit Tests: `ambari-server/target/surefire-reports/` +- Integration Tests: `ambari-server/target/failsafe-reports/` + +### Python Test Reports +- Test Results: `ambari-server/target/python-test-results/` +- Coverage Reports: `ambari-server/target/python-coverage/` + +:::tip +When debugging test failures, check these report directories for detailed test execution logs and stack traces. +::: diff --git a/versioned_docs/version-3.0.0/quick-start/installation-guide.md b/versioned_docs/version-3.0.0/quick-start/installation-guide.md index db52d1d8..485a50eb 100644 --- a/versioned_docs/version-3.0.0/quick-start/installation-guide.md +++ b/versioned_docs/version-3.0.0/quick-start/installation-guide.md @@ -2,6 +2,42 @@ This guide covers the installation and setup of Apache Ambari on bare metal, KVM, Docker, or Vagrant environments. +## Important: Firewall Configuration + +:::tip Before You Begin +Firewall settings can significantly impact cluster functionality by blocking necessary communication ports between components. Please review the following guidelines carefully. +::: + +### Development & Testing Environments +For development or testing environments, consider disabling the firewall: +```bash +systemctl stop firewalld +systemctl disable firewalld +``` + +### Production Environments + +#### For System Administrators +Configure the firewall to allow required Hadoop ecosystem ports based on your specific component deployment: + +| Component | Ports | Purpose | +|-----------|-------|---------| +| Ambari Server | 8080, 8440, 8441 | Web UI, Agent communication | +| Core Hadoop | 8020, 9000, 50070, 50075 | HDFS NameNode, DataNode HTTP | +| YARN | 8032, 8088, 19888 | ResourceManager, UI, JobHistory | +| Hive | 9083, 10000 | Metastore, HiveServer2 | + +:::note NodeManager Ports +YARN NodeManagers allocate containers on dynamically assigned ports (default range: 32768-65535). +This range can be restricted via `yarn.nodemanager.resource.ports` in `yarn-site.xml` +::: + +:::info For General Users +If you're not familiar with advanced network configuration, we recommend: +- Disabling the firewall during initial setup and cluster testing +- Consulting with a network security specialist for production deployments +::: + ## Prerequisites Ensure you have a working environment (bare metal, KVM, Docker, or [Vagrant setup](./environment-setup/vagrant-environment-setup.md)) before proceeding. @@ -204,16 +240,10 @@ Default credentials: ## Troubleshooting -1. If you encounter firewall issues, disable it (for development environments only): -```bash -systemctl stop firewalld -systemctl disable firewalld -``` - -2. Ensure proper hostname resolution by configuring `/etc/hosts` on all nodes. +1. Ensure proper hostname resolution by configuring `/etc/hosts` on all nodes. -3. For MySQL 8 connection issues, verify the JDBC URL includes the correct SSL parameters in `ambari.properties`. +2. For MySQL 8 connection issues, verify the JDBC URL includes the correct SSL parameters in `ambari.properties`. -4. Check service logs: +3. Check service logs: - Ambari Server: `/var/log/ambari-server/ambari-server.log` - Ambari Agent: `/var/log/ambari-agent/ambari-agent.log` diff --git a/versioned_docs/version-3.0.0/release-notes.md b/versioned_docs/version-3.0.0/release-notes.md index 25f84e96..f6e491a9 100644 --- a/versioned_docs/version-3.0.0/release-notes.md +++ b/versioned_docs/version-3.0.0/release-notes.md @@ -6,17 +6,92 @@ sidebar_position: 2 ## Overview +:::tip Major Release Apache Ambari 3.0.0 represents a significant milestone in the project's development, bringing major improvements to cluster management capabilities, user experience, and platform support. This release focuses on modernizing the technology stack, enhancing security features, and improving overall stability and performance. +::: + +## Acknowledgment and Important Notes + +### A Message to Our Community + +:::info Important Transition +We want to acknowledge that Ambari 3.0 represents a significant milestone as the first major release since the HDP project became closed source. Due to this transition, we have made some important decisions: + +- The 2.7.x series has reached End-of-Life (EOL) and will no longer be maintained, as we no longer have access to the HDP packaging source code. +- Version 3.0 marks the beginning of our new mainline branch, which will receive primary maintenance and development focus. +::: + +### From Our Hearts to the Community + +:::note A Journey Together +The path to this release has been long and challenging. Our community has weathered significant changes and obstacles: + +
+ +- 🔄 Community restructuring and transitions +- 👥 A significantly reduced contributor base +- 🛠 Critical infrastructure challenges, especially around RPM distribution +- 🔨 Profound organizational changes + +
+::: + +:::tip Thank You +To every user who has stood by Ambari through these times: your patience and understanding mean more to us than words can express. We know this release has been a long time coming, and we are truly sorry for keeping you waiting. +::: + +:::info With Gratitude +Our hearts are filled with gratitude for: + +
+ +- 🌟 The pioneers who laid Ambari's foundation +- ⭐️ The steadfast contributors who've stayed with us through the storms +- 💫 Every community member whose belief in open source has never wavered + +
+ +This release exists because of your unwavering support and dedication to the open source spirit. Your commitment has been our guiding light through these challenging times. +::: + +### Technical Changes + +#### Apache Bigtop Integration +:::tip New Integration +We are pleased to announce that Ambari 3.0 now uses Apache Bigtop for component packaging: + +
+ +- 📦 [Apache Bigtop](https://github.com/apache/bigtop/) is now our default packaging system +- 🔧 Bigtop Stack serves as the default project stack +- 🚀 This integration provides a more sustainable and community-driven approach to package management + +
+::: ## Release Highlights +### Major Improvements + +- **Apache Bigtop Integration**: + - Ambari 3.0.0 now uses Apache Bigtop as the default packaging system for component management. + - Bigtop Stack serves as the default project stack, providing a more sustainable and community-driven approach to package management. + +:::tip Bigtop Component Compilation +For developers interested in compiling components for the Ambari Bigtop Stack, we have prepared a **[detailed guide](ambari-dev/bigtop-guide.md)** that walks through the process step by step. +::: + ### Platform Support -- Added support for Rocky Linux 8 and 9 ([AMBARI-26133](https://issues.apache.org/jira/browse/AMBARI-26133)) -- Added support for openEuler-22.03 ([AMBARI-26126](https://issues.apache.org/jira/browse/AMBARI-26126)) -- Added Java 17 support ([AMBARI-26142](https://issues.apache.org/jira/browse/AMBARI-26142), [AMBARI-26186](https://issues.apache.org/jira/browse/AMBARI-26186)) -- Added MySQL 8 support for Hive ([AMBARI-26130](https://issues.apache.org/jira/browse/AMBARI-26130)) -- Enhanced Docker environment with bigtop/puppet:trunk-rockylinux-8 image +
+ +- ✅ Added support for Rocky Linux 8 and 9 ([AMBARI-26133](https://issues.apache.org/jira/browse/AMBARI-26133)) +- ✅ Added support for openEuler-22.03 ([AMBARI-26126](https://issues.apache.org/jira/browse/AMBARI-26126)) +- ☕️ Added Java 17 support ([AMBARI-26142](https://issues.apache.org/jira/browse/AMBARI-26142), [AMBARI-26186](https://issues.apache.org/jira/browse/AMBARI-26186)) +- 🗄 Added MySQL 8 support for Hive ([AMBARI-26130](https://issues.apache.org/jira/browse/AMBARI-26130)) +- 📈 Enhanced Docker environment with bigtop/puppet:trunk-rockylinux-8 image + +
### New Services and Components @@ -236,53 +311,73 @@ The PGP signatures can be verified using the public keys of the Apache Ambari Re ## Acknowledgments -Apache Ambari 3.0.0 is the result of significant contributions from the Apache Ambari community. We would like to thank all the contributors who have helped make this release possible: - -- jialiang -- zrain -- Peng Lu -- Mohammad Arshad -- yaruyng -- tongxiaojun -- Sandeep Kumar -- coldless177 -- Vishal Suvagia -- zhenye zhang -- yaolei -- xjmu -- Himanshu Maurya -- timyuer -- Ananya Singh -- rzuo -- Viraj Jasani -- William Horn -- vanshuhassija -- Yu Hou -- basapuram-kumar -- Prabhjyot Singh -- Brahma Reddy Battula -- Basapuram Kumar -- Shreeya Sand -- Bhavik Patel -- lupeng -- piaolingzxh -- Murali Krishna -- LiJie20190102 -- HARSHITH GANDHE -- userhimanshuverma -- Arnout Engelen -- wangda -- Dmytro Sen -- Rich Bowen -- Shubham Sharma -- Weijian Wen -- Will Guo -- guluo -- Peng Lee -- liqinwyyx -- and many others - -Special thanks to our Release Manager(s) for their dedication and hard work in preparing this release. +:::tip Our Amazing Contributors 🌟 +Apache Ambari 3.0.0 represents the collective effort of our incredible community. We are deeply honored to recognize the individuals who have shaped this milestone release: + +
+ +### Core Contributors 💫 +These dedicated individuals have made substantial contributions to this release: + +- **jialiang** 🏆 +- **zrain** 🏆 +- **Peng Lu** 🏆 +- **Mohammad Arshad** 🏆 +- **Sandeep Kumar** 🏆 +- **coldless177** 🏆 +- **Vishal Suvagia** 🏆 +- **zhenye zhang** 🏆 +- **yaolei** 🏆 +- **xjmu** 🏆 +- **Himanshu Maurya** 🏆 +- **timyuer** 🏆 +- **yaruyng** 🏆 +- **tongxiaojun** 🏆 +### Community Champions 🌠 +Their valuable contributions have helped drive the project forward: + +- **Ananya Singh** +- **rzuo** +- **Viraj Jasani** +- **William Horn** +- **vanshuhassija** +- **Yu Hou** +- **basapuram-kumar** +- **Prabhjyot Singh** +- **Brahma Reddy Battula** +- **Basapuram Kumar** +- **Shreeya Sand** +- **Bhavik Patel** + +### Technical Innovators ⚡ +Their technical expertise has been instrumental: + +- **lupeng** +- **piaolingzxh** +- **Murali Krishna** +- **LiJie20190102** +- **HARSHITH GANDHE** +- **userhimanshuverma** +- **Arnout Engelen** +- **wangda** +- **Dmytro Sen** +- **Rich Bowen** +- **Shubham Sharma** +- **Weijian Wen** +- **Will Guo** +- **guluo** +- **Peng Lee** +- **liqinwyyx** + +
+ +:::note Special Recognition 🎖 +A special note of appreciation goes to our Release Manager(s), whose tireless coordination and attention to detail have been crucial in bringing this release to fruition. +::: + +:::tip Join Our Community! 🤝 +Every contribution, no matter how small, helps make Ambari better. We welcome new contributors to join our vibrant community! +::: ## Community diff --git a/versioned_sidebars/version-3.0.0-sidebars.json b/versioned_sidebars/version-3.0.0-sidebars.json index 1551e3e6..6b26dc61 100644 --- a/versioned_sidebars/version-3.0.0-sidebars.json +++ b/versioned_sidebars/version-3.0.0-sidebars.json @@ -123,17 +123,22 @@ { "type": "category", "label": "Ambari Development", + "collapsible": true, + "collapsed": false, "link": { "type": "doc", "id": "ambari-dev/index" }, "items": [ + "ambari-dev/building-from-source", + "ambari-dev/running-tests", "ambari-dev/development-process-for-new-major-features", "ambari-dev/ambari-code-layout", "ambari-dev/apache-ambari-jira", "ambari-dev/coding-guidelines-for-ambari", "ambari-dev/how-to-commit", "ambari-dev/how-to-contribute", + "ambari-dev/bigtop-guide", "ambari-dev/code-review-guidelines", "ambari-dev/releasing-ambari", "ambari-dev/admin-view-ambari-admin-development",