Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-11630. Add Build from Maven guide #100

Open
wants to merge 9 commits into
base: HDDS-9225-website-v2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cspell.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,7 @@ words:
- UX
- devs
- CLI
- xzf
- Dskip
- Pdist
- Pnative
errose28 marked this conversation as resolved.
Show resolved Hide resolved
96 changes: 96 additions & 0 deletions docs/08-developer-guide/01-build/02-maven.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,99 @@ sidebar_label: Maven

- Cover basic Maven commands to build and run tests.
- Document all the Ozone specific Maven flags we use to speed up or skip parts of the build, and when they are useful.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

This guide explains how to build Apache Ozone from source using Maven and prepare it for deployment.

## Prerequisites
errose28 marked this conversation as resolved.
Show resolved Hide resolved

**TODO** : [HDDS-11625](https://issues.apache.org/jira/browse/HDDS-11625) Finalize the version numbers of prerequisite packages
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's see if we can finalize this as part of this change as well. The only thing I'm not sure about is the maven version. Really Ozone should define this in its pom probably using something like the Maven enforcer plugin. Maybe we should raise a PR to add this to the main Ozone repo and then update the docs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point we should take advantage of the enforcer plugin to have reproducible builds.

However, what should be considered the minimum version for Maven and Git? We depend on Hadoop and I can see if they have a minimum version specified for Maven.

On that note, it is also not clear what the minimum version of Java should be. We do build with versions 8, 11, 17. Some of the pom files specify the compiler source and target versions to Java 8. Some of the tests fail when run from the IDE with a newer JDK and need extra JVM parameters to allow Java lang and util modules and the security manager.


Before you begin, ensure you have the following installed on your build machine:

- Java 1.8 or higher
- Apache Maven 3.6.3 or higher
- Git (if building from source repository)

## Building Ozone
errose28 marked this conversation as resolved.
Show resolved Hide resolved

You can build Apache Ozone either by cloning the source code from Git or by downloading the official source tarball.

### 1. Obtain the Source Code

Choose one of the following methods to get the source code:

<Tabs>
<TabItem value="Git" label="Git" default>
```bash
git clone https://github.com/apache/ozone.git
cd ozone
```
</TabItem>
<TabItem value="Tarball" label="Tarball">
errose28 marked this conversation as resolved.
Show resolved Hide resolved
```bash
curl -OL https://dlcdn.apache.org/ozone/1.4.0/ozone-1.4.0-src.tar.gz
tar xzf ozone-1.4.0-src.tar.gz
cd ozone-1.4.0-src
```
</TabItem>
</Tabs>

### 2. Build the Project

#### Basic Build

For a basic build that skips tests:

```bash
mvn clean package -DskipTests=true
```

This command will:

- Clean previous build artifacts
- Compile the source code
- Package the compiled code into JAR files
- Create a distribution in `hadoop-ozone/dist/target/ozone-<version>`

#### Build with Tests

To run unit tests during the build:

```bash
mvn clean package
```

#### Create Distribution Tarball

To create a distribution tarball for deployment:

```bash
mvn clean package -DskipTests=true -Pdist
```

This creates a tarball in `hadoop-ozone/dist/target` that contains all necessary files for deployment.

### Maven Build Options
errose28 marked this conversation as resolved.
Show resolved Hide resolved

Several Maven options are available to customize the build process:

- `-DskipTests=true`: Skip all tests
- `-Pdist`: Enable the distribution profile to create deployment tarballs
- `-Pnative`: Build native libraries (requires additional system dependencies)
errose28 marked this conversation as resolved.
Show resolved Hide resolved
- `-T 4`: Use 4 threads for parallel building (adjust number based on your CPU)
errose28 marked this conversation as resolved.
Show resolved Hide resolved
- `-am -pl module-name`: Build a specific module and its dependencies
Copy link
Contributor

@errose28 errose28 Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs mvn install run previously to work correctly, otherwise it will try to download the dependencies and for source builds from an arbitrary commit this will not be present. The previous build commands here have only mentioned mvn package.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I update the package commands to install.

Copy link
Contributor

@adoroszlai adoroszlai Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-am is short for --also-make. It builds all modules that are required for the specified module(s). E.g. -am -pl :hdds-server-scm will build hdds-common, hdds-server-framework, etc.

Prior install is required only if you omit -am.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok we need to add some more details to get this to work properly:

  • The command must be run from the root of the repository.
    • Running it from within a module without -pl or passing it a module that exists in your current working directory fails.
    • Maybe this is common sense for more seasoned maven users but it is generally unintuitive behavior.
  • The module name must be preceded with a module specifier :

Putting this together, mvn package -am -pl :hdds-server-scm gives the expected behavior when run from the repository root.


### Build Output
errose28 marked this conversation as resolved.
Show resolved Hide resolved

The build process creates several important artifacts:

- **Distribution Directory**: `hadoop-ozone/dist/target/ozone-<version>/`
- **Distribution Tarball**: `hadoop-ozone/dist/target/ozone-<version>.tar.gz` (when using `-Pdist`)
- **Individual Module JARs**: Found in `target/` directories within each module
errose28 marked this conversation as resolved.
Show resolved Hide resolved

### Next Steps

Run the build by deploying the binary on either a [machine](../../05-administrator-guide/01-installation/03-installing-binaries.md) or on a [Docker cluster](../../08-developer-guide/02-run/02-docker-compose.md)