HDDS-11630. Add Build from Maven guide #100
Conversation
github-actions
bot
added
the
website-v2
There was a problem hiding this comment.
Thanks for working on this @ptlrs. There's a fair amount of maven options hidden in Ozone that would be good to document here too. We can probably collect input from other devs on what they are using. Some that I have saved:
- Skip pnpm build of Recon front-end:
-Dskip.npx=true - Skip shading Hadoop client jars:
-DskipShade
These options greatly speed up the build if you are a developer and know you do not need either of these components.
|
|
||
| ## Prerequisites | ||
|
|
||
| **TODO** : [HDDS-11625](https://issues.apache.org/jira/browse/HDDS-11625) Finalize the version numbers of prerequisite packages |
There was a problem hiding this comment.
Let's see if we can finalize this as part of this change as well. The only thing I'm not sure about is the maven version. Really Ozone should define this in its pom probably using something like the Maven enforcer plugin. Maybe we should raise a PR to add this to the main Ozone repo and then update the docs.
There was a problem hiding this comment.
That's a good point we should take advantage of the enforcer plugin to have reproducible builds.
However, what should be considered the minimum version for Maven and Git? We depend on Hadoop and I can see if they have a minimum version specified for Maven.
On that note, it is also not clear what the minimum version of Java should be. We do build with versions 8, 11, 17. Some of the pom files specify the compiler source and target versions to Java 8. Some of the tests fail when run from the IDE with a newer JDK and need extra JVM parameters to allow Java lang and util modules and the security manager.
| - `-Pdist`: Enable the distribution profile to create deployment tarballs | ||
| - `-Pnative`: Build native libraries (requires additional system dependencies) | ||
| - `-T 4`: Use 4 threads for parallel building (adjust number based on your CPU) | ||
| - `-am -pl module-name`: Build a specific module and its dependencies |
There was a problem hiding this comment.
I think this needs mvn install run previously to work correctly, otherwise it will try to download the dependencies and for source builds from an arbitrary commit this will not be present. The previous build commands here have only mentioned mvn package.
There was a problem hiding this comment.
I update the package commands to install.
There was a problem hiding this comment.
-am is short for --also-make. It builds all modules that are required for the specified module(s). E.g. -am -pl :hdds-server-scm will build hdds-common, hdds-server-framework, etc.
Prior install is required only if you omit -am.
There was a problem hiding this comment.
Ok we need to add some more details to get this to work properly:
- The command must be run from the root of the repository.
- Running it from within a module without
-plor passing it a module that exists in your current working directory fails. - Maybe this is common sense for more seasoned maven users but it is generally unintuitive behavior.
- Running it from within a module without
- The module name must be preceded with a module specifier
:
Putting this together, mvn package -am -pl :hdds-server-scm gives the expected behavior when run from the repository root.
|
See also some updates to the way we skip Recon frontend build in apache/ozone#7454 |
Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
| | `-T 4` | Use 4 threads for parallel building (adjust number based on your CPU) | | ||
| | `-T 2C` | Use 2 threads per core for parallel building (adjust number based on your CPU) | | ||
| | `-am -pl module-name` | Build a specific module and its dependencies | | ||
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | |
There was a problem hiding this comment.
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | | |
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | | |
| | `-DskipRecon` | Skip Recon frontend build. This saves a ton of time by e.g. skipping the Recon frontend npm build | | |
There was a problem hiding this comment.
e.g. usually means there are more examples. Is skipShade skipping anything else other than the fat jar? We should probably also specify that jar is only used on the client side.
| Use this option if you only want the released version of Ozone. | ||
| <br/>Obtain the Ozone sources from the [download](/download) page. | ||
| <br/>Next, unpack the tarball |
There was a problem hiding this comment.
| Use this option if you only want the released version of Ozone. | |
| <br/>Obtain the Ozone sources from the [download](/download) page. | |
| <br/>Next, unpack the tarball | |
| Use this option if you only want a released version of Ozone. Source code for Ozone releases can be obtained from the [download page](/download). |
Just a little more concise and IMO reads smoother without the line breaks.
|
|
||
| ## Build the Project | ||
|
|
||
| Apache Ozone uses Maven as its build system. The build process compiles the source code, runs tests, and creates deployable artifacts. The project supports various build configurations to accommodate different development and deployment needs. |
There was a problem hiding this comment.
| Apache Ozone uses Maven as its build system. The build process compiles the source code, runs tests, and creates deployable artifacts. The project supports various build configurations to accommodate different development and deployment needs. | |
| Apache Ozone uses [Maven](https://maven.apache.org/) as its build system. The build process compiles the source code, runs tests, and creates deployable artifacts. The project supports various build configurations to accommodate different development and deployment needs. |
| For a basic build that skips tests: | ||
|
|
||
| ```bash | ||
| mvn clean install -DskipTests=true |
There was a problem hiding this comment.
I think we can change these commands back to package per this discussion
| | `-T 4` | Use 4 threads for parallel building (adjust number based on your CPU) | | ||
| | `-T 2C` | Use 2 threads per core for parallel building (adjust number based on your CPU) | | ||
| | `-am -pl module-name` | Build a specific module and its dependencies | | ||
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | |
There was a problem hiding this comment.
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | | |
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the `ozone-filesystem-hadoop3` fat jar build | |
In addition to helping readability this tells the markdownlint CI check it is ok for Ozone to be lowercase in this context.
| | `-T 2C` | Use 2 threads per core for parallel building (adjust number based on your CPU) | | ||
| | `-am -pl module-name` | Build a specific module and its dependencies | | ||
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | | ||
| | `-Dmaven.artifact.threads=30` | Allow maven to download 30 artifacts at once. The default value is 5. This could speed up the build process by a lot when the maven cache was not previously populated. | |
There was a problem hiding this comment.
| | `-Dmaven.artifact.threads=30` | Allow maven to download 30 artifacts at once. The default value is 5. This could speed up the build process by a lot when the maven cache was not previously populated. | | |
| | `-Dmaven.artifact.threads=30` | Allow Maven to download 30 artifacts at once. The default value is 5. This could speed up the build process by a lot when the Maven cache was not previously populated. | |
CI correctly flagged the inconsistent capitalization of maven here.
| #### Maven Build Options | ||
|
|
||
| Several Maven options are available to customize the build process: | ||
|
|
||
| Here's the conversion to a markdown table: |
There was a problem hiding this comment.
| #### Maven Build Options | |
| Several Maven options are available to customize the build process: | |
| Here's the conversion to a markdown table: | |
| #### Common Maven Build Options |
IMO the title explains it all. This is getting published as a website and all traces of markdown will be erased.
| | Command | Description | | ||
| |---------|-------------| | ||
| | `-DskipTests=true` | Skip all tests | | ||
| | `-Dskip.installnpm -Dskip.installnpx -Dskip.installyarn -Dskip.npm -Dskip.npx -Dskip.yarn` | Skip building the Javascript frontend for Recon | |
There was a problem hiding this comment.
Let's remove this and just use the -DskipRecon option.
| | `-Pdist` | Enable the distribution profile to create deployment tarballs | | ||
| | `-T 4` | Use 4 threads for parallel building (adjust number based on your CPU) | | ||
| | `-T 2C` | Use 2 threads per core for parallel building (adjust number based on your CPU) | | ||
| | `-am -pl module-name` | Build a specific module and its dependencies | |
There was a problem hiding this comment.
| | `-am -pl module-name` | Build a specific module and its dependencies | | |
| | `-am -pl :<module-name>` | Build a specific module and its dependencies when run from the root of the project | |
| | `-T 4` | Use 4 threads for parallel building (adjust number based on your CPU) | | ||
| | `-T 2C` | Use 2 threads per core for parallel building (adjust number based on your CPU) | | ||
| | `-am -pl module-name` | Build a specific module and its dependencies | | ||
| | `-DskipShade` | Skip shading. This saves a ton of time by e.g. skipping the ozone-filesystem-hadoop3 fat jar build | |
There was a problem hiding this comment.
e.g. usually means there are more examples. Is skipShade skipping anything else other than the fat jar? We should probably also specify that jar is only used on the client side.
| You can test the result of the compilation process by running a simple Ozone command which will display the Ozone version | ||
|
|
||
| ```bash | ||
| ./hadoop-ozone/dist/target/ozone-<version>-SNAPSHOT/bin/ozone version | ||
| ``` | ||
|
|
||
| The build process creates several important artifacts: | ||
|
|
||
| - **Distribution Tarball**: `hadoop-ozone/dist/target/ozone-<version>.tar.gz` (when using `-Pdist`) | ||
| - **Distribution Directory**: `hadoop-ozone/dist/target/ozone-<version>/` | ||
| - **Individual Module JARs**: `hadoop-ozone/dist/target/ozone-<version>-SNAPSHOT/share/ozone/lib` |
There was a problem hiding this comment.
| You can test the result of the compilation process by running a simple Ozone command which will display the Ozone version | |
| ```bash | |
| ./hadoop-ozone/dist/target/ozone-<version>-SNAPSHOT/bin/ozone version | |
| ``` | |
| The build process creates several important artifacts: | |
| - **Distribution Tarball**: `hadoop-ozone/dist/target/ozone-<version>.tar.gz` (when using `-Pdist`) | |
| - **Distribution Directory**: `hadoop-ozone/dist/target/ozone-<version>/` | |
| - **Individual Module JARs**: `hadoop-ozone/dist/target/ozone-<version>-SNAPSHOT/share/ozone/lib` | |
| The build process creates several important artifacts: | |
| - **Distribution Tarball**: `hadoop-ozone/dist/target/ozone-<version>.tar.gz` (when using `-Pdist`) | |
| - **Distribution Directory**: `hadoop-ozone/dist/target/ozone-<version>/` | |
| - **Individual Module JARs**: `hadoop-ozone/dist/target/ozone-<version>/share/ozone/lib` | |
| You can test the result of the compilation process by running a simple Ozone command which will display the Ozone version: | |
| ```bash | |
| hadoop-ozone/dist/target/ozone-<version>/bin/ozone version |
I think this order of presentation makes more sense.
- I removed
SNAPSHOT, we can leave that as part of<version>. Not every build will be a snapshot build. ./is not technically required here since we are already providing a relative path to the executable.
What changes were proposed in this pull request?
Added content for the developer guide to build ozone sources using maven
Please describe your PR in detail:
What is the link to the Apache Jira?
https://issues.apache.org/jira/browse/HDDS-11630
How was this patch tested?
Ran the website locally via docker-compose
CI:
https://github.com/ptlrs/ozone-site/actions/runs/11606635045https://github.com/ptlrs/ozone-site/actions/runs/11811331336https://github.com/ptlrs/ozone-site/actions/runs/11823796643https://github.com/ptlrs/ozone-site/actions/runs/11842696659https://github.com/ptlrs/ozone-site/actions/runs/11842732645