This document is primarily targeted towards Deephaven release managers. It is meant to contain the necessary instructions for executing a Deephaven release.
Please be sure to read and understand this document in full before proceeding with a release. If you have any questions or concerns about any of the commands, please reach out.
The deephaven-core jars are the most important artifacts. They serve as the basis for the Deephaven server application and downstream consumers. They are released to Maven Central.
The deephaven-core wheel is the most important artifact for Python integration purposes. It is released to PyPi and GitHub releases.
The deephaven-server wheel is used to start the Deephaven server application from the context of python. It is released to PyPi. Note: the deephaven-server wheel is large (roughly, the same size as the Deephaven server application), so it is not re-attached to GitHub releases.
The Deephaven server application is an important artifact for running the Deephaven server with the default classpath and configuration points. It also serves as the basis for Deephaven server images. It is released to GitHub releases in both tar and zip formats.
The Deephaven server images are important artifacts for easily running the Deephaven server in docker.
The images are released to the GitHub container registry ghcr.io/.
The image building logic is separated into the deephaven/deephaven-server-docker repository.
The regular images are the server (includes Python environment) and server-slim (does not include Python environment).
The extended images are all extensions to the server image and include additional Python dependencies.
- ghcr.io/deephaven/server-all-ai
- ghcr.io/deephaven/server-nltk
- ghcr.io/deephaven/server-pytorch
- ghcr.io/deephaven/server-sklearn
- ghcr.io/deephaven/server-tensorflow
The Deephaven server software bill-of-materials is an experimental artifact that some consumers may want. It may also be useful for release management purposes to ensure that no unexpected dependencies are change from release to release. It is released to GitHub releases in the syft json format.
The Deephaven python client wheels are released on PyPi as pydeephaven and
pydeephaven-ticking.
The Deephaven go client is released as a Go package.
API documentation is generated for Java, Python and C++ implemetations for Deephaven integration. The artifacts are released to GitHub releases and are published as the following:
- Java Client/Server API
- Python Integration API
- Python Client API
- C++ Client API
- C++ Examples
- R Client API
- JavaScript/TypeScript Client API
The npm package @deephaven/jsapi-types allow TypeScript clients to use their IDE's autocomplete and compiler's typechecks.
It is released to npm.
The majority of the release procedure is controlled through the publish-ci.yml workflow.
It is kicked off by a push to a branch name that matches release/v*.
Please familiarize yourself with the steps in that workflow.
Ensure you are proceeding with a known release, and there aren't any blockers.
These release notes assume that the Deephaven repository git@github.com:deephaven/deephaven-core.git is referenced as the remote named upstream.
Please ensure your local repository is setup as such, or that you replace any commands with the appropriately named remote:
$ git remote get-url upstream
git@github.com:deephaven/deephaven-core.gitEnsure you are up-to-date with upstream/main, or at the commit that you want to start a new release from.
If you are unsure what commit to start from, please ask.
Please double-check you are on the version you expect to be releasing.
The releases have so far proceeded with release/vX.Y.Z, where X.Y.Z is the version number (this isn't a technical requirement), please replace X.Y.Z with the appropriate version.
We also separate out the release branch from upstream/main with an empty commit (this isn't a technical requirement).
$ git fetch upstream
$ git checkout upstream/main
$ ./gradlew printVersion -PdeephavenBaseQualifier= -q
$ git checkout -b release/vX.Y.Z
$ git commit --allow-empty -m "Cut for X.Y.Z"For patch releases, typically the branch will be based off of the previous release tag, and not upstream/main, and the necessary patch fixes can be cherry-picked from the upstream/main branch.
The patch release manager is also responsible for bumping the patch version numbers as appropriate (note comment block on the list of commands
below).
Here is an example going from X.Y.0 to X.Y.1:
$ git fetch upstream
$ git checkout vX.Y.0
$ git checkout -b release/vX.Y.1
$ git cherry-pick <...>
#
# Edit files, updating from X.Y.0 to X.Y.1, and git add them.
#
# Look in the last section "Version bump in preparation of next release" for a list of
# files to update to the right version you are producing.
#
# See https://github.com/deephaven/deephaven-core/issues/3466 for future improvements to this process.
$ ...
$ git commit -m "Bump to X.Y.1"
$ git --no-pager log --oneline vX.Y.0..release/vX.Y.1
#
# Compare output to expected PR list for missing or extraneous PRsIt's also best practice to ensure that the cherry-picks compile, as there can sometimes be changes that cherry-pick cleanly, but don't compile.
./gradlew quick is recommended as a quick validation that things look ok.
Triple-check things look correct, the release is a "GO", and then start the release process by pushing the release branch to upstream:
$ git show release/vX.Y.Z
$ git push -u upstream release/vX.Y.ZNote: release branches are not typically merged back into main.
The release will proceed with GitHub Actions. The specific action can be found based off of the name of the release branch: ?query=branch%3Arelease%2FvX.Y.Z.
The "Publish" step creates the artifacts and publishes the jars to a Maven Central staging repository.
The "Upload Artifacts" step uploads the Deephaven server application, the deephaven-core wheel, and the deephaven-server wheel as temporary GitHub action artifacts.
The "Publish deephaven-core to PyPi" uploads the deephaven-core wheel to PyPi. If this step fails, the deephaven-core wheel from the "Upload Artifacts" step can be uploaded manually.
The "Publish deephaven-server to PyPi" uploads the deephaven-server wheel to PyPi. If this step fails, the deephaven-server wheel from the "Upload Artifacts" step can be uploaded manually.
The "Publish pydeephaven to PyPi" uploads the pydeephaven wheel to PyPi. If this step fails, the pydeephaven wheel from the "Upload Artifacts" step can be uploaded manually.
The "Publish pydeephaven-ticking to PyPi" uploads the pydeephaven-ticking wheels to PyPi. If this step fails, the pydeephaven-ticking wheel from the "Upload Artifacts" step can be uploaded manually.
The "Publish @deephaven/jsapi-types to npmjs" uploads the TypeScript tarball to NPM. If this step fails, the deephaven-jsapi-types tarball from the "Upload Artifacts" step can be uploaded manually.
Once the workflow job is done, ensure all publication sources have the new artifacts.
Once the full publish-ci.yml worflow is done, the release artifacts can be downloaded from the GitHub Artifacts (located in the "Summary" tab of the action). Similarly, release artifacts can be downloaded from the docs-ci.yml workflow. These are currently manual steps taken from the browser. (The artifacts will be uploaded in Step #9)
There is potential in the future for QA-ing these artifacts above and beyond the integration testing that CI provides, as the release is not set in stone yet.
This step is optional, but encouraged. If you are unable to use these tools, it is possible for somebody else to do this part.
The following tools are used: syft, cyclonedx.
syft server-jetty-X.Y.Z.tar -o json > server-jetty-X.Y.Z.tar.syft.jsonCompare differences:
syft convert server-jetty-A.B.C.tar.syft.json -o cyclonedx-json=/tmp/A.B.C.cyclonedx.json
syft convert server-jetty-X.Y.Z.tar.syft.json -o cyclonedx-json=/tmp/X.Y.Z.cyclonedx.json
cyclonedx diff /tmp/A.B.C.cyclonedx.json /tmp/X.Y.Z.cyclonedx.json --component-versionsPlease post the difference to the Deephaven team to ensure there are no unexpected new dependencies, removed dependencies, or updated dependencies.
The jars are put into a Maven Central Repository Manager staging repository. You'll need your own username and password to sign in (to ensure auditability).
Arguably, the Maven Central jars are the most important artifacts - once they are officially released from the staging repository, they are released "forever". This is in contrast with PyPi where build numbers could be incremented and docker where tags can always be re-written.
If any late-breaking issues are found during the release process, but the Maven Central jars have not been released from staging, the release process can theoretically be re-done.
When ready, the staging repository will need to be "Closed" and then "Released". Once the staging repository has been "Released", there is no going back.
The jars will be visible after release at https://repo1.maven.org/maven2/io/deephaven/. Sometimes it takes a little bit of time for the jars to appear.
The vX.Y.Z tag is primarily meant for an immutable reference point in the future.
It does not kick off any additional jobs.
The release should only be tagged after the Maven Central staging repository has been "Released".
$ git tag -a -m "[Release] X.Y.Z" vX.Y.Z release/vX.Y.Z
$ git show vX.Y.Z
$ git push upstream vX.Y.ZCreate a new GitHub release and use the vX.Y.Z tag as reference.
The convention is to have the Release title of the form vX.Y.Z and to autogenerate the release notes in comparison to the previous release tag. Question: should we always generate release notes based off of the previous minor release, instead of patch? Our git release workflow suggests we may want to do it always minor to minor.
Upload the Deephaven server application, deephaven-core wheel, pydeephaven wheel, pydeephaven-ticking wheels, @deephaven/jsapi-types tarball, and SBOM artifacts. Also, upload the C++, Java, Python, R and TypeScript docs artifacts. (These are the artifacts downloaded in Step #5)
Hit the GitHub "Publish release" button.
The go client release consists of simply tagging and pushing to upstream:
$ git tag -a -m "[Release] Deephaven Go Client X.Y.Z" go/vX.Y.Z release/vX.Y.Z
$ git show go/vX.Y.Z
$ git push upstream go/vX.Y.ZVerify that Reference API Docs point to the latest version and that version X.Y.Z is present. (In the case of a patch on an old release, this may not be the version just built.)
The (non-public) deephaven.io next branch needs to be merged into main. Ping Margaret.
Follow the release process as described at deephaven-server-docker/RELEASE.md.
Ping team, ping community, ping friends - the latest Deephaven has been released!
The release branches serve a purpose for kicking off CI jobs, but aren't special in other ways. Sometime after a release, old release branches can be safely deleted.
Say we just did release 0.31.0. The next expected release is 0.32.0 We update the repository with a bump to all files that
mention the version explicitly. These files are listed below:
#
# Edit files for version change
#
gradle.properties
R/rdeephaven/DESCRIPTION
cpp-client/deephaven/CMakeLists.txt
This leaves the files "ready" for the next regular release, and also ensures any build done from a developer for testing of latest is not confused with the code just released.
In the case of a patch release these would need to be updated to a different version, like from 0.31.0 to 0.31.1.
There are a few external dependencies that Deephaven manages, but they have separate release lifecycles that don't directly impact the release of the other artifacts mentioned earlier.