Process for updates to target platforms

Target platforms should not grow or change unexpectedly.

Below is the process to follow for sanely changing the target platform to empower developers and keep churn to a minimum.

Goals:

Avoid unnecessary churn (failed builds)
Avoid adding unnecessary or incomplete dependencies
Document why and when changes occur

Process:

Create JIRA outlining the request
Create a pull request with your changes
Send email to jbosstools-dev@lists.jboss.org requesting feedback. State expected feedback window (usually 1 week).
Await feedback from build, affected team leads and project lead(s). For urgent changes, 1 project lead is OK.
Apply the pull-request

Create JIRA outlining the request

Use https://jira.jboss.org/jira/browse/JBIDE to open the issue.

Type of bug:

Feature Request if this change is introducing or removing a new feature/set of plugins
Bug if this is just to fix/update existing plugins

Component: target-platform + releated components if applicable (i.e. gwt)

Assignee: (default)

Fix version: Set it to the corresponding JBoss Tools release

Description:

Template (copy into JIRA):

*Reason:* ...
*Project page/sources:* ...
*Version:* ....
*License and owner:* ...
*Original p2 repo:* http://third-party-vendor.com/path/to/updates-site/
*JBoss mirror:* Leave empty
*Include Sources:* Yes|No
*Affected JBoss Tools components:* ... (components that will depend on this change, jbosstools-whatever, discovery/central, JBoss Tools update-site...)
*Include in JBDS:* Yes|No
*Type of dependency:* testing|central-only|distribution
*List of bundles added/removed:*
{code}
suggested diff on the .target files
{code}

If applicable:
*non-p2 repository:* ...
*non-p2 ids:* ...

Provide enough detail to explain why the change is needed, and what is required to verify. If you do not know the answer use "Unknown"; if not applicable, use "N/A".

Reason: Why is the request needed, typical reasons are: "Required by new feature described in JBIDE-xyz", "Eclipse releasetrain update from Kepler M5 to M6" or "Removal because it conflicts with JBTIS and give bug describe at JBIDE-xyz"
License and owner: Which license are these features available under and who are the maintainer? i.e. EPL, Google.
Original p2 repo: which repository are the plugins/features from? eg., http://dl.google.com/eclipse/plugin/4.2
JBoss mirror: URL to use, eg. http://downloads.jboss.org/jbosstools/updates/requirements/mars/201506111000-RC4/
- If no mirror already exists, you can:
  - leave this blank, or
  - create one yourself.
Include Sources: Are source bundles included in the mirror, or are sources available elsewhere? Use "included" or explain.
Affected projects: Which projects are this known to affect, both inside and outside JBoss Tools core.
Include in JBDS: Incubating (unstable) features should only be included in JBoss Tools.
Type of dependency: is this just for testing (incl. in TP, but not in distribution), central-only (incl. in Central ONLY, not in the JBoss Tools or JBDS target platform), distribution (incl. in installer & update sites)
List of bundles added/removed: p2diff report or pull-request preferred; if not available, please list features/plugins in alpha-sorted order

If your requirement is not available in an update site (p2 repository), eg. only available as maven artifact, please include more info:

non-p2 repository: the exact repository URL (eg., Maven central or jboss repository) where the artifact exists
non-p2 ids: list the exact ids, eg., for Maven state GAV (Group, Artifact, Version)

Non-p2 artifacts will be added to JBoss Tools Locus as p2 artifacts, so they can be included in the target platform.

Create and submit target-platform change

Fetch sources

Following instructions cover the minimal edits needed for a review (a final Pull request needs updates to both jbosstools and devstudio target platform as needed)

git clone https://github.com/jbosstools/jbosstools-target-platforms.git
cd jbosstools-target-platforms
git checkout <relevant version.x>

For the Integration Stack:

git clone https://github.com/jbosstools/jbosstools-integration-stack.git
cd jbosstools-integration-stack
git checkout <relevant version.x or master>

Build target platform(s) locally

To build, you require specific versions of Java and Maven. Also, there is some Maven setup. The How to Build JBoss Tools with Maven 3 document will guide you through that setup.

Note that you can only build the multiple/pom.xml locally since the unified/pom.xml refers to a remote location on download.jboss.org or devstudio.redhat.com which may or may not exist yet.

This command will run the build, but will NOT download the contents of the target platform to disk:

$ mvn clean verify -f multiple/pom.xml

If you want to download the contents of the target platform to disk, do this:

$ mvn clean verify -f multiple/pom.xml -Pmultiple2repo

If you want to run the build and fetch source bundles at the same time as other bundles are being resolved, do this:

$ mvn clean verify -f multiple/pom.xml -Pmultiple2repo -Dmirror-target-to-repo.includeSources=true

If you want to build both the multiple and unified target platforms (including mirroring the actual features & plugins onto disk), you need to point the unified pom at your locally created target platform:

$ mvn clean verify -f pom.xml -Pmultiple2repo -DtargetRepositoryUrl=file:///path/to/jbosstools-target-platforms/jbosstools/multiple/target/jbosstools-multiple.target.repo/

Or to just build the unified .target (after first building the multiple .target):

$ mvn clean verify -f unified/pom.xml -DtargetRepositoryUrl=file:///path/to/jbosstools-target-platforms/jbosstools/multiple/target/jbosstools-multiple.target.repo/

Caution

Partial target platforms such as those for JBoss Central cannot be materialized with their sources included. If you require sources, you must use a complete (merged) target platform which includes ALL upstream dependencies, but bear in mind this will produce a MUCH larger p2 repo, including all upstream dependencies and those dependencies' sources, too.

If you want to run the build and not fail if there’s a problem w/ validation, do this:

$ mvn clean verify -Pmultiple2repo -Dvalidate-target-platform.failOnError=false

If you just want to check if things compiles/builds you can run:

$ mvn clean verify -Pmultiple2repo -DskipTest=true

But do not push changes without having the new and existing unit tests pass!

Update the .target file(s)

Edit jbosstools/multiple/jbosstools-multiple.target. You may update an existing location or add a new one:

    <location includeAllPlatforms="false" includeMode="slicer" type="InstallableUnit" includeSource="true">
      <repository location="http://download.jboss.org/jbosstools/updates/requirements/tm/4.1.0.201606052351/"/>
      <unit id="org.eclipse.rse.terminals.feature.group" version="3.8.0.201505221634"/>
    </location>

Caution

UNSTATED transitive dependencies could be resolved from any of the available URLs in the target platform.

To avoid resolving the wrong (too old, too new) version of a dependency, make sure that you clearly STATE all dependencies in the <location> that contains on the IU. Using 0.0.0 may resolve to the wrong version.

For example, if docker tools depends on a specific version of httpclient, make sure that the version needed is available on the <location> from which you’re trying to resolve it.

    <location includeAllPlatforms="false" includeMode="slicer" type="InstallableUnit" includeSource="true">
      <repository location="http://download.jboss.org/jbosstools/updates/requirements/orbit/S20170306214312/"/>
      <unit id="org.apache.commons.httpclient" version="3.1.0.v201012070820"/>
      <unit id="org.apache.httpcomponents.httpclient" version="4.5.2.v20170210-0925"/>
      ...
    </location>

For JBoss Central, edit jbtcentraltarget/multiple/jbtcentral-multiple.target and/or jbtearlyaccesstarget/multiple/jbtearlyaccess-multiple.target.

For the Integration Stack, edit the applicable .target files in target-platform/.

Note	If you do not have a mirrored download.jboss.org location use the vendor’s URL, so that the .target file can be easily tested.

Now build and validate the changed target platform.

Update versions of IUs in the .target files

When moving from one version of the target to another, the steps are:

If this is a new target platform (rather than changes to an existing SNAPSHOT) increment the target platform versions contained in all the pom.xml and *.target files.
Manually update the URLs contained in the *.target files (as noted above).

Validate the target platform. You can use a script to automate these next 4 steps.

Caution

Regenerate the IU versions, using org.jboss.tools.tycho-plugins:target-platform-utils, and validate the results.
- mvn -U org.jboss.tools.tycho-plugins:target-platform-utils:fix-versions -DtargetFile=jbosstools/multiple/jbosstools-multiple.target
Resolve & verify the new target platform, optionally including sources (which will produce a MUCH larger on-disk footprint).
- mvn (clean) install -Pmultiple2repo or
- mvn (clean) install -Pmultiple2repo -Dmirror-target-to-repo.includeSources=true
Install the new target platform into a clean Eclipse JEE bundle to verify that everything can be installed, using a headless p2.director script. More info.
Produce a p2diff report.

If applicable, test your locally built target platform in another build.
- mvn verify -Dtpc.targetKind=multiple -Dtpc.version=<version-you-built>
  1. Note that tpc.version is the version listed in the pom.xml file of the target platform ( eg., 4.40.0.Beta4-SNAPSHOT) and tpc.targetKind is set to multiple since the default (unified) will not work before the target platform (.target and site) are published to Nexus and download.jboss.org
  2. If everything builds OK, can be installed (see install validation step in Update versions of IUs in the .target files), and you can use the new functionallity needed by the change, commit your change to a topic branch and submit a pull request for your update.
Follow the release guidelines for how to commit to a topic branch, and announce target platform changes.
Check in updated target files & push to the branch.

Produce a p2diff report

As noted above, you can create a p2diff report to clarify what IUs will change if your PR is applied. For example:

git clone https://github.com/irbull/p2diff.git
pushd p2diff/org.eclipse.equinox.p2.example.p2diff.releng/ && mvn install && popd
pushd p2diff/org.eclipse.equinox.p2.example.p2diff.packaging/target/products/org.eclipse.equinox.p2.example.p2diff.product
cd linux/gtk/x86_64 # or your preferred platform
./p2diff \
 file://${HOME}/tru/jbosstools-target-platforms/jbosstools/multiple/target_with_com.google/jbosstools-multiple.target.repo/ \
 file://${HOME}/tru/jbosstools-target-platforms/jbosstools/multiple/target/jbosstools-multiple.target.repo/
popd

< com.google.gwt.eclipse.sdkbundle.feature.feature.group [2.5.1]
< com.google.gwt.eclipse.core [3.4.2.v201310081834-rel-r43]
< com.google.gdt.eclipse.core [3.4.2.v201310081834-rel-r43]
< com.google.gdt.eclipse.platform [3.4.2.v201310081834-rel-r43]
< com.google.gwt.eclipse.sdkbundle.feature.feature.jar [2.5.1]
< com.google.gwt.eclipse.oophm [3.4.2.v201310081834-rel-r43]
< com.google.gwt.eclipse.sdkbundle [2.5.1]
=== Summary ===
file:///${HOME}/tru/jbosstools-target-platforms/jbosstools/multiple/target_with_com.google/jbosstools-multiple.target.repo/ contains 7 unique IUs

Request feedback and review

From the whole team

In most cases, the change can impact multiple components so the whole team should be notified about the change request to verify the suggested change doesn’t break them.

Send feedback request email to jbosstools-dev@lists.jboss.org, listing the PR, JIRA link(s), steps to use, and additional info if desired.

Past examples:

For 4.40.0.Beta2: Remove GPE/GWT, fix Atlassian
For 4.40.0.Beta1: Add Sapphire

Using the template below will get you started, but you will still need to fill in these details:

PR # and JIRA(s)
p2diff report

Template:

# configure these to generate the email body
TARGET_PLATFORM_VERSION=4.50.0.Beta1-SNAPSHOT
TOPIC_BRANCH=JBIDE-19776 # use 4.50.x or similar branch if PR is already applied
JIRA_NUM=JBIDE-19776
GITHUB_USER=mickaelistra # or nickboldt, etc. Use "origin" if PR is already applied
DEADLINE="within 7 days" # or "within 2 days" or "within 24 hours" or "as soon as possible"
PR_NUM=142
PR_URL=https://github.com/jbosstools/jbosstools-target-platforms/pull/${PR_NUM}/
P2DIFF="https://issues.jboss.org/secure/attachment/12389427/jbosstools.p2diff.txt"

echo "
Subject:

ACTION REQUIRED: Update target platform ${TARGET_PLATFORM_VERSION}
or
Proposed change to target platform ${TARGET_PLATFORM_VERSION}

Body:

Here is a proposal for a change to the JBoss Tools and Red Hat CodeReady Studio ${TARGET_PLATFORM_VERSION} target platforms.

${PR_URL}

It consists in the following change(s):

* ${JIRA_NUM}: Create and use Mars M7 target-platform


p2diff reports:

${P2DIFF}

Please review the above PR(s), as it will be applied *${DEADLINE}*.
You can use the following to build & test the target-platform locally against your component(s).

Build target-platform:
  cd /path/to/jbosstools-target-platforms/jbosstools/multiple
  git checkout ${PR_URL}
  mvn clean install

Or, without hub:
  cd /path/to/jbosstools-target-platforms/jbosstools/multiple
  git fetch origin pull/${PR_NUM}/head && git checkout FETCH_HEAD
  mvn clean install

If you'd like to build the "unified" target platform too:
  cd /path/to/jbosstools-target-platforms/jbosstools/unified
  mvn clean install -DtargetRepositoryUrl=file:///path/to/jbosstools-target-platforms/jbosstools/multiple/jbosstools-multiple.target.repo/

Then, to test the new "multiple" target platform against your component's build:
  cd /path/to/your/jbosstools-component
  mvn clean verify -Dtpc.version=${TARGET_PLATFORM_VERSION} -Dtpc.targetKind=multiple

Or, to test the new "unified" target platform against your component's build:
  cd /path/to/your/jbosstools-component
  mvn clean verify -Dtpc.version=${TARGET_PLATFORM_VERSION}
--

If you want to perform a scripted install of the entire target platform
into your local Eclipse or JBDS instance, you can now do so with this
script:

https://github.com/jbosstools/jbosstools-build-ci/blob/master/util/installFromTarget.sh

Usage is documented in the above script, and in this README:

https://github.com/jbosstools/jbosstools-target-platforms/tree/master#updating-versions-of-ius-in-target-files

--

More info:

  (if applicable)
"

You can also use a Jenkins job to send the mail.

Then Await feedback: Generally, developers should have 7 days to review a proposed change. However, for more urgent changes (such as to fix a broken contribution from a previous PR or to remove something on which no projects still depend), 1 or 2 days may be sufficient. For components that are known to be potentially affected, directly ping their developers via the mailing-list or IRC to make sure they try it.

Be sure to state the deadline in the email so devs & project leads know the urgency.

or, from requester only

When the following conditions are met:

Only one component is affected
The requester is the lead of the only affected component
There is no IU added/removed
It’s only an update of a project known to be snapshot

Then, we can assume that the requester’s feedback is enough, and ask for review directly on the Jira. If requesters gives a +1 for a given PR.

Notify the team

When the change is applied and that TP has been rebuilt and made available, send a mail to jbosstools-dev to notify about that change.

You can also use a Jenkins job to send the mail.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

target_platforms_updates.adoc

target_platforms_updates.adoc

Process for updates to target platforms

Create JIRA outlining the request

Create and submit target-platform change

Fetch sources

Build target platform(s) locally

Update the .target file(s)

Update versions of IUs in the .target files

Produce a p2diff report

Request feedback and review

From the whole team

or, from requester only

Notify the team

Files

target_platforms_updates.adoc

Latest commit

History

target_platforms_updates.adoc

File metadata and controls

Process for updates to target platforms

Create JIRA outlining the request

Create and submit target-platform change

Fetch sources

Build target platform(s) locally

Update the .target file(s)

Update versions of IUs in the .target files

Produce a p2diff report

Request feedback and review

From the whole team

or, from requester only

Notify the team