Get Releasebuild running again after ESH integration and BND migration

[pfink]

In this issue, I'll track efforts to get releasebuild running again after ESH integration:

• Initial Build fails because openhab-core reactor pom does not inherit anymore from features/p2/poms/pom.xml, so unleash plugin is not available. Doing unleash / releasebuild configuration twice again does not make sense IMO. Therefore, I carved out unleash into Super POM and inherit unleash from there: Carve out infrastructure stuff into an openHAB Super POM openhab-core#555. It's WIP currently, I'd like to get the build green once before its merged. Workaround/Hotfix: Add unleash plugin to pom.xml openhab-core#675
• Releasebuild fails with a NPE by unleash, but I don't think that it's related to the Super POM carve out. Created NPE on setReleaseVersionsTycho step shillner/unleash-maven-plugin#115 -> Fixed in Handle itests as part of the reactor during release openhab-core#673.
• Containing the 2 PRs from above, the build now fails with: xtend compile errors during releasebuild openhab-core#672. Fix: Skip spy plugin injection #896.
• Build fails with: https://github.com/openhab/openhab2-addons/pull/5194#issuecomment-476111204 (fixed)
• To be merged: Add openHAB Super POM infrastructure#4
• To be merged: Move dependency check to second unleash phase #914
• To be merged: Update unleash plugin version openhab1-addons#5862
• To be merged: Skip tycho versions step for all projects except openhab1-addons #916
• Build now fails due to the temporary POM structure of the openhab2-addons repo that was made for the Tycho -> BND migration (https://github.com/openhab/openhab2-addons/issues/5005). I'll wait until the migration is finished to get it running once the final POM structure is established.
• Super POM to be released in Maven Central
• To be merged after Super POM upload to Maven Central: https://github.com/openhab/openhab2-addons/pull/5582
• To be merged: inherit openhab-webui from Super POM (Inherit from openHAB Super POM openhab-webui#73)
• To be merged: inherit openhab-distro from Super POM (Inherit from openHAB Super POM #919)
• To be merged: Adapt Jenkinsfile to new repository structure #923

[pfink]

I could fix the NPE now. @kaikreuzer: Could you take a look at the next issue?

openhab/openhab-core#672

[wborn]

Since all UIs moved to openhab-webui, there now also needs to be a pipeline step for releasing that project.

[shillner]

Hi, let me also say a word to the issues ;)
First of all, thanks for using the plugin! I'm glad that the plugin is considered to be useful and is taken into account.

I'll have a look at this NPE issue these days.
I'm also interested in your ideas, findings and implementations of a Jenkins pipeline step for unleash. Please keep me up-to-data once you have a solution. Unfortunately I can hardly assist in that task since the time is too raw currently.

[pfink]

@shillner: No need to assist, issue was already solved, see here: shillner/unleash-maven-plugin#115 (comment) :)

[pfink]

Status did not change, but I updated the summary within the first post of this issue so that its clearer now. Help regarding openhab/openhab-core#672 would be highly appreciated. Also, I marked the hyperlinked builds as "keep forever" so that they aren't lost anymore after a week or so.

@wborn: Sure! But currently, the release already fails at the very beginning at openhab-core, so I try to get this running first.

[pfink]

Build now fails due to the temporary POM structure of the openhab2-addons repo that was made for the Tycho -> BND migration (openhab/openhab2-addons#5005). I tried for 3 hours to fix issues there, but IMO this does not make sense anymore. There too many issues with this temporary POM structure and it makes no sense to invest further effort here as it'll change again anyway. I'll wait until the migration is finished to get it running once the final POM structure is established.

[kaikreuzer]

That's bad news. I cannot judge when the bnd migration might be complete and we have already missed 2 milestone builds in Feb and March...

I am not sure whether we really have a "temporary pom structure" - I thought we have the tycho one (which used to work in the past for the release builds) and the new bnd stuff, which must work in the future for the release builds.
So are you sure that the problems are due to some "intermediate code" and will automatically vanish once the migration is completed? I am fearing that we rather postpone the solution by weeks and then notice that the problems are still the same as today as they are caused by the changes that openhab/openhab2-addons#5005 introduced.

Can you provide any more details on the issues you encounter? Maybe @maggu2810 or others can help addressing those as part of openhab/openhab2-addons#5005.

[pfink]

As stated, there are a bunch of issues: E.g. everything (metadata, plugins, ...) was deleted from the root POM which is now temporary. After fixing that, I encountered errors related to the versioning checks (snapshot parent version, ..) which are probably caused by the temporary hierarchy that was made to support tycho & bnd in parallel during the migration. I skipped that by manually updating versions and encountered NPEs by the unleash plugin. NPEs can only be tracked with debugging unleash internals and I have to deep-dive a lot into the plugin internals which could take hours and days (just as for the last NPE ;-) ). As all changes I made so far would also just be temporary ones and probably obsolete after migration is finished, I decided that it could be the same for the NPE topic, that's why I decided to not spend this effort.

I hope that the bnd migration will be finished in 2-3 weeks - I could not promise anyway that I can get it running with the current structure in the meantime as I'm not sure how complex it would be and how many issues are specific to the temporary structure.

I'll now continue with the other repositories (zwave, webui, ..) so that we don't lose so much time.

[maggu2810]

I never used the unleash plugin for my projects I maintain or the ones I work with. So I don't know why it is necessary for the release process and why it is not working. Sorry.

@pfink Would it help if you don't use the base POM of openhab2-addons that's only intention is to provide one reactor for the Bnd and Tycho based Maven build? Perhaps you could instead use two jobs, one that is using the base Tycho POM and one that is using the base Bnd POM.

[pfink]

That would be an idea. What will happen when the migration is finished? The current base bnd POM will get the root POM? Then I would prefer to concentrate on this and simulating the final structure already in the releasebuild. It makes no sense to spend effort into something which will not exist anymore in 1 month...

Btw, fyi, the unleash plugin orchestrates the release process and takes over many tasks here (checking for snapshot dependencies which are not allowed, version bump [release version, next dev version, ..], tagging, ..).

[wborn]

We initially used 2 reactors (tycho, bnd) when we started the bnd migration in openhab2-addons. However the "Deploy artifacts to Artifactory" post-build step couldn't handle that (https://github.com/openhab/openhab2-addons/pull/5018#issuecomment-471171941). This resulted in the artifacts of only one reactor being deployed to Artifactory. So you may face the same issues when using a pipeline command that's backed by the same logic.

[maggu2810]

@wborn I assume you used one maven job that executes multiple build steps.
Why not using two jobs?

[maggu2810]

checking for snapshot dependencies which are not allowed, version bump [release version, next dev version, ..], tagging, ..

Hm, isn't that already working with Jenkins and Maven projects without that plugin?

[wborn]

Two jobs probably also worked but it would also be more complex. I'd rather then ditch the Jenkins post build action and replace it with the execution of some Maven goal. 😉

[pfink]

Hm, isn't that already working with Jenkins and Maven projects without that plugin?

Sure, there are many different ways to do the release process. I chose to do it with unleash. Of course, it has advantages and disadvantages. There is much time invested in the current solution, and you would need also much time to do another solution. A fully automated openHAB release process is not something that you can setup in 10 minutes with 2 Jenkins jobs.

The huge advantage of the unleash plugin is that you can do a release with just one command even without Jenkins, at least in a perfect world. Especially, every developer with a fork could easily do his own release on his own workspace. If something goes wrong with the build, SCM operations etc. are automatically rolled back. Unfortunately, in our world, it's still much more complex. Anyhow, we're not so far away and in the background I'm still working to make our world a perfect world ;-)

Another advantage of the unleash plugin is that it has a quite good amount of builtin sanity checks which can unveal many issues within the POM structure, dependencies and so on. Especially during introduction of the plugin, it unvealed quite a lot flaws that we fixed.

If you already have the single magic command that does everything, I'd be curious to see it! :P

[kaikreuzer]

@pfink All Tycho has vanished from openhab2-addons, so I think it is time to make the release build working again!

[kaikreuzer]

I've reactivated openhab2-addons in the release build, but unfortunately, it is still pretty red...

[J-N-K]

Apart from the unleash-plugin you probably need openhab2-addons#5570 to get the addons-build green.

[pfink]

I could agree with @maggu2810 on the lowest common denominator regarding openhab/openhab-core#555. I'll continue with this one now and make further PRs which will fix the current issue on the addons build.

[pfink]

Added

• Add openHAB Super POM infrastructure#4 and
• https://github.com/openhab/openhab2-addons/pull/5582

[bjoernbrings]

Is there anything that could be done by the community to support in getting releasebuilds up and running again (like support in migrating the addons was in the past)?

[pfink]

Is there anything that could be done by the community to support in getting releasebuilds up and running again (like support in migrating the addons was in the past)?

You can work on the issues linked within this thread or try to get it running on your own. The Jenkins script can be found here: https://github.com/openhab/openhab-distro/blob/master/launch/Jenkinsfile

This one is our testing job: https://ci.openhab.org/view/Sandbox/job/sandbox-openhab2-release/

[pfink]

Today I additionally raised #914.

[pfink]

Added

• Update unleash plugin version openhab1-addons#5862
• Skip tycho versions step for all projects except openhab1-addons #916

[bjoernbrings]

You can work on the issues linked within this thread or try to get it running on your own. The Jenkins script can be found here: https://github.com/openhab/openhab-distro/blob/master/launch/Jenkinsfile

I tired to get this pipeline running locally, but failed due to several parameters, etc.
Is there anywhere a instruction how to get this script running (as far as it is in the sandbox)?

[kaikreuzer]

@pfink I have manually uploaded the super pom to https://bintray.com/openhab/mvn/openhab-super-pom/1.0.0, so that we can move on here.

The pipeline build nontheless still fails with

[ERROR] No plugin found for prefix 'unleash' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/var/jenkins_home/workspace/sandbox-openhab2-release/.repository), jcenter (https://jcenter.bintray.com), openhab-artifactory-release (https://openhab.jfrog.io/openhab/libs-release), central (https://repo.maven.apache.org/maven2)] -> [Help 1]

What needs to be done here?

[kaikreuzer]

@pfink Just to mention it: The main issue description looks outdated.

[pfink]

I updated the main issue description now!

[wborn]

I see the latest pipeline build succeeded! 👍 🥇
Is there anything still to be done for this besides celebrating it? 😉

[kaikreuzer]

Hooray 🎉!
I would hope that this means that we can again build milestones - but let's have @pfink the final word on this.
Next step would be that we feel comfortable enough with the current snapshot quality to actually build a milestone from it 🤔...

[pfink]

Did a short checkup now and everything looks very good :) Just one last PR to be merged (reflects the latest pipeline script changes): #923

I also fixed the openHAB2-Bundles Build now!

Some final notes:

• all repositories except of openhab1-addons are version-bumped with the versions plugin instead of the tycho versions plugin now.
• sandbox release can be found here (for those not familiar what "sandbox release" means: this is no real release, just a "simulation" how the release would look like if it had been done last night):
  • Release: https://openhab.jfrog.io/openhab/sandbox-all/org/openhab/distro/openhab/2.5.0
  • Next Snapshot: https://openhab.jfrog.io/openhab/sandbox-all/org/openhab/distro/openhab/2.6.0-SNAPSHOT
• Automatic SCM actions (version bumps, tags, ..) can be checked here: https://github.com/openhab-sandbox

[wborn]

I also fixed the openHAB2-Bundles Build now!

Great! Shall we also add such a pre-step to the other builds that have the same issue?

[wborn]

The same fix has now been applied to the other builds so they should be successful again.
There's also a new Distro build again. 😄

[kaikreuzer]

Thank you both @pfink and @wborn!

[cweitkamp]

Very nice. I am really looking forward for a next milestone. Thank you all for your effort.

[bjoernbrings]

Now the build is down since some days again (the regular, not the sandbox)

[kaikreuzer]

No, the release build is all green: https://ci.openhab.org/view/Sandbox/job/sandbox-openhab2-release/

[Gozilla01]

@kaikreuzer
latest build #1​60​7 4-giu-2019 16.59
Should a new one be released?

[bjoernbrings]

So the snapshots (that are currently not working after a failure 5 days ago) have not been migrated so far? Thought there is a link.
But you are right, the sandbox is still green.

[kaikreuzer]

This issue here is about release builds, not snapshot builds.

[martinvw]

The same fix has now been applied to the other builds so they should be successful again.
There's also a new Distro build again. 😄

Could it be that the ZWave and ZigBee builds were missed? Is the exact change just adding the pre-build step? Could @wborn or @pfink take care of this or could someone hint which is the best example to copy from for those builds.

[wborn]

The issue is that Jenkins uses an embedded Maven that hasn't been updated for handling parent version ranges. The workaround for the other projects was adding the pre build step where it executes:

mvn versions:resolve-ranges -B -DprocessParent=true

That only resolves ranges for dependencies/parents in the project itself. In case of zigbee/zwave the parent with the range (openhab-super-pom) is defined in its parent (org.openhab.addons:org.openhab.addons.reactor) so it cannot update that POM with the resolved range.

See also: https://github.com/openhab/openhab2-addons/pull/5582

To workaround this outdated embedded Maven version in Jenkins, we could change these Jobs from "Maven project" to "Freestyle projects"/"Pipeline" builds (https://github.com/openhab/openhab2-addons/pull/5582#issuecomment-498383737).

[martinvw]

What would be the best? @pfink WDYT, should we go for the freestyle in this case?

[pauxus]

Just my two cents: I would advise using pipelines with the takari maven wrapper. This would reduce the need to apply upcoming changes again to all relevant builds. Rather than that, all relevant information could be included in the code itself (and could be changed in a branch, if necessary. this would also allow to verify and accept actual build changes via a PR).

[pfink]

I also think pipeline would be better. This way we can manage the build source code within Git so that everyone can do PRs against it.

For the bundles build, I'm not so sure. With a pipeline job, you don't have the easy overview which module failed...

[peterloron]

Yes, pipelines are definitely the way to go. The takari wrapper works pretty well too.