Move away from Bintray

[kaikreuzer]

Bintray is about to be closed: https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

We will therefore need to remove any links to Bintray and find a suitable substitution.

The simplest option atm seems to be the move to Artifactory (openhab.jfrog.io) - the risk here is though that it is operated by the same company as Bintray, so we might be warned that this could turn out to be a fragile option as well. Since we have all our snapshot and milestone artifacts there already. I'd nonetheless suggest to go this way. It might be a worthwhile effort to check whether we can set up a CNAME DNS entry, so that we would at least be under control of the DNS name and could point it to somewhere else in case the free JFrog Artifactory is ended some day.

A better option for releases would be to directly publish them to Maven Central - I am not sure how easy and stable we could get that into our release build jobs.

This issue is meant as an aggregate to track the efforts that are needed.

• Change "update" script in distro - Removed Bintray and JFrog dependency from update url #1271
• Change website download location
• Remove mentions of Bintray from documentation/website
• Change "distributionManagement" section in all pom-xmls
• Adapt build plans
• Adapt Docker build/containers (Download milestones and releases using openHAB redirect openhab-docker#356)
• inform users to change download locations e.g. in their linux distributions (apt/rpm)
• Change openhab-osgiify to not require artifacts from Bintray
• Change artifactory to not include Bintray locations anymore

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/bintray-jcenter-sunset-on-may-1st-2021/116391/3

[BClark09]

Also commenting here to track and to mention that Bintray is our stable host for the Linux packages.

Fortunately, our stable packages are available on Artifactory so I suspect a notification to transfer and documentation change is all that's needed here.

We have openHAB 1.x packages available from Bintray. I know that's going back a bit, but should we transfer these?

[kaikreuzer]

For the zipped distro and KARs, I am just pushing the latest+major releases (2.5.12, 2.5.0, 2.4.0, ... 2.0.0, 1.8.3) to Maven Central, see https://repo1.maven.org/maven2/org/openhab/distro/openhab/

If it is possible to move the linux packages as well (I guess our Artifactory is the only real target option, right?), it might be nice not to loose them. Is there some mechanism you could use to transfer them?

[Orifahrer]

I am trying to set up openhab 2.5.12 in my new docker env (using OH2 on Raspi until now). As my docker setup still needs some tweaking, I reinstalled the container several times to improve my settings. This seemed to be working until yesterday, but today I am getting a whole bunch of errors when I try to do the recommended setup to install PaperUI.

openhab.log says:
[ERROR] [core.karaf.internal.FeatureInstaller] - Failed installing 'openhab-ui-homebuilder, openhab-ui-basic, openhab-ui-habpanel, openhab-ui-paper': Error:
Error downloading mvn:commons-collections/commons-collections/3.2.2

I find these errors in the container stdout log:
Error downloading mvn:commons-collections/commons-collections/3.2.2
...
Caused by: java.io.IOException: Error resolving artifact commons-collections:commons-collections:jar:3.2.2: [Could not find artifact commons-collections:commons-collections:jar:3.2.2 in openhab (https://openhab.jfrog.io/openhab/libs-release/)]

Could it be related to this issue (as I cannot find anything related to common-collections under the given URL)?

[kaikreuzer]

@Orifahrer That seems due to cleanups on Artifactory - I hope that this will be fixed again when Artifactory hit cache expires after a couple of hours. If the problem persists tomorrow, please open a dedicated issue about it here in this repo.

[BClark09]

I guess our Artifactory is the only real target option, right?

I think so, although would we run into any storage constraints?

Is there some mechanism you could use to transfer them?

I'll have a look, might be a case of designing a new script for it so that we can get those 1.8.3 packages on and signed. There's a .deb package for each add-on so might take a bit of time.

[mvalla]

• is moving OH to GitHub packages maybe another option?
• the OH3 openwebnet binding links a lib that is currently published to bintray/JCenter. I would like to move the lib to either maven central or to GitHub packages. What would you suggest? Can GitHub packages be added in the OH build process as a source for dependency libs? How?

Sorry maybe question is stupid but I am not full knowledgeable about OH build process

[kaikreuzer]

is moving OH to GitHub packages maybe another option?

The last time I checked, it still was required to authenticate even for public packages - so this is imho currently no option.

I would like to move the lib to either maven central or to GitHub packages. What would you suggest?

I'd definitely suggest to push it to Maven Central. As long as Bintray/JCenter is available, it is actually quite easy to get it pushed to MC.

[pfink]

Please add

• inform users to change download locations e.g. in their linux distributions (apt/rpm)

to the TODO list.

One question: It seems that we assume Artifactory already provides everything we need. I'm wondering, what was the reason to use Bintray in the first place then? Just to understand if we did use it to solve a specific problem with artifactory and if we would face this problem again in case we migrate everything to artifactory.

If it is possible to move the linux packages as well (I guess our Artifactory is the only real target option, right?),

I've written to packagecloud, they state on their website that they support open source projects, maybe this would also be an option in case artifactory is not sufficient. I'll keep you posted regarding the result of this inquiriy.

JFrog did setup a call with me because I asked their chatbot which migration path they would recommend for FOSS projects. Results of this call: They are still dedicated to support FOSS projects (they referred this website: https://jfrog.com/open-source/#programs). They have a replacement solution for the software distribution use case (e.g. apt/rpm, download from website, ...) called "JFrog Distribution", but they can't tell whether this could be offered to us free of charge. They requested to write an e-mail with all details regarding our project. I'm not sure whether we really need JFrog distribution or if artifactory is already sufficient.

@kaikreuzer Could you check which JFrog subscription is currently assigned to us? This seems only visible in the customer portal where I don't have access: https://my.jfrog.com/ Maybe we have already access to JFrog Distribution? https://openhab.jfrog.io/ui/bundles/target

Also, which transfer quota do we currently have and how much of it is used? The current "free" subscription by JFrog only allows 10GB/month. I think we have to have more currently, otherwise we would have already issues ;)

It might be a worthwhile effort to check whether we can set up a CNAME DNS entry

Yes, let's try that. Not sure whether this service is included in our subscription (on the pricing page it's included as of "Enterprise" level), anyhow I'm happy to support here if you like.

A better option for releases would be to directly publish them to Maven Central - I am not sure how easy and stable we could get that into our release build jobs.

Probably it would be better to setup an extra job for this, comparably to linux package jobs. This allows to check the result in JFrog first before pushing it to Maven Central. I think somewhere I have already a script for copying defined artifacts from one repo to another, maybe we could use that. I remember there were two modes:

a) Copying all artifacts based on an existing maven execution (artifact list is completely taken from console output of the Jenkins job, script is reading from the "Uploading..." lines that maven prints into the log)
b) Copying a pre-defined list of artifacts specified within a configuration file

[pfink]

Update from packagecloud: They would offer us all their services for free if we confirm that we're not backed by any company.

[kaikreuzer]

It seems that we assume Artifactory already provides everything we need. I'm wondering, what was the reason to use Bintray in the first place then?

Simply: Bintray was there in the first place and we always used it for our downloads. It especially provides a CDN, which makes it more efficient than Artifactory.

For Artifactory, we already have a free OSS plan, so assuming that will stay, we could stick to it. I'll follow up on the CNAME with them, let's see.

[kaikreuzer]

@wborn I have added our latest releases 3.0.1 and 2.5.12 to Artifactory, so that they are now available under the new location https://www.openhab.org/download/releases (like e.g. https://www.openhab.org/download/releases/org/openhab/distro/openhab/3.0.1/openhab-3.0.1.zip).
I do not plan to move any older releases, though.

This should make it possible to start working on adapting the Docker build right away.

[kaikreuzer]

@BClark09 May I ask you to look into what is necessary to do for the linux-packages?

I'll have a look, might be a case of designing a new script for it so that we can get those 1.8.3 packages on and signed. There's a .deb package for each add-on so might take a bit of time.

We should be careful with the disk space on Artifactory. My suggestion would be to only keep the latest version of every (old) release, i.e. openHAB Distro 1.8.3, 2.5.12, 3.0.1, openHAB Addons 1.14.0, 2.5.12, 3.0.1.

More important than moving the old ones is to make sure that any update scripts etc. are changed on time. After April, we do not have any write access to Bintray anymore, so whatever we need to deploy there in order to push out required changes to the users, must be done within the next weeks.

[kaikreuzer]

I'll follow up on the CNAME with them, let's see.

This feels too complicated and I think openhab/website#278 is a good solution for our problem. With this, we can make sure that our download location for updates can stay the same, even if we have to move binaries in the future. For the build itself, we had already introduced the openhab-super-pom, which allows us to "retrofit" any changes in repositories on the existing builds.

[wborn]

I have added our latest releases 3.0.1

Thanks but it looks like the KAR file didn't get properly named. 🤔

https://openhab.jfrog.io/artifactory/libs-release-local/org/openhab/distro/openhab-addons/3.0.1/

We should be careful with the disk space on Artifactory.

OK that explains why it might not be feasible to upload all releases there right now.

There are also some scripts in openhab-syno-spk (@cniweb) and openhab-snap (@kubiko) that need to be updated before they can download artifacts using these new https://www.openhab.org/download/releases URLs

[kaikreuzer]

Thanks but it looks like the KAR file didn't get properly named.

Oops. Fixed!

[wborn]

Hi @kaikreuzer! We probably still need to make some changes for openhab-osgiify. I cannot build it with an empty repo.

The build also seems to have some issues downloading content, see:

https://ci.openhab.org/view/All/job/openhab-osgiify/22/

[kaikreuzer]

Build issues were seemingly just temporarily - now it is green.
But yes, as it downloads stuff from Bintray, we will have to take care of this since it will otherwise break next month.

[wborn]

I only build the new project in that build as workaround for the scheduled brown-out today, see:

https://status.bintray.com/

[kaikreuzer]

Got it.
I've transferred the required files to Artifactory and created openhab/openhab-osgiify#20 and openhab/openhab-osgiify#21, which together should solve the build issue.

[kaikreuzer]

I guess this can be closed now.