Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move away from Bintray #1256

Closed
9 tasks done
kaikreuzer opened this issue Feb 6, 2021 · 33 comments
Closed
9 tasks done

Move away from Bintray #1256

kaikreuzer opened this issue Feb 6, 2021 · 33 comments

Comments

@kaikreuzer
Copy link
Member

kaikreuzer commented Feb 6, 2021

Bintray is about to be closed: https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

We will therefore need to remove any links to Bintray and find a suitable substitution.

The simplest option atm seems to be the move to Artifactory (openhab.jfrog.io) - the risk here is though that it is operated by the same company as Bintray, so we might be warned that this could turn out to be a fragile option as well. Since we have all our snapshot and milestone artifacts there already. I'd nonetheless suggest to go this way. It might be a worthwhile effort to check whether we can set up a CNAME DNS entry, so that we would at least be under control of the DNS name and could point it to somewhere else in case the free JFrog Artifactory is ended some day.

A better option for releases would be to directly publish them to Maven Central - I am not sure how easy and stable we could get that into our release build jobs.

This issue is meant as an aggregate to track the efforts that are needed.

@openhab-bot
Copy link
Collaborator

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/bintray-jcenter-sunset-on-may-1st-2021/116391/3

@BClark09
Copy link
Member

BClark09 commented Feb 6, 2021

Also commenting here to track and to mention that Bintray is our stable host for the Linux packages.

Fortunately, our stable packages are available on Artifactory so I suspect a notification to transfer and documentation change is all that's needed here.

We have openHAB 1.x packages available from Bintray. I know that's going back a bit, but should we transfer these?

@kaikreuzer
Copy link
Member Author

For the zipped distro and KARs, I am just pushing the latest+major releases (2.5.12, 2.5.0, 2.4.0, ... 2.0.0, 1.8.3) to Maven Central, see https://repo1.maven.org/maven2/org/openhab/distro/openhab/

If it is possible to move the linux packages as well (I guess our Artifactory is the only real target option, right?), it might be nice not to loose them. Is there some mechanism you could use to transfer them?

@Orifahrer
Copy link

I am trying to set up openhab 2.5.12 in my new docker env (using OH2 on Raspi until now). As my docker setup still needs some tweaking, I reinstalled the container several times to improve my settings. This seemed to be working until yesterday, but today I am getting a whole bunch of errors when I try to do the recommended setup to install PaperUI.

openhab.log says:
[ERROR] [core.karaf.internal.FeatureInstaller] - Failed installing 'openhab-ui-homebuilder, openhab-ui-basic, openhab-ui-habpanel, openhab-ui-paper': Error:
Error downloading mvn:commons-collections/commons-collections/3.2.2

I find these errors in the container stdout log:
Error downloading mvn:commons-collections/commons-collections/3.2.2
...
Caused by: java.io.IOException: Error resolving artifact commons-collections:commons-collections:jar:3.2.2: [Could not find artifact commons-collections:commons-collections:jar:3.2.2 in openhab (https://openhab.jfrog.io/openhab/libs-release/)]

Could it be related to this issue (as I cannot find anything related to common-collections under the given URL)?

@kaikreuzer
Copy link
Member Author

@Orifahrer That seems due to cleanups on Artifactory - I hope that this will be fixed again when Artifactory hit cache expires after a couple of hours. If the problem persists tomorrow, please open a dedicated issue about it here in this repo.

@BClark09
Copy link
Member

BClark09 commented Feb 6, 2021

I guess our Artifactory is the only real target option, right?

I think so, although would we run into any storage constraints?

Is there some mechanism you could use to transfer them?

I'll have a look, might be a case of designing a new script for it so that we can get those 1.8.3 packages on and signed. There's a .deb package for each add-on so might take a bit of time.

@mvalla
Copy link

mvalla commented Feb 7, 2021

  • is moving OH to GitHub packages maybe another option?
  • the OH3 openwebnet binding links a lib that is currently published to bintray/JCenter. I would like to move the lib to either maven central or to GitHub packages. What would you suggest? Can GitHub packages be added in the OH build process as a source for dependency libs? How?

Sorry maybe question is stupid but I am not full knowledgeable about OH build process

@kaikreuzer
Copy link
Member Author

is moving OH to GitHub packages maybe another option?

The last time I checked, it still was required to authenticate even for public packages - so this is imho currently no option.

I would like to move the lib to either maven central or to GitHub packages. What would you suggest?

I'd definitely suggest to push it to Maven Central. As long as Bintray/JCenter is available, it is actually quite easy to get it pushed to MC.

@pfink
Copy link
Contributor

pfink commented Mar 9, 2021

Please add

  • inform users to change download locations e.g. in their linux distributions (apt/rpm)

to the TODO list.

One question: It seems that we assume Artifactory already provides everything we need. I'm wondering, what was the reason to use Bintray in the first place then? Just to understand if we did use it to solve a specific problem with artifactory and if we would face this problem again in case we migrate everything to artifactory.

If it is possible to move the linux packages as well (I guess our Artifactory is the only real target option, right?),

I've written to packagecloud, they state on their website that they support open source projects, maybe this would also be an option in case artifactory is not sufficient. I'll keep you posted regarding the result of this inquiriy.

JFrog did setup a call with me because I asked their chatbot which migration path they would recommend for FOSS projects. Results of this call: They are still dedicated to support FOSS projects (they referred this website: https://jfrog.com/open-source/#programs). They have a replacement solution for the software distribution use case (e.g. apt/rpm, download from website, ...) called "JFrog Distribution", but they can't tell whether this could be offered to us free of charge. They requested to write an e-mail with all details regarding our project. I'm not sure whether we really need JFrog distribution or if artifactory is already sufficient.

@kaikreuzer Could you check which JFrog subscription is currently assigned to us? This seems only visible in the customer portal where I don't have access: https://my.jfrog.com/ Maybe we have already access to JFrog Distribution? https://openhab.jfrog.io/ui/bundles/target

Also, which transfer quota do we currently have and how much of it is used? The current "free" subscription by JFrog only allows 10GB/month. I think we have to have more currently, otherwise we would have already issues ;)

It might be a worthwhile effort to check whether we can set up a CNAME DNS entry

Yes, let's try that. Not sure whether this service is included in our subscription (on the pricing page it's included as of "Enterprise" level), anyhow I'm happy to support here if you like.

A better option for releases would be to directly publish them to Maven Central - I am not sure how easy and stable we could get that into our release build jobs.

Probably it would be better to setup an extra job for this, comparably to linux package jobs. This allows to check the result in JFrog first before pushing it to Maven Central. I think somewhere I have already a script for copying defined artifacts from one repo to another, maybe we could use that. I remember there were two modes:

a) Copying all artifacts based on an existing maven execution (artifact list is completely taken from console output of the Jenkins job, script is reading from the "Uploading..." lines that maven prints into the log)
b) Copying a pre-defined list of artifacts specified within a configuration file

@pfink
Copy link
Contributor

pfink commented Mar 19, 2021

Update from packagecloud: They would offer us all their services for free if we confirm that we're not backed by any company.

@kaikreuzer
Copy link
Member Author

It seems that we assume Artifactory already provides everything we need. I'm wondering, what was the reason to use Bintray in the first place then?

Simply: Bintray was there in the first place and we always used it for our downloads. It especially provides a CDN, which makes it more efficient than Artifactory.

For Artifactory, we already have a free OSS plan, so assuming that will stay, we could stick to it. I'll follow up on the CNAME with them, let's see.

kaikreuzer added a commit to openhab/website that referenced this issue Mar 25, 2021
This gives us a chance to not hard-code jfrog urls anywhere in our distro.

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
ghys pushed a commit to openhab/website that referenced this issue Mar 26, 2021
This gives us a chance to not hard-code jfrog urls anywhere in our distro.

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/infrastructure that referenced this issue Mar 26, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
@wborn wborn pinned this issue Mar 26, 2021
wborn pushed a commit to openhab/infrastructure that referenced this issue Mar 26, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/website that referenced this issue Mar 26, 2021
Introducing separate redirects for milestones and releases, so that we are flexible with future routing of requests.

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
ghys pushed a commit to openhab/website that referenced this issue Mar 26, 2021
Introducing separate redirects for milestones and releases, so that we are flexible with future routing of requests.

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
@kaikreuzer
Copy link
Member Author

@wborn I have added our latest releases 3.0.1 and 2.5.12 to Artifactory, so that they are now available under the new location https://www.openhab.org/download/releases (like e.g. https://www.openhab.org/download/releases/org/openhab/distro/openhab/3.0.1/openhab-3.0.1.zip).
I do not plan to move any older releases, though.

This should make it possible to start working on adapting the Docker build right away.

@kaikreuzer
Copy link
Member Author

@BClark09 May I ask you to look into what is necessary to do for the linux-packages?

I'll have a look, might be a case of designing a new script for it so that we can get those 1.8.3 packages on and signed. There's a .deb package for each add-on so might take a bit of time.

We should be careful with the disk space on Artifactory. My suggestion would be to only keep the latest version of every (old) release, i.e. openHAB Distro 1.8.3, 2.5.12, 3.0.1, openHAB Addons 1.14.0, 2.5.12, 3.0.1.

More important than moving the old ones is to make sure that any update scripts etc. are changed on time. After April, we do not have any write access to Bintray anymore, so whatever we need to deploy there in order to push out required changes to the users, must be done within the next weeks.

@kaikreuzer
Copy link
Member Author

I'll follow up on the CNAME with them, let's see.

This feels too complicated and I think openhab/website#278 is a good solution for our problem. With this, we can make sure that our download location for updates can stay the same, even if we have to move binaries in the future. For the build itself, we had already introduced the openhab-super-pom, which allows us to "retrofit" any changes in repositories on the existing builds.

@wborn
Copy link
Member

wborn commented Mar 27, 2021

I have added our latest releases 3.0.1

Thanks but it looks like the KAR file didn't get properly named. 🤔

https://openhab.jfrog.io/artifactory/libs-release-local/org/openhab/distro/openhab-addons/3.0.1/

We should be careful with the disk space on Artifactory.

OK that explains why it might not be feasible to upload all releases there right now.

There are also some scripts in openhab-syno-spk (@cniweb) and openhab-snap (@kubiko) that need to be updated before they can download artifacts using these new https://www.openhab.org/download/releases URLs

@kaikreuzer
Copy link
Member Author

Thanks but it looks like the KAR file didn't get properly named.

Oops. Fixed!

kaikreuzer added a commit to openhab/org.openhab.binding.zigbee that referenced this issue Apr 20, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit that referenced this issue Apr 20, 2021
Related to #1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
wborn pushed a commit that referenced this issue Apr 20, 2021
Related to #1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/openhab-addons that referenced this issue Apr 20, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/openhab-webui that referenced this issue Apr 20, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/org.openhab.binding.zwave that referenced this issue Apr 20, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/org.openhab.binding.zigbee that referenced this issue Apr 20, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/openhab-addons that referenced this issue Apr 20, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/openhab-webui that referenced this issue Apr 20, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/org.openhab.binding.zwave that referenced this issue Apr 20, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit to openhab/org.openhab.binding.zigbee that referenced this issue Apr 20, 2021
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
kaikreuzer added a commit that referenced this issue Apr 20, 2021
Related to #1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
@wborn
Copy link
Member

wborn commented Apr 26, 2021

Hi @kaikreuzer! We probably still need to make some changes for openhab-osgiify. I cannot build it with an empty repo.

The build also seems to have some issues downloading content, see:

https://ci.openhab.org/view/All/job/openhab-osgiify/22/

@kaikreuzer
Copy link
Member Author

Build issues were seemingly just temporarily - now it is green.
But yes, as it downloads stuff from Bintray, we will have to take care of this since it will otherwise break next month.

@wborn
Copy link
Member

wborn commented Apr 26, 2021

I only build the new project in that build as workaround for the scheduled brown-out today, see:

https://status.bintray.com/

@kaikreuzer
Copy link
Member Author

Got it.
I've transferred the required files to Artifactory and created openhab/openhab-osgiify#20 and openhab/openhab-osgiify#21, which together should solve the build issue.

hubsif pushed a commit to hubsif/openhab-webui that referenced this issue May 6, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
themillhousegroup pushed a commit to themillhousegroup/openhab2-addons that referenced this issue May 10, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
Signed-off-by: John Marshall <john.marshall.au@gmail.com>
@kaikreuzer
Copy link
Member Author

I guess this can be closed now.

@wborn wborn unpinned this issue May 16, 2021
computergeek1507 pushed a commit to computergeek1507/openhab-addons that referenced this issue Jul 13, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
frederictobiasc pushed a commit to frederictobiasc/openhab-addons that referenced this issue Oct 26, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
thinkingstone pushed a commit to thinkingstone/openhab-addons that referenced this issue Nov 7, 2021
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
xiweilej added a commit to xiweilej/openhab that referenced this issue Jan 21, 2022
Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
marcfischerboschio pushed a commit to marcfischerboschio/openHABaddon that referenced this issue Apr 20, 2022
* Adapted url for Artifactory

Related to openhab/openhab-distro#1256

Signed-off-by: Kai Kreuzer <kai@openhab.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants