Provide and publish ppc64le images

[wohali]

A few requests have come in for ppc64le images, see #66, #60, etc.

This is blocked until:

• Jenkins CI is updated to include an ppc64le build target, see Add multi-platform support to CI configuration couchdb#1121 for progress on this
• CI workflow is "all-green" on a regular basis
• PR Support multi-arch image building for couchdb docker image #60 is merged

[abdasgupta]

Hi @wohali Currently I am pursuing this task. Can I get an update on the progress of this issue??

[wohali]

Still blocked by everything stated in my last update.

[abdasgupta]

Hi @wohali , Can we work to solve the first problem of the list at first? Can I know what is to be done yet to close the first problem?

[wohali]

Start with: If you have access to a ppc64le machine, run make check and confirm that all tests pass. If some fail, log them here:

https://github.com/apache/couchdb/issues

as ppc64le specific failures.

[abdasgupta]

Hi @wohali raised a issue after observing a test case failure: apache/couchdb#1351 I am currently working to progress further.

[abdasgupta]

Hi @wohali So make check is working fine on ppc64le. Can you guide me to next step to bring couchdb on power?

[abdasgupta]

Hi @wohali any update??

[wohali]

@abdasgupta We need to add regression testing on ppc64le so that this good result continues. This requires ASF Jenkins work. Are you an ASF contributor/member?

[abdasgupta]

Hi @wohali , We have two ppc64le nodes on ASF jenkins. Please follow the link, https://issues.apache.org/jira/browse/INFRA-15887 . I am a contributor of Mesos of ASF. Can you please let me know what do I need to do to enable Jenkins CI on ppc64le?

[wohali]

We need to update Jenkinsfile on apache/couchdb to run on this platform.

Currently all builds on Jenkins are done in docker containers themselves to avoid having to install the prerequisites on the Jenkins nodes directly.

Have a look at apache/couchdb-ci and see if you can set up the scripts to build a testing environnment on ppc64le. Pick the OS you think is most supported there - my guess would be Ubuntu 16.04 LTS or possibly 18.04. (I have no access to one of these systems so I can't tell you for sure).

Make a PR on that repo and I'll review the code, and can push an image up to the couchdbdev Docker Hub organisation for the team.

After that, make a PR on apache/couchdb that uses the image in Jenkinsfile. Name your branch jenkins- and we'll be able to test it before it is committed to master.

Once we have proper CI on ppc64le, so we can look at failure situations with every commit to master, I'll greenlight updating this repo to have ppc64le support and we can push an apache/couchdb image that is ppc64le flavoured.

[sarveshtamba]

@wohali @abdasgupta
Do we have couchdb support for a ppc64le CI yet? I was checking https://builds.apache.org/job/CouchDB/ and saw that the two ppc64le nodes(ubuntu-ppc64le and hadoop-ppc64le-1) have no Projects tied to them. Please let me know in case I am missing something here.

[wohali]

@sarveshtamba We have an issue with performance in our CI environment through which we cross-compile and test things. We're working the issues out with arm (aarch64) first, because there is more demand for that platform - then will add ppc64le next.

See apache/couchdb#2030 for progress and problems. Basically we need to up the timeouts across all of the CouchDB tests across the board for this to work.

If you know Erlang and can help the team, we'd be appreciative of support. We don't need any more "platform porters" right now.

[wohali]

Just to be clear - there's issues with the native node availability, and since CI runs are intended to gate checkins to the master branch of CouchDB, we shouldn't be relying on native aarch or ppc64le machines on checkin suites to gate merging PRs. (We already have enough issues keeping those runs green.) If we have no alternative, because cross-compiled tests fail, then so be it, but I don't want to rely on a single machine ubuntu-ppc64le (we're not allowed to use the hadoop one) that could potentially block every single commit to the project. Make sense?

[sarveshtamba]

Hi @wohali, thanks for the response.

From the above I think we have two issues here:-

1.) Performance issue:-
Currently Im trying to run the test suite for CouchDB using make check on a local ppc64le node. Once that succeeds, can we have couchdb support for a ppc64le CI, or will it atleast help expedite things?

2.) Native node availability:-
In case you are short of ubuntu based ppc64le nodes, would it help if we help in providing that hardware?

Let me know if my understanding is correct.

[wohali]

I am out until Monday but briefly, we need the tests to be able to pass on cross-compile, not just native hardware. Donations of HW would be appreciated but will not ungate this effort. You finding any failing `make check` test cases and reporting them is also critical, but again, will not ungate this effort.

[sarveshtamba]

@wohali could you please let us know the process for providing the hardware for ppc64le?

[wohali]

@sarveshtamba Generally, ASF Infrastructure expects to manage the hardware themselves, using their puppet infrastructure. See https://github.com/apache/infrastructure-puppet/tree/deployment/modules/build_slaves and https://github.com/apache/infrastructure-puppet/blob/deployment/modules/build_slaves/files/toolchains.xml for detail.

If this is not possible, there is an alternative where you can admin the machine, ensure the build dependencies are installed, and run the Jenkins agent yourself, and be authorized to connect the worker to Jenkins - but Infra will be asking if that hardware can be used by projects other than just CouchDB to remove the ppc64le single point of failure for all ASF projects. You need to be prepared to answer this question.

If you can give me a heads up to the question above, I can put you in touch with the right people.

[sarveshtamba]

@wohali Tested CouchDB v2.3(current master) with Erlang v21 and Elixir v1.8 on ppc64le, the build completes successfully and the entire test suite executes successfully with no test failures. Let us know if we still need additional dedicated ppc64le hardware for supporting CouchDB CI.

[wohali]

@sarveshtamba Yes, we need 2x ppc64le machines against which to test. The hadoop machine is hands-off for us, and we can't rely on just the one other machine.

I can push out packages for 2.3.1 but until we have those 2x machines I can't agree to a 3.0.0 package or image release.

@kocolosk will the VM credits IBM is providing be able to cover 2x ppc64le Jenkins nodes? If so then there's no need for @sarveshtamba to look any further.

[kocolosk]

Sure we can cover that.

[wohali]

ppc64le binary packages for debian stretch are now published. These are the packages we need for the current Docker image.

I'm unable to build using qemu for ppc64le for debian buster, unfortunately; I'm getting a qemu segmentation fault every time python is invoked. I'll try upgrading my qemu install to the latest (direct from source) and see if that works later - it's not a blocker here.

[wohali]

If anyone watching this can test the debian stretch packages on real POWER hardware with a real CouchDB workload, we'd appreciate it.

On a machine with debian stretch installed, follow the instructions here:

http://docs.couchdb.org/en/latest/install/unix.html#installation-using-the-apache-couchdb-convenience-binary-packages

[wohali]

Once again, we are in desperate need of someone to actually test these images and tell me if they work for your needs.

I am starting to think no one actually cares about these builds, and that IBM only pushed for these builds to be available for publicity reasons, not because they have actual use cases in these spaces.

Prove me wrong.

/cc @sarveshtamba

[sarveshtamba]

@wohali someone from my team will try to test this out and will let you know.

[snehlatamohite]

Hi All,

I have started building dockerfile - - https://github.com/apache/couchdb-docker/blob/master/2.3.1-ubi7/Dockerfile on Power , however observed that installing couchdb steps are failing with following error . I have checked that http://apache.bintray.com/couchdb-rpm/el7/ has only x86_64/ folder and this repo is not available for Power.

Error while building dockerfile -
Loaded plugins: ovl, product-id, search-disabled-repos
http://apache.bintray.com/couchdb-rpm/el7/ppc64le/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article
https://access.redhat.com/articles/1320623
If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

Thanks,
Snehlata Mohite.

[willholley]

@snehlatamohite as per the comments above, only Debian stretch is supported at the moment. If you can follow the instructions for Debian at http://docs.couchdb.org/en/latest/install/unix.html#installation-using-the-apache-couchdb-convenience-binary-packages on a POWER machine, that would be helpful.

[snehlatamohite]

@willholley - I will check. Thanks.

[willholley]

passing on from an internal IBM team testing on Power:

I have run "Verify CouchDB Installation" from couchdb URL - http://:5984/_utils/#/verifyinstall and observed that status for "Create View" is passed

[sarveshtamba]

I am trying to install couchdb on RHEL 7 for ppc64le as per http://docs.couchdb.org/en/latest/install/unix.html#installation-using-the-apache-couchdb-convenience-binary-packages

RedHat 7: Place the following text into /etc/yum.repos.d/bintray-apache-couchdb-rpm.repo:

[bintray--apache-couchdb-rpm]
name=bintray--apache-couchdb-rpm
baseurl=http://apache.bintray.com/couchdb-rpm/el7/$basearch/
gpgcheck=0
repo_gpgcheck=0
enabled=1

However after this step I see the following error:-

[root@p006n04 install]# sudo yum -y install epel-release && sudo yum -y install couchdb
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
http://apache.bintray.com/couchdb-rpm/el7/ppc64le/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.



 One of the configured repositories failed (bintray--apache-couchdb-rpm),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=bintray--apache-couchdb-rpm ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable bintray--apache-couchdb-rpm
        or
            subscription-manager repos --disable=bintray--apache-couchdb-rpm

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=bintray--apache-couchdb-rpm.skip_if_unavailable=true

failure: repodata/repomd.xml from bintray--apache-couchdb-rpm: [Errno 256] No more mirrors to try.
http://apache.bintray.com/couchdb-rpm/el7/ppc64le/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
[root@p006n04 install]#

http://apache.bintray.com/couchdb-rpm/el7/ppc64le/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Seems to be missing.

[willholley]

@sarveshtamba ppc64le packages are currently only provided for Debian. You'll need to build from source for RHEL/ppc64le.

[wohali]

Images are now posted for apache/couchdb:2.3.1 and will be part of future releases. We will track merging this for the top-level couchdb image separately.

[wohali]

This is now done in the downstream couchdb image as well:

docker-library/official-images#7524