Support for Bitbucket, Sonar and Squash

[Avfdo1994]

Is it possible to use perceval with bitbucket, sonar and squash?

[acs]

The current supported data sources are described at:

https://github.com/chaoss/grimoirelab-perceval/blob/master/README.md

There is no support for those data sources.

[acs]

http://www.squashtest.org/en squash data source is this one?

[Avfdo1994]

@acs Yes, it's http://www.squashtest.org/en

[Avfdo1994]

Is there any alternative/workaround of using Bitbucket with Grimoire?

[jgbarah]

Is there any alternative/workaround of using Bitbucket with Grimoire?

If you're interested only in git repos, you can easily use the git backend. If you're interested in pull requests, they are not supported yet, a Perceval backend would be needed.

[sergiocasaleiro]

If you're interested only in git repos, you can easily use the git backend.

Does this mean interested on using Perceval with Bitbucket without the PRs?
If so, how can we specify the authentication?
Thanks!

[jgbarah]

Does this mean interested on using Perceval with Bitbucket without the PRs?

Yes.

If so, how can we specify the authentication?

If the repository is public, nothing to do, since Perceval uses git for accessing it, and there are no calls too the Bitbucket API. If the repository is private, if you already have your account configured to access it, it should just work, because Perceval is basically running git clone. Please, let us know if it does not work in either of these cases: to be honest, my experience with analyzing Bitbucket is not that large.

[aaronnickovich]

I'd like to contribute a bitbucket interface for this project. bitbucket has similar APIs to github.

[sduenas]

That would be great! Please take as reference github.py and gitlab.py as reference.

[valeriocos]

Hi @aaronnickovich, thank you for your interest! If you need some help don't hesitate to ping us! Feel also free to share your branch and we can work on it together!

[sarthak0906]

Hello everyone,
Is anyone working on the sonarqube interface for the project? If it is still open I would like to contribute to it.

[valeriocos]

Hi @sarthak0906! You're contribution is more than welcome.

Please have a look to the github.py and gitlab.py as reference. You can also have a look at a prototype of the sonarqube backend here. Don't hesitate to share your branch or to write us if you have any questions.

Thanks for contributing to GrimoireLab!

[aaronnickovich]

I have a question for the community. I've been working on a bitbucket implementation and the public domain bitbucket uses an API V2. However, enterprise bitbucket is typically using the v1 API. They are very different.
Would anyone like to help suggest if Perceval should have a bitbucketv1 and a bitbucket, or would it be better to do a bitbucket interface with something like an --api-version flag that will select the api version? I believe there is a deprecation warning issued by bitbucket so I'm worried about maintanence as well. Please look at my branch.

[jgbarah]

From my point of view, having a --api (or similar) flag would be the way to go (even if we start with a single API supported). This happens not only in Bitbucket, we have other data sources with different APIs too.

Maybe a reason for not having that flag would be that it can be automagically inferred from a simple query to the API which version it is implementing.

[aaronnickovich]

From my point of view, having a --api (or similar) flag would be the way to go (even if we start with a single API supported). This happens not only in Bitbucket, we have other data sources with different APIs too.

Maybe a reason for not having that flag would be that it can be automagically inferred from a simple query to the API which version it is implementing.

These data sources you refer to appear to be minor changes in how to fetch something with a slightly different server version. For instance, mediawiki and gerrit. When I talk about bitbucket APIs, it's actually two different products offered by Atlassian:
Bitbucket Server: https://docs.atlassian.com/bitbucket-server/rest/6.7.1/bitbucket-rest.html
Bitbucket Cloud: https://developer.atlassian.com/bitbucket/api/2/reference/resource

Bitbucket SERVER has different versions like 5.10 and 6.7, and the rest interface do not seem to change much even between the major versions.

The cloud vs server is, however, so vast it's like trying to combine github and gitlab together. It'd honestly be similar to switch between the two classes like this:
if( --api == 1.0):
use github.backend and github.client
elif (--api == 2.0):
use gitlab.backend and gitlab.client

What I know absolutely today is this:
bitbucket.org (CLOUD) === API v2
bitbucket.{some-enterprise.com} (SERVER) == API v1

What I don't know now or in future plans:
bitbucket.{some-enterprise.com} == v2

To begin I thought perceval should have a "bitbucket" data collection implementation for both cloud and server, but it's so different that it's not very nice using if, else to separate them out.

With respect to integration with SirMordred, auto-magic is nice to have.

[sarthak0906]

Hey everyone.
I have a doubt regarding the sonarqube backend, What all data should be fetched from the sonar database.

[valeriocos]

Hi @sarthak0906, sorry for the late reply.
I can have a closer look to the sonar API, but it seems that following endpoint should be enough https://sonarcloud.io/web_api/api/measures. The backend could receive the name of a component and then iterate over the measures for each metric, WDYT?

[sduenas]

I have a question for the community. I've been working on a bitbucket implementation and the public domain bitbucket uses an API V2. However, enterprise bitbucket is typically using the v1 API. They are very different.
Would anyone like to help suggest if Perceval should have a bitbucketv1 and a bitbucket, or would it be better to do a bitbucket interface with something like an --api-version flag that will select the api version? I believe there is a deprecation warning issued by bitbucket so I'm worried about maintanence as well. Please look at my branch.

I have a doubt. Does data have the same fields and structure depending which version you use?

[aaronnickovich]

I have a question for the community. I've been working on a bitbucket implementation and the public domain bitbucket uses an API V2. However, enterprise bitbucket is typically using the v1 API. They are very different.
Would anyone like to help suggest if Perceval should have a bitbucketv1 and a bitbucket, or would it be better to do a bitbucket interface with something like an --api-version flag that will select the api version? I believe there is a deprecation warning issued by bitbucket so I'm worried about maintanence as well. Please look at my branch.

I have a doubt. Does data have the same fields and structure depending which version you use?

Hi @sduenas,
No the input and output of each have about as much in common as gitlab and github. It's actually bitbucket server and bitbucket cloud which look like 2 different product offerings.

The field for 'name' in cloud is 'slug' in bitbucket server. Same goes for 'uuid' vs 'id', respectively. Cloud offers a CI/CD build tool and an issues page which are missing from the server implementation.

I haven't worked on ELK but I understand it adds some metadata. I think I see where you're going with this. If these two produce different data, then it should be treated as a different perceval agent.

[sarthak0906]

Hi @sarthak0906, sorry for the late reply.
I can have a closer look to the sonar API, but it seems that following endpoint should be enough https://sonarcloud.io/web_api/api/measures. The backend could receive the name of a component and then iterate over the measures for each metric, WDYT?

That does sound amazing
Thanks for the help
And sorry for late reply

[sarthak0906]

Hey, I had a doubt,
How can I test for a new backend from the command line?
Also is there any gitter chat or any other platform to connect and ask a few more doubts if needed.

[valeriocos]

HI @sarthak0906,

You can install Perceval with the new backend in a virtual env (see details below) or test it from pycharm (you should create a configuration and pass the params, details about how to run Perceval with Pycharm can be found here: https://github.com/inishchith/chaoss-microtasks/tree/master/microtask-1, an example of configuration is at [1])

python3 -m venv /tmp/mytest
cd /tmp/mytest
source bin/activate
cd <perceval-path>
pip install . -e    
perceval --help <--- check that perceval has been installed

[1]

We can use the IRC channel (#chaoss-community irc channel in FreeNode.net , more info at https://chaoss.community/participate/) or set up a Skype/Google Hangouts call. When would you like to meet?

[aaronnickovich]

Hi @sarthak0906 and @valeriocos,
I actually just started adding tests for one of the backends this week. I figured out a test workflow for myself that works for me. I like to run tests with code coverage reports to see what areas I'm missing in the test cases so I'm not making redundant tests.

Once you have run the steps mentioned by valeriocos above to setup your environment:

python3 -m venv /tmp/mytest
cd /tmp/mytest
source bin/activate
cd <perceval-path>
pip install . -e    
perceval --help <--- check that perceval has been installed

I like to run tests like this using cli:

pip3 install pytest-cov coverage
cd <base_directory_of_perceval>/perceval/backends/core
py.test ../../../tests/test_bitbucketserver.py --cov=perceval.backends.core.bitbucketserver --cov-report html --cov-report term-missing
firefox ./htmlcov/index.html          <---- view graphical test coverage 

the test coverage outputs an index.html where I can browse for covered, missed, or skipped code like this: https://nedbatchelder.com/files/sample_coverage_html/index.html

[valeriocos]

Nice pointer @aaronnickovich!

I use just coverage and execute it twice:

• coverage/__main__.py run run_tests.py --source=perceval to generate the .coverage file
• coverage/__main__.py report -m to get a textual output from .coverage as the one below

/home/slimbook/Escritorio/sources/venv/bin/python3 /usr/local/lib/python3.6/dist-packages/coverage/__main__.py report -m
Name                                                                                                                                             Stmts   Miss  Cover   Missing
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
...
/home/slimbook/Escritorio/sources/perceval/perceval/backend.py                                                                                     394      6    98%   384, 500, 610-613
/home/slimbook/Escritorio/sources/perceval/perceval/backends/__init__.py                                                                             1      0   100%
/home/slimbook/Escritorio/sources/perceval/perceval/backends/core/__init__.py                                                                        2      0   100%
/home/slimbook/Escritorio/sources/perceval/perceval/backends/core/askbot.py                                                                        230      3    99%   306-307, 435
/home/slimbook/Escritorio/sources/perceval/perceval/backends/core/bugzilla.py                                                                      230      2    99%   224-225

[sarthak0906]

I'm still trying to complete the task before I go on to test it,
but it definitely is very helpful and will remember it

[imnitishng]

Hi @valeriocos. I see no activity regarding this issue, I have also been learning about Bitbucket API for some time. I believe implementing a Bitbucket backend similar to github for retrieving Issues and Pull requests might be of use. I have made some progress in implementing the same.
Hence, can I take up this issue?

[valeriocos]

Thank you @imnitishng ! Before starting please have a look at the API doc. If I remember correctly there were two different versions of the API, which have some pretty big differences. Thanks!

[imnitishng]

Well yes there are different versions, but what I am aiming for as of now is the basic support of retrieving issues and PRs from a bitbucket repository (just like github).
The API looks pretty straightforward for doing that task. I will get to you as I discover more or face any problems.
Do we open a new issue or discussion is fine here?

[valeriocos]

Great @imnitishng !

Do we open a new issue or discussion is fine here?

This thread is fine, thanks!

[vchrombie]

The bitbucket backend created by @imnitishng is ported to a new repository perceval-backends/grimoirelab-perceval-bitbucket. The work is still pending, but you can try the backend.

Feel free to contribute if you are interested.

[sduenas]

Closing this because there's no activity on this issue. As @vchrombie said, Bitbucket is already available.