README

Important notice (2018-09-19)
-----------------------------

This project lacked active development for a long while, and it is very
unlikely that it gets develpment attention in the future. For those
interested in retrieving and processing data from issue tracking repositories
(bug tracking repositories), please consider checking GrimoireLab [1] and
GrimoireLab-Perceval [2] 

[1] https://chaoss.github.io/grimoirelab
[2] https://github.com/chaoss/grimoirelab-perceval


Description
-----------

Bicho is a command line-based tool used to parse bug/issue tracking
systems. It gets all the information associated with issues and stores
them in a relational database. (It is part of the MetricsGrimoire suite,
which produces data for vizGrimoire to analyze and visualize.)

Currently Bicho supports:
- Bugzilla
- Sourceforge.net (abandoned)
- JIRA
- Launchpad
- GitHub
- Maniphest
- Redmine
- Gerrit
- Allura (unstable)
- Google Code (abandoned)
- Trac


 License
---------

Bicho is licensed under GNU General Public License (GPL), version 2 or later.


 Download
----------

Home page:
* http://metricsgrimoire.github.com/Bicho/

Releases:
* https://github.com/MetricsGrimoire/Bicho/downloads

Latest version:
* git://github.com/MetricsGrimoire/Bicho.git


 Requirements
------------

 * Python >= 2.4
 * Python Storm. You'll also need the following Python libraries:
   - mysqldb (default engine should be set to MYISAM)
   - python-launchpadlib (only for for Launchpad backend)
 * Beautiful Soup library: error-tolerant HTML parser for Python
 * python-feedparser
 * dateutil


 Installation
-------------

 You can install Bicho running the setup.py script:

  # python setup.py install

 For the impatients:

  $ bicho --help


 Running Bicho
--------------

This is a quick list of example commands you would run in your terminal to
get bug data from various kinds of bug trackers and capture it in a
database on your local machine. The general format is:

$ bicho --db-user-out=[YOUR DATABASE USERNAME] --db-password-out=[YOUR DATABASE PASSWORD] --db-database-out=[NAME OF DATABASE] -d [DELAY BETWEEN REQUESTS IN NUMBER OF SECONDS] -b [ABBREVIATION FOR BACKEND] -u [BUGTRACKER URL IN QUOTES]

For more guidance, please see doc/UserManual.txt .

It is very important to use a delay. If you run Bicho against big sites
without a delay between bug queries, your IP address could be banned!

E1. Getting information from a project that uses Bugzilla, like Bicho ;)

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -d 15 -b bg -u "https://bugzilla.libresoft.es/buglist.cgi?product=bicho"

E2. Getting information from a project hosted on sourceforge.net

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -d 15 -b sf -u "http://sourceforge.net/tracker/?atid=516295&group_id=66938"

E3. Getting information from a project using JIRA

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -d 15 -b jira -u "http://support.petalslink.com/browse/PETALSMASTER"

E4. Getting information from a project using Launchpad

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -d 15 -b lp -u "https://bugs.launchpad.net/openstack"

E5. Getting information from a project using Allura

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -d 15 -b allura -u "http://sourceforge.net/rest/p/allura/tickets"

E6. Getting information from a project using GitHub

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -b github -u "https://api.github.com/repos/composer/composer/issues" --backend-token=[API TOKEN]

E7. Getting information from a project using Redmine

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] --backend-user=[REDMINE USER] --backend-password=[REDMINE PASSWORD] -d 1 -b redmine -u "https://www.bitergia.net/"

E8. Getting information from Maniphest

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] --backend-token=[API TOKEN] -b maniphest -u https://phabricator.wikimedia.org [--no-resume]

E9. Getting information from Trac

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -b trac -u https://fedorahosted.org/freeipa/

E10. Getting information from Review Board

$ bicho --db-user-out=[DB USER] --db-password-out=[DB PASS] --db-database-out=[DB NAME] -n 25 -b reviewboard -u https://reviews.apache.org/groups/geode/


Known issues
------------

Newer versions of MySQL server may raise the next error message:
```
Error Code: 1406. Data too long for column
```

To avoid this problem, please update your MySQL configuration removing `STRICT_TRANS_TABLES` value from `sql-mode` parameter.


 Roadmap
---------

0.93:
* The updated list of bugs to be fixed can be found here https://github.com/MetricsGrimoire/Bicho/issues?milestone=1&page=1&state=open
* Incremental support broken by issues updated during the download bug #28
* Incorrect order downloading issues from Bugzilla #20
* Incoherent number of issues after webkit analysis bug bugzilla support #26
* Error in database character sets while comparing dates #8
* Problem cloning repo in case insensitive systems #12
* Incremental feature doesn't support multiple projects in the same database #30

1.0:
* https://github.com/MetricsGrimoire/Bicho/issues?milestone=2&page=1&state=open
* issues_log for bugzilla and launchpad
** Launchad support for issues_log table enhancement launchpad support #24
** More efficient and cleaner code for the table issues_log for bugzilla
* New table with information about executions (date, issues downloaded, etc ..)
* Tests, tests and tests
* Improved debug mode with more useful details
* Network fault tolerance (in order to survive to connection issues)
* New backends:
** FusionForge


 Improving Bicho
----------------

Source code, wiki and ITS available on GitHub:
* https://github.com/MetricsGrimoire/Bicho

Please write to the developers mailing at
* metrics-grimoire _at _ lists.libresoft.es

If you want to receive updates about new versions, and keep in touch
with the development team, consider subscribing to the list. It is a
very low traffic list (< 1 msg a day):

* https://lists.libresoft.es/listinfo/metrics-grimoire


 Credits
--------

Bicho has been originally developed by the GSyC/LibreSoft group at the
Universidad Rey Juan Carlos in Mostoles, near Madrid (Spain). It is
part of a wider research on libre software engineering, aiming to gain
knowledge on how libre software is developed and maintained.


 FAQ
----

F1. Bicho crashed with 'UnicodeEncodeError' exception

UnicodeEncodeError appears when it is not possible to write the data in the
database with the encoding used by this one. To avoid that, set your database
to use UTF-8. For instance:

CREATE DATABASE [DB NAME] CHARACTER SET utf8 COLLATE utf8_unicode_ci;

F2. What is the database schema?

There is a nice PNG schema in the directory /doc/database .

F3. How can I create a new backend?

Tell us through the contact information above that you want to create a new
backend. We'll try to give you as much information as possible.

Whenever possible, we want to use available APIs. However, some old
bugtrackers used to not have APIs, and even today, some APIs don't give us
all of the information that we want. So a new backend needs to be able to
fall back to HTML scraping and parsing via Beautiful Soup in cases where the
bug tracker's API doesn't exist (maybe an old version of Bugzilla/JIRA/etc.)
or doesn't provide stuff we want.

While writing a new backend, please also write tests per tests/README.md .

F4. How can I submit a bug report?

Use the GitHub issue tracker: https://github.com/MetricsGrimoire/Bicho/issues .