issues data dump

[karlcow]

It could be useful for statistics purpose to have a dump of the full DB of issues.
So someone could do matching with a local db, or extract the information about their own system for importing.

Not urgent.

[miketaylr]

Right now 100% of the data on issues can be extracted from https://developer.github.com/v3/issues/. We may want to store data in a local db in the future, however (the only thing we store in the db is username and avatar logo URI)

[karlcow]

Related to @miketaylr comment
0a61cb0

[miketaylr]

I guess we could store a copy of all API responses from GitHub in a NOSQL db or something like PostGres which can do JSON. This could be how we implement what I described in #239 (comment)... maybe.

[karlcow]

to karl explore to do the data dump using github repo.

[hallvors]

https://github.com/webcompat/issue_parser/blob/master/dump_webcompat_to_db.py creates a database with fields id, summary, URL and fills it with all data from our existing issues.

(It needs to be copied along with the extract_id_title_url.py script to somewhere it sees the webcompat module to import the 'engine' database pointer).

The plan is to

• Write code to import some data about existing issues to db
• Move the issue tagger bot to run on webcompat server - move bot action to webcompat server miketaylr/web-bugs-issue-tagger#10
• Update the issue tagger bot with code that will dump data to our local DB
• Maybe also generate / update a web-visible JSON file to be used for the JS that will power [Feature] Create UI for a "detect bugs with similar domain names" feature #307

[hallvors]

Should we add a field having the full body to this DB? It's not hard.

[hallvors]

This branch:
https://github.com/webcompat/webcompat.com/tree/dbdump
has code that dumps data about new bugs to a local SQLite database file called issues.db

It's tested locally and seems to work fine. It might lack a tiny bit of polish, I'll do pep8 and flake8 and such shortly before doing a pull request.

It has a field called "url" but should perhaps have a "domain" field?

The webhooks stuff on that branch covers new issues. Additionally, there's code here:
https://github.com/webcompat/issue_parser/blob/master/dump_webcompat_to_db.py
that can be run at a suitable point in time to grab all the data for existing issues and adding them to the db. This is more of a one-off operation, you would place that file and extract_id_title_url.py next to the webcompat folder and run.py, log in with ssh and run the dump_webcompat_to_db.py script, and remove those two again.

Comments welcome, but feel free to ignore small issues that pep8/flake8 will complain about :)

[karlcow]

Hmm. @hallvors took the ownership. Let me change that.

[hallvors]

Indeed I took over this, and I didn't even plan to, just happened to think that the stuff in issue_parser repo could be extended slightly to make a DB dump - and went from there.. :)

Now, we're still not grabbing several details that matter: labels, comments.. It's probably easyish to do, but just as a sort of backup feature. The issues DB we'll be using to generate RSS and JSON data, but a comments DB would just be an "in case we want to move off GitHub" thing. So I don't think that's a priority..

[hallvors]

(Of course one use case for more data in the DB is if we need to or ought to use GitHub's APIs less. This might also make stuff faster and make it easier to build more markup on the server and use less JS? We'd have potential syncing complications though..)

[miketaylr]

@hallvors we assigned this to @karlcow in Paris. I'm sure he's got plenty of other things to do right now, but in the future can you give a heads up or ask if he's not in the middle of something before taking over? 🚔

[hallvors]

Sure. I'll try to recognise the line between "exploring an idea" and "taking over a task" ;)

[karlcow]

To note that this issue was about creating an "issues data dump", which is not totally related to what @hallvors is trying to do here, which is a lot more complicated.

The initial issue is

It could be useful for statistics purpose to have a dump of the full DB of issues.
So someone could do matching with a local db, or extract the information about their own system for importing.

Somehow this is more but in a different way :)
https://github.com/webcompat/issue_parser/blob/master/dump_webcompat_to_db.py
(which I didn't know before the comment here).

What @hallvors is trying to do (and is useful) would deserve a separate issue (something like Live Issues DB) and relates a lot more to my comment in #327 (comment)

Right now the proposed code is one way.

1. someone enters data
2. data gets pushed to a DB on the opened event through the webhook.

Some cases for out of sync of the db:

• If someone fixes the data on github
• if someone deletes the issue on github
• if someone fills the issue directly on github

With the Live Issues DB, we could indeed send templates with more information and resync if it's slightly out of date, etc. aka using the xhr stuff for its purpose. That would be neat.

[karlcow]

As for

Sure. I'll try to recognise the line between "exploring an idea" and "taking over a task" ;)

nitpick. Recommendations:

1. Read the initial comment of the issue
2. Ask about it in the comments
3. Discuss about it in the issue through the comments.
4. Nothing is really urgent, allow time for people to reply, think about it.

;)

[karlcow]

Replying to my own comment:

What @hallvors is trying to do (and is useful) would deserve a separate issue (something like Live Issues DB) and relates a lot more to my comment in #327 (comment)

This is partly #327 in fact.

[hallvors]

[Edit: removing process discussion, let's keep this on topic]

• Creating an issue directly on GitHub should still trigger the webhook, so no problems there
• To keep our "slave db" in sync we also need webhooks for issue deletion and updates - should be easy though. But that's better handled in a different issue, like @karlcow suggests. So I've opened Set up live "backup" database on webcompat.com server, updated from suitable webhooks #788 to continue the discussion there.

[hallvors]

(Given this state of affairs though - this bug could be considered fixed by the dump_webcompat_to_db.py script. If someone wants a local DB with a dump of all the issues in webcompat/web-bugs/issues all they need to do is to clone that repo and run the script AFAIK. @karlcow , thoughts?)

[hallvors]

Closing this since code to create a local "data dump" has been written.

[deepthivenkat]

@karlcow @hallvors I'm trying to run the dump_webcompat_to_db.py to dump the data from github to issues.db

I tried the below steps.

1. I have cloned webbugs, issueparser and webcompat repo.
2. As mentioned above, I have the copy of dump_to_db.py and extract_id_title_url.py inside the webcompat.com repo.
3. When i ran dump_to_db.py, I get the below error.

Am i missing anything here?

[karlcow]

@deepthivenkat

The error message says it:
ImportError: no module named db.

Basically you got a module which is not installed. BUT

1. Avoid to clutter your current webcompat working space with other scripts, not part of the projet, you might end up adding and committing them to the project.
2. create a specific working environment for this script.
3. If you want I can look at https://github.com/webcompat/issue_parser and straighten it a bit to make it more beginner friendly before you work on it. This could avoid future mistakes.