graph-builder: add plugin to fetch openshift/cincinnati-graph-data from github

[steveej]

Add a github_openshift_secondary_metadata_scraper plugin:

• Implement github v3 API helper module with deserialization tests
• Implement plugin logic
• Test to fetch production data from
  https://github.com/openshift/cincinnati-graph-data

This plugin is in preparation for a subsequent plugin which will be able
to parse the fetched data.

---

• Plugin to fetch and unpack repository
  • Impl
  • Network dependent tests
  • Independent tests? e.g. mock Github API and have a custom generated tarball?
• Add the legacy metadata to a raw metadata JSON in the data repository (raw/metadata.json: add historical metadata from quay cincinnati-graph-data#74)
• Double-check if any blocking code should be moved to the blocking thread-pool
• Change graph-builder default configuration to use these plugins instead of the quay secondary metadata fetcher moved to graph-builder: use openshift/cincinnati-graph-data instead of quay labels #232.

[steveej]

the OWNERS files of the test fixtures were tripping up prow.

/verify-owners

[openshift-ci-robot]

The following users are mentioned in OWNERS file(s) but are not members of the openshift org.

Once all users have been added as members of the org, you can trigger verification by writing /verify-owners in a comment.

• cincinnati-graph-data-approvers
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/blocked-edges/OWNERS
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/channels/OWNERS
• cincinnati-graph-data-reviewers
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/blocked-edges/OWNERS
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/channels/OWNERS
• release-admins
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/blocked-edges/OWNERS
  • graph-builder/src/plugins/openshift_secondary_metadata_parser/test_fixtures/20200220.104838/cincinnati-graph-data/channels/OWNERS

[steveej]

/retest

[vrutkovs]

LGTM other than a few typos

[vrutkovs]

I'm not too happy about binding Cincinnati to Github. Could it be reworked into a generic git repo (use clone instead of fetching data as a tarball)? This would make disconnected installs easier

[steveej]

I'm not too happy about binding Cincinnati to Github. Could it be reworked into a generic git repo (use clone instead of fetching data as a tarball)? This would make disconnected installs easier

I understand your concern and I can assure you that making GitHub a hard requirement is not a goal.
The next step from here is to finish the graph-builder plugin configuration support.
That will allow us to configure the graph-builder so that the github scraper is disabled, and only the data parser is enabled. The data repository can be transferred and provided to the graph-builder by any means.
This doesn't exclude building additional plugins which download the archive from a fixed URL, shell out to git, use a git library, etc.

[vrutkovs]

The data repository can be transferred and provided to the graph-builder by any means.

Ah, that makes sense (since data would be basically checked out files if I understood correctly)

[steveej]

Ah, that makes sense (since data would be basically checked out files if I understood correctly)

Yes. The plugin in this PR downloads and extracts them, while the plugin in #231 parses the extracted files. I will pick up #74 soon which will implement the missing piece.

[steveej]

/retest

[vrutkovs]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: steveeJ, vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [steveeJ,vrutkovs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment