graph-builder/plugins/github scraper: add token authorization handling #233

steveej · 2020-02-27T15:49:56Z

This implements reading a token from a file and setting the authorization header on the request towards the GitHub API.

It also adjusts the deployment template.

Github API token for CI cincinnati: change to new secrets release#7356
Github API token for stage https://gitlab.cee.redhat.com/service/app-interface/merge_requests/3392
Based on settings: use occurrences notation for file settings #245

dist/openshift/cincinnati.yaml

graph-builder/src/plugins/github_openshift_secondary_metadata_scraper/plugin.rs

steveej · 2020-02-27T18:19:28Z

e2e flake: level=fatal msg="failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply using Terraform"

/retest

steveej · 2020-02-27T21:06:58Z

more flakes: level=error msg="Error: Error waiting for AMI (ami-0b526b366cf36f573) to be ready: timeout while waiting for state to become 'available' (last state: 'pending', timeout: 40m0s)"

/retest

Required to use github token in e2e runs, see openshift/cincinnati#233

vrutkovs · 2020-03-02T16:58:13Z

dist/openshift/cincinnati.yaml

@@ -174,7 +176,7 @@ objects:
          volumes:
            - name: secrets
              secret:
-                secretName: cincinnati-registry-credentials
+                secretName: cincinnati-credentials


hack/e2e.sh is still using cincinnati-registry-credentials.

I think this needs two template options:

CREDENTIALS_SECRET_NAME (set by default to stage github token secret)

CINCINNATI_GITHUB_SCRAPER_OAUTH_TOKEN_PATH (set to github token path by default)

and have hack/e2e.sh adjusted to create a dummy secret and pass -p CINCINNATI_GITHUB_SCRAPER_OAUTH_TOKEN_PATH="" in oc new-app.

This way we can have:

CREDENTIALS_SECRET_NAME in PR e2e set to docker credentials and CINCINNATI_GITHUB_SCRAPER_OAUTH_TOKEN_PATH set to none. e2e test for PRs won't use github token for authentication

stage / prod deploy would mount the secret and use token-based authentication

This is similar to RUST_BACKTRACE env var

Since CINCINNATI_GITHUB_SCRAPER_OAUTH_TOKEN_PATH="" will lead to the error, I tried to find a way to leave this variable unset in the e2e environment. Please see the latest commit and let me know if it's a viable approach or not.

That would work, but wouldn't template look cleaner if Cincinnati support empty path in CINCINNATI_GITHUB_SCRAPER_OAUTH_TOKEN_PATH?

In my mind that's a workaround and I would prefer to indicate the absence of the secret by the absence of the environment variable, instead of special casing the "" value.

If you feel strongly against this exercise I can be convinced otherwise.

steveej · 2020-03-09T16:07:29Z

/retest

openshift-ci-robot · 2020-03-11T11:05:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, steveeJ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [LalatenduMohanty,steveeJ]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vrutkovs · 2020-03-11T13:26:13Z

/hold cancel

LGTM

In preparation for consuming the 'version' file that graph-data grew in [1]. This gets us the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

Consuming the 'version' file that graph-data grew in [1], so the logs will contain more useful error messages when the plugin is fed an incompatible schema version. Bumping the graph-data commit used for CI moves us to the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

In preparation for consuming the 'version' file that graph-data grew in [1]. This gets us the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

Consuming the 'version' file that graph-data grew in [1], so the logs will contain more useful error messages when the plugin is fed an incompatible schema version. Bumping the graph-data commit used for CI moves us to the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

In preparation for consuming the 'version' file that graph-data grew in [1]. This gets us the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

Consuming the 'version' file that graph-data grew in [1], so the logs will contain more useful error messages when the plugin is fed an incompatible schema version. Bumping the graph-data commit used for CI moves us to the first main-line commit to include the 'version' file: cincinnati-graph-data$ git --no-pager show --date=short --format='%ad %h %s' d980578d2e b87e7c2782 2020-03-18 d980578 Merge pull request openshift#125 from marun/unique-service-ca-serial 2020-05-11 b87e7c2 Merge pull request openshift#233 from wking/version-file [1]: openshift/cincinnati-graph-data#233

steveej requested review from vrutkovs and LalatenduMohanty February 27, 2020 15:49

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 27, 2020

openshift-ci-robot requested review from jottofar and tparikh February 27, 2020 15:50

steveej mentioned this pull request Feb 27, 2020

graph-builder: use openshift/cincinnati-graph-data instead of quay labels #232

Merged

6 tasks

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 27, 2020

vrutkovs reviewed Feb 27, 2020

View reviewed changes

dist/openshift/cincinnati.yaml Outdated Show resolved Hide resolved

vrutkovs reviewed Feb 27, 2020

View reviewed changes

graph-builder/src/plugins/github_openshift_secondary_metadata_scraper/plugin.rs Outdated Show resolved Hide resolved

steveej force-pushed the pr/plugin-gh-scraper-tokenauth branch from 2534549 to 3113fb0 Compare February 27, 2020 21:08

vrutkovs mentioned this pull request Feb 28, 2020

WIP Cincinnati: mount credentials secret in e2e test container openshift/release#7370

Closed

vrutkovs added a commit to vrutkovs/release that referenced this pull request Mar 2, 2020

Cincinnati: mount credentials secret in e2e test container

b733b31

Required to use github token in e2e runs, see openshift/cincinnati#233

vrutkovs reviewed Mar 2, 2020

View reviewed changes

steveej force-pushed the pr/plugin-gh-scraper-tokenauth branch 2 times, most recently from dbad675 to 32c5ab5 Compare March 2, 2020 19:09

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 6, 2020

steveej force-pushed the pr/plugin-gh-scraper-tokenauth branch from 32c5ab5 to c4f443d Compare March 9, 2020 14:31

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 9, 2020

steveej force-pushed the pr/plugin-gh-scraper-tokenauth branch 5 times, most recently from 30fe23d to 109c3b3 Compare March 10, 2020 09:35

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 11, 2020

openshift-merge-robot merged commit f234448 into openshift:master Mar 11, 2020

steveej deleted the pr/plugin-gh-scraper-tokenauth branch March 11, 2020 13:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

graph-builder/plugins/github scraper: add token authorization handling #233

graph-builder/plugins/github scraper: add token authorization handling #233

steveej commented Feb 27, 2020 •

edited

Loading

steveej commented Feb 27, 2020

steveej commented Feb 27, 2020

vrutkovs Mar 2, 2020

steveej Mar 2, 2020

vrutkovs Mar 2, 2020

steveej Mar 2, 2020

steveej commented Mar 9, 2020

openshift-ci-robot commented Mar 11, 2020

vrutkovs commented Mar 11, 2020

graph-builder/plugins/github scraper: add token authorization handling #233

graph-builder/plugins/github scraper: add token authorization handling #233

Conversation

steveej commented Feb 27, 2020 • edited Loading

steveej commented Feb 27, 2020

steveej commented Feb 27, 2020

vrutkovs Mar 2, 2020

Choose a reason for hiding this comment

steveej Mar 2, 2020

Choose a reason for hiding this comment

vrutkovs Mar 2, 2020

Choose a reason for hiding this comment

steveej Mar 2, 2020

Choose a reason for hiding this comment

steveej commented Mar 9, 2020

openshift-ci-robot commented Mar 11, 2020

vrutkovs commented Mar 11, 2020

steveej commented Feb 27, 2020 •

edited

Loading