Add support for generating based on scraping the Github API.

[coderanger]

Includes general support for repo hosts and a specific implementation for Github.

I'm not sold on the naming of "repo host" but I couldn't think of anything better to cover future expansion to GitLab, Bitbucket, etc.

Closes #216

[CLAassistant]

All committers have signed the CLA.

[coderanger]

An example YAML:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: testing
  namespace: default
spec:
  generators:
  - repoHost:
      github:
        organization: myorg
        tokenRef:
          name: github-token
          key: token
      filters:
      - repositoryMatch: ^service-.*
        pathExists: kubernetes/kustomization.yaml
  template:
    metadata:
      name: '{{ repository }}'
    spec:
      source:
        repoURL: '{{ url }}'
        targetRevision: '{{ branch }}'
        path: kubernetes/
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: default

[OmerKahani]

Can you please also add docs, an example, and E2E tests?

Very nice generator and use case!

[OmerKahani]

@coderanger try to rebase / merge master, #207 should have fixed the problem in "Verify generated manifests"

[coderanger]

A question raised from a friend: should this support discovering branches as well as repos. Right now its hardwired to only consider the default branch of a repo. Seems like a thing that could be added in the future, another filter and a flag for allBranches: true or something.

[jgwest]

Really nice PR @coderanger , thanks for contributing this! I've got some suggestions around name changes, and some doc/comment tweaks below, but nothing major... let me know what you think 😄 .

[coderanger]

Updated those three todos so they now all exist and work :)

[coderanger]

Okay, I renamed everything to "SCM provider" as the overall subsystem name and reworked the multi-filter behavior to be simpler (the OR/AND approach discussed above).

[jgwest]

@coderanger - FYI if you see an E2E test failure in the PR check, it's not anything wrong with your code, it's this: #227 (I'm working on a fix).

[jgwest]

Looking great, one tweak to the example and good to go (unless you had anything else you wanted to add)

[coderanger]

I think that's everything I can think of to add to start with :) I'm sure when folks with other use cases get their hands on it, there will be more options and filters that we can ponder but I think this covers the bulk of it.

[jgwest]

Really nice, thanks for working on this! It adds a powerful new tool into the ApplicationSet tool belt, and should really enable folks doing GitOps over multiple repositories in an organization to best automate that using ApplicationSets 🎉.