doc: add scorecard documentation (#942)

**Description of the change:** Docs for the new scorecard subcommand


**Motivation for the change:** Closes #936

CHANGELOG.md

@@ -22,6 +22,7 @@
 - Ansible operator proxy added the cache handler which allows the get requests to use the operators cache. [#760](https://github.com/operator-framework/operator-sdk/pull/760)
 - Ansible operator proxy added ability to dynamically watch dependent resource that were created by ansible operator. [#857](https://github.com/operator-framework/operator-sdk/pull/857)
 - Ansible-based operators have leader election turned on by default. When upgrading, add environment variable `POD_NAME` to your operator's Deployment using the Kubernetes downward API. To see an example, run `operator-sdk new --type=ansible ...` and see file `deploy/operator.yaml`.
+- A new command [`operator-sdk scorecard`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#scorecard) which runs a series of generic tests on operators to ensure that an operator follows best practices. For more information, see the [`Scorecard Documentation`](doc/test-framework/scorecard.md)
 
 ### Changed
 

doc/dev/testing/travis-build.md

@@ -40,6 +40,7 @@ The Go, Ansible, and Helm tests then differ in what tests they run.
     4. Run `test local` with `--up-local` flag.
     5. Run `test local` with both `--up-local` and `--kubeconfig` flags.
     6. Create all test resources with kubectl and run `test local` with `--no-setup` flag.
+    7. Run `scorecard` subcommand and check that expected score matches actual score.
 4. Run [go e2e tests][go-e2e].
     1. Use `operator-sdk` to create and configure a new `memcached-operator` project and install the memcached CRD in the cluster.
     2. Run cluster test (namespace is autogenerated and deleted by test framework).

doc/sdk-cli-reference.md

@@ -307,7 +307,7 @@ should use `up local` instead.
 
 #### Example
 
-```bash
+```console
 $ operator-sdk run ansible --watches-file=/opt/ansible/watches.yaml --reconcile-period=30s
 ```
 
@@ -324,10 +324,57 @@ should use `up local` instead.
 
 #### Example
 
-```bash
+```console
 $ operator-sdk run helm --watches-file=/opt/helm/watches.yaml --reconcile-period=30s
 ```
 
+## scorecard
+
+Run scorecard tests on an operator
+
+### Flags
+
+* `basic-tests` - Enable basic operator checks (default true)
+* `cr-manifest` string - (required) Path to manifest for Custom Resource
+* `csv-path` string - (required if `olm-tests` is set) Path to CSV being tested
+* `global-manifest` string - Path to manifest for Global resources (e.g. CRD manifests)
+* `init-timeout` int - Timeout for status block on CR to be created, in seconds (default 10)
+* `kubeconfig` string - Path to kubeconfig of custom resource created in cluster
+* `namespace` string - Namespace of custom resource created in cluster
+* `namespaced-manifest` string - Path to manifest for namespaced resources (e.g. RBAC and Operator manifest)
+* `olm-tests` - Enable OLM integration checks (default true)
+* `proxy-image` string - Image name for scorecard proxy (default "quay.io/operator-framework/scorecard-proxy")
+* `proxy-pull-policy` string - Pull policy for scorecard proxy image (default "Always")
+* `verbose` - Enable verbose logging
+* `-h, --help` - help for scorecard
+
+### Example
+
+```console
+$ operator-sdk scorecard --cr-manifest deploy/crds/cache_v1alpha1_memcached_cr.yaml --csv-path deploy/memcachedoperator.0.0.2.csv.yaml
+Checking for existence of spec and status blocks in CR
+Checking that operator actions are reflected in status
+Checking that writing into CRs has an effect
+Checking for CRD resources
+Checking for existence CR example
+Checking spec descriptors
+Checking status descriptors
+Basic Operator:
+        Spec Block Exists: 1/1 points
+        Status Block Exist: 1/1 points
+        Operator actions are reflected in status: 1/1 points
+        Writing into CRs has an effect: 1/1 points
+OLM Integration:
+        Owned CRDs have resources listed: 1/1 points
+        CRs have at least 1 example: 0/1 points
+        Spec fields with descriptors: 1/1 points
+        Status fields with descriptors: 0/1 points
+
+Total Score: 6/8 points
+SUGGESTION: Add an alm-examples annotation to your CSV to pass the CRs have at least 1 example test
+SUGGESTION: Add a status descriptor for nodes
+```
+
 ## test
 
 ### Available Commands

doc/test-framework/scorecard.md

@@ -0,0 +1,94 @@
+# Operator Scorecard
+
+The Operator Scorecard is a testing utility included in the `operator-sdk` binary that guides users towards operator best practices
+by checking the correctness of their operators and CSVs. While the Scorecard is in an early
+stage of development, it will gain more functionality and stabilize over time.
+
+## How It Works
+
+The scorecard works by creating all resources required by CRs and the operator. For the operator
+deployment, it also adds another container to the operator's pod that is used to record calls to the API server,
+which are analyzed by the scorecard for various tests. The scorecard will also analyze the CR object itself,
+modifying spec fields and monitoring how the operator responds.
+
+## Requirements
+
+- An operator made using the `operator-sdk` or an operator that uses a config getter that supports reading from the `KUBECONFIG` environment variable (such as the `clientcmd` or `controller-runtime` config getters). This is required for the scorecard proxy to work correctly.
+- Resource manifests for installing/configuring the operator and custom resources (see the [Writing E2E Tests][writing-tests] doc for more information on the global and namespaced manifests).
+- (OLM tests only) A CSV file for your operator.
+
+## Running the tests
+
+The scorecard currently uses a large amount of flags to configure the scorecard tests. You can see
+these flags in the `scorecard` subcommand help text, or in the [SDK CLI Reference][cli-reference] doc. Here, we will highlight a few important
+flags:
+
+- `--cr-manifest` - this is a required flag for the scorecard. This flag must point to the location of the manifest for the custom resource you are currently testing.
+- `--csv-path` - this flag is required if the OLM tests are enabled (the tests are enabled by default). This flag must point to the location of the operators' CSV file.
+- `--namespaced-manifest` - if set, this flag must point to a manifest file with all resources that run within a namespace. By default, the scorecard will combine `service_account.yaml`, `role.yaml`, `role_binding.yaml`, and `operator.yaml` from the `deploy` directory into a temporary manifest to use as the namespaced manifest.
+- `--global-manifest` - if set, this flag must point to all required resources that run globally (not namespaced). By default, the scorecard will combine all CRDs in the `deploy/crds` directory into a temporary manifest to use as the global manifest.
+- `--namespace` - if set, which namespace to run the scorecard tests in. If it is not set, the scorecard will use the default namespace of the current context set in the kubeconfig file.
+
+To run the tests, simply run the `scorecard` subcommand from your project root with the flags you want to
+use. For example:
+
+```console
+$ operator-sdk scorecard --cr-manifest deploy/crds/app_operator_cr.yaml --csv-path deploy/app_operator-0.0.2.yaml
+```
+
+## What Each Test Does
+
+There are 8 tests the scorecard can run:
+
+### Basic Operator
+
+#### Spec Block Exists
+
+This test checks the Custom Resource that is created in the cluster to make sure that it has a spec block. This test
+has a maximum score of 1.
+
+#### Status Block Exists
+
+This test checks the Custom Resource that is created in the cluster to make sure that it has a status block. This
+test has a maximum score of 1.
+
+#### Operator Action Are Reflected In Status
+
+This test makes modifications to each field in the Custom Resources spec block, waits, then verifies that the
+operator updates the status block. This test has a maximum score of 1.
+
+Note: This test is somewhat prone to breakage as it can potentially change a spec field to an
+invalid value. We plan to partially replace this test with user defined tests in the future.
+
+#### Writing Into CRs Has An Effect
+
+This test reads the scorecard proxy's logs to verify that the operator is making `PUT` and/or `POST` requests to the
+API server, indicating that it is modifying resources. This test has a maximum score of 1.
+
+### OLM Integration
+
+#### Owned CRDs Have Resources Listed
+
+This test makes sure that the CRDs listed in the [`owned` CRDs section][owned-crds] of the CSV have a `resources` subsection. This
+test has a maximum score equal to the number of CRDs listed in the CSV.
+
+Note: In the future, this test will verify that all resources modified by the operator are listed in the resources section.
+
+#### CRs Have At Least 1 Example
+
+This test checks that the CSV has an [`alm-examples` section][alm-examples] in its metadatas' annotations. This test has a maximum score of 1.
+
+#### Spec Fields With Descriptors
+
+This test verifies that every field in the Custom Resource's spec section has a corresponding descriptor listed in
+the CSV. This test has a maximum score equal to the number of fields in the spec section of your Custom Resource.
+
+#### Status Fields With Descriptors
+
+This test verifies that every field in the Custom Resource's status section has a corresponding descriptor listed in
+the CSV. This test has a maximum score equal to the number of fields in the status section of your Custom Resource.
+
+[cli-reference]: ../sdk-cli-reference.md#scorecard
+[writing-tests]: ./writing-e2e-tests.md
+[owned-crds]: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md#owned-crds
+[alm-examples]: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md#crd-templates