Update Docs (#74)

* updating landing readme

* removes the quick start from the homepage, merges main

* refactors how the docs are configured to run with make docs-serve

* renames the object section to resources, fixes a README typo, cleans up the getting started guides

* checkpoints the deployment guide before it starts to get truly messy

* checkpoint deployment guide

* lander update

finished writing the "five easy steps" though it's really not that easy

* deployment.md 

Reworded, reorg'ed and reduced repeats

* Update README.md

style tings

Co-authored-by: Thomas La Piana <tal103020@icloud.com>

Makefile

@@ -183,4 +183,5 @@ compose-build:
 
 .PHONY: docs-serve
 docs-serve:
+	@docker-compose build docs
 	@docker-compose up docs

README.md

@@ -5,46 +5,135 @@
 [![Code style: black][black-image]][black-url]
 [![Checked with mypy][mypy-image]][mypy-url]
 
-## Overview
-
-Fides (Latin: Fidēs) enables engineers and data teams to declaratively enforce data privacy requirements within the Software Development Life-Cycle.
-
-With Fides, anyone working with risky types of data (e.g. personally identifiable information), can declare or describe their data intentions and Fides will continually evaluate compliance and warn users of unsafe changes _before_ they make it into production.
-
-This approach ensures that privacy is described within your source code, thereby making privacy easier to manage and a proactive part of your existing software development practices.
-
-## Principles
-
-* Data Lineage Declarations
-* Privacy controls at the CI layer
-* Predefined Privacy Taxonomy
-* Translation layer between engineers and lawyers
-
-## Quick Guide
-
-To make things more concrete, the following is a brief overview of the steps required to set up a new project with Fides as used by a monorepo:
-
-1. Create a new directory for your Fides objects to live in, for examples `fides_manifests/`.
-
-1. The next step is to define Fides objects as manifest files. This would include defining datasets, extending the privacy classifiers, and anything else needed to describe the state of the project's privacy.
-
-1. Apply the manifests using `fidesctl apply fides_manifests/`. This command will create/update objects via the Fides API.
-
-1. Set up a CI pipeline to run when the system file is changed. It should use the `fidesctl dry-evaluate <system_manifest> <system_key>` command to check that a system is still valid after it has been update.
-
-1. Upon merge to the main branch, a pipeline should run to re-apply the `fides_manifests/` folder.
-
-For more information on getting started, see the [tutorial docs]().
-
-## Resources
+## :zap: Overview
+
+Fides (*fee-dez*, Latin: Fidēs) is the modern framework for data teams to implement data privacy requirements using all your existing CI/CD tools.
+
+- **A Privacy Grammar for Engineers.** Fides is a way for you to declare when, where and how you plan to use risky types of data (e.g. personally identifiable information) directly in your code. 
+
+- **Privacy Policies That Aren't Just for Lawyers.** Fides allows you to make a privacy policy that's *actually* enforced at time of integration and deployment.
+
+- **CI/CD/CP.** Update your policies and data systems as frequently as you (or your legal team!) needs. The updated policies will continuously evaluate compliance and warn users of unsafe changes _before_ they make it into production.
+
+- **Built to Scale.** Lots of databases? Tons of microservices? Large distributed infrastructure? Fides defines the data privacy taxonomy that allows for both lawyers and engineers to work together with a common language, so that the policies and rules can be applied across the entire data ecosystem.
+
+
+## :rocket: Getting Started 
+
+We recommend getting started with [our tutorial here](https://github.com/ethyca/fides/blob/main/docs/fides/docs/tutorial.md), but it's just as easy to jump right in with 5 easy steps:
+
+1. Install the fides CLI using Docker ([detailed instructions here](https://github.com/ethyca/fides/blob/main/docs/fides/docs/getting_started/docker.md))
+```
+docker pull ethyca/fidesctl:latest
+```
+
+2. Create a directory for your Fides resources to live in, like `fides_resources/`.
+```
+mkdir fides_resources/
+```
+
+3. Let's make our first policy, you don't even need a lawyer :wink:  using our template, modify and add what ever rules you'd like
+  <details>
+    <summary>Here's an example policy .yaml to get you started</summary>
+
+    ```yaml
+  policy:
+    - organizationId: 1
+      fidesKey: "primaryPrivacyPolicy"
+      name: "Primary Privacy Policy"
+      description: "The main privacy policy for the organization."
+      rules:
+        - organizationId: 1
+          fidesKey: "rejectTargetedMarketing"
+          name: "Reject Targeted Marketing"
+          description: "Disallow marketing that is targeted towards users."
+          dataCategories:
+            inclusion: "ANY"
+            values:
+              - profiling_data
+              - account_data
+              - derived_data
+              - cloud_service_provider_data
+          dataUses:
+            inclusion: ANY
+            values:
+              - market_advertise_or_promote
+              - offer_upgrades_or_upsell
+          dataSubjects:
+            inclusion: ANY
+            values:
+              - trainee
+              - commuter
+          dataQualifier: pseudonymized_data
+          action: REJECT
+        - organizationId: 1
+          fidesKey: rejectSome
+          name: "Reject Some Marketing"
+          description: "Disallow some marketing that is targeted towards users."
+          dataCategories:
+            inclusion: ANY
+            values:
+              - user_location
+              - personal_health_data_and_medical_records
+              - connectivity_data
+              - credentials
+          dataUses:
+            inclusion: ALL
+            values:
+              - improvement_of_business_support_for_contracted_service
+              - personalize
+              - share_when_required_to_provide_the_service
+          dataSubjects:
+            inclusion: NONE
+            values:
+              - trainee
+              - commuter
+              - patient
+          dataQualifier: pseudonymized_data
+          action: REJECT
+    ```
+  </details>
+
+
+4. And now, create a data system for Fides to check your data privacy policy against. 
+  <details>
+      <summary>Here's an example system .yaml to get you started</summary>
+
+    ```yaml
+  system:
+    - organizationId: 1
+      fidesKey: "demoSystem"
+      name: "Demo System"
+      description: "A system used for demos."
+      systemType: "Service"
+      privacyDeclarations:
+        - name: "Analyze Anonymous Content"
+          dataCategories:
+            - "account_data"
+          dataUse: "provide"
+          dataQualifier: "anonymized_data"
+          dataSubjects:
+            - "anonymous_user"
+          datasetReferences:
+            - "sample_db_dataset.Email"
+      systemDependencies: []
+    ```
+  </details>
+
+
+5. Send all your resources to the server using `fidesctl apply fides_manifests/` and that's it! See how your data set stacks up against the policy by using `fidesctl evaluate`
+
+And ICYMI, we really recommend doing [the tutorial](https://github.com/ethyca/fides/blob/main/docs/fides/docs/tutorial.md)? It's helpful to contextualize how you can use Fides in your organization, today.
+
+## :book: Resources
 
 ### Documentation
 
-Fides' documentation is available [here]().
+Fides' documentation is available [here](https://github.com/ethyca/fides/tree/main/docs/fides/docs).
 
 ### Contributing
 
-Read about the Fides [community]() or dive in to the [development guides]() for information about contributions, documentation, code style, testing and more.
+Read about the Fides [community](https://github.com/ethyca/fides/tree/main/docs/fides/docs/community) or dive in to the [development guides](https://github.com/ethyca/fides/tree/main/docs/fides/docs/development) for information about contributions, documentation, code style, testing and more.
 
 [pypi-image]: https://img.shields.io/pypi/v/fidesctl.svg
 [pypi-url]: https://pypi.python.org/pypi/fidesctl/

docker-compose.yml

@@ -51,7 +51,9 @@ services:
       - env_files/mysql_db.env
 
   docs:
-    image: squidfunk/mkdocs-material
+    build:
+      context: docs/fides/
+      dockerfile: Dockerfile
     volumes:
       - ./docs/fides:/docs
     expose:

docs/README.md

@@ -1,5 +1,3 @@
 # Fides Docs
 
-1. `pip install -r requirements.txt`
-1. `cd fides`
-1. `mkdocs serve`
+This is where the docs site for Fides lives. To run the docs locally, use `make docs-serve` and visit `localhost:8000`

docs/fides/Dockerfile

@@ -0,0 +1,19 @@
+FROM python:3.8-slim-buster
+
+# Install auxiliary software
+RUN apt-get update
+RUN apt-get install -y \
+    git \
+    vim
+
+# Update pip and install requirements
+RUN pip install -U pip
+COPY requirements.txt requirements.txt
+RUN pip install -r requirements.txt
+
+# Copy in the application files and install it locally
+COPY ./docs /docs
+WORKDIR /docs
+
+EXPOSE 8000
+CMD ["mkdocs", "serve", "--dev-addr=0.0.0.0:8000"]

docs/fides/docs/deployment.md

@@ -1,11 +1,42 @@
 # Deployment
 
-The easiest way to deploy Fides is with `Docker`. There are separate containers for `Fidesctl`, the `Server` and the `DB`.
+We recommend deploying Fides with `Docker` using the included docker-compose file [found here in the repo](https://github.com/ethyca/fides/blob/main/docker-compose.yml). There are separate containers for `Fidesctl`, the `API` and the `DB`.
 
-## Application Variables
+## Requirements
 
-These are the environment variables that can be set to configure Fidesctl for a specific deployment:
+1. Install MySQL 8
+1. Install Python 3.8 or newer (including pip)
+1. Install Docker
 
-* `FIDES_API_URL` - The URL of the Fides webserver
+## Setup
 
-After deploying the API and DB, you need to inject the `FIDES_SERVER_URL` that points to wherever the API got deployed. Fidesctl will check for this environment variable first when running any commands instead of requiring the `-u` flag.
+The setup is done via Docker, with configuration values injected at runtime into the container.
+
+1. Spin up a MySQL DB with the following env vars:
+
+    ```env
+    MYSQL_ROOT_PASSWORD="<root_password>"
+    MYSQL_USER="<user>"
+    MYSQL_PASSWORD="<user_password>"
+    MYSQL_DATABASE="<db>"
+    ```
+
+1. Start the FidesAPI -> `docker run -p "127.0.0.1:8080:8080/tcp" --env FIDES_DB_JDBC_URL="jdbc:mysql://<server_address>:3306/<db>" --env FIDES_DB_JDBC_USER="<user>" --env FIDES_DB_JDBC_PASSWORD="<user_password>" ethyca/fidesapi:latest /bin/bash -c "sbt flywayMigrate && sbt ~jetty:start"`
+1. Install Fidesctl -> `pip install fidesctl`
+1. Configuration of Fidesctl will be done via an `ini` file that will then be mounted onto the docker container. Fidesctl will automatically look for a file called `fides.ini` in the local directory, or at a location set by the `FIDES_CONFIG_PATH` environment variable. 
+    <details>
+        <summary>Here's an example ini file to get you started</summary>
+        
+    ```ini
+    [user]
+    user_id = 1
+    api_key = test_api_key
+
+    [cli]
+    server_url = http://fidesapi:8080
+    ```
+    </details>
+
+    Write your `fides.ini` file and set your `FIDES_CONFIG_PATH` environment variable as needed
+
+5. Run `fidesctl` to see a list of possible commands!

docs/fides/docs/fides_objects/object_types.md → docs/fides/docs/fides_resources.md

@@ -4,7 +4,7 @@ This page describes the various objects that make up the Fides platform.
 
 ## Object Relationship Diagram
 
-![alt text](../img/Object_Relations.svg "Fides Manifest Workflow")
+![alt text](img/Object_Relations.svg "Fides Manifest Workflow")
 
 ## Organization
 

docs/fides/docs/getting_started/docker.md

@@ -2,21 +2,22 @@
 
 ---
 
-The easiest way to get started with Fides is to launch it using the supplied `make` commands, which in turn call Docker to do the heavy lifting.
+The recommended way to get Fides running is to launch it using the supplied `make` commands. The make commands wrap docker-compose commands that will spin up each piece of the project.
 
 ## Requirements
 
 1. Install Make
-1. Install Docker
+1. Install Docker (Docker compose is bundled with Docker in current versions)
 1. Clone the [Fides repo](https://github.com/ethyca/fides)
 
 ## Docker Setup
 
 The following commands should all be run from the top-level Fides directory (where the Makefile is)
 
-1. `make cli` -> this will build the required images, spin up the database, and open a shell inside of a container with `fidesctl` installed
-1. About 15 seconds after the `fidesctl` shell initializes, run the `fidesctl ping` command to verify that `fidesctl` can communicate with the server. If it fails, wait a bit longer and try again until you get a successful response.
-1. `fidesctl` -> this command will list all of the possible `fidesctl` commands
+1. `make compose-build` -> This will build all of the required images
+1. `make init-db` -> Spins up the database and runs the initialization scripts
+1. `make cli` -> This will spin up the entire project and open a shell within the Fidesctl container, with the FidesAPI being accessible. This command will "hang" for a bit, as Fidesctl will wait for the FidesAPI service to be healthy before spinning up the terminal.
+1. `fidesctl` -> Lists all of the possible `fidesctl` commands! You're now ready to start enforcing privacy with Fides!
 
 ## Next Steps
 

docs/fides/docs/getting_started/local.md

@@ -1,6 +1,6 @@
 # Running Fides Locally
 
-Fides can also be spun up locally without relying on Docker or Make, however there are significantly more prerequisites.
+Fides can also be spun up locally without relying on Docker or Make, however there are significantly more prerequisites. This is __not the recommended way to run Fides!__ Please see the [Getting Started with Docker](docker.md) guide for the recommended setup experience.
 
 ## Local Requirements
 
@@ -18,8 +18,9 @@ Fides can also be spun up locally without relying on Docker or Make, however the
 1. Update the `application.conf` in `src/main/resources/` with the database credentials
 1. `sbt flywayMigrate`
 1. `sbt ~jetty:start` -> You now have a Fides Server instance up and running powered by MySQL!
-1. In a new shell -> `pip install fidesctl`
-1. Set the `FIDES_SERVER_URL` environment variable to `localhost:8080`
+1. In a new shell -> `cd fidesctl/ && pip install -e .`
+1. Set the `FIDES_SERVER_URL` environment variable to `localhost:8080` and restart your shell as needed
+1. Set the `FIDES_CONFIG_PATH` environment variable and write a config based off of the `fidesctl/example_config.ini`, changing fields as needed.
 1. You can now run `fidesctl ping` to verify that your installation is set up properly
 
 ## Next Steps