Filtering type of alerts to export and Dependabot alerts support GHES (#44)

* select features; dependabot for GHES 3.8;

* linter fixes

* linter fixes

Author: theztefan

.env-sample

@@ -0,0 +1,8 @@
+## GHES 
+#export GITHUB_API_URL="https://GHES-instance.net/api/v3"
+#export GITHUB_SERVER_URL="https://GHES-instance.net"
+
+export GITHUB_PAT="ghp_TOKEN"
+export GITHUB_REPORT_SCOPE="repository"
+export SCOPE_NAME="org/repo-name"
+export FEATURES="secretscanning,dependabot"

.gitignore

@@ -133,3 +133,6 @@ swap.md
 
 # CSV files
 *.csv
+
+# Ignore macos files
+.DS_Store

CONTRIBUTING.md

@@ -21,6 +21,31 @@ Here are a few things you can do that will increase the likelihood of your pull
 - Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
 - Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
 
+## Local development
+
+After cloning your forked repository, you can set up your development environment by running the following commands:
+
+```bash
+# Create a virtual environment
+python3 -m venv venv
+source venv/bin/activate
+
+# Install dependencies
+pip3 install -r requirements.txt
+```
+
+To execute the python script `main.py` locally you will need to setup the required environment variables. You can do this by making a copy of `.env-sample` into `.env` file in the root of the project and edit the values to match your environment.
+
+```bash
+source .env
+```
+
+You can now run the script locally:
+
+```bash
+python3 main.py
+```
+
 ## Resources
 
 - [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)

README.md

@@ -69,13 +69,36 @@ Or for an enterprise:
           SCOPE_NAME: "enterprise-slug-goes-here"
 ```
 
+The list of all available options that can be set as environmental variables is below:
+
+- `GITHUB_API_URL`: The URL of the GitHub API. Default value: `https://api.github.com`.
+- `GITHUB_SERVER_URL`: The URL of the GitHub server. Default value: `https://github.com`.
+- `GITHUB_PAT` or `GITHUB_TOKEN`: The personal access token (PAT) or token for authenticating with the GitHub API. If `GITHUB_PAT` is not set, the value of `GITHUB_TOKEN` is used if it is set. If neither is set, an error occurs.
+- `GITHUB_REPORT_SCOPE`: The scope of the report to generate. Valid values are `repository` (default), `organization`  or `enterprise`.
+- `SCOPE_NAME` or `GITHUB_REPOSITORY`: The name of the repository, organization or enterprise to generate the report for. If `SCOPE_NAME` is not set, the value of `GITHUB_REPOSITORY` is used if it is set. If neither is set, an error occurs.
+- `FEATURES`: A comma-separated list of features to include in the report. Valid values are `codescanning`, `secretscanning`, `dependabot` or simply `all`. Default value: `all`.
+
+The first two are only needed if you're running this in a GitHub Enterprise Server or GitHub AE environment.  The last one is useful if you only want to get data on a specific feature.  For example, if you only want to get data on secret scanning, you can set `FEATURES` to `secretscanning`. Here's just another example how you would configure this on a GitHub Enterprise Server:
+
+```yaml
+      - name: CSV export
+        uses: advanced-security/ghas-to-csv@v2
+        env:
+          GITHUB_PAT: ${{ secrets.PAT }}
+          GITHUB_API_URL: "https://github.example.com/api/v3"
+          GITHUB_SERVER_URL: "https://github.example.com"
+          GITHUB_REPORT_SCOPE: "enterprise"
+          SCOPE_NAME: "enterprise-slug-goes-here"
+          FEATURES: "secretscanning,codescanning"
+```
+
 ## Reporting
 
 |   | GitHub Enterprise Cloud | GitHub Enterprise Server (3.5+) | GitHub AE (M2) | Notes |
 | --- | --- | --- | --- | --- |
 | Secret scanning | :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise |  :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise | :white_check_mark: Repo<br>:x: Org<br>:x: Enterprise | [API docs](https://docs.github.com/en/enterprise-cloud@latest/rest/reference/secret-scanning) |
-| Code scanning |  :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise | :white_check_mark: Repo<br>:white_check_mark: Org<br>:curly_loop: Enterprise |  :white_check_mark: Repo<br>:x: Org<br>:curly_loop: Enterprise | [API docs](https://docs.github.com/en/enterprise-cloud@latest/rest/reference/code-scanning) |
-| Dependabot | :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise | :x: | :x: | [API docs](https://docs.github.com/en/enterprise-cloud@latest/rest/dependabot/alerts) |
+| Code scanning |  :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise | :white_check_mark: Repo<br>:white_check_mark: Org<br>:curly_loop: Enterprise (3.5, 3.6) <br>:white_check_mark: Enterprise (3.7+) |  :white_check_mark: Repo<br>:x: Org<br>:curly_loop: Enterprise | [API docs](https://docs.github.com/en/enterprise-cloud@latest/rest/reference/code-scanning) |
+| Dependabot | :white_check_mark: Repo<br>:white_check_mark: Org<br>:white_check_mark: Enterprise | :white_check_mark: Repo (3.8+)<br>:white_check_mark: Org (3.8+)<br>:white_check_mark: Enterprise (3.8+)  | :x: | [API docs](https://docs.github.com/en/enterprise-cloud@latest/rest/dependabot/alerts) |
 
 :information_source:  All of this reporting requires either public repositories or a GitHub Advanced Security license.
 
@@ -143,7 +166,8 @@ nginx-pid/
 
 ## But it doesn't do THIS THING
 
-The API docs are [here](https://docs.github.com/en/enterprise-cloud@latest) and pull requests are welcome! :heart:
+The API docs are [here](https://docs.github.com/en/enterprise-cloud@latest) and pull requests are welcome! :heart:.
+See [CONTRIBUTING](CONTRIBUTING.md) for more information.
 
 ## Other notes
 

main.py

@@ -8,18 +8,22 @@
 - PAT of appropriate scope (assumes the workflow token if not specified)
 - Report scope ("enterprise", "organization", "repository")
 - Enterprise slug OR organization name OR repository name
+- Features to run (comma separated list of "secretscanning", "codescanning", "dependabot")
 
 Outputs:
 - CSV file of secret scanning alerts
 - CSV file of code scanning alerts
-
-TODO: dependabot alerts
+- CSV file of Dependabot alerts 
 """
 
 # Import modules
 from src import code_scanning, dependabot, enterprise, secret_scanning
 import os
 
+
+# Define the available features
+FEATURES = ["secretscanning", "codescanning", "dependabot"]
+
 # Read in config values
 if os.environ.get("GITHUB_API_URL") is None:
     api_endpoint = "https://api.github.com"
@@ -46,72 +50,95 @@
 else:
     scope_name = os.environ.get("SCOPE_NAME")
 
+if os.environ.get("FEATURES") is None:
+    features = FEATURES
+else:
+    if os.environ.get("FEATURES") == "all":
+        features = FEATURES
+    else:
+        features = os.environ.get("FEATURES").split(",")
+        for f in features:
+            if f not in FEATURES:
+                print(
+                    f"Invalid feature: {f}. Proceeding without. Valid features are: {FEATURES}"
+                )
+                features.remove(f)
+
+
 # Do the things!
 if __name__ == "__main__":
+    print("Starting GitHub security report...")
+    # enterprise scope
     if report_scope == "enterprise":
         # secret scanning
-        secrets_list = secret_scanning.get_enterprise_secret_scanning_alerts(
-            api_endpoint, github_pat, scope_name
-        )
-        secret_scanning.write_enterprise_secrets_list(secrets_list)
-        # code scanning
-        if enterprise.get_enterprise_version(api_endpoint) != "GHEC":
-            repo_list = enterprise.get_repo_report(url, github_pat)
-            cs_list = code_scanning.list_enterprise_server_code_scanning_alerts(
-                api_endpoint, github_pat, repo_list
-            )
-            code_scanning.write_enterprise_server_cs_list(cs_list)
-        else:
-            cs_list = code_scanning.list_enterprise_cloud_code_scanning_alerts(
+        if "secretscanning" in features:
+            secrets_list = secret_scanning.get_enterprise_secret_scanning_alerts(
                 api_endpoint, github_pat, scope_name
             )
-            code_scanning.write_enterprise_cloud_cs_list(cs_list)
+            secret_scanning.write_enterprise_secrets_list(secrets_list)
+        # code scanning
+        if "codescanning" in features:
+            version = enterprise.get_enterprise_version(api_endpoint)
+            # for GHES version 3.5 and 3.6 we need to loop through each repo and use the repo level api to get the code scanning alerts
+            # for 3.7 and above we use the enterprise level api to get the code scanning alerts
+            if version.startswith("3.5") or version.startswith("3.6"):
+                repo_list = enterprise.get_repo_report(url, github_pat)
+                cs_list = code_scanning.list_enterprise_server_code_scanning_alerts(
+                    api_endpoint, github_pat, repo_list
+                )
+                code_scanning.write_enterprise_server_cs_list(cs_list)
+            else:
+                cs_list = code_scanning.list_enterprise_cloud_code_scanning_alerts(
+                    api_endpoint, github_pat, scope_name
+                )
+                code_scanning.write_enterprise_cloud_cs_list(cs_list)
         # dependabot alerts
-        if enterprise.get_enterprise_version(api_endpoint) == "GHEC":
+        if "dependabot" in features:
             dependabot_list = dependabot.list_enterprise_dependabot_alerts(
                 api_endpoint, github_pat, scope_name
             )
             dependabot.write_org_or_enterprise_dependabot_list(dependabot_list)
-        else:
-            pass
-
+    # organization scope
     elif report_scope == "organization":
         # code scanning
-        cs_list = code_scanning.list_org_code_scanning_alerts(
-            api_endpoint, github_pat, scope_name
-        )
-        code_scanning.write_org_cs_list(cs_list)
+        if "codescanning" in features:
+            cs_list = code_scanning.list_org_code_scanning_alerts(
+                api_endpoint, github_pat, scope_name
+            )
+            code_scanning.write_org_cs_list(cs_list)
         # dependabot alerts
-        if enterprise.get_enterprise_version(api_endpoint) == "GHEC":
+        if "dependabot" in features:
             dependabot_list = dependabot.list_org_dependabot_alerts(
                 api_endpoint, github_pat, scope_name
             )
             dependabot.write_org_or_enterprise_dependabot_list(dependabot_list)
-        else:
-            pass
+
         # secret scanning
-        secrets_list = secret_scanning.get_org_secret_scanning_alerts(
-            api_endpoint, github_pat, scope_name
-        )
-        secret_scanning.write_org_secrets_list(secrets_list)
+        if "secretscanning" in features:
+            secrets_list = secret_scanning.get_org_secret_scanning_alerts(
+                api_endpoint, github_pat, scope_name
+            )
+            secret_scanning.write_org_secrets_list(secrets_list)
+    # repository scope
     elif report_scope == "repository":
         # code scanning
-        cs_list = code_scanning.list_repo_code_scanning_alerts(
-            api_endpoint, github_pat, scope_name
-        )
-        code_scanning.write_repo_cs_list(cs_list)
+        if "codescanning" in features:
+            cs_list = code_scanning.list_repo_code_scanning_alerts(
+                api_endpoint, github_pat, scope_name
+            )
+            code_scanning.write_repo_cs_list(cs_list)
         # dependabot alerts
-        if enterprise.get_enterprise_version(api_endpoint) == "GHEC":
+        if "dependabot" in features:
             dependabot_list = dependabot.list_repo_dependabot_alerts(
                 api_endpoint, github_pat, scope_name
             )
             dependabot.write_repo_dependabot_list(dependabot_list)
-        else:
-            pass
+
         # secret scanning
-        secrets_list = secret_scanning.get_repo_secret_scanning_alerts(
-            api_endpoint, github_pat, scope_name
-        )
-        secret_scanning.write_repo_secrets_list(secrets_list)
+        if "secretscanning" in features:
+            secrets_list = secret_scanning.get_repo_secret_scanning_alerts(
+                api_endpoint, github_pat, scope_name
+            )
+            secret_scanning.write_repo_secrets_list(secrets_list)
     else:
         exit("invalid report scope")