Updated docs (#288)

* Fix up markdownlint warnings * Fix up docs build warnings * Update ToC * Ensure all the util docs are complete * Use sphinx_autodoc_typehints and update intersphinx mapping * Update all sphinx-related tooling to the latest * Fix up spelling problems * Improve the documentation of the types module * Add missing Union import * Improve the scanner docstrings * Finish up the types docstrings * Fix docstrings for util * Shift more configuration options from features -> config * Update config docs * Add "Upgrading" documentation This basically expands the CHANGELOG content from 3.x into a few sets of * What will I see different? * What do I do about it? topics that should get users over the 2-to-3 hump even if they read nothing else. * Update docs/source/configuration.rst Co-authored-by: Scott Bailey <72747501+rbailey-godaddy@users.noreply.github.com> * Address review comments * Hide unsupported options * Add "oops not there yet" section Co-authored-by: Sushant Mimani <sushant.mimani@gmail.com> Co-authored-by: Scott Bailey <scott.bailey@godaddy.com>
godaddy · Jan 5, 2022 · 6424683 · 6424683
1 parent 32be379
commit 6424683
Show file tree

Hide file tree

Showing 15 changed files with 662 additions and 301 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -72,7 +72,7 @@ Features:
 
 * [#227](https://github.com/godaddy/tartufo/pull/227) - Report findings incrementally
   as scan progresses instead of holding all of them until it has completed. This
-  is a reimplementation of [#108](https://github.com/godaddy/tartufo/pull/108);
+  is a re-implementation of [#108](https://github.com/godaddy/tartufo/pull/108);
   thanks to @dclayton-godaddy for showing the way.
 * [#244](https://github.com/godaddy/tartufo/pull/244) - Drops support for
   `--fetch/--no-fetch` option for local scans
@@ -88,11 +88,19 @@ Misc:
 * [#255](https://github.com/godaddy/tartufo/issues/255) - Removed deprecated flags
   --include-paths and --exclude-paths
 
-v2.10.0 - 3 November 2021
+v2.10.1 - 27 December 2021
 ------------
 
 Bug fixes:
 
+*[#309](https://github.com/godaddy/tartufo/pull/309) Fixes an issue where verbose output display
+would error out if the new-style entropy exclusion pattern was used
+
+v2.10.0 - 3 November 2021
+-------------------------
+
+Bug fixes:
+
 * [#247](https://github.com/godaddy/tartufo/issues/247) All versions of tartufo from
   v2.2.0 through v2.9.0 inclusive mishandle `scan-remote-repo`. Only the repository's
   default branch was scanned, and secrets present in other branches would not be
@@ -126,7 +134,7 @@ Features:
   This is now done via a TOML table, rather than a specifically patterned string.
   Users who have the old style configuration will now receive a
   `DeprecationWarning` stating that the old behavior will go away with v3.0.
-* Fixed up warning handling so that we can display `DeprecationWarning`s to users
+* Fixed up warning handling so that we can display `DeprecationWarnings` to users
   more easily.
 * [#223](https://github.com/godaddy/tartufo/pull/223) New flags
   (`-b64`/`--b64-entropy-score` and `-hex`/`--hex-entropy-score`) allow for user

diff --git a/docs/source/CONTRIBUTING.md b/docs/source/CONTRIBUTING.md
@@ -11,18 +11,19 @@ project by reading these Contribution Guidelines.
 to [GoDaddy's Open Source Software Slack][slack], or the
 [Tartufo Mailing list].
 
-Table of Contents
-=================
+## Table of Contents
 
 * [Answering Questions](#answering-questions)
 * [Reporting Bugs](#reporting-bugs)
 * [Triaging bugs or contributing code](#triaging-bugs-or-contributing-code)
 * [Code Review](#code-review)
 * [Attribution of Changes](#attribution-of-changes)
 * [Writing Code](#writing-code)
-    * [Setting Up A Development Environment](#setting-up-a-development-environment)
-    * [Code Style](#code-style)
+  * [Setting Up A Development Environment](#setting-up-a-development-environment)
+  * [Code Style](#code-style)
 * [Running tests](#running-tests)
+* [Contributing as a Maintainer](#contributing-as-a-maintainer)
+  * [Issuing a New Release](#issuing-a-new-release)
 * [Additional Resources](#additional-resources)
 
 ## Answering Questions
@@ -43,11 +44,11 @@ software and can be easily found through [PyPI].
 
 Submit a ticket for your issue, assuming one does not already exist:
 
-- Create it on our [Issue Tracker][issues]
-- Clearly describe the issue by following the template layout
-    - Make sure to include steps to reproduce the bug.
-    - A reproducible (unit) test could be helpful in solving the bug.
-    - Describe the environment that (re)produced the problem.
+* Create it on our [Issue Tracker][issues]
+* Clearly describe the issue by following the template layout
+  * Make sure to include steps to reproduce the bug.
+  * A reproducible (unit) test could be helpful in solving the bug.
+  * Describe the environment that (re)produced the problem.
 
 > For a bug to be actionable, it needs to be reproducible. If you or
 > contributors can't reproduce the bug, try to figure out why. Please take care
@@ -61,11 +62,11 @@ reasoning about a longer sample.
 
 From a practical perspective, contributions are as simple as:
 
-- Forking the repository on GitHub.
-- Making changes to your forked repository.
-- When committing, reference your issue (if present) and include a note about the fix.
-- If possible, and if applicable, please also add/update unit tests for your changes.
-- Push the changes to your fork and submit a pull request to the 'main' branch of the projects' repository.
+* Forking the repository on GitHub.
+* Making changes to your forked repository.
+* When committing, reference your issue (if present) and include a note about the fix.
+* If possible, and if applicable, please also add/update unit tests for your changes.
+* Push the changes to your fork and submit a pull request to the 'main' branch of the projects' repository.
 
 If you are interested in making a large change and feel unsure about its overall effect, please make sure to first
 discuss the change and reach a consensus with core contributors through [slack]. Then ask about the best way to go about
@@ -83,11 +84,11 @@ Any open source project relies heavily on code review to improve software qualit
 Sometimes code reviews will take longer than you would hope for, especially for larger features. Here are some accepted
 ways to speed up review times for your patches:
 
-- Review other people's changes. If you help out, others will be more willing to do the same for you. Good will is our
+* Review other people's changes. If you help out, others will be more willing to do the same for you. Good will is our
   currency.
-- Split your change into multiple smaller changes. The smaller your change, the higher the probability that somebody
+* Split your change into multiple smaller changes. The smaller your change, the higher the probability that somebody
   will take a quick look at it.
-- Ping the change on [slack]. If it is urgent, provide reasons why it is important to get this change landed. Remember
+* Ping the change on [slack]. If it is urgent, provide reasons why it is important to get this change landed. Remember
   that you're asking for valuable time from other professional developers.
 
 **Note that anyone is welcome to review and give feedback on a change, but only people with commit access to the
@@ -206,25 +207,25 @@ This process is thankfully mostly automated. There are, however, a handful of ma
 off that automation. It is all built this way to help ensure that issuing a release is a very conscious decision,
 requiring peer review, and cannot easily happen accidentally. The steps involved currently are:
 
-- Create a new branch locally for the release, for example:
+* Create a new branch locally for the release, for example:
 
   ```console
   > git checkout -b releases/v2.1.0
   ```
 
-- Tell Poetry to [bump the version]:
+* Tell Poetry to [bump the version]:
 
   ```console
   > poetry version minor
   Bumping version from 2.0.1 to 2.1.0
   ```
 
-    - Note: All this is doing, is updating the version number in the
-      `pyproject.toml`. You can totally do this manually. This command just might be a bit quicker. And it's nice to
-      have a command to do it for you. Yay automation!
-- Update the CHANGELOG with the appropriate new version number and release date.
-- Create a pull request for these changes, and get it approved!
-- Once your PR has been merged, the final piece is to actually create the new release.
+  * Note: All this is doing, is updating the version number in the
+    `pyproject.toml`. You can totally do this manually. This command just might be a bit quicker. And it's nice to
+    have a command to do it for you. Yay automation!
+* Update the CHANGELOG with the appropriate new version number and release date.
+* Create a pull request for these changes, and get it approved!
+* Once your PR has been merged, the final piece is to actually create the new release.
 
     1. Go to the `tartufo` [releases page] and click on `Draft a new release`.
     2. Enter an appropriate tag version (in this example, `v2.1.0`).
@@ -237,8 +238,8 @@ Congratulations, you've just issued a new release for `tartufo`. The automation
 
 ## Additional Resources
 
-- [General GitHub Documentation](https://help.github.com/)
-- [GitHub Pull Request documentation](https://help.github.com/send-pull-requests/)
+* [General GitHub Documentation](https://help.github.com/)
+* [GitHub Pull Request documentation](https://help.github.com/send-pull-requests/)
 
 [black]: https://github.com/psf/black
 [black-editors]: https://black.readthedocs.io/en/stable/integrations/editors.html

diff --git a/docs/source/api.rst b/docs/source/api.rst
@@ -26,7 +26,6 @@ tartufo.types
 
 .. automodule:: tartufo.types
    :members:
-   :undoc-members:
 
 tartufo.util
 ------------

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -10,7 +10,6 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-import codecs
 import datetime
 import os
 import pathlib
@@ -49,6 +48,7 @@
     "sphinx.ext.autodoc",
     "sphinx.ext.intersphinx",
     "sphinx.ext.viewcode",
+    "sphinx_autodoc_typehints",
 ]
 autodoc_typehints = "description"  # show type hints in doc body instead of signature
 autoclass_content = "both"  # get docstring from class level and init simultaneously
@@ -92,6 +92,6 @@
 # Set up linking to external Sphinx documetation
 intersphinx_mapping = {
     "click": ("https://click.palletsprojects.com/en/7.x/", None),
-    "gitpython": ("https://gitpython.readthedocs.io/en/stable/", None),
     "python": ("https://docs.python.org/3", None),
+    "pygit2": ("https://pygit2.readthedocs.io/en/stable/", None),
 }
diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
@@ -39,14 +39,15 @@ this:
    regex = true
    entropy = true
    exclude-path-patterns = [
-       'poetry.lock',
-       # To not have to escape `\` in regexes, use single quoted
-       # TOML 'literal strings'
-       'docs/source/(.*)\.rst',
+    {path-pattern = 'poetry\.lock'},
+    {path-pattern = 'pyproject\.toml'},
+    # To not have to escape `\` in regexes, use single quoted
+    # TOML 'literal strings'
+    {path-pattern = 'docs/source/(.*)\.rst'},
    ]
    exclude-signatures = [
-       "62f22e4500140a6ed959a6143c52b0e81c74e7491081292fce733de4ec574542",
-       "ecbbe1edd6373c7e2b88b65d24d7fe84610faafd1bc2cf6ae35b43a77183e80b",
+       {signature = "62f22e4500140a6ed959a6143c52b0e81c74e7491081292fce733de4ec574542"},
+       {signature = "ecbbe1edd6373c7e2b88b65d24d7fe84610faafd1bc2cf6ae35b43a77183e80b"},
    ]
 
 Note that all options specified in a configuration file are treated as
@@ -55,6 +56,112 @@ defaults, and will be overridden by any options specified on the command line.
 For a full list of available command line options, check out the :doc:`usage`
 document.
 
+.. _exclude-signatures:
+
+Excluding Signatures
+--------------------
+
+You might see the following header in the output for an issue:
+
+.. image:: _static/img/issue-signature.png
+
+Looking at this information, it's clear that this issue was found in a test
+file, and it's probably okay. Of course, you will want to look at the actual
+body of what was found and determine that for yourself. But let's say that this
+really is okay, and we want tell ``tartufo`` to ignore this issue in future
+scans. To do this, you can either specify it on the command line...
+
+.. code-block:: sh
+
+    > tartufo -e 2a3cb329b81351e357b09f1b97323ff726e72bd5ff8427c9295e6ef68226e1d1
+    # No output! Success!
+    >
+
+Or you can add it to your config file, so that this exclusion is always
+remembered!
+
+.. code-block:: toml
+
+    [tool.tartufo]
+    exclude-signatures = [
+      "2a3cb329b81351e357b09f1b97323ff726e72bd5ff8427c9295e6ef68226e1d1",
+    ]
+
+Done! This particular issue will no longer show up in your scan results.
+
+As of version 3.0, a new format for specifying exclusion signatures has been added.
+
+.. code-block:: toml
+
+    [tool.tartufo]
+    exclude-signatures = [
+        {signature = "2a3cb329b81351e357b09f1b97323ff726e72bd5ff8427c9295e6ef68226e1d1", reason = "reason for exclusion"},
+    ]
+
+.. note::
+
+    Currently all 3 formats of signature exclusions are supported. However, only `TOML`_ `array of tables`_ format
+    will be supported in future versions.
+
+.. _limiting-scans-by-paths:
+
+Limiting Scans by Path
+----------------------
+You can include or exclude paths for scanning using
+Python Regular Expressions (regex) and the `--include-path-patterns` and
+`--exclude-path-patterns` options.
+
+.. warning::
+
+   Using include patterns is more dangerous, since it's easy to miss the
+   creation of new secrets if future files don't match an existing include
+   rule. We recommend only using fine-grained exclude patterns instead.
+
+.. code-block:: toml
+
+   [tool.tartufo]
+   include-path-patterns = [
+      'src/',
+      'gradle/',
+      # regexes must match the entire path, but can use python's regex syntax
+      # for case-insensitive matching and other advanced options
+      '(.*/)?id_[rd]sa$',
+      # Single quoted strings in TOML don't require escapes for `\` in regexes
+      '(?i).*\.(properties|conf|ini|txt|y(a)?ml)$',
+   ]
+   exclude-path-patterns = [
+      '(.*/)?\.classpath$',
+      '.*\.jmx$',
+      '(.*/)?test/(.*/)?resources/',
+   ]
+
+The filter expressions can also be specified as command line arguments.
+Patterns specified like this are merged with any patterns specified
+in the config file:
+
+.. code-block:: sh
+
+   > tartufo \
+     --include-path-patterns 'src/' -ip 'gradle/' \
+     --exclude-path-patterns '(.*/)?\.classpath$' -xp '.*\.jmx$' \
+     scan-local-repo file://path/to/my/repo.git
+
+As of version 3.0, a new format for specifying paths has been added.
+.. code-block:: toml
+
+   [tool.tartufo]
+   include-path-patterns = [
+      {path-pattern = 'src/', reason='reason for inclusion'},
+   ]
+   exclude-path-patterns = [
+      {path-pattern = 'poetry\.lock', reason='reason for exclusion'},
+   ]
+
+.. note::
+
+    Currently all 3 formats are supported. However, only `TOML`_ `array of tables`_ format
+    will be supported in future versions.
+
 Configuration File Exclusive Options
 ------------------------------------
 
@@ -65,6 +172,8 @@ available only in the configuration file. This is due to the nature of their
 construction, and the fact that they would be exceedingly difficult to
 represent on the command line.
 
+.. _rule-patterns:
+
 Rule Patterns
 +++++++++++++
 
@@ -105,5 +214,54 @@ Option 2: Separating each rule out into its own table.
     Only ``reason`` and ``pattern`` are required. If no ``path-pattern`` is
     specified, then the pattern will be used to scan against all files.
 
+.. _entropy-exclusion-patterns:
+
+Entropy Exclusion Patterns
+++++++++++++++++++++++++++
+
+Entropy scans can produce a high number of false positive matches such as git
+SHAs or MD5 digests. To avoid these false positives, you can use the
+``exclude-entropy-patterns`` configuration option. These patterns will be
+applied to and matched against any strings flagged by entropy checks. As above,
+this directive utilizes an `array of tables`_, enabling two forms:
+
+Option 1:
+
+.. code-block:: toml
+
+    [tool.tartufo]
+    exclude-entropy-patterns = [
+        {path-pattern = 'docs/.*\.md$', pattern = '^[a-zA-Z0-9]$', reason = 'exclude all git SHAs in the docs'},
+        {path-pattern = '\.github/workflows/.*\.yml', pattern = 'uses: .*@[a-zA-Z0-9]{40}', reason = 'GitHub Actions'}
+    ]
+
+Option 2:
+
+.. code-block:: toml
+
+    [[tool.tartufo.exclude-entropy-patterns]]
+    path-pattern = 'docs/.*\.md$'
+    pattern = '^[a-zA-Z0-9]$'
+    reason = 'exclude all git SHAs in the docs'
+
+    [[tool.tartufo.exclude-entropy-patterns]]
+    path-pattern = '\.github/workflows/.*\.yml'
+    pattern = 'uses: .*@[a-zA-Z0-9]{40}'
+    reason = 'GitHub Actions'
+
+
+There are 5 relevant keys for this directive, as described below.
+
+============ ======== ============================ ==============================================================
+Key          Required Value                        Description
+============ ======== ============================ ==============================================================
+pattern      Yes      Regular expression           The pattern used to check against the match
+path-pattern No       Regular expression           A pattern to specify to what files the exclusion will apply
+reason       No       String                       A plaintext reason the exclusion has been added
+match-type   No       String ("match" or "scope")  Whether to perform a `search or match`_ regex operation
+scope        No       String ("word" or "line")    Whether to match against the current word or full line of text
+============ ======== ============================ ==============================================================
+
 .. _TOML: https://toml.io/
 .. _array of tables: https://toml.io/en/v1.0.0#array-of-tables
+.. _search or match: https://docs.python.org/3/library/re.html#search-vs-match