pre commit hook

Run Credential Digger before commits

Credential Digger can be used as a pre-commit hook with the pre-commit framework. The hook scans the staged files of a repository looking for possible leaks before they get committed. If the commit is clean (i.e., it does not contain discoveries), the hook will be successful and the files will be commited as expected. In case the commit contains leaks, the hook will ask the user a confirmation whether to ignore the alerts or block the commit. It is possible also to execute silently the commit verification avoiding any interaction, and causing the hook to fail and abort the commit in case of discoveries. Credential Digger hook can also be used in CLI.

TL;DR

• pip install pre-commit to install the pre-commit framework
• create a .pre-commit-config.yaml in the root of your repository as follows:

repos:
-   repo: https://github.com/SAP/credential-digger
    rev: main
    hooks:
    -   id: credential-digger-hook
        args: [hook, --rules=/path/to/rules/my_hook_rules.yml, --sqlite=/path/to/db/my_hook_db.db]

You can copy the rules.yml here

• Use: pre-commit install to install the hook in the current repo
• Use: pre-commit run to install the environment (first time only) and to test the hook (stage some files to actually run Credential Digger)

Description

The hook is fully functioning when committing via terminal. Please note that committing through VS Code or some other IDE can cause problems, e.g., a pre-commit not found message.

The pre-commit framework installs the hook in ~/.cache/pre-commit/repo. The cache is not automatically deleted but it can be done easily. To change the installation folder of the hooks set PRE_COMMIT_HOME (ref., Managing CI caches).

The hook uses postgreSQL by default which needs to be already up and running. It is possible to use SQLite by specifying the db location e.g., ~/.local/data.db but please do not use relative paths.

Requirements

Install pre-commit with pip install pre-commit

Installation

There are two ways to set-up Credential Digger hook:

• Via the pre-commit framework
• Cloning Credential Digger repository

First set-up option: Via pre-commit framework

Create a file .pre-commit-config.yaml in the root of your repository with the following structure:

repos:
-   repo: https://github.com/SAP/credential-digger
    rev: main
    hooks:
    -   id: credential-digger-hook

Second set-up option: Cloning Credential Digger repository

It is possible to use the hook also cloning Credential Digger

git clone https://github.com/SAP/credential-digger

Then, change the repo field of .pre-commit-config.yaml in repo: /path/to/repo/credentialdigger

Install the hook

Independently of the set-up option used, the hook must be installed.

Use pre-commit install to install the hook in repo/.git/hooks/pre-commit

At the first commit, the hook will install Credential Digger and all its dependencies in the same environment of the repository (that will be re-used for the following commits). It is possible to install immediately the environment using pre-commit install --install-hooks. The environment set up will require several minutes.

To avoid the warning of rev mutable reference, use rev: *release* e.g., rev: v4.8.0

Please note the hook uses a db and a set of rules. If you use the hook for the first time (or change the database), be it either with the pre-commit framework or by CLI, you have to set the rules you wish to use.

Hook arguments

It is possible to choose different parameters to configure the hook:

• --sqlite: If specified, scan the repo using the sqlite client passing as argument the path of the db. Otherwise, use postgres (must be up and running)
• --rules: Specify the yaml file path containing the scan rules e.g., /path/to/rules.yaml
• --no_interaction: Flag used to remove the interaction i.e., do not prompt if the commit should continue in case of discoveries. If specified, the hook will fail in case of discoveries.

An example:

repos:
-   repo: https://github.com/SAP/credential-digger
    rev: main
    hooks:
    -   id: credential-digger-hook
        args: [hook, --rules=/path/to/rules/my_hook_rules.yml, --sqlite=/path/to/db/my_hook_db.db]

If you wish to use only the --rules or --sqlite option, remove the other one but keep the hook argument in the first position. The hook, in case of discoveries will ask whether to continue or to abort the commit. To remove this interaction, modify the args field: args: [hook, ..., --no_interaction, ...]

Uninstall

To uninstall the hook use: pre-commit uninstall -t pre-commit.

To delete the pre-commit environment use pre-commit clean. This action will delete pre-commit modules used by the pre-commit framework, not the hook. If you have not uninstalled the hook, the next commit will reinstall Credential Digger hook.

Install globally

It is possible to install the hook globally, for current and future repositories. This procedure allows to have the hook without the need to run pre-commit install.

If git global has not been set yet: git config --global init.templateDir ~/.git-template

Then: pre-commit init-templatedir ~/.git-template in the same folder where is .pre-commit-config.yaml

git init will initialize a repository with the credential digger hook already installed (a .pre-commit-config.yaml has to be in the repo)

---

Skip the hook

If the hook is configured, then it will used every commit. It is possible to commit without using the hook (so skipping it) with one of these two commands:

• git commit -m "commit msg" -n It will bypass pre-commit and commit-msg hooks
• SKIP=credential-digger-hook git commit -m "commit msg" It will bypass only credential-digger-hook

Test the hook

Credential Digger hook can be tested in 3 ways:

• Using a temp environment
• Using a real environment
• Using CLI

Whatever test you choose, if --no_interaction is not specified, you will be asked to confirm or not the commit (in case of leaks). This is the expected behavior, there is no commit in place.

Test: temp environment

A temp environment is created on uncommitted changes and Credential Digger hook is installed and run. To use this configuration, set-up the file .pre-commit-config.yaml as explained before, and run:

pre-commit try-repo /path/to/crendentialdigger_repo

Note that every time you execute this test a new environment is created, and therefore the hook needs to be installed from scratch. So, it will take several minutes.

Test: real environment

Follow the installation instruction specified above. Then, to test the hook, use: pre-commit run

Using a temp environment pre-commit try-repo /path/to/crendentialdigger_repo
Using real environment pre-commit run

Test: CLI

You can launch the hook by CLI, refer to that section for further information.

CLI hook

Credential Digger hook can be also tested without the pre-commit framework. First install Credential Digger with pip install credentialdigger

Use credentialdigger hook to run the scan on staged files.

Supported options:

-h, --help         show this help message and exit
--dotenv DOTENV    The path to the .env file which will be used in all
                   commands. If not specified, the one in the current
                   directory will be used (if present).
--sqlite SQLITE    If specified, scan the repo using the sqlite client
                   passing as argument the path of the db.
                   Otherwise, use postgres (must be up and running)
--rules RULES      Specify the yaml file path containing the scan rules
                   e.g., /path/to/rules.yaml
--no_interaction   Flag used to remove the interaction i.e.,
                   do not prompt if the commit should continue
                   in case of discoveries. If specified, the hook will
                   fail in case of discoveries.

The hook will scan the staged files in the current repository. For example:

$ pip install credentialdigger

$ cd /path/to/repo

$ git add somefile

$ credentialdigger hook --rules=~/.local/rules.yaml --sqlite=~/.local/new_data.db  # First time usage of db, specify the rules