Add file caching to improvement performance when running uv-secure mu…

…ltiple times (#31)

* Added gitleaks pre-commit

* Making some corrections and tweaks to the CONTRIBUTING guide

* First pass at implementing file caching for httpx requests

* Moved cache configuration into Configuration object

* Refactored the way CLI args are applied and add a new flag to disable caching

* Bumped minor release and updated README

* Added some more documentation around file caching

* Removed uvloop and winloop as required dependencies and left them as optional dependencies

* Fixed some example config comments

.pre-commit-config.yaml

@@ -21,6 +21,10 @@ repos:
         exclude: '\.ipynb$'
       - id: mixed-line-ending
         args: [--fix=lf]
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.23.1
+    hooks:
+      - id: gitleaks
   - repo: https://github.com/igorshubovych/markdownlint-cli
     rev: v0.43.0
     hooks:
@@ -33,7 +37,7 @@ repos:
         additional_dependencies:
           - pydantic
   - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.9.2
+    rev: v0.9.3
     hooks:
       - id: ruff
         args: [ --fix, --exit-non-zero-on-fix ]

CONTRIBUTING.md

@@ -11,8 +11,8 @@ they don't see multiple contributors.
 
 - Follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
   for all code contributions.
-- Use **type hinting** consistently throughout the codebase (I'm a huge fan of strong-typing
-- and runtime typehinting - i.e. Pydantic style)...
+- Use **type hinting** consistently throughout the codebase (I'm a huge fan of
+  strong-typing and runtime typehinting - i.e. Pydantic style)...
 - Use pre-commit to run linters and type checkers on all changes.
 - MyPy (run by pre-commit) runs the type checking - it is prone to some false positives
   so use comments to disable checks if all else fails (but don't resort to unnecessary
@@ -21,27 +21,31 @@ they don't see multiple contributors.
 ## Testing
 
 - **Aim to maintain 100% test coverage** Ensure all changes are covered with appropriate
-  unit or integration tests. Not there is some platform / Python version specific logic
+  unit or integration tests. Note there is some platform / Python version specific logic
   so you'll only see full test coverage in CI which merges coverage across those
   dimensions.
 - Prefer integration tests for checking CLI input all the way through to CLI output.
 - Use [pytest](https://pytest.org/) as the testing framework.
 - To run tests, execute:
 
-  ```bash
+  ```shell
   uv run pytest
   ```
 
 - Use the `tests` directory for organizing test cases. The file and folder structure of
-  the tests should match the src folder to the extent that there are test modules that
-  map to specific src modules.
+  the tests directory should match the src folder to the extent that there are test
+  modules that map to specific src modules.
+- The test module for a src module should have the same name as the src module except
+  the test module will have a `test_` prefix. I want to use this scheme to make it easy
+  to find the tests for any given logic.
 
 ## Development Environment
 
-- I aim to support all the currently supported stable versions (3.9 through to 3.13).
+- I aim to support all the currently supported stable versions of Python (3.9 through to
+  3.13).
 - Install dependencies and development tools using [uv](https://docs.astral.sh/uv/):
 
-  ```bash
+  ```shell
   uv sync --dev
   ```
 
@@ -56,7 +60,7 @@ they don't see multiple contributors.
 
 - Create a descriptive branch for your changes:
 
-     ```bash
+     ```shell
      git checkout -b feature/short-description
      ```
 
@@ -69,7 +73,7 @@ they don't see multiple contributors.
 
 - Run all tests and ensure high coverage:
 
-     ```bash
+     ```shell
      uv run pytest
      ```
 
@@ -78,14 +82,14 @@ they don't see multiple contributors.
 - If you don't already have pre-commit installed, you only need to run this command
   once:
 
-     ```bash
+     ```shell
      uv tool install pre-commit
      ```
 
 - After checking out the repository for the first time, set up the pre-commit hooks
   by running:
 
-     ```bash
+     ```shell
      pre-commit install
      ```
 
@@ -94,7 +98,7 @@ they don't see multiple contributors.
 
 Developers can also force pre-commit to run on all files at any time by running:
 
-```bash
+```shell
 pre-commit run --all-files
 ```
 
@@ -104,7 +108,7 @@ This ensures consistency across the entire codebase and still executes quite fas
 
 - Push your branch to your fork:
 
-     ```bash
+     ```shell
      git push origin feature/short-description
      ```
 

README.md

@@ -45,6 +45,30 @@ pipx install uv-secure
 you can optionally install uv-secure as a development dependency in a virtual
 environment.
 
+## Optional Dependencies
+
+uv-secure uses highly asynchronous code to request multiple API responses or file opens
+concurrently. You can install uvloop on Linux/Mac or winloop on Windows to speed up the
+asynchronous event loop (at the expense of debuggability if you want to develop
+uv-secure yourself). Also note, winloop is a relatively young package and may give you
+some stability issues on particular versions of Python
+
+If you want to install the optional dependency with uv do it like this:
+
+```shell
+uv tool install uv-secure --with uvloop
+```
+
+or with pipx like this:
+
+```powershell
+pipx install uv-secure
+pipx inject uv-secure winloop
+```
+
+uv-secure will automatically use uvloop or winloop if it finds them in the same
+environment as itself.
+
 ## Usage
 
 After installation, you can run uv-secure --help to see the options.
@@ -68,6 +92,8 @@ After installation, you can run uv-secure --help to see the options.
 │                                     the vulnerabilities table                        │
 │ --desc                              Flag whether to include vulnerability detailed   │
 │                                     description in the vulnerabilities table         │
+│ --disable-cache                     Flag whether to disable caching for              │
+│                                     vulnerability http requests                      │
 │ --ignore              -i      TEXT  Comma-separated list of vulnerability IDs to     │
 │                                     ignore, e.g. VULN-123,VULN-456                   │
 │                                     [default: None]                                  │
@@ -101,19 +127,53 @@ uv-secure can read configuration from a toml file specified with the config opti
 
 ```toml
 ignore_vulnerabilities = ["VULN-123"]
-aliases = true
-desc = true
+aliases = true # Defaults to false
+desc = true # Defaults to false
+
+[cache_settings]
+cache_path = "~/.uv-secure" # Defaults to ~/.cache/uv-secure if not set
+ttl_seconds = 60.0 # Defaults to one day (86400 seconds) if not set
+disable_cache = false # Defaults to false if not set
 ```
 
 ### pyproject.toml
 
 ```toml
 [tool.uv-secure]
 ignore_vulnerabilities = ["VULN-123"]
-aliases = true
-desc = true
+aliases = true # Defaults to false
+desc = true # Defaults to false
+
+[tool.uv-secure.cache_settings]
+cache_path = "~/.uv-secure" # Defaults to ~/.cache/uv-secure if not set
+ttl_seconds = 60.0 # Defaults to one day (86400 seconds) if not set
+disable_cache = false # Defaults to false if not set
 ```
 
+### File Caching
+
+File caching is enabled by default to speed up subsequent runs of uv-secure. By default,
+cache results are saved to:
+
+```shell
+~/.cache/uv-secure
+```
+
+or on Windows
+
+```powershell
+%USERPROFILE%\.cache\uv-secure
+```
+
+This can be configured to another location if you wish.
+
+#### Cache Performance on Windows
+
+I'm unsure about other operating systems, but I found on Windows unless I excluded the
+cache directory from the _Virus & threat protection settings_ the file caching only made
+a minimal performance improvement on subsequent runs (whereas it can speed up subsequent
+runs over 50% if you add the cache directory as an exclusion).
+
 ### Configuration discovery
 
 If the ignore and config options are left unset uv-secure will search for configuration
@@ -123,7 +183,7 @@ uv-secure tries to follow
 [Ruff's configuration file discovery strategy](https://docs.astral.sh/ruff/configuration/#config-file-discovery)
 
 Similar to Ruff, pyproject.toml files that don't contain uv-secure configuration are
-ignored. Currently if multiple uv-secure configuration files are defined in the same
+ignored. Currently, if multiple uv-secure configuration files are defined in the same
 directory upstream from a uv.lock file the configurations are used in this precedence
 order:
 
@@ -150,7 +210,7 @@ uv-secure can be run as a pre-commit hook by adding this configuration to your
 
 ```yaml
   - repo: https://github.com/owenlamont/uv-secure
-    rev: 0.6.0
+    rev: 0.7.0
     hooks:
       - id: uv-secure
 ```
@@ -167,13 +227,11 @@ Or manually check the latest release and update the _rev_ value accordingly.
 
 Below are some ideas (in no particular order) I have for improving uv-secure:
 
+- Add package metadata checks, i.e. age of most recent release threshold
 - Package for conda on conda-forge
-- Create contributor guide and coding standards doc
 - Add rate limiting on how hard the PyPi json API is hit to query package
   vulnerabilities (this hasn't been a problem yet, but I suspect may be for uv.lock
   files with many dependencies)
-- Explore some local caching for recording known vulnerabilities for specific package
-  versions to speed up re-runs
 - Add support for other lock file formats beyond uv.lock
 - Support some of the other output file formats pip-audit does
 - Consider adding support for scanning dependencies from the current venv
@@ -221,31 +279,16 @@ installed by default, so I request PyCharm _Install packaging tool_ in the
 _Python Interpreter_ settings (I may just add these in future are dev dependencies to
 reduce the friction if this causes others too much pain). I have also encountered some
 test failures on Windows if you use winloop with setuptools and pip - so you probably do
-want  to switch to the asyncio eventloop if installing those (I'm hoping to continue
-using winloop, but it's a relatively young project and has some rough edges - I may drop
-it as a dependency on Windows if it causes to many issues).
+want to remove winloop if debugging in that environment if you added it.
 
 #### Debugging Async Code
 
 Given uv-secure is often IO bound waiting on API responses or file reads I've tried to
-make it as asynchronous as I can. uv-secure also uses uvloop and winloop which should be
-more performant than the vanilla asyncio event loop - but they don't play nice with
-Python debuggers. The hacky way at present to use asyncio event loop when debugging is
-uncommenting the run import in run.py:
-
-```python
-if sys.platform in ("win32", "cygwin", "cli"):
-    from winloop import run
-else:
-    from uvloop import run
-# from asyncio import run  # uncomment for local dev and debugging
-```
-
-I definitely want to come up with a nicer scheme in the future. Either make the import
-depend on an environment variable to set local development, or perhaps make uvloop and
-winloop extra dependencies with asyncio event loop being the fallback so you can choose
-not to include them (I need to research best/common practise here some more and pick
-something).
+make it as asynchronous as I can. uv-secure also uses uvloop and winloop if installed
+which should be more performant than the vanilla asyncio event loop - but they don't
+play nice with Python debuggers. If you intend to do debugging I suggest leaving them
+out of the virtual environment. By default, winloop or uvloop won't be installed the
+repo venv unless you explicitly add them.
 
 ## Related Work and Motivation
 

pyproject.toml

@@ -32,15 +32,13 @@ classifiers = [
 dependencies = [
     "anyio>=4.7.0",
     "asyncer>=0.0.8",
+    "hishel>=0.1.1",
     "httpx>=0.28.1",
     "inflect>=7.4.0",
     "pydantic>=2.10.3",
-    "pytest-asyncio>=0.25.2",
     "rich>=13.9.4",
     'tomli; python_version < "3.11"',
     "typer>=0.15.1",
-    "uvloop>=0.21.0 ; sys_platform != 'win32'",
-    "winloop>=0.1.7 ; sys_platform == 'win32'",
 ]
 
 [project.scripts]
@@ -98,7 +96,7 @@ init_typed = true
 warn_required_dynamic_aliases = true
 
 [tool.pytest.ini_options]
-asyncio_default_fixture_loop_scope = "function"
+asyncio_default_fixture_loop_scope = "session"
 filterwarnings = [
     "error",
     "ignore::ResourceWarning" # Disabling as I think pytest-httpx is somehow causing
@@ -200,6 +198,7 @@ convention = "google"
 dev = [
     "coverage>=7.6.9",
     "pytest>=8.3.4",
+    "pytest-asyncio>=0.25.2",
     "pytest-cov>=6.0.0",
     "pytest-httpx>=0.35.0",
     "pytest-mock>=3.14.0",

src/uv_secure/__version__.py

@@ -1 +1 @@
-__version__ = "0.6.0"
+__version__ = "0.7.0"

src/uv_secure/configuration/__init__.py

@@ -1,8 +1,20 @@
 from uv_secure.configuration.config_factory import (
     config_cli_arg_factory,
     config_file_factory,
+)
+from uv_secure.configuration.configuration import (
+    CacheSettings,
     Configuration,
+    override_config,
+    OverrideConfiguration,
 )
 
 
-__all__ = ["Configuration", "config_cli_arg_factory", "config_file_factory"]
+__all__ = [
+    "CacheSettings",
+    "Configuration",
+    "OverrideConfiguration",
+    "config_cli_arg_factory",
+    "config_file_factory",
+    "override_config",
+]