How to centralise datastore

[Praful]

I'm trying to centralise the datastore so that it doesn't exist in mulitple directories.

I've created this alias for docker then initialised the datastore:

alias noseyparker='docker run -v "$PWD":/scan ghcr.io/praetorian-inc/noseyparker:latest'
cd $TMPDIR
noseyparker datastore init

This creates a datastore in $TMPDIR/datastore.np/

When I run

noseyparker scan --datastore $TMPDIR/datastore.np/ .

I get error

Error: Failed to open datastore at /home/praful/tmp/datastore.np/: unable to open database file: /home/praful/tmp/datastore.np/datastore.db: Error code 14: Unable to open the database file

even though the file does exist.

How should I create a centralised datastore and use it for scanning and reporting?

Thanks

[bradlarsen]

The Docker bind mounts complicate the invocation for sure. Presumably $TMPDIR is not a subdirectory of $PWD for you.

What happens if you try this invocation instead?

cd $TMPDIR
noseyparker datastore init
noseyparker scan --datastore datastore.np/ .

Of course, in the above, the . scan target would be $TMPDIR, and probably not what you actually want to scan...

[Praful]

Thanks for replying. Running from $TMPDIR works if I want to scan the $TMPDIR. If I try another folder ie

cd $TMPDIR
noseyparker datastore init
noseyparker scan --datastore datastore.np/ $HOME/Downloads

I get the following error:

2024-07-21T10:15:20.350681Z ERROR input_enumerator: Failed to get entry: /home/praful/Downloads: No such file or directory (os error 2); skipping
Scanned 0 B from 0 blobs in 0 seconds (0 B/s); 0/0 new matches

As you said Docker bind complicates things. I may compile locally (but the article that pointed me to the app said it took a while hunting down the dependencies) or just write a short bash function that runs scan/report then deletes the datastore.

[bradlarsen]

@Praful So, I think the issue that you're running into is purely a Docker thing.

A Docker container runs with its own view of the filesystem. If you want to read files from the host or write files back to the host, you have to map those paths into paths in the Docker container.

Here's your noseyparker alias again:

alias noseyparker='docker run -v "$PWD":/scan ghcr.io/praetorian-inc/noseyparker:latest'

The Docker invocation maps $PWD — the current directory — from the host to /scan within the container using a bind mount. This only exposes the current directory and its child entries on the host under /scan in the container.

In your latest test, you switched to $TMPDIR:

cd $TMPDIR
noseyparker datastore init
noseyparker scan --datastore datastore.np/ $HOME/Downloads

When you run noseyparker from there, $PWD is $TMPDIR, which presumably is some path not beneath $HOME for you.

If you expand the values of $TMPDIR and $HOME, it may be clearer what's going on. My guess is that they have values something like this:

TMPDIR=/tmp
HOME=/Users/praful

In this case, what you are trying to scan (/Users/praful/Downloads) is not beneath /tmp/, which is exposed as /scan within the Docker container.

Does this make sense?

P.S. From your use case, it sounds like it might be useful to have some Nosey Parker mode that scans and reports all in one step, without saving the intermediate datastore. 🤔

[Praful]

@bradlarsen thank you for explaining what's going on with docker. You've guessed sufficiently correctly to help me find a workable solution. Everything is a subdir of $HOME. $TMPDIR is $HOME/tmp. So I could make $HOME the $PWD for docker and provide relative paths to that for scanning and the datastore. It's, perhaps, not ideal to expose the whole of $HOME to a third party docker container!

Your understanding of my use case is also correct: I'd prefer a combined scan/report function!

I appreciate that scanning and reporting are logically distinct operations. I guess scan is a "summary" view and "report" a detailed view of the results and the app is storing the results of the scan in case the user later wants a detailed view and another scan could be too time-consuming? Perhaps there's a historical element to the feature set but an output switch could be provided for those who want to save the results, either in "summary" or "detailed" mode?

Thanks again for engaging with my issue 😊

[bradlarsen]

I guess scan is a "summary" view and "report" a detailed view of the results and the app is storing the results of the scan in case the user later wants a detailed view and another scan could be too time-consuming?

The scan operation is what actually finds content given the initial input seeds (filesystem paths, Git URLs, GitHub usernames, etc.) and runs the detection mechanisms to find likely secrets. Though Nosey Parker tends to be orders of magnitude faster than similar secret detection tools, this phase can be expensive on large inputs (e.g., ~hours when dealing with ~terabytes of input).

The scan operation records each likely secret it encounters, and some metadata, to the datastore. Most of the information there is stored in a sqlite database, though this is an implementation detail and subject to change (so don't rely on the database schema being stable).

At the end of the scan operation, a summary of findings is printed to stdout. This is the same output as you get from the summarize operation.

The report operation simply renders the content recorded from the earlier scan operation in one of a handful of formats. This should be quick to execute, as only the sqlite database in the datastore needs to be consulted -- no other file I/O is performed.

I guess scan is a "summary" view and "report" a detailed view of the results and the app is storing the results of the scan in case the user later wants a detailed view and another scan could be too time-consuming?

Yes, that's exactly the reason for the separation of scanning and reporting! In the glorious future, perhaps Nosey Parker will have some additional filtering capabilities to its report operation, allowing for iterative filtering of output.

[Praful]

Thanks, again, for the background to some of the design decisions.