Add trufflehog for secret detection #150

hugo-syn · 2024-04-19T12:47:57Z

Hi,

I've discovered your tool via secator it's very nice thanks for your work !

In this PR I'm adding 2 things, the first big improvement is the use of trufflehog for secret detection. Trufflehog can detect a looot of credentials, the complete list can be found here. This change make the tool slower for secret detection but improve the secret detection.

I've also add a command line option to ignore some url for secret scaning. For example scaning png files for secrets can be avoided:

$ cariddi  -s -secret-extension-filter pdf,svg,png,jpeg
...

… file could be avoided

…not present in trufflehog like s3 buckets.

edoardottt · 2024-04-19T13:06:53Z

Hi @hugo-syn , thank you so much for improving cariddi! Appreciated :)

I don't have a lot of time and since this is a big PR I'll take some time to review it.
For now try to resolve the conflicts with the devel branch for go.mod and go.sum files.

hugo-syn · 2024-04-19T13:18:10Z

Hi, @edoardottt I've fixed the merge conflicts, I've updated some packages.

For the PR I understand, don't hesitate if you have questions or remarks regarding my code :)

edoardottt · 2024-04-19T13:22:42Z

If possible (there could be issues with deps) revert back go.mod to using go 1.21 and remove toolchain info (the linter action fails :,( ).

hugo-syn · 2024-04-19T13:34:31Z

The trufflehog package requires go 1.22.0:

$ go build ./cmd/cariddi
go: github.com/trufflesecurity/trufflehog/v3@v3.73.0 requires go >= 1.22.0 (running go 1.21.0)

I can bump the golang version in the different CI files if you want

hugo-syn · 2024-04-19T13:51:26Z

And I've checked it looks good with go 1.22.0

edoardottt · 2024-04-22T10:28:09Z

Hi @hugo-syn,

And I've checked it looks good with go 1.22.0

the go build action is failing:

go build -v ./...
  shell: /usr/bin/bash -e {0}
  env:
    GOROOT: /opt/hostedtoolcache/go/1.18.10/x64
go: errors parsing go.mod:
/home/runner/work/cariddi/cariddi/go.mod: invalid go version '1.22.0': must match format 1.23
/home/runner/work/cariddi/cariddi/go.mod: unknown directive: toolchain

https://github.com/edoardottt/cariddi/actions/runs/8754239537/job/24025715013?pr=150#step:6:8

I think this is caused by github actions, they must be set to the correct go version too (all of them: go, linting...).

hugo-syn · 2024-04-29T13:07:37Z

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

edoardottt · 2024-04-29T13:20:53Z

I've change the golang version in the workflows to 1.22.0 the build pipeline is ok now but there are errors with the golang-ci

I've seen. See if locally everything is okay with golangci-lint run. If you need help I'm here. In the next days I'm gonna review it. Since it's a huge PR I guess there will be a lil bit of work to be done... :/ Let's see.

This change make the tool slower

Can you quantify this? I'd like to comprehend how much this will be slower.

cc @hugo-syn . Let me know if you need assistance :)

hugo-syn · 2024-05-17T12:30:43Z

Hi @edoardottt Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Can you quantify this? I'd like to comprehend how much this will be slower.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

edoardottt · 2024-05-17T12:50:28Z

Unfortunately I didn't manage to fix the CI there are new errors with code that I did not modify :/

Ok, so don't worry about golangci-lint errors. I'll fix them. Go build is the important one and it's not failing.

It's difficult to say but since it runs all the secret detection of trufflhog the secret search will take more time but it will find lots of new secrets

Ok, I'll merge in devel and perform some tests.

For now it's okay so, wait for my PR review :)

hugo-syn · 2024-05-17T12:57:06Z

Do not hesitate if you have any questions ! :)

edoardottt

Thanks @hugo-syn !

Requesting some changes
Left some thoughts, let me know yours.

I'm here if you have any doubt or need assistance :)

edoardottt · 2024-05-17T12:53:09Z

cmd/cariddi/main.go

+		Rua:                   flags.Rua,
+		Proxy:                 flags.Proxy,
+		SecretsFlag:           flags.Secrets,
+		SecretExtensionFilter: strings.Split(flags.SecretExtensionFilter, ","),


Better to perform this in another part, maybe in pkg/input/flags.go. Consider that it may be necessary to trim spaces too (TrimSpace() function).

edoardottt · 2024-05-17T12:56:23Z

pkg/crawler/utils.go

@@ -69,3 +71,25 @@ func intensiveOk(target string, urlInput string, debug bool) bool {

 	return root == target
 }
+
+// Return the extension of an URL
+func GetFileExtensionFromUrl(rawUrl string) (string, error) {


We need some tests for this function since it's an important one. You can use the ones you find in this repo as example.
Try to insert as many test cases as possible (use double dots, query parameters, fragment, invalid URL...).

where do you want to do tests ?

edoardottt · 2024-05-17T12:57:46Z

pkg/input/flags.go

@@ -120,6 +122,8 @@ func ScanFlag() Input {
 	secretsPtr := flag.Bool("s", false, "Hunt for secrets.")
 	secretsFilePtr := flag.String("sf", "", "Use an external file (txt, one per line)"+
 		" to use custom regexes for secrets hunting.")
+	secretsExtensionFilterPtr := flag.String("secret-extension-filter", "", "Comma"+


Can we have a shorter name for this? Like sef... do you like it?

yeah sure !

edoardottt · 2024-05-17T12:58:33Z

pkg/input/flags.go

@@ -163,6 +167,7 @@ func ScanFlag() Input {
 		*proxyPtr,
 		*secretsPtr,
 		*secretsFilePtr,
+		*secretsExtensionFilterPtr,


Modify the README.md file according to the modifications made. In particular the new flag provided.

edoardottt · 2024-05-17T13:02:35Z

pkg/crawler/scan.go

+
+		wgScanners.Wait()
+
+		// keep legacy code, the original list contains secrets that are not present in trufflehog like s3 buckets.


I'm not fully understanding this. I guess now the code performs both trufflehog and cariddi secrets detections right? Maybe we should find a solution for this.

A simple solution could be move the S3 regexes to info of the cariddi engine and run only the trufflehog engine for the secrets. But I'm open to suggestions if you have them.

Also, we have to take in mind that the user can input a file with custom regexes, I don't know if trufflehog can do this. If not, maybe we could leave the custom cariddi engine for the custom secrets.

Let me know your thoughts on this.

yes I did not know so I kept both way.

I don't know if you can add custom regexes with trufflehog when used ad a library. I think you should keep both.

edoardottt

PR review adjustments @hugo-syn

edoardottt · 2024-05-17T13:06:12Z

go.mod


 require (
 	github.com/fatih/color v1.17.0
 	github.com/gocolly/colly v1.2.0
+	github.com/trufflesecurity/trufflehog/v3 v3.73.0


I'm leaving this comment here but it's not related to this line specifically.

The goreleaser CI is giving these errors:

Error: ../../../go/pkg/mod/github.com/trufflesecurity/trufflehog/v3@v3.73.0/pkg/gitparse/gitparse.go:200:18: cannot use defaultMaxDiffSize (untyped int constant 2147483648) as int value in struct literal (overflows) Error: ../../../go/pkg/mod/github.com/trufflesecurity/trufflehog/v3@v3.73.0/pkg/gitparse/gitparse.go:201:18: cannot use defaultMaxCommitSize (untyped int constant 2147[48](https://github.com/edoardottt/cariddi/actions/runs/9128423415/job/25100742967?pr=150#step:4:49)3648) as int value in struct literal (overflows)

Do you have any clue about what they mean?

no clue on this I'm not used to this

hugo-syn · 2024-08-05T12:15:38Z

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

edoardottt · 2024-08-06T08:09:42Z

Hi @edoardottt I'm not used to dev practices what should I do ? feel free to rewrite what I did to fit your way :)

Hi @hugo-syn . I don't have write access to your fork + this PR is huge + it seems it was created a while ago and since this would be a big improvement I'll proceed in this way:
I'll create a PR by myself to merge in devel and you can review that for me. Maybe this will speed up the process.

hugo-syn · 2024-08-06T12:19:15Z

Oh ok I thought there would be a mechanism so that you could modify the code directly. Sounds good to me. Do not hesitate if you have any question

ocervell · 2024-11-20T15:00:04Z

@hugo-syn great idea to add trufflehog !
@edoardottt for what it's worth, maintainers can make changes on PRs using the GitHub CLI by running gh pr checkout 150

edoardottt · 2024-11-20T15:46:17Z

Hi @ocervell !
The changes were applied in https://github.com/edoardottt/cariddi/tree/trufflehog.
However it produces a lot of false positives, it's far than acceptable.

hugo-syn added 3 commits April 19, 2024 12:23

feat: add support for trufflehog

2792613

feat: add extension list for secret scanning for exemple scanning png…

cdc202e

… file could be avoided

feat: add legacy code as the original list contains secrets that are …

55900cb

…not present in trufflehog like s3 buckets.

auto-assign bot requested a review from edoardottt April 19, 2024 12:48

edoardottt changed the base branch from main to devel April 19, 2024 13:03

edoardottt self-assigned this Apr 19, 2024

Merge branch 'devel' into trufflehog

a4c17f1

edoardottt added enhancement New feature or request Go labels Apr 22, 2024

edoardottt mentioned this pull request Apr 22, 2024

Add trufflehog for secret detection #152

Open

edoardottt linked an issue Apr 22, 2024 that may be closed by this pull request

Add trufflehog for secret detection #152

Open

edoardottt mentioned this pull request Apr 22, 2024

Add -secret-extension-filter #153

Open

edoardottt linked an issue Apr 22, 2024 that may be closed by this pull request

Add -secret-extension-filter #153

Open

ci: bump golang version to 1.22.0

c1f64b8

Merge branch 'devel' into trufflehog

3519c94

edoardottt requested changes May 17, 2024

View reviewed changes

edoardottt reviewed May 17, 2024

View reviewed changes

Merge branch 'edoardottt:main' into trufflehog

32cbfda

fix: filter secrets

00d38c6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add trufflehog for secret detection #150

Add trufflehog for secret detection #150

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 22, 2024 •

edited

Loading

hugo-syn commented Apr 29, 2024

edoardottt commented Apr 29, 2024 •

edited

Loading

hugo-syn commented May 17, 2024

edoardottt commented May 17, 2024

hugo-syn commented May 17, 2024

edoardottt left a comment •

edited

Loading

edoardottt May 17, 2024

edoardottt May 17, 2024

hugo-syn Aug 5, 2024

edoardottt May 17, 2024

hugo-syn Aug 5, 2024

edoardottt May 17, 2024

edoardottt May 17, 2024

hugo-syn Aug 5, 2024

edoardottt left a comment •

edited

Loading

edoardottt May 17, 2024

hugo-syn Aug 5, 2024

hugo-syn commented Aug 5, 2024

edoardottt commented Aug 6, 2024

hugo-syn commented Aug 6, 2024

ocervell commented Nov 20, 2024 •

edited

Loading

edoardottt commented Nov 20, 2024


		wgScanners.Wait()

		// keep legacy code, the original list contains secrets that are not present in trufflehog like s3 buckets.

Add trufflehog for secret detection #150

Are you sure you want to change the base?

Add trufflehog for secret detection #150

Conversation

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

hugo-syn commented Apr 19, 2024

edoardottt commented Apr 22, 2024 • edited Loading

hugo-syn commented Apr 29, 2024

edoardottt commented Apr 29, 2024 • edited Loading

hugo-syn commented May 17, 2024

edoardottt commented May 17, 2024

hugo-syn commented May 17, 2024

edoardottt left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edoardottt left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hugo-syn commented Aug 5, 2024

edoardottt commented Aug 6, 2024

hugo-syn commented Aug 6, 2024

ocervell commented Nov 20, 2024 • edited Loading

edoardottt commented Nov 20, 2024

edoardottt commented Apr 22, 2024 •

edited

Loading

edoardottt commented Apr 29, 2024 •

edited

Loading

edoardottt left a comment •

edited

Loading

edoardottt left a comment •

edited

Loading

ocervell commented Nov 20, 2024 •

edited

Loading