Parallel package catalog processing

[Mikcl]

Closes #1353

This introduces a new option SYFT_PARALLELISM=N environement variable and syft config file value. Which will use at most N workers to process the package catalogers in parallel. The default will be 1.

• uses the sync library to create a wait group, waiting for all package catalogers to finish before proceeding.
• effectively has no real change in behaviour to users calling syft today, as the default is to create one worker.

---

example performance benchmark comparision

Open to recommendations on performance benchmarking but early results on laptop seem promising:

Created an exmple directory with venv and node_modules

➜  syft git:(mikcl/concurrent-catalog) SYFT_PARALLELISM=1 time go run cmd/syft/main.go packages  --file ../dump  /path/to/project
[indexing truncated]
 ✔ Indexed /Users/mikcl/Documents/junk/benchmark-concurrent
 ✔ Cataloged packages      [758 packages]

go run cmd/syft/main.go packages --file ../dump   13.29s user 1.75s system 114% cpu 13.179 total
➜  syft git:(mikcl/concurrent-catalog) SYFT_PARALLELISM=4 time  go run cmd/syft/main.go packages   --file ../dump  /path/to/project
[indexing truncated]
 ✔ Indexed /Users/mikcl/Documents/junk/benchmark-concurrent
 ✔ Cataloged packages      [758 packages]
go run cmd/syft/main.go packages --file ../dump   14.16s user 1.94s system 174% cpu 9.235 total

1 worker: 13.29s user 1.75s system 114% cpu 13.179 total
4 workers: 14.16s user 1.94s system 174% cpu 9.235 total

uname -a
Darwin Kernel Version 21.6.0: [date redacted]; root:xnu-8020.141.5~2/RELEASE_ARM64_T8101 arm64

For larger file systems and cpu's with more cores, this may be useful :)

[kzantow]

This seems like a great idea! I wonder if it could be simplified a bit (see some initial comments). Also, I wonder if there would be a way to extract this to some more "generic" function like runInParallel(workers, func()...)...

[kzantow]

A friendly reminder that you'll need to sign-off your commits (https://github.com/anchore/syft/blob/main/CONTRIBUTING.md#sign-off-your-work)

[Mikcl]

Thanks for the initial review @kzantow (i think i may take this out of draft now)

This is the concurrency pattern that i have been following. I will assume for it idiomatic?

Summarise the approach / requirements:
Given N jobs and W workers,
we would need to return two things from the job (1. discovered packages dynamically, 2. "results" to aggregate from all jobs)

I think the sync/WaitGroup approach is suitable here.

• Set up W workers to independently read from the jobs channel.
• publish N catalogers to the jobs channel.
• worker w picks up the next job, and does necessary processing and result publishing to desried channels.
  • discoveredPackages to publish dynamic values from this process.
  • results for later aggregation
• once wg.Wait is ready then we can aggregate

Are we aligned with the general steps? I'm open to exploring other concurrency patterns, however the waitgroup seems advantageous here due to Wait and our need to aggregate?

wdyt?

[wagoodman]

My preference would be to introduce only an application configuration option first and not a CLI option. That is, keep the SYFT_PARALLELISM env var / parallelism config var and remove the --parallelism CLI switch.

[wagoodman]

Open question / asking for folks opinion: why introduce a number of logical concurrent workers? We could lean on the default GOMAXPROCS value and simply start go routines for all catalogers without setting an artificial limit.

[Mikcl]

• Have hidden the cli --parallelism flag. (such that it will not show up as incouraged interface/in the --help text, but still functional).

SYFT_PARALLELISM is the corresponding env variable.

Do we want to remove the flag entriely?

We could lean on the default GOMAXPROCS?

This pr currently setting 1 as the default (i.e "same" as without). Would encourage keeping it the same in initial release before increasing / settting different default.

My (non maintainer) pov:

• we could set GOMAXPROCS in a following PR as the default.
  • however we dont necessarily know how this is used by other users / whether this would impact their workflows in anyway. Something which could be comunicated in advance?
  • Do not think there is much risk invovled but configurability would be good.

[Mikcl]

Any updates on @wagoodman open question and interface comments above from the syft team.

[Mikcl]

just bumping in case there are any new updates on this from syft team.

cc @kzantow

[wagoodman]

Sorry for the radio silence -- I thought a little more about this. I agree having a more explicit config option (SYFT_PARALLELISM) instead of leaning on GOMAXPROCS would be more obvious.

I do think we should make this an app-config-only update and still not introduce a CLI flag. That is, I see that you hid the flag, but ideally it wouldn't be on the CLI at all. I can help with those updates if you'd like?

[Mikcl]

thanks for the update and thoughts on the direction @wagoodman

• updated to be an environment variable (SYFT_PARALLELISM) and in yaml config option only.
• (removed cli flag option).
• updated tests and PR text to reflect the change.

[Mikcl]

hny!, what will it take to get this across the line?

[Mikcl]

Is this still progressing? happy with waiting, would be good to get any feedback if possible.

understand that there may be other priorities so will leave this open for now :)

[wagoodman]

Sorry for the radio silence -- I'm going to push a couple minor tweaks, then I think it'll be go. Nice work @Mikcl !

[wagoodman]

nice enhancement @Mikcl !

[Mikcl]

Thanks for the logging changes, looks good

(some tests failed due to the logging change, made a commit to update it).

I feel like it is useful to assert some of the logging (There were tests for it which failed due to logging update commit)

This last commit should fix the tests that failed.

Please feel free to auto-enable-merge / merge when available if approved changes.

[Mikcl]

this is green now 🚀

Thanks for your patience on this PR @wagoodman

[wagoodman]

My bad, thanks for the fix!