feat(gcp): Ability to skip project ids in configuration

[TinLe]

Currently config.hcl allow one to give specific project or folder, along with all ('*'). Same for list of resources to fetch, e.g. you can list a bunch of resources.

It would be nice to be able to do the reverse, e.g. denylist.

For project, we can give a list of one or more projects and folder to not fetch data from.

For resources, we can give a list of one or more resources to not fetch data from.

Is your feature request related to a problem? Please describe.

When you have many projects and resources, it can be painful to list all projects EXCEPT for the denied one that you do not want. The same applies to resources, when you have to list all resources except for the one or two that you do not want.

Describe the solution you'd like

An example using GCP provider config:

provider "gcp" {
configuration {
// Nice to be able to put denied folders here, so fetch data from all folders except these.
// folder_ids_denylist = [ "organizations/<ORG_ID>", "folders/<FOLDER_ID>" ]

// project_ids_denylist = [<CHANGE_THIS_TO_YOUR_PROJECT_ID>]

}

// list of resources to NOT fetch
resources_denylist = [
"resource_manager.projects",
]
}

Describe alternatives you've considered

Without denylist, the current solution is to list all projects, except for the denied ones that we do not want to fetch. This does not work for us as we have thousands of projects.

Same idea for resources. Very painful.

Additional context

It's a nice to have, make configuration easier to manage.

[bbernays]

@TinLe we do already support the not_resources Attribute for specifying which resources not to fetch. I am working on the documentation for it tomorrow, so once that is done I will send you a link to it

[TinLe]

@bbernays Did you meant "skip_resources" ? I found that in the code. I did not realize that option exists. Thank you for pointing me to it.

It would be nice also to have something similar for projects and folders. We have too many projects, and it is painful to list all the ones we want to scan vs just listing the smaller number that we do not want.

[bbernays]

Seeing as we already support what you are looking for at the core level (skipping of resources) I am going to move this feature request to the GCP provider so that we can look into supporting skipping projects/folders

[erezrokah]

Thanks for opening the issue @TinLe. I renamed blacklist to denylist as it can considered a more inclusive term. See more in https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white

[bbernays]

@TinLe - Here is the updated documentation for the skip_resources

[erezrokah]

Hi @TinLe thank you for opening the issue and @bbernays for responding.

We've since moved from HCL to YAML and released CloudQuery v1.
With YAML configuration format I think it's easier to handle this request by doing the following:

# Get projects list
export CQ_PROJECTS=$(gcloud projects list --format="json" | jq '[.[].projectId]' | jq -c '. - ["project-1-to-skip","project-2-to-skip"]')

kind: source
spec:
  name: gcp
  path: cloudquery/gcp
  ...
  spec:
    project_ids: ${CQ_PROJECTS}

A similar approach can be used to skip folder_ids, WDYT?

[bbernays]

In V1 we expose the project_filter property that allows users to specify their own queries/filter functionalities and then the filtering is handled on the server side of GCP...For example this allows users to exclude any project that begins with TEST*

https://www.cloudquery.io/docs/plugins/sources/gcp/configuration#gcp-spec

https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects/list

[erezrokah]

n V1 we expose the project_filter property that allows users to specify their own queries/filter functionalities and then the filtering is handled on the server side of GCP...For example this allows users to exclude any project that begins with TEST*

Thanks @bbernays I noticed that but couldn't find an example to negate the pattern. Do you mind posting one here?

[erezrokah]

Kudos to @bbernays, this is the syntax for the project_filter https://cloud.google.com/sdk/gcloud/reference/topic/filters which supports negated patterns.

Pending our comments I'll close the issue. @TinLe if the solutions provided doesn't solve your use case, please comment and I'll re-open