Add support to discover content packages based on datasets

[mrodm]

Closes #1336

This PR adds support to discover packages based on the datasets defined in the discovery parameter:

discovery:
  datasets:
    - name: nginx.access
    - name: nginx.error

Example of query:

GET /search?discovery=datasets:nginx.error`

If there are multiple discovery parameters in the query, like:

GET /search?discovery=datasets:nginx.error&discovery=fields:process.pid`

packages must match all filters at the same time. In this example:

• packages must just define process.pid as its only discovery field.
• packages must at least define nginx.error in their discovery datasets.

Author's checklist

• Update tests including discovery.datasets field.
• Ensure that it can be defined more than once discovery query parameter.
• Add changelog entry
• Test that it can be used with discovery.fields filter.

How to test this PR locally

mage build

./package-registry

# in other terminal
# it should return "good_content" content package
curl -s "http://localhost:8080/search?discovery=datasets:good_content.access&prerelease=true"

# it should not return any package
curl -s "http://localhost:8080/search?discovery=datasets:good_content.none&prerelease=true"

# using `datasets` and `fields` discovery filters, it should return also "good_content" package
# using both filter, packages must match in all filters
curl -s "http://localhost:8080/search?discovery=datasets:good_content.access&prerelease=true&discovery=fields:process.pid"

[prodsecmachine]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[mrodm]

Change required to allow get the values of all the discovery query parameter if it is set several times.

[mrodm]

The way it has been implemented, in order to add a package into a response, the filter must match the given package for fields and for datasets.

So in a query like

GET /search?discovery=fields:process.pid&discovery=datasets:nginx.error

The packages added to the response must fulfill these two conditions:

• define just process.id in their discovery.fields,
• define at least nginx.error in their discovery.datasets.

[mrodm]

This key is shown because discovery is not empty:

• Package manifest:

discovery:
  fields: []
  datasets: []

• Field definition in BasePackage struct:

	Discovery               *Discovery           `config:"discovery,omitempty" json:"discovery,omitempty" yaml:"discovery,omitempty"`

[jsoriano]

Would omitzero help here?

[mrodm]

It looks it could be used to allow not show discovery: {} in the response.

I'll push a change to include this tag for json.
Thanks!

[mrodm]

Pushed change in f6825ec

[jsoriano]

Would omitzero help here?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0c10032

History

• 💚 Build #1088 succeeded 7916f34
• 💚 Build #1086 succeeded f6825ec
• 💚 Build #1077 succeeded 49106f9
• 💚 Build #1073 succeeded 2a14ac4
• 💚 Build #1066 succeeded dc6fed9
• 💚 Build #1065 succeeded 1554fd7

cc @mrodm