Filebeat's Snyk module only supports retrieval of 10 items

[michaelhyatt]

• Version: 7.14.0
• Operating System: Any
• Steps to Reproduce: Configure and start filebeat with Snyk module with -e.

I can only see 10 items being sent to Elasticsearch although there should be way more. In the log file, there is a line indicating that only one call has been made to the backend. The perPage parameter is set to 10, which suggests that it is just not paginating beyond retrieving the first 10 items. I also don't see any subsequent calls to Snyk APIs to retrieve more items.

2021-08-10T16:40:07.233+1000	INFO	[input.httpjson-cursor]	v2/request.go:210	request finished: 10 events published	{"id": "FEE286D5CD835789", "input_source": "https://snyk.io/api/v1/reporting/issues/?page=1&perPage=10&sortBy=issueTitle&order=asc&groupBy=issue", "input_url": "https://snyk.io/api/v1/reporting/issues/?page=1&perPage=10&sortBy=issueTitle&order=asc&groupBy=issue"}

^^ @papicella

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[P1llus]

It seems that we need to add the on_template_error to pagination, to prevent it from failing during pagination, will start to apply a fix for this