feat: Collect DNS records from NS1

[akash1810]

What does this change, and why?

The NS1 source plugin (https://github.com/guardian/cq-source-ns1) is written in TypeScript, and distributed as a Docker image.

Previous attempts (see #337) to use this plugin failed, because ECS containers cannot interact with Docker (see aws/containers-roadmap#1356).

In this change, we run the NS1 source plugin directly, in its own container, resulting in a GRPC server on localhost:7777. We then configure the CloudQuery sync container to connect to this endpoint.

Collecting this data provides more information about our running services. For example, we can identify which services use Fastly, CloudFront, or no CDN.

We can also use this data to identify "dangling DNS records". That is, records whose answer no longer exist, and thus should be deleted, saving on the limited number of records available to us.

Resulting architecture

Standard task

flowchart TD
    A[Fire Lens] --> B(CloudQuery Sync)

Loading

NS1 task

flowchart TD
    A[Fire Lens] --> C(CloudQuery Sync to localhost:7777)
    B[CloudQuery NS1] --> C(CloudQuery Sync to localhost:7777)

Loading

How has it been verified?

Deployed to CODE, ran the task, and seen NS1 data in Grafana CODE 🎉.

Note

The NS1 API key used in CODE has been copied from https://github.com/guardian/dns-validation-lambda in CODE. This key has been limited to particular zones. We'd need to generate a new API key, with read-only access to all zones.

[github-actions]

Deploy build 2968 of deploy::service-catalogue to CODE

All deployment options

• Deploy build 2968 of deploy::service-catalogue to CODE
• Deploy parts of build 2968 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.

[AshCorr]

Nice 👌

[AshCorr]

Suggested change

	dockerDistributedPluginImage?: RepositoryImage;
	additionalPluginImage?: RepositoryImage;

or

Suggested change

	dockerDistributedPluginImage?: RepositoryImage;
	pluginContainer?: RepositoryImage;

Think this would work with any kind of Container image right? Not just Docker images?

[akash1810]

I used docker here as a way to distinguish CloudQuery plugins written in Go, vs not-Go. The former is distributed as a binary, and the latter as a Docker image, see https://docs.cloudquery.io/docs/developers/creating-new-plugin/javascript-source#releasing-and-deploying-your-plugin.

[AshCorr]

I think this container would be essential no? I guess the main CloudQuery container will crash either way so probably not that important.

[akash1810]

I got an error from this previously... I can't remember the exact error, I've a vague memory that CloudFormation failed to deploy the change.

[mchv]

A new read-only API key has now been created and is ready to be used.

[akash1810]

In this change, we run the NS1 source plugin directly, in its own container, resulting in a GRPC server on localhost:7777. We then configure the CloudQuery sync container to connect to this endpoint.

Confirming this is how CloudQuery suggest doing this too! https://docs.cloudquery.io/docs/advanced-topics/using-cloud-query-docker-registry-plugins-inside-a-containerized-environment