The SQS queue is used to store the URLs that need to be crawled.
DynamoDB is used to store the URLs that have been crawled and the URLs that have been found on the page.
Neptune is used to store the relationships between the URLs that have been found on the page.
The tests are written in Go and can be run using the following command:
bash go test ./...