investigate services TTS systems are having to use to supplement cloud.gov

[afeld]

Background information

cloud.gov offers a number of managed services, which is great for teams not having to worry about managing that infrastructure, the compliance, additional user accounts and permissions, etc. That said, there have been a number of cases where a TTS system needs a service that isn't offered, and then need to manage it on its own.

Example limitations from TTS Programs:

• Federalist: queueing (e.g. AWS SQS)
• Touchpoints: transactional email (e.g. AWS SES) - cc https://github.com/18F/tts-tech-portfolio-private/issues/880
• Data.gov: No Solr brokered service (e.g. no self-managed backend service option)
• Search.gov: Custom High Availability Elasticsearch cluster (e.g. no self-managed backend service option)
• USA.gov: Hard to manage customization and state for traditional CMS applications (e.g. drupal)

Generalized Examples:

• No Vulnerability scanning as-a-service offerings (e.g. SonarQube)
• No Container Registries as-a-service offerings (e.g. Harbor)
• No Git as-a-service offerings (e.g. Gitlab)
• No CI/CD as-a-service offerings (e.g. drone.io, jenkins, etc)
• No queuing backend services offerings beyond Redis (e.g. Kafka, Rabbitmq, etc.)
  • Requires dev work to migrate to redis (if at all possible)
• 6GB disk quota limit for Applications (e.g. No service broker offering for Networked File System (NFS))
• Limited brokered services for AWS managed services (only RDS is offered)
• Brokering GCP/Azure services is not an option to self-authorize a supplemental IaaS provider
• Kubernetes-as-a-Service nor bosh-deployments are offered to customers - self-hosting open source application stacks are nearly impossible to deploy via a cf push given the complexity around the orchestration of backend service requirements.
  • These project communities are generally maintaining their suggested deployments as docker-compose and/or helm charts. (examples - Gitlab CE - helm chart only support | Defectdojo - helm/docker-compose only | harbor container registry - helm or bosh only)

User stories

• As a developer on a TTS system, I want to manage as little infrastructure (and corresponding compliance) as possible so that I can focus on my application.
• As someone responsible for TTS budget, I don't want duplicate effort of teams spending money building out the same things.

Implementation

• Get an inventory of what services are being leveraged by which systems - cc Draft a Technology Radar for TTS 18F/g-devops#10
• Eliminate services from the list already brokered by cloud.gov
• Eliminate services from the list that aren't easily brokerable, such as if they:
  • Need UI access
• Prioritize remaining services by how widely they are used in TTS
• Present to cloud.gov team
• Break out issues for follow-up work, either by the Tech Portfolio or cloud.gov

Acceptance criteria

• We have a prioritized list of services that would TTS programs would benefit from having available

[afeld]

Discussion in an Engineering @ Solutions meeting

[mogul]

Limited brokered services for AWS managed services (only RDS is offered)

cloud.gov switched to brokering AWS Elasticache (Redis) and AWS Elasticsearch in place of their own k8s-hosted Redis and Elasticsearch, which are now deprecated.

[afeld]

A few projects with immediate needs here:

• Data.gov with Solr
• Federalist with SES/SQS
• FormService with MongoDB
• Touchpoints with SES

@JJediny will be reaching out to get the teams together to see if we can combine efforts and expand some brokering offerings.

[RonWilliams]

@afeld This is my first time seeing this, but we're onboard with prioritizing the needs identified. We're seeking investment to extend cloud.gov further, but having a list of specific items with clearly identified users is helpful for justification.

Thanks for pushing this research forward!

[mogul]

In the Security and Compliance Guild meeting today we also discussed this topic at length. We identified these as other potentially brokerable gap-fillers:

• minimal stack for alerting on logs/metrics. Options that can run entirely in cloud.gov:
  • Existing log-cache + Monit app (no persistence)
  • promtail app + Grafana Loki app + brokered S3
  • Possibly New Relic's FedRAMP'd logging service?
• transactional email (eg SES)
  • Not only for app needs, but also for alerting
• Moderate+ social coding and Moderate+ CI/CD
  • GitHub.com is Li-SaaS, need more than that for supporting Moderate-or-higher teams
  • login.gov is putting GitLab inside their boundary, but it could be brokerable

data.gov is also in need of a transactional email solution (eg SES). We're tackling adding SES to the Supplementary Service Broker(SSB) this week (see the "Sketch" section), so hopefully we'll have something reusable soon.

[ryanwoldatwork]

I'd like to see guidance on email conventions and DNS configs required, that are related considerations to getting SES working.

I'll shareback what we learn.

I've learned that it is not permitted to send mail using an @gsa.gov address from an external source.

So, now I'm considering a @touchpoints.app.cloud.gov email or a @touchpoints.digital.gov email. Likely the latter.

[ryanwoldatwork]

I recently got AWS SES up and running finally.

Technically, its not too bad. DNS records need to be set and verified which typically takes some coordination.

I got hung up on selecting and configuring the "right" email to use. I tried to use an existing @gsa.gov email, which turned out to be a lot more effort (for the org) than using a @touchpoints.digital.gov email.

Early, I pursued the gsa.gov email because it existed and we didn't have a touchpoints.digital.gov email, nor the need for one. But ultimately, I was able to verify the ..digital.gov DNS records successfully; whereas, I could not verify gsa records.

SES could be a good candidate to be brokered. A cloud.gov help page would probably fill the gap as well.

[adborden]

@JJediny and I think the next step is to have a meeting with cloud.gov (and maybe leadership). Now that we have a rough idea of a list of services that could be implemented to help TTS programs, we want to answer:

• who should do that work?
• which order should they be implemented?
• who will own the service once launched?

There are several options, and we should discuss with partners before moving forward. For example, Tech Portfolio could own and manage services. For services that would be good for all cloud.gov customers, beyond TTS, the Tech Portfolio could implement services to hand-off to cloud.gov.

Agenda:

• Introduction and background of this work
• What is cloud.gov's capacity for implementing and managing new brokered services?
• If a hand-off to cloud.gov was desired, what would a human-centered design approach look like?

[adborden]

The SES service has already been implemented by data.gov. In terms of a high-value, low effort brokerable service from the list that would help TTS, this might be the best option.

[mogul]

"has already" is a bit of an overstatement... AWS just changed their domain identity verification method, so I'm having to redo some of that work. Also as delivered, it only supports creating a random @ssb.data.gov address to send from, which isn't as useful as it could be.

[afeld]

Features that GIVE needs from cloud.gov to ATO

[afeld]

This is done; see infrastructure needs.