Example terraform plan for Faktory setup?

[mperham]

Would it be useful to provide a Terraform plan for setting up Faktory on cloud providers? Does anyone have a real one used in production they wish to share as a starting point?

I'm thinking something that a developer could use to provision Faktory within seconds to test drive it, for example.

[jbielick]

As a starting place, maybe it would be useful to provide a golden AMI similar to the docker image that's offered? Terraforming an instance with an AMI id that would get you up and running would be a great start IMO. I assumed you're talking about a VM-based deployment, is that correct?

[scottrobertson]

If going down the AMI route, it's probably good to use something like https://www.packer.io. Try not to hardcode it to AWS, as not everyone uses it.

[mperham]

My understanding of terraform is that it is supposed to make Faktory setup cloud independent, tying in an Amazon image defeats that purpose. I’d like this to support Azure, DigitalOcean, etc. is that a pipe dream?

…

On May 24, 2020, at 06:58, Josh Bielick ***@***.***> wrote:  As a starting place, maybe it would be useful to provide a golden AMI similar to the docker image that's offered? Terraforming an instance with an AMI id that would get you up and running would be a great start IMO. I assumed you're talking about a VM-based deployment, is that correct? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jbielick]

Terraform would only fulfill the infrastructure as code portion. That is, you could define the existence of an EC2 instance in AWS, the existence of a compute instance in GCP, but those definitions are not abstract / cloud agnostic.

An EC2 in AWS:

provider "aws" {
  region = "us-west-2"
}

# the below does a search for "most recent ubuntu 14.04 machine image for amd64  published by
# Canonical themselves". The ID is then used as the machine image for the ec2 instance.
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

# you could imagine this definition being a faktory server, 
# but the AMI would need to have faktory # installed on it, 
# hence building a golden AMI.
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}

A compute instance in GCP:

resource "google_compute_instance" "default" {
  name         = "test"
  machine_type = "n1-standard-1"
  zone         = "us-central1-a"

  tags = ["foo", "bar"]

  boot_disk {
    initialize_params {
      # the boot os / machine image of this compute instance. Again, would be 
      # really valuable to have a pre-built Faktory instance image to use (pre-configured)
      image = "debian-cloud/debian-9"
    }
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral IP
    }
  }

  metadata_startup_script = "echo hi > /test.txt"
}

Would the terraform definitions help someone? I guess so, those the examples in the docs are often enough to get started. To provide more value you might have to include definitions like adding security groups and open ports, EBS volume, etc, but that's a lot to assume about someone's network topology, etc.

IMO the real value proposition (save me time as an infra engineer) would be providing machine images that are preconfigured. An AMI in AWS, a GCP machine image, etc (I'm not familiar with what this looks like in every cloud provider). At least at that point I wouldn't have to run configuration management tooling on the instance to install faktory, redis, openssl, etc. If you wanted to provide that golden machine image, you'd have to configure and create it for each cloud provider AFAIK—that's where Packer comes in. You could, maybe, use packer to define a builder for each cloud provider you're targeting. Provision the linux instance of your choosing, configure it via the configuration management tool of your choice, then snapshot it into a machine image for use on that cloud provider's repository. Packer will help automate all of that—you'll just need to fill in the blanks (like what you need to do to configure the instance [apt-get install, etc]).

I think the machine image is half the value. The other half is setting up the actual cloud provider VM with a persistent disk mounted in the right place, firewall rules (anything else?). I guess you could provide some example terraform definitions for all of that. Those definitions won't be cloud agnostic, though. You would need a set of definitions for each cloud provider.

This all feels really similar to the "Marketplace" offerings in AWS / GCP (not familiar with others, but I bet they have something analogous).

Amazon Marketplace

AWS Marketplace is a digital catalog with thousands of software listings from independent software vendors that make it easy to find, test, buy, and deploy software that runs on AWS

GCP Marketplace

Explore, launch, and manage solutions in just a few clicks. Marketplace lets you quickly deploy software on Google Cloud Platform

Azure Marketplace

Provision end-to-end solutions quickly and reliably

Those are effectively "one-click" deploy solutions. That may target a slightly different audience than you're trying to. I only point those out to say that they're the whole shebang—machine image, instance, peripheral, and network component definitions. You're up and running in no time. It's a different paradigm than terraform and the setup for something like that is extremely vendor-specific.

All that said, I think that's why I said it might be a good starting place to create and offer a machine image on various cloud providers. Excuse my use of "AMI"—I didn't mean that you'd only create an Amazon Machine Image, but AFAIK there's no "machine image that can be used across all providers". At that point you're just describing a docker image.

So for VMs, does this make sense? I say all this assuming the goal is "make Faktory easy to deploy". It might be helpful to describe the exact persona you're targeting for this user story: an infra engineer looking to try this out, a production-ready deployment someone can setup without knowing a lot about infra, a try-it-out style deployment to get familiar and grow from?

A note on container-based deployments

If anyone finds this thread and is more concerned with a container-based deployment of Faktory, there might be a few ways of running containerized Faktory in the cloud which could be covered by the following (not exhaustive) list:

• Kubernetes deployment via Helm

The Helm chart uses the official Faktory docker image, defines persistent storage, networking, and monitoring (thanks, @Envek). This is essentially cloud-agnostic.

• HashiCorp Nomad

Probably fairly easy to provide an example definition for this.

• Docker Swarm
• Mesos

[mperham]

Yeah, this all came about when I saw the DO Marketplace and recalled that Mastodon provided a Marketplace setup.

Since we have Docker images for everything, I'm wondering if there is a generic Ubuntu LTS+Docker image already where you can plug in a docker image id "contribsys/faktory:1.4.0" along with a persistent volume. That'll get us 90% of what we need. I was also thinking about a more advanced recipe that sets up a smaller, secondary instance with a Redis replica for Faktory Pro but one step at a time...

We could create example AWS and GCP terraforms, with the understanding that instance type needs will vary greatly based on expected load (i.e. 10 jobs/min or 1,000 jobs/sec?).

[Envek]

For now the most simple way to run faktory is Kubernetes.

You just do helm install faktory like this:

helm repo add adwerx https://adwerx.github.io/charts
helm install test-faktory adwerx/faktory --set "password=VerySecurePass,persistence.size=100Mi"

1 minute and voila: faktory is running with some automatically allocated persistent storage.

And if you need Kubernetes, you can use Terraform to create some managed cluster for you. Here is my example to create Amazon EKS with Terraform for reference: terraform-eks-example

[ibrahima]

This is definitely very late, but I think one way to make this reasonably simple would be to use the cloud provider's "serverless" (I hate that term) container offering. E.g. with AWS ECS Fargate, you can point it at the Faktory Docker image and define a service without having to stand up an EC2 cluster behind it. I wouldn't be surprised if other cloud providers have a similar service. But the Terraform needed to spin up the service would be cloud-provider specific as others have mentioned.

Using managed Kubernetes as mentioned above would also be relatively provider-agnostic but how to set up the managed cluster would still be provider-specific.

Oh hmm, looks like there's a Wiki page for roughly what I described, for AWS ECS: https://github.com/contribsys/faktory/wiki/AWS-ECS. Sorry for the random noise (I came across this issue looking for something else, and thought I had an idea to contribute 😕)