DC/OS generation

[longshorej]

This modifies the CLI to be able to generate DC/OS Marathon configuration.

• Registry without namespace bugfix
• Manually sketch out groups/apps workflow
• CLI subcommand with empty config generation
• Generate an actual config
• Health/Readiness checks
  - [ ] Feature Parity (Secrets, a few others) (I won't have time to do this one)
• MarathonLB labels
• Misc testing

[longshorej]

fair bit to go, but some progress:

 $ rp generate-marathon-configuration     "front-end:0.0.1" --namespace chirper
{
    "apps" : [
        {
            "instances" : 1,
            "container" : {
                "image" : "front-end:0.0.1",
                "network" : "BRIDGE",
                "portMappings" : [
                    {
                        "containerPort" : 0,
                        "servicePort" : 0,
                        "protocol" : "tcp",
                        "name" : "http"
                    }
                ]
            },
            "id" : "/chirper/front-end",
            "labels" : {
                "APP_NAME" : "front-end",
                "APP_NAME_VERSION" : "front-end-v0-0-1"
            },
            "env" : {
                "RP_ENDPOINT_HTTP_BIND_PORT" : "$PORT_HTTP",
                "RP_ENDPOINT_0_BIND_PORT" : "$PORT_HTTP",
                "RP_ENDPOINTS" : "HTTP",
                "RP_APP_VERSION" : "0.0.1",
                "RP_JAVA_OPTS" : "-Dconfig.resource=rp-application.conf",
                "RP_PLATFORM" : "marathon",
                "RP_MODULES" : "common,play-http-binding,service-discovery",
                "RP_ENDPOINT_HTTP_HOST" : "$HOST",
                "RP_ENDPOINT_HTTP_PORT" : "$PORT_HTTP",
                "RP_APP_NAME" : "front-end",
                "RP_ENDPOINT_0_PORT" : "$PORT_HTTP",
                "RP_APP_TYPE" : "lagom",
                "RP_ENDPOINT_0_BIND_HOST" : "$HOST",
                "RP_ENDPOINT_HTTP_BIND_HOST" : "$HOST",
                "RP_ENDPOINT_0_HOST" : "$HOST",
                "RP_ENDPOINTS_COUNT" : "1",
                "RP_NAMESPACE" : "chirper"
            }
        }
    ]
}

-> 0

[longshorej]

Here's one with clustering and marathon-lb:

$ rp generate-marathon-configuration chirp-impl:0.0.1 --registry-disable-https --registry-force-pull true  --env JAVA_OPTS="-Dplay.http.secret.key=asdf -Dplay.filters.hosts.allowed.0=."  --instances 2 --transform-output '.mem = 512' 
{
    "instances" : 2,
    "container" : {
        "docker" : {
            "image" : "chirp-impl:0.0.1",
            "forcePullImage" : true,
            "network" : "BRIDGE",
            "portMappings" : [
                {
                    "containerPort" : 0,
                    "servicePort" : 0,
                    "protocol" : "tcp",
                    "name" : "http"
                },
                {
                    "containerPort" : 0,
                    "servicePort" : 0,
                    "protocol" : "tcp",
                    "name" : "akkaremote"
                },
                {
                    "containerPort" : 0,
                    "servicePort" : 0,
                    "protocol" : "tcp",
                    "name" : "akkamgmthttp"
                }
            ]
        }
    },
    "healthChecks" : [
        {
            "path" : "/platform-tooling/healthy",
            "gracePeriodSeconds" : 60,
            "intervalSeconds" : 60,
            "protocol" : "HTTP",
            "portName" : "akkamgmthttp"
        }
    ],
    "id" : "/chirpservice-v0-0-1",
    "labels" : {
        "APP_NAME" : "chirpservice",
        "APP_NAME_VERSION" : "chirpservice-v0-0-1",
        "HAPROXY_GROUP" : "external",
        "HAPROXY_0_PATH" : "/api/chirps/history /api/chirps/live"
    },
    "mem" : 512,
    "upgradeStrategy" : {
        "maximumOverCapacity" : 0,
        "minimumHealthCapacity" : 0.5
    },
    "args" : [
        "/rp-start",
        "bin/chirp-impl"
    ],
    "killSelection" : "YOUNGEST_FIRST",
    "readinessChecks" : [
        {
            "path" : "/platform-tooling/ready",
            "gracePeriodSeconds" : 60,
            "intervalSeconds" : 15,
            "protocol" : "HTTP",
            "portName" : "akkamgmthttp"
        }
    ],
    "env" : {
        "RP_ENDPOINT_HTTP_BIND_PORT" : "$PORT_HTTP",
        "RP_ENDPOINT_AKKA_MGMT_HTTP_BIND_PORT" : "$PORT_AKKAMGMTHTTP",
        "JAVA_OPTS" : "-Dplay.http.secret.key=asdf -Dplay.filters.hosts.allowed.0=.",
        "RP_ENDPOINT_AKKA_REMOTE_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_0_BIND_PORT" : "$PORT_HTTP",
        "RP_ENDPOINTS" : "HTTP,AKKA_REMOTE,AKKA_MGMT_HTTP",
        "RP_ENDPOINT_AKKA_MGMT_HTTP_PORT" : "$PORT_AKKAMGMTHTTP",
        "RP_APP_VERSION" : "0.0.1",
        "RP_JAVA_OPTS" : "-Dconfig.resource=rp-application.conf -Dakka.discovery.method=marathon-api -Dakka.management.cluster.bootstrap.contact-point-discovery.effective-name=chirpservice -Dakka.management.cluster.bootstrap.contact-point-discovery.required-contact-point-nr=2 -Dakka.discovery.marathon-api.app-label-query=APP_NAME==%s",
        "RP_ENDPOINT_1_PORT" : "$PORT_AKKAREMOTE",
        "RP_PLATFORM" : "mesos",
        "RP_MODULES" : "akka-cluster-bootstrapping,akka-management,common,play-http-binding,service-discovery,status",
        "RP_ENDPOINT_AKKA_REMOTE_BIND_PORT" : "$PORT_AKKAREMOTE",
        "RP_ENDPOINT_AKKA_REMOTE_HOST" : "$HOST",
        "RP_ENDPOINT_2_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_HTTP_HOST" : "$HOST",
        "RP_ENDPOINT_2_BIND_PORT" : "$PORT_AKKAMGMTHTTP",
        "RP_ENDPOINT_HTTP_PORT" : "$PORT_HTTP",
        "RP_APP_NAME" : "chirpservice",
        "RP_ENDPOINT_1_BIND_PORT" : "$PORT_AKKAREMOTE",
        "RP_ENDPOINT_1_HOST" : "$HOST",
        "RP_ENDPOINT_2_PORT" : "$PORT_AKKAMGMTHTTP",
        "RP_ENDPOINT_0_PORT" : "$PORT_HTTP",
        "RP_ENDPOINT_AKKA_MGMT_HTTP_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_AKKA_MGMT_HTTP_HOST" : "$HOST",
        "RP_ENDPOINT_2_HOST" : "$HOST",
        "RP_APP_TYPE" : "lagom",
        "RP_ENDPOINT_0_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_HTTP_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_1_BIND_HOST" : "0.0.0.0",
        "RP_ENDPOINT_0_HOST" : "$HOST",
        "RP_ENDPOINTS_COUNT" : "3",
        "RP_ENDPOINT_AKKA_REMOTE_PORT" : "$PORT_AKKAREMOTE"
    }
}
-> 0

[longshorej]

Okay, this is ready.

• There's a bit of duplication with args and envs between Kubernetes/Marathon but I think it's cleaner and clearer that way. A lot of these things look similar but vary in slight ways that I posit would be unworth the increase in complexity to refactor to some common base class.
• Tested with Chirper on DC/OS 1.10, with and without namespaces
• Requires More DC/OS fixes reactive-lib#55 and Fixes for DC/OS sbt-reactive-app#98
• Doesn't test the final output of the DC/OS configuration, only the various building blocks. As this output may change in the near future as this stabilizes, I think this is fine for now. See Reactivate Deployment Unit Tests #54 which covers the same situation but for Kubernetes.
• There's probably a fair bit of the Main.scala stuff that can be simplified, but I wanted to introduce the minimal amount of changes right now. Refactoring for this project is General clean-up/refactoring #102
• Secrets are not currently supported; I suggest they be done in a separate PR.
• Deployment type for Marathon is Rolling by default, unlike Canary on Kubernetes. This is because DC/OS has a more limited service naming model.

[dbrinegar]

Quick scan looks good, appreciate the commentary above. Time permitting will do some smoke testing tomorrow. Curious to get @mitkus point of view too.

[longshorej]

I think someday these should be annotations that are sourced from the Docker image, with sbt/maven settings to override the defauts.

[mitkus]

Good point, I'll make an issue.

[mitkus]

Good point, I'll make an issue.