Provide remote configuration API

[mperham]

Faktory's conf.d filesystem configuration is looking increasingly dated as more users and customers use Docker and containers where the local filesystem is not easy to provide or change. Instead, provide a Web API where TOML files can be POST'd via curl by an application deployment. Something like:

for x in config/faktory/*.toml; do
  curl -T $x http://localhost:7420/config/upload/
end && curl -X POST http://localhost:7420/config/reload

(Basic auth elided for readability)

That curl command will POST foo.toml to /config/update/foo.toml. After all of the TOML are loaded, we call reload to have Faktory atomically reload its config.

Configuration will be stored in Redis so it persists across Faktory boots. The config will track last modification time and SHAs for each config item in order to minimize unnecessary changes or reloads.

0. Uploads will return an error if the TOML is malformed.
1. Uploads will stage into a temporary Redis key. Calling reload will swap with the live config, just like calling HUP today. Otherwise the temp key should expire after N minutes.
2. For backwards compatibility, an existing Faktory at boot will load conf.d files into Redis if any exist and their modification time is > the last config update modtime.
3. Partial config update is not allowed. Every TOML file must be POST'd before reload. This is to prevent renaming a TOML and having the old filename hang around in Redis. For maximum reliability, POST one big TOML.

For sanity's sake, it is strongly recommended to use either conf.d or /config/upload exclusively. Don't mix the two. The Web UI will not provide a manual upload page as this feature is intended for deployment automation.

[profsmallpine]

Just wanting to sanity check that this functionality does not yet exist as it would help us out?

[mperham]

@profsmallpine It does not exist and I'm focused on Sidekiq for the next month or two. You are welcome to spike or prototype it if you want to dive in.

[profsmallpine]

We are finally getting around to a move from ec2 to aws serverless. I went ahead and pushed this responsibility of configuration to the docker container. We will have to replace the image and restart our fargate task which isn't as clean as reloading the config, but keeps deployments and devops simpler. The dockerfile:

FROM docker.contribsys.com/contribsys/faktory-ent:1.6.2
COPY /config /etc/faktory/conf.d
EXPOSE 7419 7420
CMD ["/faktory", "-w", ":7420", "-b", ":7419"]

1. do you (@mperham) see any issues with this approach?
2. We are using CDK written in go to set all of this up. It's a beefier part of the infra given the need to setup sidecar for metrics and the efs volume. I don't mind sharing that code for anyone else on a similar path. Is there an ideal location to post that?

[mperham]

Looks good to me. If you want to publish something in the wiki, it is publicly editable.

[andycox]

Where do you see this on the horizon? I wish I had the Go chops to give it a try myself, but unfortunately I don't. I'd like to deploy our Faktory server to Fargate, but none of the existing workarounds to its lack of HUPability seem desirable as they would require some Faktory server downtime as a new image is deployed to Fargate or the existing one is redeployed.

Configuration changes seem far more common than upgrading the Faktory server itself, so I'm willing to live with running it on an EC2 instance so that we can easily tweak the config and HUP to reload.

[Jcambass]

👋 I'm starting to take a look at implementing this. While I can't commit to actually finishing it yet, I do have some questions around the specs:

Partial config update is not allowed. Every TOML file must be POST'd before reload. This is to prevent renaming a TOML and having the old filename hang around in Redis. For maximum reliability, POST one big TOML.

AFAIK there is currently no enforcement on how you split your conf.d configs, which makes the definition of Every TOML file must be POST'd before reload unclear.

This is unless we require you to upload the same file names that your conf.d configs have. This seems like a no-go given how we want to discourage users from using both features together and we only support it for backwards compatibility.

For backwards compatibility, an existing Faktory at boot will load conf.d files into Redis if any exist and their modification time is > the last config update modtime.

I assume that the plan is to use the filesystems "last modified" information to make this decision for the conf.d configs.

I also assume that we're ok with re-reading the conf.d after a deployment as the "last modified" information of the files might be the time of deployment, so they are newer than the previously POSTed toml. The assumption is that you will POST them again as part of the deployment, right?

---

As long as we don't either enforce a specific set of config files or enforce that you use the same set of config file in conf.d and in the /config endpoint, I'm not sure we can make this work in a reliable way.

A few alternative thoughts:
Option 1: Could we get away with not storing the settings read from conf.d in redis and only ever reading them if there is no config in redis? This would not allow using conf.d and the endpoint together but provide a migration path without breaking existing setups.

Option 2: Stick to the idea in the original description of this issue. But combine all conf.d in one big file to store in redis and use the highest modified time across all the conf.d files.

Personally I like Option 1. But it depends on what we want to achieve. The way "Partial config update is not allowed" is described makes me believe that the mental model is one config file per feature/sub-system. So that one sub-system always gets the config from either conf.d or /config. But since there is nothing enforcing this separation of settings into different files this isn't enough. If we really care about all settings either coming from conf.d or /config, then Option 1 makes this more clear.

---

I know this is a bit a 🧠 dump. If I misunderstood any of this let me know! If not, please let me know what you think is the best way forward.

[mperham]

I'm not tied to the "spec" I wrote three years ago, you're welcome to suggest alternatives.

The main thing that everyone is asking for is upgrading Faktory without downtime. I don't see a lot of requests for this remote config functionality.