Foreground process model

[mars]

Continuation of process model discussion that started in #920

Background

Currently, kong start is a daemonizing command. It exits after spawning its services in the background, nginx, dnsmasq, & serf.

This daemon process model is not compatible with container platforms like Docker & Heroku, as these containers depend on the start command remaining attached to the calling process, in the foreground. Refer to the official docs:

• Heroku foreground execution
• Docker examples demonstrating foreground execution:
  • Postgres
    • CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D"…]
    • -D means no daemonize, remain in the foreground
  • NodeJS
    • CMD ["node", "/src/index.js"]
    • Node remains in the foreground, only exits the event loop when there are no more callbacks to perform (About Node.js®)
  • Redis
    • ENTRYPOINT ["/usr/bin/redis-server"]
    • redis-server remains in the foreground

The official Kong Docker container itself does not actually run Kong in the foreground. It's handed off to stream the error log.

CMD kong start && tail -f /usr/local/kong/logs/error.log

What happens if one of the background processes spawned by kong start exits or crashes?

Proposal: Process Supervision

To solve this problem, we need to implement a process supervisor that starts each service as a subprocess, remaining attached in the foreground, so that upon exit or crash, the supervisor can restart the subprocess or exit so that its parent process (Docker daemon or Heroku dyno manager) can restart the container.

External process management solutions:

• runit with Heroku
• Supervisor with Docker

Kong's current service startup in Lua could also be crafted into a proper process manager using Lua modules like:

• subprocess
• lua-subprocess

I've so far failed to get either of these Lua modules working locally, much less as part of a service deployment. They seem to be old experiments without any activity to make them easily installable as a Lua rock.

Challenges: Configuration & Log streams

So, I've been experimenting with using an external process manager to run nginx, dnsmasq, & serf as foreground processes with stdout/stderr log streams.

The main challenge is calling into Kong's Lua services modules to prepare the database and service configurations, before the services are started by the process manager.

The Path Forward

The external process management model would be the most robust solution, using proven Posix tools to supervise the processes of Kong. Building a process manager in Lua seems like internalization of a problem already solved elsewhere.

Separating Kong configuration calls from process management will be the biggest challenge.

What do you all think?

[thibaultcha]

Hi Mars!

Great, thanks for the detailed feedback!

Yes, process management could definitely be handled better by Kong's CLI and avoid things like this or that or even sending signals through a shell invocation as well as handle cases when Kong needs not be daemonized.

It is worth noting that foreground mode as been discussed in various places already, notably in Kong/docker-kong#16 (run Nginx in foreground for Docker) and #647 (Using LuaPosix), which back in the day got ignored because we did not use to manage so many subprocesses.

I see two benefits if we switch to another solution:

• Better handling of Kong's subprocesses (such as the waiting example or signal sending) in general. That means a more reliable Kong and better restart/reload commands (for example iirc, the current upgrade from 0.5 to 0.6 requires people to run kong restart simply to start the subprocesses, instead of handling that from reload and avoiding downtime).
• Possibility to keep a process in foreground without using the Nginx's daemon off; directive (which btw once it ends because of a kong stop has the oddity of printing [OK] started, which is somewhat funny)

However, if not written in Lua, it has a major drawback:

• Including one more dependency in Kong's stack, which is already quite huge (Java, Cassandra, Openssl, PCRE, Nginx, Openresty, LuaJIT, Luarocks, Serf, dnsmasq and I might forget some).

Using Lua scripts to build Kong's CLI is also a major advantage in the sense that it can reuse all of Kong's internal APIs, which is done as of now and will be even much more in the future for commands such as kong add-api <...> etc... (See #810 for even more compatibility with Kong's internal APIs).

Maybe we could write Lua bindings for a C process manager (do you know of any?) or use an existing one and keep Lua. This would also be very useful for our test suite which starts and stops Kong many, many times.

[mars]

Hi @thibaultcha!

So true that running the processes from Lua would provide a minimalism & directness to the models at play inside Kong.

After investigating LuaPosix and the "subprocess" rocks mentioned above, writing a robust process control layer in Lua is going to be significant feature. LuaPosix (or a tiny part of it) is the foundation of those subprocess modules, which demonstrate how challenging this will be ⚡

So, I've been continuing down the path of reading the service configurations out of Kong. I've written a Lua script that exports all the process params as environment variables.

This Procfile is what I have working today with those environment variables:

dnsmasq: dnsmasq -p $DNSMASQ_PORT -N -o --listen-address=127.0.0.1 --keep-in-foreground
nginx: nginx -p $NGINX_WORKING_DIR -c $NGINX_CONFIG
serf: serf agent -bind $SERF_CLUSTER_LISTEN -rpc-addr $SERF_CLUSTER_LISTEN_RPC -encrypt $SERF_ENCRYPT -log-level err -profile wan -node $SERF_NODE_NAME -event-handler $SERF_EVENT_HANDLER

Making the minimal changes to Kong to support external supervision of the processes (nginx, dnsmasq, serf)

[mars]

One of the beautiful things about this process model, as that all of their stderr streams flow through the parent process, merged into a unified stream.

Hopefully the plan for a native/Lua solution would include streaming logs instead of writing to various hardcoded log files in the Nginx temp dir. Let the host's process manager decided whether to redirect log streams to a local file or send them out to a collector (syslog, etc.)

[mars]

To make Kong compatible with an external process supervisor, a few changes would be required.

You can see in this external-supervisor branch comparison:

• the KONG_EXTERNAL_SUPERVISE environment variable/flag to:
  • short-circuit is_running() checks (the external supervisor takes care of keeping processes up)
  • perform service bootstrapping from Kong.init() (only serf:_autojoin() at this time)
• the previously proposed separation of prepare_all() & start_all() to:
  • permit configuration of all services without implicitly starting them

That external-supervisor branch is WIP/research into making this work. Ideally we could apply these solutions using a more sound architectural plan, like:

• Add an after_start() method to each of the kong.cli.services so that each service clearly defines bootstrapping after the process starts.
• Add appropriate integration tests for when KONG_EXTERNAL_SUPERVISE flag is present

[ahmadnassri]

@mars

[mars]

That experimental external-supervisor fork of Kong mentioned above can now be seen in context of a complete deployment on Heroku:

• Heroku app, which uses…
• Heroku buildpack

[sonicaghi]

[mars]

@sinzone great point! We will eventually list this buildpack in Elements.

Before we make it publicly discoverable through heroku.com, we need to resolve this cluster runtime issue: Cassandra error from kong.core.cluster.

I will be following-up/working on at least the temporary solution that @thibaultcha & I discussed in that issue:

…to define the configuration & dao globals inside each of the kong.core.cluster timers.

[magnusja]

So I have seen, that now 'kong start' is run as a foreground process, but now I do not have access or error logs. How can I tell kong that I want these logs to be printed to stdout, to make it accessable via 'docker logs'?

[nguilford]

Removing third-party dependencies #835

Does the desire to remove 3p dependencies trump this issue? i.e. will mashape consider merging a PR that implements what @mars is suggesting in the interim?

@mars, I see the last time you did work on this was about a month ago. Are you planning on completing the supervised process integration or are you just trying things out? I have a desire to get this working properly with docker and, assuming that a PR will be considered here, I may be inclined to help out if it's not something someone like yourself already has a solution for.

[mars]

@mjdev Mashape's Kong does not send logs to stdout. Unless you're willing to go renegade experimenting with these external process supervision patches & running a supervisor like runit, tailing Nginx & other services' log files is the way.

@nguilford The work on this has been experimental to enable the Heroku Kong buildpack & app which is not integrated back into Mashape's Kong.

[mars]

Has process supervision been addressed with the 0.9.0 CLI improvements? Or do Kong's processes still run unattached?

[thibaultcha]

They still run unattached.

[subnetmarco]

Finally Kong 0.11+ will only consist of an nginx process (dnsmasq and serf are gone), so that this becomes a non-issue. The new Docker image for 0.11 will directly run the nginx process.

A simple example can be seen at https://github.com/Mashape/docker-kong/blob/0.11.0/Dockerfile#L16

Closing this now.

[mars]

Managed to get Kong 0.11 deployed to Heroku 💜 https://github.com/heroku/heroku-kong

Made possible by all your work on the new single-process model, prepare & run semantics, dramatically simplified clustering, and Postgres support.

[thibaultcha]

@mars Great news! We're glad to hear! Things are only going to improve on this front from now on! The next big leap forward will be #2355, if you are feeling curious ;)

[subnetmarco]

@mars do you think we can list it in Heroku Elements now?

[mars]

After some usage of the Kong buildpack & deploy button, it should begin to show-up in Heroku Elements. Things get indexed when they're used.

So, get more folks to try out Kong on Heroku with this button: