Fixes #30803: Bind to socket for Puma and Apache #883

ehelms · 2020-09-08T19:13:26Z

The goal of this change is to switch to using local unix sockets to bind Puma to, and also to have Apache reverse proxy to increasing the security of the communication from the reverse proxy to the socket.

I am looking to understand if I need to enforce permissions and user-group on the socket to enhance security. There is also currently SELinux denials that are associated with this change we will need to account for. On my test system they are:

type=AVC msg=audit(1599591946.533:10070): avc:  denied  { connectto } for  pid=6405 comm="httpd" path="/run/puma.sock" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:foreman_rails_t:s0 tclass=unix_stream_socket

SELinux policy changes needed to address -- theforeman/foreman-selinux#113

ehelms · 2020-09-08T19:14:38Z

Combines with theforeman/foreman#7972

ehelms · 2020-09-08T19:22:10Z

Note: I expect tests to fail but I want feedback on the approach before I do a bunch of updates.

templates/foreman.service-overrides.erb

manifests/config.pp

templates/foreman.socket-overrides.erb

ekohl

Since the current implementation depends on a Foreman change, we'll need to be careful about compatibility. Right now the module is compatible with Foreman 2.0, as specified in README.md. We want to support current stable + development.

Ideally we'd write any Foreman patch in such a way that it's compatible with both the old FOREMAN_BIND (without a protocol + port) and the new (with). Then cherry pick that into a stable branch. In the compatibility notes we'd then write that you need at least Foreman 2.1.x (2.1.2 at the time of writing?). I'd also like to cherry pick this specific patch to 2.1 when it's finished.

manifests/init.pp

templates/foreman.socket-overrides.erb

manifests/config.pp

manifests/config/apache.pp

ehelms · 2020-09-10T23:42:21Z

Unit tests are now passing but the acceptance tests won't pass until theforeman/foreman#7972 is available as an RPM

manifests/config/apache.pp

ekohl · 2020-10-20T19:16:05Z

Given this does not happen on the other OSes I am wondering if this is a difference in systemd versions causing this.

Unintentionally I just reproduced this on Fedora 32. I had a foreman.socket and a foreman.service which were both running. Then a systemctl restart foreman.socket triggered it as well.

ehelms · 2020-10-20T20:20:20Z

restart

Was it running localhost:3000 and then switched to Unix Socket? Or it was simply running and you restart foreman.socket and had it blow up?

ekohl · 2020-10-20T21:17:15Z

From my journal:

Oct 20 21:02:47 wisse systemd[1267]: Starting Foreman...
Oct 20 21:02:48 wisse foreman[746290]: => Booting Puma
Oct 20 21:02:48 wisse foreman[746290]: => Rails 6.0.3.3 application starting in production
Oct 20 21:02:48 wisse foreman[746290]: => Run `rails server --help` for more startup options
Oct 20 21:02:49 wisse foreman[746290]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Oct 20 21:02:52 wisse foreman[746290]: Failed to load systemd plugin
Oct 20 21:02:52 wisse foreman[746290]: ["tcp://0.0.0.0:5000"]
Oct 20 21:02:52 wisse foreman[746290]: [746290] Puma starting in cluster mode...
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Version 5.0.2 (ruby 2.7.2-p137), codename: Spoony Bard
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Min threads: 0, max threads: 16
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Environment: production
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Process workers: 2
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Preloading application
Oct 20 21:02:52 wisse foreman[746290]: [746290] * Closing unused activated socket: tcp:0.0.0.0:5000
Oct 20 21:02:52 wisse foreman[746290]: [746290] Use Ctrl-C to stop
Oct 20 21:02:52 wisse foreman[746290]: [746290] - Worker 0 (pid: 746296) booted, phase: 0
Oct 20 21:02:52 wisse foreman[746290]: [746290] - Worker 1 (pid: 746300) booted, phase: 0
Oct 20 21:05:20 wisse systemd[1267]: foreman.socket: Succeeded.
Oct 20 21:05:20 wisse systemd[1267]: Closed Foreman HTTP Server Accept Sockets.
Oct 20 21:05:20 wisse systemd[1267]: Stopping Foreman HTTP Server Accept Sockets.
Oct 20 21:05:20 wisse systemd[1267]: foreman.socket: Socket service foreman.service already active, refusing.
Oct 20 21:05:20 wisse systemd[1267]: Failed to listen on Foreman HTTP Server Accept Sockets.
Oct 20 21:05:20 wisse systemd[1267]: Dependency failed for Foreman.
Oct 20 21:05:20 wisse systemd[1267]: foreman.service: Job foreman.service/start failed with result 'dependency'.
Oct 20 21:05:25 wisse foreman[746290]: [746290] - Gracefully shutting down workers...
Oct 20 21:05:26 wisse foreman[746290]: Exiting
Oct 20 21:05:26 wisse systemd[1267]: foreman.service: Succeeded.

I was working on a different patch, but I think what's similar is that a file descriptor was open for the correct bind (0.0.0.0:5000) but puma didn't consume it. Perhaps you're hitting a similar thing where one is restarted before the other was really reloaded/restarted.

ehelms · 2020-10-21T19:34:39Z

I think, from some testing, the fundamental problem I am running into is that foreman.service is being activated by the packaging and this seems to be exacerbated on the Debian side. Given the foreman.socket has control of activating the service, I think we need to move away from the service being activated to focusing more on the socket. I did some searching I am not sure where this happens in Debian packaging. @ekohl @evgeni can one of you point me in the direction of this?

evgeni · 2020-10-22T10:04:15Z

I think this happens here:
https://github.com/theforeman/foreman-packaging/blob/deb/develop/debian/buster/foreman/rules#L36

docs: https://manpages.debian.org/testing/debhelper/dh_systemd_enable.1.en.html

I am not sure if passing --no-enable is sufficient. Probably.

evgeni · 2020-10-22T10:17:08Z

However, I don't exactly understand what the problem with the service vs the socket is.
The service can bind the port itself, or get activated by foreman.socket, both are valid scenarios and starting the socket while the service is running should be effectively a noop?

evgeni · 2020-10-22T10:18:35Z

Maybe we want dh_systemd_start --no-start? https://manpages.debian.org/testing/debhelper/dh_systemd_start.1.en.html

ehelms · 2020-10-22T12:13:03Z

So, I think what happens is:

foreman package installs
triggers activation of foreman.service running with a bind to host:port
puppet runs configuring the foreman.socket
puppet attempts to start the foreman.socket which tries to activate the foreman.service which is already activated with a different bind
puppet fails

The annoying part is, on Debian 10 the above happens if the foreman.service Service requires foreman.socket Service. If I drop the requires, then this fails on Ubuntu 1804. This never fails on EL7.

ekohl · 2020-10-22T12:20:42Z

I'm starting to wonder if we should wait for Puma 5.1.0 which should include puma/puma#2362. That will simplify the configuration required. However, it may complicate support for older versions.

ehelms · 2020-10-22T12:55:35Z

I think Puma 5.1.0 will clean things up but @ekohl you have expressed the hardening this provides and I think that's a good thing to have. I am less familiar in this area, but if I look at how EL and Debian states are post install, pre-installer run, we have variation there and I think we'd benefit from commonality:

On EL7, both foreman.service and foreman.socket come installed with foreman-service package. Both are installed from packaging as loaded and inactive:

[root@pipeline-foreman-server-nightly-centos7 vagrant]# yum install foreman-service -y
[root@pipeline-foreman-server-nightly-centos7 vagrant]# systemctl status foreman.service
● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://theforeman.org
[root@pipeline-foreman-server-nightly-centos7 vagrant]# systemctl status foreman.socket
● foreman.socket - Foreman HTTP Server Accept Sockets
   Loaded: loaded (/usr/lib/systemd/system/foreman.socket; disabled; vendor preset: disabled)
   Active: inactive (dead)
   Listen: 0.0.0.0:3000 (Stream)

On Debian 10 and Ubuntu 18.04, the foreman.service comes with the install of foreman, and the foreman.socket comes with the install of foreman-service package. The foreman.service is loaded and active, the foreman.socket is loaded and active :

root@pipeline-foreman-server-nightly-debian10:/home/vagrant# systemctl status foreman.service
● foreman.service - Foreman
   Loaded: loaded (/lib/systemd/system/foreman.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2020-10-22 12:28:17 UTC; 3min 21s ago
     Docs: https://theforeman.org
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/foreman.service

Oct 22 12:28:17 pipeline-foreman-server-nightly-debian10 systemd[1]: Starting foreman.service...
Oct 22 12:28:17 pipeline-foreman-server-nightly-debian10 foreman[31396]: foreman not configured to start. Please edit /etc/default/foreman to enable.
Oct 22 12:28:17 pipeline-foreman-server-nightly-debian10 systemd[1]: Started foreman.service.
Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: Dependency failed for Foreman.
Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: foreman.service: Job foreman.service/start failed with result 'dependency'.
root@pipeline-foreman-server-nightly-debian10:/home/vagrant# systemctl status foreman.socket
● foreman.socket - Foreman HTTP Server Accept Sockets
   Loaded: loaded (/lib/systemd/system/foreman.socket; enabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: 0.0.0.0:3000 (Stream)

Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: foreman.socket: Socket service foreman.service already active, refusing.
Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: Failed to listen on Foreman HTTP Server Accept Sockets.
Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: foreman.socket: Socket service foreman.service already active, refusing.
Oct 22 12:29:44 pipeline-foreman-server-nightly-debian10 systemd[1]: Failed to listen on Foreman HTTP Server Accept Sockets.

ekohl · 2020-10-22T13:16:55Z

I think Puma 5.1.0 will clean things up but @ekohl you have expressed the hardening this provides and I think that's a good thing to have.

Yes. The ever lasting struggle.

Maybe we want dh_systemd_start --no-start? https://manpages.debian.org/testing/debhelper/dh_systemd_start.1.en.html

I think it makes sense not to start the service by default. We have an installer and there are no migration scripts to create the PostgreSQL DB. While it's normally the Debian policy to automatically start services, we're sufficiently different that it makes sense not to.

I would also suggest to cherry pick that to 2.2 if it works in nightly. Not starting a service by default is the default on Red Hat based systems so the installer should handle it perfectly well.

spec/spec_helper_acceptance.rb

ehelms · 2020-10-27T14:25:16Z

This PR seems to be hitting this -- #897

ehelms · 2020-10-27T15:52:18Z

Woohoo, the Debian 10 tests passed with the packaging updates

ehelms · 2020-11-18T20:31:09Z

I rebased this on latest to re-run the tests given it has sat for a bit.

wbclark

I installed nightly on centos7 with this PR using Forklift. Playing around with it a bit, I can see the socket at /run/foreman.sock and the drop in configuration with expected values at /etc/systemd/system/foreman.socket.d/installer.conf(I had to read some docs to understand the purpose of the blank ListenStream=). I tested basic functionality and everything seems working as expected.

I couldn't install nightly on ubuntu 1804 but the failure is with redis.service and seems wholly unrelated to changes here.

evgeni · 2020-11-26T09:02:30Z

I think this broke EL8 installs:

% grep 'AVC.*foreman.sock' var/log/audit/audit.log
type=AVC msg=audit(1606342776.512:2783): avc:  denied  { getattr } for  pid=15093 comm="rails" path="/run/foreman.sock" dev="tmpfs" ino=121834 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1606342862.770:2796): avc:  denied  { write } for  pid=14897 comm="httpd" name="foreman.sock" dev="tmpfs" ino=121834 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file permissive=0
type=AVC msg=audit(1606342862.804:2797): avc:  denied  { write } for  pid=14900 comm="httpd" name="foreman.sock" dev="tmpfs" ino=121834 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=sock_file permissive=0

there are also other denials:

% grep 'AVC.*vda' var/log/audit/audit.log
type=AVC msg=audit(1606342756.671:2781): avc:  denied  { write } for  pid=15093 comm="rails" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1606342756.671:2782): avc:  denied  { write } for  pid=15093 comm="rails" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1606342815.244:2789): avc:  denied  { write } for  pid=15466 comm="sidekiq" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1606342815.244:2790): avc:  denied  { write } for  pid=15466 comm="sidekiq" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1606342839.039:2792): avc:  denied  { write } for  pid=15574 comm="sidekiq" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1606342839.039:2793): avc:  denied  { write } for  pid=15574 comm="sidekiq" name="home" dev="vda1" ino=100696036 scontext=system_u:system_r:foreman_rails_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir permissive=0

evgeni · 2020-11-26T09:19:50Z

why is /run/foreman.sock system_u:object_r:var_run_t:s0? foreman-selinux has a file context that makes it system_u:object_r:foreman_var_run_t,s0 (and that would be allowed to be written by httpd)

evgeni · 2020-11-26T09:21:30Z

hah, /run/foreman\.sock regular file system_u:object_r:foreman_var_run_t:s0 -- well, it's not a regular file.

theforeman-bot added Needs testing Not yet reviewed labels Sep 8, 2020

adamruzicka reviewed Sep 9, 2020

View reviewed changes

templates/foreman.service-overrides.erb Outdated Show resolved Hide resolved

evgeni mentioned this pull request Sep 9, 2020

Refs #30803: Enable unix socket binding theforeman/foreman#7972

Merged

evgeni reviewed Sep 9, 2020

View reviewed changes

manifests/config.pp Outdated Show resolved Hide resolved

ekohl reviewed Sep 9, 2020

View reviewed changes

templates/foreman.socket-overrides.erb Outdated Show resolved Hide resolved

ekohl mentioned this pull request Sep 9, 2020

Fixes #30535 - Use HTTP headers in puma theforeman/foreman#7909

Merged

ehelms force-pushed the fixes-30803 branch 2 times, most recently from 1437eaf to a308294 Compare September 9, 2020 18:24

ekohl reviewed Sep 9, 2020

View reviewed changes

ehelms force-pushed the fixes-30803 branch from a308294 to f0fe0ed Compare September 9, 2020 18:58

ehelms commented Sep 9, 2020

View reviewed changes

manifests/init.pp Outdated Show resolved Hide resolved

ehelms force-pushed the fixes-30803 branch from f0fe0ed to b0827e6 Compare September 9, 2020 20:20

ehelms commented Sep 10, 2020

View reviewed changes

templates/foreman.socket-overrides.erb Outdated Show resolved Hide resolved

ehelms force-pushed the fixes-30803 branch from b0827e6 to 77efe65 Compare September 10, 2020 01:15

ehelms commented Sep 10, 2020

View reviewed changes

manifests/config.pp Outdated Show resolved Hide resolved

ehelms force-pushed the fixes-30803 branch 2 times, most recently from 80ba3dc to e5c2e03 Compare September 10, 2020 12:14

ekohl reviewed Sep 10, 2020

View reviewed changes

manifests/config/apache.pp Outdated Show resolved Hide resolved

ehelms force-pushed the fixes-30803 branch from e5c2e03 to 59b0926 Compare September 10, 2020 23:41

ehelms force-pushed the fixes-30803 branch from 59b0926 to 252cdf4 Compare September 29, 2020 19:33

ekohl requested changes Sep 29, 2020

View reviewed changes

manifests/config/apache.pp Outdated Show resolved Hide resolved

theforeman-bot added Waiting on contributor and removed Not yet reviewed labels Sep 29, 2020

ehelms force-pushed the fixes-30803 branch from 252cdf4 to af871ec Compare September 30, 2020 12:05

theforeman-bot added Needs re-review and removed Waiting on contributor labels Sep 30, 2020

ehelms mentioned this pull request Oct 26, 2020

Do not start foreman.service or foreman.socket on package install theforeman/foreman-packaging#5900

Merged

ehelms force-pushed the fixes-30803 branch from 2783880 to 841f421 Compare October 26, 2020 20:52

ekohl reviewed Oct 27, 2020

View reviewed changes

spec/spec_helper_acceptance.rb Outdated Show resolved Hide resolved

ehelms force-pushed the fixes-30803 branch from 841f421 to eec580c Compare October 27, 2020 12:18

ehelms force-pushed the fixes-30803 branch from eec580c to 6643d93 Compare October 28, 2020 13:51

ehelms requested review from ekohl and evgeni November 12, 2020 16:39

Fixes #30803: Bind to socket for Puma and Apache

b82c7fa

ehelms force-pushed the fixes-30803 branch from 6643d93 to b82c7fa Compare November 18, 2020 20:30

wbclark approved these changes Nov 23, 2020

View reviewed changes

theforeman-bot removed the Needs re-review label Nov 23, 2020

ehelms merged commit 2473950 into theforeman:master Nov 24, 2020

evgeni mentioned this pull request Nov 26, 2020

make /run/foreman.sock match ALL files, not only regular theforeman/foreman-selinux#118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #30803: Bind to socket for Puma and Apache #883

Fixes #30803: Bind to socket for Puma and Apache #883

ehelms commented Sep 8, 2020 •

edited

Loading

ehelms commented Sep 8, 2020

ehelms commented Sep 8, 2020

ekohl left a comment

ehelms commented Sep 10, 2020

ekohl commented Oct 20, 2020

ehelms commented Oct 20, 2020

ekohl commented Oct 20, 2020

ehelms commented Oct 21, 2020

evgeni commented Oct 22, 2020

evgeni commented Oct 22, 2020

evgeni commented Oct 22, 2020

ehelms commented Oct 22, 2020

ekohl commented Oct 22, 2020

ehelms commented Oct 22, 2020

ekohl commented Oct 22, 2020

ehelms commented Oct 27, 2020

ehelms commented Oct 27, 2020

ehelms commented Nov 18, 2020

wbclark left a comment

evgeni commented Nov 26, 2020

evgeni commented Nov 26, 2020

evgeni commented Nov 26, 2020

Fixes #30803: Bind to socket for Puma and Apache #883

Fixes #30803: Bind to socket for Puma and Apache #883

Conversation

ehelms commented Sep 8, 2020 • edited Loading

ehelms commented Sep 8, 2020

ehelms commented Sep 8, 2020

ekohl left a comment

Choose a reason for hiding this comment

ehelms commented Sep 10, 2020

ekohl commented Oct 20, 2020

ehelms commented Oct 20, 2020

ekohl commented Oct 20, 2020

ehelms commented Oct 21, 2020

evgeni commented Oct 22, 2020

evgeni commented Oct 22, 2020

evgeni commented Oct 22, 2020

ehelms commented Oct 22, 2020

ekohl commented Oct 22, 2020

ehelms commented Oct 22, 2020

ekohl commented Oct 22, 2020

ehelms commented Oct 27, 2020

ehelms commented Oct 27, 2020

ehelms commented Nov 18, 2020

wbclark left a comment

Choose a reason for hiding this comment

evgeni commented Nov 26, 2020

evgeni commented Nov 26, 2020

evgeni commented Nov 26, 2020

ehelms commented Sep 8, 2020 •

edited

Loading