Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #30465 - Use libexec wrappers for SELinux #116

Merged
merged 1 commit into from
Oct 9, 2020

Conversation

ekohl
Copy link
Member

@ekohl ekohl commented Jul 22, 2020

In python3-pulpcore 3.7.1-2 the /usr/libexec/pulpcore wrappers have been introduced to enter the proper SELinux domain.

It has also been cherry picked to 3.6.3-2 but in the SELinux policy is incomplete so it has no effect. The main benefit of that cherry pick is to keep the module compatible with both 3.6 and 3.7.

The README is updated with a support policy. Since 3.7 is not being tested in CI yet, I have not yet added it to the supported versions.

@ekohl ekohl marked this pull request as ready for review July 23, 2020 15:06
@ekohl
Copy link
Member Author

ekohl commented Jul 23, 2020

So it turns out #115 is not needed for this. I verified this manually on an EL7 VM (using the beaker tests).

@ekohl ekohl force-pushed the 30465-selinux branch 2 times, most recently from 8a70a4a to 2eeb921 Compare July 23, 2020 15:56
-w pulpcore.tasking.worker.PulpWorker -n resource-manager \
--pid=/var/run/pulpcore-resource-manager/resource-manager.pid \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These pid files gave SELinux errors and https://python-rq.org/patterns/systemd/ doesn't suggest it needs pid files.

Probably because /var/run is not being labeled correctly since it's a symlink to /run making https://github.com/pulp/pulpcore-selinux/blob/64b99b463186305a371498ce1da8d1f3bb956a20/pulpcore.fc#L12-L16 useless.

@@ -8,9 +8,10 @@ Environment="DJANGO_SETTINGS_MODULE=pulpcore.app.settings"
Environment="PULP_SETTINGS=<%= scope['pulpcore::settings_file'] %>"
Environment="PULP_STATIC_ROOT=<%= scope['pulpcore::pulp_static_root'] %>"
User=<%= scope['pulpcore::user'] %>
PIDFile=/run/pulpcore-api.pid
Group=<%= scope['pulpcore::group'] %>
WorkingDirectory=%t/pulpcore-api
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think %t resolves to /run when it runs as a system service. I'm a bit unsure about this. I've considered setting it to ~ or explicit /var/lib/pulp. That way systemd also automatically ensures /var/lib/pulp is present and mounted. I'm not sure why these services should have a different workdir.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unclear from the comment what we want to end up with. If /var/lib/pulp is the proper working directory then I vote set it explicitly to that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think %t resolves to /run for system services. This is to preserve the previous /var/run (and on all modern systems that's a symlink to /run. I'm leaning to ~ or /var/lib/pulp because systemd will ensure it's mounted which can be very useful if it comes from NFS or a separate partition (which is very common), but I might missing something.

Here I'd like some feedback from Pulp upstream since I don't know what their preference is.

Copy link

@mikedep333 mikedep333 Sep 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Pulp upstream, we never considered the $PWD of the pulp processes AFAIK.

I agree with ~ (pulp_installer defaults the pulp home dir to /var/lib/pulp, but lets users change it), in case we do end up writing anything there, and so that it gets mounted.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(pulp_installer defaults the pulp home dir to /var/lib/pulp, but lets users change it)

So does this module, but since it also manages the user, the homedir will be updated and that causes service restarts. Technically I think that Linux doesn't allow changing the homedir if a user is logged in, so you need to manually stop all services before, but after that it should work. Of course there's no data migration either.

@ekohl
Copy link
Member Author

ekohl commented Jul 23, 2020

When running this, you do see AVCs:

# grep type=AVC /var/log/audit/audit.log 
type=AVC msg=audit(1595519054.448:2345): avc:  denied  { search } for  pid=28778 comm="gunicorn" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519054.448:2346): avc:  denied  { search } for  pid=28778 comm="gunicorn" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519054.723:2347): avc:  denied  { read } for  pid=28822 comm="sh" name="meminfo" dev="proc" ino=4026532028 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file permissive=0
type=AVC msg=audit(1595519058.657:2361): avc:  denied  { search } for  pid=28799 comm="rq" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519058.657:2362): avc:  denied  { search } for  pid=28799 comm="rq" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0

manifests/service.pp Outdated Show resolved Hide resolved
@ekohl
Copy link
Member Author

ekohl commented Jul 27, 2020

@mikedep333 mind having a look as well? In particular the systemd unit files.

manifests/config.pp Outdated Show resolved Hide resolved
content => "#!/bin/bash\nexec ${bin} \"$@\"\n",
owner => 'root',
group => 'root',
mode => '0755',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is worldable executable applicable in our case?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't hurt since there's nothing secret in there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it imply any other user could run those commands? Starting any rogue services?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but they can already do that today using /usr/bin/gunicorn so there is no difference. I have a suggestion to use unix sockets instead of TCP ports but the current SELinux policy in Pulpcore doesn't allow that now (https://pulp.plan.io/issues/7189). By using systemd socket activation and letting systemd create /run/pulpcore-api.sock (or similar) owned by apache:root and mode 0600 we can prevent any local user from spinning up a rogue service. It also prevents local users from impersonating admin by connecting directly to the TCP port.

@ehelms
Copy link
Member

ehelms commented Jul 27, 2020

I'm still not clear on why the wrappers are needed versus using the system paths.

@ekohl
Copy link
Member Author

ekohl commented Jul 27, 2020

I'm still not clear on why the wrappers are needed versus using the system paths.

What you execute needs to have some SELinux label. IMHO Pulpcore doesn't own /usr/bin/{gunicorn,rq} so a label there feels wrong.

@ehelms
Copy link
Member

ehelms commented Jul 27, 2020

This is just my lack of knowledge in the SELinux area. Is it common to create wrappers? It feels generally safe to assume, or rather to enforce that Pulpcore is the only thing using /usr/bin/gunicorn,rq on a box that it is installed.

This question is geared at:

  1. Are we following standard SELinux practices
  2. Are we deviating from upstream Pulp too much

I'm checking on these more than I am trying to comment on them one way or another.

@ekohl
Copy link
Member Author

ekohl commented Jul 27, 2020

This is just my lack of knowledge in the SELinux area. Is it common to create wrappers?

Yes:

# ls -lZ /usr/share/foreman/bin/rails /usr/libexec/foreman/sidekiq-selinux
-rwxr-xr-x. root root system_u:object_r:foreman_rails_exec_t:s0 /usr/libexec/foreman/sidekiq-selinux
-rwxr-xr-x. root root system_u:object_r:foreman_rails_exec_t:s0 /usr/share/foreman/bin/rails

Note that setting a context on a file is not security thing. As a normal non-privileged user you can set it:

$ touch test
$ chcon -t foreman_rails_exec_t test
$ ls -lZ test
-rw-rw-r--. ekohl ekohl unconfined_u:object_r:foreman_rails_exec_t:s0 test

It feels generally safe to assume, or rather to enforce that Pulpcore is the only thing using /usr/bin/gunicorn,rq on a box that it is installed.

I don't think it is. You may need another Python service with another SELinux context. By not claiming any specific resources in a shared path, you keep it open in the future.

Are we following standard SELinux practices

I think we are.

Are we deviating from upstream Pulp too much

I suggested the wrappers and upstream replied with "This is a really good idea IMO.". See https://pulp.plan.io/issues/7178#note-3 (last paragraph).

It also allows upstream to use the same wrapper locations in their setup with a virtualenv. This allows further standardization of the systemd unit files. I still hope that one day we can share those between both projects rather than having them duplicated in two places.

@ehelms
Copy link
Member

ehelms commented Jul 28, 2020

Thanks for the explanations. All sounds good. Are we waiting on Pulp to merge this?

@ekohl
Copy link
Member Author

ekohl commented Jul 28, 2020

I'm hoping to get an answer about WorkingDirectory and the generated denials.

Then I'll also introduce the execdir variable.

@ekohl ekohl force-pushed the 30465-selinux branch 2 times, most recently from e8c30ba to 6673e62 Compare September 1, 2020 17:21
@ekohl
Copy link
Member Author

ekohl commented Sep 1, 2020

@dkliban @mikedep333 could you have a look at these 2? I should have mentioned this earlier today

I'm hoping to get an answer about WorkingDirectory and the generated denials.

@mikedep333
Copy link

When running this, you do see AVCs:

# grep type=AVC /var/log/audit/audit.log 
type=AVC msg=audit(1595519054.448:2345): avc:  denied  { search } for  pid=28778 comm="gunicorn" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519054.448:2346): avc:  denied  { search } for  pid=28778 comm="gunicorn" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519054.723:2347): avc:  denied  { read } for  pid=28822 comm="sh" name="meminfo" dev="proc" ino=4026532028 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file permissive=0
type=AVC msg=audit(1595519058.657:2361): avc:  denied  { search } for  pid=28799 comm="rq" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1595519058.657:2362): avc:  denied  { search } for  pid=28799 comm="rq" name="httpd" dev="vda1" ino=34201106 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_config_t:s0 tclass=dir permissive=0

@zpytela is working on updating our policies right now. He should be able to help.

I'm running pulplift now with your nightly RPM packages to see if I get this error.

@ekohl
Copy link
Member Author

ekohl commented Sep 7, 2020

Updated. There is now a parameter libexecdir and the WorkingDirectory in the services is set to ~, meaning systemd will now also ensure the FS is mounted before starting services.

With this, I think it's ready to merge. I think those SELinux AVCs don't need to block merging.

Maybe we first want to get nightly green with 3.6 before adding more changes though. @jlsherrill?

@jlsherrill
Copy link
Contributor

jlsherrill commented Sep 7, 2020 via email

@ekohl
Copy link
Member Author

ekohl commented Sep 14, 2020

Rebased to resolve the merge conflict, no real change.

@ekohl
Copy link
Member Author

ekohl commented Sep 16, 2020

Looks like I had a bad rebase: it managed the assets dir twice. Should be fixed now.

@ekohl
Copy link
Member Author

ekohl commented Sep 16, 2020

@wbclark tests are green now and the pipeline is ready for it. Please have a look and approve/merge it if you think it's good.

@ehelms
Copy link
Member

ehelms commented Sep 21, 2020

I get errors testing this on current nightly. The issues look SELinux related:

[root@pipeline-katello-server-nightly-centos7 vagrant]# audit2allow -a


#============= pulpcore_t ==============
allow pulpcore_t httpd_config_t:dir search;
allow pulpcore_t httpd_sys_content_t:file getattr;
allow pulpcore_t httpd_sys_rw_content_t:dir search;
allow pulpcore_t proc_t:file read;

Or possibly this actually broken in nightly:

Sep 21 14:07:43 pipeline-katello-server-nightly-centos7.war.example.com pulpcore-worker-1[4730]: File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 250, in <module>
Sep 21 14:07:43 pipeline-katello-server-nightly-centos7.war.example.com pulpcore-worker-1[4730]: "CONTENT_ORIGIN is a required setting but it was not configured. This may be caused "
Sep 21 14:07:43 pipeline-katello-server-nightly-centos7.war.example.com pulpcore-worker-1[4730]: django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.

@ekohl
Copy link
Member Author

ekohl commented Sep 21, 2020

There was a big pulpcore-selinux update last week, but I'm not sure if that was already released. I'd like to test with that.

@zpytela
Copy link

zpytela commented Sep 21, 2020

@ehelms adding these permissions should be fairly easy. I just need to ask: does pulpcore need the access to objects with the specified types? httpd_config_t httpd_sys_content_t httpd_sys_rw_content_t
httpd_config_t can be /etc/pulp as well as /etc/httpd.
Could you share the complete AVC denials?

@ekohl
Copy link
Member Author

ekohl commented Sep 21, 2020

@zpytela I don't think your patch made it into nightly RPMs yet so I think that would be a good first step. Since it now labels /usr/bin/{rq,unicorn}, the libexec wrappers aren't even needed. I still don't think it's great to claim those files for Pulp only though.

@ehelms
Copy link
Member

ehelms commented Sep 21, 2020

Here ya go:

type=SYSCALL msg=audit(1600707522.684:93449): arch=c000003e syscall=4 success=no exit=-13 a0=7f202b7c1638 a1=7ffda81f5e50 a2=7ffda81f5e50 a3=24 items=0 ppid=1 pid=6277 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="rq" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707522.684:93449): avc:  denied  { getattr } for  pid=6277 comm="rq" path="/etc/pulp/settings.py" dev="vda1" ino=2491257 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=file
type=SYSCALL msg=audit(1600707522.338:93434): arch=c000003e syscall=2 success=no exit=-13 a0=7f59a26eb3a6 a1=80000 a2=1b6 a3=24 items=0 ppid=6277 pid=6283 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="sh" exe="/usr/bin/bash" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707522.338:93434): avc:  denied  { read } for  pid=6283 comm="sh" name="meminfo" dev="proc" ino=4026532028 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file
type=SYSCALL msg=audit(1600707521.865:93432): arch=c000003e syscall=4 success=no exit=-13 a0=7f203aa7d578 a1=7ffda81fbd70 a2=7ffda81fbd70 a3=24 items=0 ppid=1 pid=6277 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="rq" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707521.865:93432): avc:  denied  { search } for  pid=6277 comm="rq" name="pulp" dev="vda1" ino=790580 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir

manifests/init.pp Outdated Show resolved Hide resolved
@zpytela
Copy link

zpytela commented Sep 23, 2020

Here ya go:

type=SYSCALL msg=audit(1600707522.684:93449): arch=c000003e syscall=4 success=no exit=-13 a0=7f202b7c1638 a1=7ffda81f5e50 a2=7ffda81f5e50 a3=24 items=0 ppid=1 pid=6277 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="rq" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707522.684:93449): avc:  denied  { getattr } for  pid=6277 comm="rq" path="/etc/pulp/settings.py" dev="vda1" ino=2491257 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=file

So should we label settings.py with the new label pulpcore_etc_t as it was in the policy previously? If this file is for pulp3 only, the answer is probably yes. What about the keys, private_key.pem and public_key.pem?

type=SYSCALL msg=audit(1600707522.338:93434): arch=c000003e syscall=2 success=no exit=-13 a0=7f59a26eb3a6 a1=80000 a2=1b6 a3=24 items=0 ppid=6277 pid=6283 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="sh" exe="/usr/bin/bash" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707522.338:93434): avc:  denied  { read } for  pid=6283 comm="sh" name="meminfo" dev="proc" ino=4026532028 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file

This one is to add, likely with kernel_read_all_proc().

type=SYSCALL msg=audit(1600707521.865:93432): arch=c000003e syscall=4 success=no exit=-13 a0=7f203aa7d578 a1=7ffda81fbd70 a2=7ffda81fbd70 a3=24 items=0 ppid=1 pid=6277 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=(none) ses=4294967295 comm="rq" exe="/usr/bin/python3.6" subj=system_u:system_r:pulpcore_t:s0 key=(null)
type=AVC msg=audit(1600707521.865:93432): avc:  denied  { search } for  pid=6277 comm="rq" name="pulp" dev="vda1" ino=790580 scontext=system_u:system_r:pulpcore_t:s0 tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir

No information about the path; also search may not be the only permission needed. The permissive= entry was stripped off the record? Supposing 0 anyway; could you run again with setenforce 0 and if possible also with full path auditing enabled?
And in general, should pulpcore have read access to http content, labeled httpd_sys_content_t and/or httpd_sys_rw_content_t?

@ekohl
Copy link
Member Author

ekohl commented Oct 5, 2020

I extracted the service changes to #125. With the updated policy in 3.7 the libexec change isn't really needed and those service benefits should go in already.

@ekohl ekohl marked this pull request as draft October 5, 2020 09:48
@ekohl
Copy link
Member Author

ekohl commented Oct 5, 2020

I'm not entirely happy with the libexec changes. Marking it as a draft now.

manifests/service.pp Outdated Show resolved Hide resolved
In python3-pulpcore 3.7.1-2 the /usr/libexec/pulpcore wrappers have been
introduced to enter the proper SELinux domain.

It has also been cherry picked to 3.6.3-2 but in the SELinux policy is
incomplete so it has no effect. The main benefit of that cherry pick is
to keep the module compatible with both 3.6 and 3.7.
@ekohl ekohl changed the title Fixes #30465 - Run services in SELinux enforcing mode Fixes #30465 - Use libexec wrappers for SELinux Oct 9, 2020
@ekohl
Copy link
Member Author

ekohl commented Oct 9, 2020

Updated the PR to use the libexec wrappers provided by packaging.

@ekohl ekohl marked this pull request as ready for review October 9, 2020 11:30
@ekohl ekohl merged commit d9eec93 into theforeman:master Oct 9, 2020
@ekohl ekohl deleted the 30465-selinux branch October 9, 2020 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants