server: Add optional auth token #736

cgwalters · 2019-05-10T20:26:52Z

Since the bootstrap node serves Ignition to master, and today we
don't support scaling up/down masters, just disable access to
the master Ignition config in the in-cluster MCS by default.

runcom · 2019-05-10T20:50:08Z

Code lgtm, guess this can't hurt BYO either now since we just support workers there. Looks sane

pkg/server/cluster_server.go

cgwalters · 2019-05-10T21:47:15Z

Just for general info, I just discovered Ignition will do a retry loop infinitely on http status >= 500 but if you give it a 403 it gives up, so you need to de-provision the machine (e.g. oc -n openshift-machine-api delete machine/$x) if you were testing auth.

aws ec2 --region=us-east-2 get-console-output --instance-id i-0bc13fd82d10aee7a is handy to debug things like this.

cgwalters · 2019-05-10T21:54:04Z

Just writing up some more steps for testing/using this. After you do something like this:
oc -n openshift-machine-config-operator create secret generic --from-literal=worker=$(pwgen 32 1) ignition-auth

Your next step is to oc -n openshift-machine-api edit secret/worker-user-data - base64 decode it to get the JSON, then ensure that the Ignition request has the same auth key you generated in the secret, e.g.:

{"ignition":{"config":{"append":[{"source":"https://api-int.mycluster.example.com:22623/config/worker?auth=pxzbz5n2zgfcd4kff8krv78z8wwdndvj","verification":{}}]}, ...

runcom · 2019-05-10T21:54:58Z

pkg/server/server.go

@@ -31,10 +31,30 @@ type kubeconfigFunc func() (kubeconfigData []byte, rootCAData []byte, err error)
 // appenderFunc appends Config.
 type appenderFunc func(*ignv2_2types.Config) error

+// Error is returned by the GetConfig API
+type Error struct {


we might want to call it GetConfigError or something like that (doesn't need uppercase to be exported also, does it?)

(@runcom I thought upper to be an exported type)

like this naming suggestion btw.

yeah, I realized only after commenting that I uppercased my suggestion 😄

Or possibly configError to be explicit.

kikisdeliveryservice · 2019-05-10T22:00:48Z

aws route53 issues

/retest

cgwalters · 2019-05-10T22:01:23Z

Now a much stronger elaboration of this PR would be for the MCO to support one-time-use tokens.

Rather than having a secret, an admin (or automation) would do e.g.:

oc -n openshift-machine-config-operator create secret generic --from-literal=auth=$(pwgen 32 1) ignition-auth-worker-us-east-2a-sks8v

The secret name is the concatenation of ignition-auth- with the machineAPI object name.

Then the HTTP request would have e.g. ?n=worker-us-east-2a-sks8v&auth=<secret>.

The MCS would check for this secret's existence, and if it existed, accept the request and then delete the secret. This would also mean access to the EC2 metadata API from a pod couldn't gain a useful token.

The machine API would need to learn how to generate a per-instance userdata.

cgwalters · 2019-05-11T12:01:40Z

/retest

cgwalters · 2019-05-12T16:26:47Z

OK, all tests passed on this one, confirming my manual testing before submitting the PR that nothing depends on access to the /config/master endpoint. And we have the ability to gate access to /worker, but turning it on by default will require an installer PR.

I'd like to consider landing this as we're pretty sure it's not going to break anything, and gives us the mechanism to gate access to /config/worker.

This is an optional hardening for access to Ignition; the installer generates a random key (separately for master/worker pool) and installs it into the `openshift-machine-config-operator` namespace. If the MCS finds an `ignition-auth` secret with the `master/worker` keys, it will use it: openshift/machine-config-operator#736 This PR just generates those secrets, so we can land it before the MCO PR as well.

cgwalters · 2019-05-12T16:39:49Z

Installer PR: openshift/installer#1740

runcom · 2019-05-12T20:17:00Z

/approve

This lgtm, can this land separately from the installer PR? do we need a go-on from Auth team?

ashcrow

nits but those are nits 😄

openshift-ci-robot · 2019-05-13T13:52:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashcrow, cgwalters, runcom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ashcrow,cgwalters,runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

squeed · 2019-05-13T14:07:16Z

pkg/server/cluster_server.go

-func (cs *clusterServer) GetConfig(cr poolRequest) (*ignv2_2types.Config, error) {
-	mp, err := cs.machineClient.MachineConfigPools().Get(cr.machineConfigPool, metav1.GetOptions{})
+func (cs *clusterServer) GetConfig(cr poolRequest, auth string) (*ignv2_2types.Config, error) {	
+	authSecretObj, err := cs.kubeClient.CoreV1().Secrets(componentNamespace).Get(ignitionAuth, metav1.GetOptions{})


The ability to configure multiple secrets would make later rotation much easier.

squeed · 2019-05-13T14:10:15Z

pkg/server/cluster_server.go

+
+	// If there's a secret, require that it was passed as a query parameter.
+	if authEnabled {
+		authSecret := string(authSecretObj.Data[name])


If an administrator forgets to configure an authsecret for a new machine pool, this will fail open.

That's intentional for backcompat reasons.

crawford · 2019-05-13T17:28:04Z

/hold

This doesn't but us much in terms of security and it makes disaster recovery more difficult.

cgwalters · 2019-05-13T17:34:32Z

This doesn't but us much in terms of security

If access to the EC2 metadata endpoint is shut off (which it needs to be anyways), or in cases where the bootstrap config isn't accessible at all, I think this is fairly strong security. It entirely shuts down the problem.

How is this not buying much?

and it makes disaster recovery more difficult.

Slightly...and if desired we can switch this to just landing code to enable auth tokens for both master/worker without actually enabling it by default. In other words, drop the change to disable the master pool.

Further, keep in mind that this code is a step towards further integration with machineAPI.

Ignition may contain secret data; pods running on the cluster shouldn't have access. This adds opt-in support for denying serving that data. It is disabled by default so we can check whether this would happen in any CI scenarios to start. Run `oc -n openshift-machine-config-operator create configmap machine-config-server provision-check=yes` to switch to enforcing mode. First, we deny any request that appears to come from the pod overlay network. This closes off a lot of avenues without any risk. However, we can't guarantee all in-cluster requests appear to originate from the pod network; in some cases according to the SDN team, particularly for machines that have multiple NICs. Hence, this PR also closes off access to any IP that responds on port 22, as that is a port that is: - Known to be active by default - Not firewalled A previous attempt at this was to have an [auth token](openshift#736); but this fix doesn't require changing the installer and people's PXE setups. In the future we may reserve a port in the 9xxx range and have the MCD respond on it so that admins who disable/firewall SSH don't have indirectly reduced security.

openshift-ci-robot · 2020-09-14T18:13:50Z

@cgwalters: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws	`5e6a17b`	link	`/test e2e-aws`
ci/prow/e2e-aws-op	`5e6a17b`	link	`/test e2e-aws-op`
ci/prow/e2e-aws-disruptive	`5e6a17b`	link	`/test e2e-aws-disruptive`
ci/prow/e2e-gcp-op	`5e6a17b`	link	`/test e2e-gcp-op`
ci/prow/e2e-aws-upgrade	`5e6a17b`	link	`/test e2e-aws-upgrade`
ci/prow/e2e-gcp-upgrade	`5e6a17b`	link	`/test e2e-gcp-upgrade`
ci/prow/e2e-vsphere	`5e6a17b`	link	`/test e2e-vsphere`
ci/prow/build-rpms-from-tar	`5e6a17b`	link	`/test build-rpms-from-tar`
ci/prow/e2e-ovn-step-registry	`5e6a17b`	link	`/test e2e-ovn-step-registry`
ci/prow/e2e-aws-proxy	`5e6a17b`	link	`/test e2e-aws-proxy`
ci/prow/e2e-upgrade	`5e6a17b`	link	`/test e2e-upgrade`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-merge-robot · 2020-11-05T21:59:42Z

@cgwalters: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-agnostic-upgrade	`5e6a17b`	link	`/test e2e-agnostic-upgrade`
ci/prow/e2e-aws-serial	`5e6a17b`	link	`/test e2e-aws-serial`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cgwalters · 2020-11-23T21:55:26Z

Closing in favor of #2223

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 10, 2019

openshift-ci-robot requested review from jlebon and kikisdeliveryservice May 10, 2019 20:27

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 10, 2019

cgwalters changed the title ~~server: Disable access to the master Ignition by default~~ server: Disable access to the master Ignition by default, add optional auth token May 10, 2019

cgwalters force-pushed the mcs-master-worker-separation branch from 1af70b9 to 3bf7431 Compare May 10, 2019 20:49

cgwalters force-pushed the mcs-master-worker-separation branch from 3bf7431 to cd24821 Compare May 10, 2019 21:01

kikisdeliveryservice reviewed May 10, 2019

View reviewed changes

pkg/server/cluster_server.go Show resolved Hide resolved

runcom reviewed May 10, 2019

View reviewed changes

cgwalters mentioned this pull request May 12, 2019

Generate an ignition-auth key and provide it to the MCS openshift/installer#1740

Closed

ashcrow approved these changes May 13, 2019

View reviewed changes

squeed reviewed May 13, 2019

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 13, 2019

cgwalters mentioned this pull request May 13, 2019

RFE: Move user-data secret creation from the installer into the MCO #683

Closed

kikisdeliveryservice added the team-mco label Nov 18, 2020

cgwalters closed this Nov 23, 2020

cgwalters mentioned this pull request Jan 14, 2022

[RFE] Multi-node Request for Enhacement openshift/microshift#498

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: Add optional auth token #736

server: Add optional auth token #736

cgwalters commented May 10, 2019

runcom commented May 10, 2019

cgwalters commented May 10, 2019

cgwalters commented May 10, 2019

runcom May 10, 2019

kikisdeliveryservice May 10, 2019

kikisdeliveryservice May 10, 2019

runcom May 10, 2019

ashcrow May 13, 2019

kikisdeliveryservice commented May 10, 2019

cgwalters commented May 10, 2019 •

edited

Loading

cgwalters commented May 11, 2019

cgwalters commented May 12, 2019

cgwalters commented May 12, 2019

runcom commented May 12, 2019

ashcrow left a comment

openshift-ci-robot commented May 13, 2019

squeed May 13, 2019

squeed May 13, 2019

cgwalters May 13, 2019

crawford commented May 13, 2019

cgwalters commented May 13, 2019

openshift-ci-robot commented Sep 14, 2020

openshift-merge-robot commented Nov 5, 2020

cgwalters commented Nov 23, 2020

server: Add optional auth token #736

server: Add optional auth token #736

Conversation

cgwalters commented May 10, 2019

runcom commented May 10, 2019

cgwalters commented May 10, 2019

cgwalters commented May 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kikisdeliveryservice commented May 10, 2019

cgwalters commented May 10, 2019 • edited Loading

cgwalters commented May 11, 2019

cgwalters commented May 12, 2019

cgwalters commented May 12, 2019

runcom commented May 12, 2019

ashcrow left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented May 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crawford commented May 13, 2019

cgwalters commented May 13, 2019

openshift-ci-robot commented Sep 14, 2020

openshift-merge-robot commented Nov 5, 2020

cgwalters commented Nov 23, 2020

cgwalters commented May 10, 2019 •

edited

Loading