AWS STS (Security Token Service) API provides a comfortable way to avoid storing credentials in configuration files or environment variables like used with aws cli tool. This Lab will guide you through the setup process step-by-step and will tackle following technologies:
- LDAP, used as backend for user storage and credential based authentication
- Red Hat Single Sign On, used as web based integration flow for token layer authentication
- Red Hat Ceph Storage, used as S3 Storage frontend utilizing STS roles and policies for granting access
NOTE
This Lab is not supported by Red Hat in anyway
and should also not be used as production implementation.
The Lab is based upon following requirements
- Red Hat Enterprise Linux 9 base images
- Red Hat Enterprise Linux 9 Workstation (hosting podman containers)
- access to registry.redhat.io (get your credentials at console.redhat.com)
- subscription-manager access to the repositories
- rhceph-6-tools-for-rhel-9-x86_64-rpms
- rhceph-7-tools-for-rhel-9-x86_64-rpms
- aws cli (install instructions)
- KVM virtualization capabilities (for Red Hat Ceph Storage)
- DNS (dnsmasq) configuration pointing to the names of the instances deployed
- 150GB diskspace (100GB for Red Hat Ceph OSD storage, images and temporary data)
- +4 Cores
- +16GB Memory
-
the Lab is not meant for Production purpose
-
Red Hat Ceph Storage will be deployed as Single Node instance
-
Red Hat Build of Keycloak will be deployed without a Database
-
ds389 will be deployed without persistent storage
-
even though all Containers can run rootless you need to set
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
NOTE rebooting any instance makes it neccessary to re-iterate through the setup steps.
- all passwords are set to
changeme
to ensure we have a trusted
path between all involved components we are going to deploy our own CA and a certificate with following names to match:
-
sso.example.com
-
ldap.example.com
-
rgw.example.com
-
s3.example.com
-
Execute the following steps to create the neccessary certificates
openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -key ca.key -sha256 -days 30 -out ca.crt -subj '/CN=STS Root CA/C=US/ST=STS/L=STS/O=STS' openssl req -new -nodes -out tls.csr -newkey rsa:4096 -keyout tls.key -subj '/CN=*.example.com/C=US/ST=STS/L=STS/O=STS' cat <<'EOF'> tls.v3.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = sso.example.com DNS.2 = ldap.example.com DNS.3 = rgw.example.com DNS.4 = s3.example.com EOF openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 30 -sha256 -extfile tls.v3.ext
-
Update you local trust bundle if you want to use your Workstation for connecting to the services
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust
To ensure SSO to LDAP connections will be trusted, we need to create our custom Java truststore.
-
Initialize a new truststore
keytool -keystore examplestore -genkey -alias example
-
Import the example CA
keytool -import -keystore examplestore -file ca.crt -alias exampleCA
-
Import the wildcard Certificate
keytool -import -keystore examplestore -file tls.crt -alias wildcard-example
This truststore will later be used when initializing the SSO deployment.
Even though Red Hat Ceph Storage 7 is shipped containerized, we utilize a Virtual Machine to have a clean separation between the Hypervisor resources and the Lab required resources. Download the image Red Hat Enterprise Linux 9.4 KVM Guest image from Red Hat.
-
Prepare your Workstation to be a Hypervisor for Containers and Virtual machines.
sudo dnf -y install \ virt-manager \ virt-install \ podman \ skopeo \ guestfs-tools \ xorriso
-
Create virt-customize instructions to prepare Red Hat Ceph installation
# choose your Ceph version by exporting the approriate repository export CEPH_VERSION=rhceph-6-tools-for-rhel-9-x86_64-rpms export CEPH_VERSION=rhceph-7-tools-for-rhel-9-x86_64-rpms # export your subscription-manager credentials export USERNAME=... export PASSWORD=... cat <<EOF> cmd.txt subscription-manager register \ --username=${USERNAME} \ --password=${PASSWORD} \ --name=ceph \ --auto-attach \ --servicelevel='Self-Support' subscription-manager repos --enable ${CEPH_VERSION} dnf install cephadm podman -y useradd -G wheel cephadmin sed -i -e " s#^%wheel.*#%wheel ALL=(ALL) NOPASSWD: ALL#; " /etc/sudoers mkdir /root/.config/containers -p runuser -u cephadmin -- ssh-keygen -N '' -f /home/cephadmin/.ssh/id_rsa EOF
The cmd.txt file will be executed during the customization process and ensure we have the neccessary packages and configurations in place to just bootstrap a Singel Node Ceph Cluster in partticular it does
- Registeres the System with subscription-manager
- Enables the Ceph repository (v6 or v7)
- Installs packages:
- cephadm
- podman
- Adds an unprivileged user for Ceph called
cephadmin
into the Groupwheel
- Updates sudoers to not prompt for passwords for members of Group
wheel
- Creates container related folders (auth.json)
- Creates a RSA ssh key for Ceph deployments (necessary even in Single Node)
-
Store your registry access https://console.redhat.com/openshift/install/pull-secret token as
auth.json
in the same directory as your Red Hat Enterprise Linux 9.4 KVM Guest image. -
Customize the image with the following command
virt-customize -a rhel-9.4-x86_64-kvm.qcow2 \ --run cmd.txt \ --copy-in auth.json:/root/.config/containers/
- You can as well include bootstrap and rollout definitions in the base image by extending the command with multiple
--copy-in
statements
- You can as well include bootstrap and rollout definitions in the base image by extending the command with multiple
-
Sysprep the image to be re-useable
virt-sysprep -a rhel-9.4-x86_64-kvm.qcow2 \ --operations machine-id,bash-history,logfiles,tmp-files,net-hostname,net-hwaddr
-
Create a cloud-init ISO for customizing login and access accordingly
touch meta-data cat <<EOF> user-data #cloud-config user: cloud-user password: changeme chpasswd: expire: False ssh_authorized_keys: - $(cat ~/.ssh/id_rsa.pub) EOF genisoimage -input-charset utf-8 -output cloudinit.iso -volid cidata -joliet -rock user-data meta-data
- The default user will be named
cloud-user
- The default password will be
changeme
- If existing your SSH public key will be added for access to the cloud-user
- The default user will be named
-
Create a Virtual Machine with the customized image and two additional empty disk and the cloudinit ISO
- You might need to adjust the qemu bridge configuration if you hit a
access denied by acl file
error
# export the name of the Bridge you want to use export bridge=br0 echo "allow ${bridge}" >>/etc/qemu-kvm/bridge.conf # initialize the Virtual Machine virt-install \ --vcpus 4 \ --memory 16384 \ --name ceph \ --import \ --disk rhel-9.4-x86_64-kvm.qcow2 \ --disk size=50 \ --disk size=50 \ --network network=${bridge},model=virtio \ --disk cloudinit.iso,device=cdrom \ --os-variant rhel9.3
- You might need to adjust the qemu bridge configuration if you hit a
-
Access the Virtual Machine as
cloud-user
with passwordchangeme
- You can access the System through SSH or console
- You can grant
root
access by-
updating the /etc/ssh/sshd_config
-
updating the /root/.ssh/authorized_keys accordingly
sudo sed -i -e ' s/#PermitRootLogin.*/PermitRootLogin yes/; ' /etc/ssh/sshd_config sudo systemctl restart sshd sudo cp ~cloud-user/.ssh/authorized_keys /root/.ssh/authorized_keys
-
-
Access the Virtual System and escalate privileges
-
Set the hostname of the system
sudo hostnamectl hostname ceph.example.com
- NOTE you need to re-login to reflect the change on the shell
-
Copy the Certificates to the Red Hat Ceph Storage
scp ca.crt tls.crt tls.key cloud-user@ceph.example.com:
-
update the trust bundle on the Ceph Host
sudo cp ca.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust
-
Escalate the
cloud-user
privilegessudo -i
-
Export the Host IP for bootstrapping
export CEPHIP=$(ip -br a | awk '/eth0/ { print $3 }' | cut -f1 -d'/')
-
Export the Hostname for bootstraping
export RGWHOST=$(hostname)
-
Export the Certifiacate and Key for RGW
export CERTIFICATE=$(cat ~cloud-user/tls.crt ~cloud-user/tls.key | sed -e ' s#^# #g;')
-
Create your rollout.yaml file
# ensure to update the device section according to your virtualization bus cat <<EOF> rollout-tmpl.yaml service_type: host addr: ${CEPHIP} hostname: ${RGWHOST} labels: - mon - mgr - osd - rgw --- service_type: mon placement: label: mon --- service_type: mgr placement: label: mgr --- service_type: rgw service_id: default placement: label: rgw spec: rgw_frontend_port: 443 rgw_frontend_ssl_certificate: | ${CERTIFICATE} ssl: true extra_container_args: - -v - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem --- service_type: osd service_id: default_osd_group placement: hosts: - ${RGWHOST} data_devices: paths: - /dev/sdb # update to your deployment disk system - /dev/sdc # update to your deployment disk system EOF
-
-
Create your Ceph cluster through cephadm
# expand variables in the yaml template to a Ceph orchestrator yaml cat rollout-tmpl.yaml | envsubst > rollout.yaml # bootstrap the Ceph cluster cephadm bootstrap \ --mon-ip ${CEPHIP} \ --initial-dashboard-user admin \ --initial-dashboard-password changeme \ --ssh-private-key /home/cephadmin/.ssh/id_rsa \ --ssh-public-key /home/cephadmin/.ssh/id_rsa.pub \ --ssh-user cephadmin \ --dashboard-password-noupdate \ --allow-fqdn-hostname \ --allow-overwrite \ --apply-spec ./rollout.yaml \ --single-host-defaults
-
Execute a cephadm shell to configure following STS
-
OIDC user for the OIDC provider
-
STS Key
-
RGW to use STS
-
STS Roles
-
Role Policies
cephadm shell # export OIDC access and secret export OIDCUID=oidc export OIDCPWD=oidc # create the RGW user for the OpenID Provider configuration radosgw-admin user create \ --uid=${OIDCUID} \ --display-name="OpenID Provider user" \ --access-key=${OIDCUID} \ --secret-key=${OIDCPWD} # Grant the oidc-provider CAPS (* for Lab simplification) radosgw-admin caps add \ --uid=${OIDCUID} \ --caps="oidc-provider=*" # Create a 16 char random string as STS Key ceph config set client.rgw rgw_sts_key $(tr -dc A-Za-z0-9 </dev/urandom | head -c 16; echo) # Configure RGW to use STS as authentication mechanism ceph config set client.rgw rgw_s3_auth_use_sts true # Restart the RGW instance as it doesn't pick up the changes properly otherwise ceph orch restart rgw.default
-
Creating a Role for Administrator access through STS and
group
assignment Depending on our SSO implementation we need to understand theCondition
matching. With LDAP as backend and SSO federating those, it is mandatory to understand that settings in the SSO realm affect theCondition
matching.-
When federating LDAP groups in SSO, we can select the Groups to be mapped to the Tokens with
Full group path
or without. TheFull group path
will expose the groups in the token with a leading slash/
. Nested groups are the reason for that as they will for example show up like/admin/rgwadmin/engineering
which reflects three groupsadmin
sub grouprgwadmin
and a subgroup of rgwadminengineering
. -
For the Lab we do not utilize nested groups and we syntax every configuration to NO
Full group path
configurations. -
execute follwing in our cephadm shell to create the role and policy
export SSO=sso.example.com export SSOREALM=ceph cat <<EOF | python3 -m json.tool --compact > /tmp/role.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": [ "arn:aws:iam:::oidc-provider/${SSO}/realms/${SSOREALM}" ] }, "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Condition": { "StringLike": { "${SSO}/realms/${SSOREALM}:groups":["rgwadmins"] } } } ] } EOF
-
the command will create a compact one-line json content as radosgw-admin does not accept a json-nice formatted input
-
-
execute follwing in our cephadm shell to create the policy for our admin role
cat <<EOF | python3 -m json.tool --compact > /tmp/policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": "arn:aws:s3:::*" } ] } EOF
- that policy will grant all
rgwadmins
full access
- that policy will grant all
-
-
we now add the role and policy to our RGW instance
radosgw-admin role create \ --role-name rgwadmins \ --assume-role-policy-doc=$(cat /tmp/role.json) radosgw-admin role policy put \ --role-name=rgwadmins \ --policy-name=admin \ --policy-doc=$(cat /tmp/policy.json)
-
verify that everything is setup correctly by executing
radosgw-admin role list
-
Start a LDAP instance as follows
podman run -d \ --name ds389 \ -p 389:3389 \ -p 636:3636 \ --rm \ -v $(pwd)/tls.crt:/data/tls/server.crt:z \ -v $(pwd)/tls.key:/data/tls/server.key:z \ -v $(pwd)/ca.crt:/data/tls/ca/ca.crt:z \ -e DS_DM_PASSWORD=changeme \ -e DS_SUFFIX_NAME=dc=example,dc=com \ docker.io/389ds/dirsrv:latest
- HINT the initial startup will take a minute
-
Create a Directory content LDIF file
cat <<'EOF'> example.com.ldif dn: dc=example,dc=com objectClass: top objectClass: domain dc: example description: dc=example,dc=com dn: ou=People,dc=example,dc=com ou: People objectClass: top objectClass: organizationalUnit dn: ou=Groups,dc=example,dc=com ou: Groups objectClass: top objectClass: organizationalUnit dn: cn=Lisa Collins,ou=People,dc=example,dc=com cn: Lisa Collins uid: lisa uidNumber: 90698 gidNumber: 90698 loginShell: /sbin/nologin homeDirectory: /home/huynhbrittany displayName: Lisa Collins shadowExpire: 1 shadowInactive: 1 shadowLastChange: 19176 shadowMax: 99999 shadowMin: 0 shadowWarning: 7 objectClass: top objectClass: nsPerson objectClass: nsAccount objectClass: nsOrgPerson objectClass: posixAccount objectClass: shadowaccount objectClass: inetuser userPassword:: e1BCS0RGMl9TSEEyNTZ9QUFBSUFBSThXb0N4UFhUOUdzNUxMM1QvZEM5eE5LZS9 KZmNBT3E1OEpUQU56dkRJTG9XYkJCR1Roc0NrdkpLOW02TDM4M1JkVXVqdGw5WUtjNWgrUDVKTHNS NU5tNE1mQlYrT3VHT1diUHNQOUZsdFdleHU3UjIzTldSRi9hRys5ZDM4azJRaWRVOGhXaitYYmhqe GppTlROS1drVXcvT2lzTzVBUWdxRTB6eU1KWXJ3a1RqdGk5QllFclREb2NzYnl4OHgyc1RRcGlkV2 tBKy9Uc1BWemc1dXZ0UElqZGNDWEFhSE54T28wTnUvcmNuZlkrSm12MUN1R3BNdVpYWDY5RUp4Nis wL3M2QkZDNnVNZC9vWWVQVUt6b2pmakRnT3ZKUnk0N3BVVzFCUVM4Q3o0VzhSWDZPaFlCVFRTalFV dHlqaklUb3Y4KzlJVkRwSitJNlk2Y3RRZ0NLWDlHR2VIWGdPMzJJTmE3R1lmb2w3TStuUE83UDVTd 2s3bmVpQmJWUFNud3FwSG45eU5KK0syNWxKQVRIajZWTE9CYnhzVWxpQ3dKeE8wd1h3cE9U dn: cn=Kimberly Weeks,ou=People,dc=example,dc=com cn: Kimberly Weeks uid: kimberly uidNumber: 93578 gidNumber: 93578 loginShell: /sbin/nologin homeDirectory: /home/melissa80 displayName: Kimberly Weeks shadowExpire: 1 shadowInactive: 1 shadowLastChange: 19176 shadowMax: 99999 shadowMin: 0 shadowWarning: 7 objectClass: top objectClass: nsPerson objectClass: nsAccount objectClass: nsOrgPerson objectClass: posixAccount objectClass: shadowaccount objectClass: inetuser userPassword:: e1BCS0RGMl9TSEEyNTZ9QUFBSUFBSThXb0N4UFhUOUdzNUxMM1QvZEM5eE5LZS9 KZmNBT3E1OEpUQU56dkRJTG9XYkJCR1Roc0NrdkpLOW02TDM4M1JkVXVqdGw5WUtjNWgrUDVKTHNS NU5tNE1mQlYrT3VHT1diUHNQOUZsdFdleHU3UjIzTldSRi9hRys5ZDM4azJRaWRVOGhXaitYYmhqe GppTlROS1drVXcvT2lzTzVBUWdxRTB6eU1KWXJ3a1RqdGk5QllFclREb2NzYnl4OHgyc1RRcGlkV2 tBKy9Uc1BWemc1dXZ0UElqZGNDWEFhSE54T28wTnUvcmNuZlkrSm12MUN1R3BNdVpYWDY5RUp4Nis wL3M2QkZDNnVNZC9vWWVQVUt6b2pmakRnT3ZKUnk0N3BVVzFCUVM4Q3o0VzhSWDZPaFlCVFRTalFV dHlqaklUb3Y4KzlJVkRwSitJNlk2Y3RRZ0NLWDlHR2VIWGdPMzJJTmE3R1lmb2w3TStuUE83UDVTd 2s3bmVpQmJWUFNud3FwSG45eU5KK0syNWxKQVRIajZWTE9CYnhzVWxpQ3dKeE8wd1h3cE9U dn: cn=Kenneth Clay,ou=People,dc=example,dc=com cn: Kenneth Clay uid: kenneth uidNumber: 35153 gidNumber: 35153 loginShell: /sbin/nologin homeDirectory: /home/austinmiller displayName: Kenneth Clay shadowExpire: 1 shadowInactive: 1 shadowLastChange: 19176 shadowMax: 99999 shadowMin: 0 shadowWarning: 7 objectClass: top objectClass: nsPerson objectClass: nsAccount objectClass: nsOrgPerson objectClass: posixAccount objectClass: shadowaccount objectClass: inetuser userPassword:: e1BCS0RGMl9TSEEyNTZ9QUFBSUFBSThXb0N4UFhUOUdzNUxMM1QvZEM5eE5LZS9 KZmNBT3E1OEpUQU56dkRJTG9XYkJCR1Roc0NrdkpLOW02TDM4M1JkVXVqdGw5WUtjNWgrUDVKTHNS NU5tNE1mQlYrT3VHT1diUHNQOUZsdFdleHU3UjIzTldSRi9hRys5ZDM4azJRaWRVOGhXaitYYmhqe GppTlROS1drVXcvT2lzTzVBUWdxRTB6eU1KWXJ3a1RqdGk5QllFclREb2NzYnl4OHgyc1RRcGlkV2 tBKy9Uc1BWemc1dXZ0UElqZGNDWEFhSE54T28wTnUvcmNuZlkrSm12MUN1R3BNdVpYWDY5RUp4Nis wL3M2QkZDNnVNZC9vWWVQVUt6b2pmakRnT3ZKUnk0N3BVVzFCUVM4Q3o0VzhSWDZPaFlCVFRTalFV dHlqaklUb3Y4KzlJVkRwSitJNlk2Y3RRZ0NLWDlHR2VIWGdPMzJJTmE3R1lmb2w3TStuUE83UDVTd 2s3bmVpQmJWUFNud3FwSG45eU5KK0syNWxKQVRIajZWTE9CYnhzVWxpQ3dKeE8wd1h3cE9U dn: cn=rgwadmins,ou=Groups,dc=example,dc=com objectClass: top objectClass: groupOfNames cn: rgwadmins member: cn=Lisa Collins,ou=People,dc=example,dc=com dn: cn=rgwwriters,ou=Groups,dc=example,dc=com objectClass: top objectClass: groupOfNames cn: rgwwriters member: cn=Kimberly Weeks,ou=People,dc=example,dc=com dn: cn=rgwreaders,ou=Groups,dc=example,dc=com objectClass: top objectClass: groupOfNames cn: rgwreaders member: cn=Kenneth Clay,ou=People,dc=example,dc=com EOF
-
Execute following command to create the Backend for the LDAP data
podman exec -ti ds389 dsconf localhost \ backend create \ --suffix dc=example,dc=com \ --be-name userRoot
-
Add the LDIF to the LDAP backend
LDAPTLS_CACERT=ca.crt ldapadd \ -x -H ldaps://ldap.example.com:636 \ -D 'cn=Directory Manager' \ -w changeme \ -f example.com.ldif
-
Create your SSO instance
NOTE you cannot run this in your homeDirectory due to SELinux prevention. Create a subfolder with the necessary files
# ensure Keycloak can read the cert key file (rootless issue) chmod +r tls.key podman run --rm --userns keep-id --name sso -d -p 443:8443 -p 80:8080 \ -e KC_HOSTNAME=sso.example.com \ -e KD_HOSTNAME_ADMIN_URL=https://sso.example.com:443 \ -e KC_HTTP_ENABLED=true \ -e KC_HTTP_PORT=8080 \ -e KC_HTTPS_PORT=8443 \ -e KC_HTTPS_CERTIFICATE_FILE=/mnt/certificates/tls.crt \ -e KC_HTTPS_CERTIFICATE_KEY_FILE=/mnt/certificates/tls.key \ -e KC_PROXY=passthrough \ -e KEYCLOAK_ADMIN=admin \ -e KEYCLOAK_ADMIN_PASSWORD=changeme \ -v $(pwd):/mnt/certificates/:z \ registry.redhat.io/rhbk/keycloak-rhel9:24-7 start \ --optimized \ --spi-truststore-file-file=/mnt/certificates/examplestore \ --spi-truststore-file-password=changeme \ --spi-truststore-file-hostname-verification-policy=ANY
- Login to SSO at https://sso.example.com with credentials
admin
andchangeme
- Select the
Create realm
button- Enter the name
ceph
asRealm name
- Enter the name
- Select
Realm settings
- Select the tab
Keys
- Select the
Add providers
section under the list of tabs - Select
rsa-geenerate
(RSA signature keys ...) Provider - Update the
Priority
to105
to be the first listed when returning tokens
- Select the
- Select the tab
- Select
User Federation
in the Navigation Bar on the left- Select
Add Ldap providers
- Select
Red Hat Directory Server
inVendor
selection - Enter
ldaps://ldap.example.com:636
in theConnection URL
field- Click
Test connection
to ensure SSL trustchain is working
- Click
- Enter
cn=Directory Manager
in theBind DN
field - Enter
changeme
in theBind credentials
field- Click
Test authentication
to ensure authentication is working
- Click
- Select
READ_ONLY
in theEdit mode
field - Enter
ou=People,dc=example,dc=com
in theUsers DN
field - Enter
nsPerson
in theUser object classes
field as only value - Select
Subtree
in theSearch scope
field
- Select
- Click
Save
to configure the LDAP federation accordingly - Click
ldap
in the List of Federation providers- in the
Action
dropbox at the top corner right selectSync all users
- Select the
Mappers
Tab- Click
Add mapper
to create a new group mapper- Enter
groups
ini theName
field - Select
group-ldap-mapper
in theMapper type
field - Enter
ou=Groups,dc=example,dc=com
in theLDAP Roles DN
field - Click
Save
to store the configuration - Select
groups
from the list of Mappers- in the
Action
dropbox at the top corner right selectSync LDAP groups to Keycloak
- in the
- Enter
- Click
- in the
- Select
Clients
in the Navigation Bar on the left- Click
Create client
- Enter
ceph
in theClient ID
field- Click
Next
- Click
- Turn on
Client authentication
- Select
Service accounts roles
in the Authentication flow section- Click
Next
- Click
- Enter
*
in theValid redirect URIs
field - Enter
*
in theValid post logout redirect URIs
field - Enter
*
in theWeb origins
field- Click
Next
- Click
- Enter
- Select the
Credentials
tab to retrieve theClient Secret
value- Store this secret for the scripts used later
- Select the
Client scopes
tab- Select the
ceph-dedicated
Scope (Dedicated scope and mappers for this client)- Click
Add mapper
- Select
By configuration
- Select
User Property
- Enter
username
in theName
field - Enter
username
in theProperty
field - Enter
sub
in theToken Claim Name
- Click
Save
- Click
- Enter
- Select
- Select
- Click
- Select the
ceph-dedicated
Scope (Dedicated scope and mappers for this client)- Click
Add mapper
- Select
By configuration
- Select
Group Membership
* Entergroups
in theName
field - Enter
groups
in theToken Claim Name
- Turn off
Full group path
- Click
Save
- Click
- Select
- Select
- Click
- Select the
ceph-dedicated
Scope (Dedicated scope and mappers for this client)- Click
Add mapper
- Select
By configuration
- Select
Audience
- Enter
audience
in theName
field - Select
ceph
in theIncluded Client Audience
dropdown - Turn on
Add to ID token
- Click
Save
- Click
- Enter
- Select
- Select
- Click
- Select the
- Click
- Select
-
Export the neccessary configuration
export SSO=sso.example.com export SSOREALM=ceph
-
Retrieve the Signing fingerprint
export FPRINT=$(curl --cacert ca.crt -s $(curl --cacert ca.crt -s https://${SSO}/realms/${SSOREALM}/.well-known/openid-configuration | jq -r .jwks_uri ) | \ jq -r ' .keys[] | select(.use=="sig") | .x5c[] | ["-----BEGIN CERTIFICATE-----", ., "-----END CERTIFICATE-----"] | join("\n")' | \ openssl x509 -noout -fingerprint | \ cut -f2 -d'=' | \ sed -e 's#:##g; ')
-
NOTE it is mandatory to understand the the returned content will contain more than one certificate data. One is used for encrypting and the other one is used for signatures. In the SSO configuration I guided you through setting the Priority of the Signature Certificate higher than the encryption Certificate to get for STS and for Clients always the expected Certificate first. The Snipped above handles it through
select(.use=="sig")
and you will get RGW error log entries withinvalid padding
andAn error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown
Client errors if the Certificate is the wrong one. -
NOTE if your TLS chain is broken you will see
-13
errors in RGW's log. -
Turn on RGW debugging with
cephadm shell ceph config set client.rgw debug_rgw 20/5 cephadm shell ceph config set client.rgw debug_ms 1/5
-
-
Create the Open-id-connect Provider in RGW
export AWS_CA_BUNDLE=ca.crt export AWS_REGION=${REGION} export AWS_ACCESS_KEY_ID=oidc export AWS_SECRET_ACCESS_KEY=oidc export SSO=sso.example.com export SSOCLIENT=ceph export RGW=rgw.example.com aws --endpoint-url https://${RGW} iam create-open-id-connect-provider \ --url https://${SSO}/realms/${SSOREALM} \ --thumbprint-list ${FPRINT} \ --client-id-list ${SSOCLIENT}
-
Verify the Open-id-connect Provider
aws --endpoint-url https://${RGW} \ iam get-open-id-connect-provider \ --open-id-connect-provider-arn \ arn:aws:iam:::oidc-provider/${SSO}/realms/${SSOREALM}
- NOTE when ever you have already utilized STS you need to unset
AWS_SESSION_TOKEN
to useAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
again.
- NOTE when ever you have already utilized STS you need to unset
-
create a wrapper to simplify OAuth token retreival
cat <<'EOF'> token export SSO=sso.example.com export SSOREALM=ceph export CLIENT_SECRET=${CLIENT_SECRET:-"invalid"} USERNAME=$1 PASSWORD=$2 KC_ACCESS_TOKEN=$(curl -s --cacert ca.crt -k -q -L -X POST "https://${SSO}/realms/${SSOREALM}/protocol/openid-connect/token" \ -H 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode "client_id=${SSOREALM}" \ --data-urlencode 'grant_type=password' \ --data-urlencode "client_secret=${CLIENT_SECRET}" \ --data-urlencode 'scope=openid' \ --data-urlencode "username=${USERNAME}" \ --data-urlencode "password=${PASSWORD}" | \ jq -r .access_token) export KC_ACCESS_TOKEN EOF
-
create a wrapper to simplify STS authentication and AWS credentials exchange
cat <<'EOF'> sts-role #!/bin/bash export AWS_CA_BUNDLE=ca.crt unset AWS_ACCESS_KEY_ID unset AWS_SECRET_ACCESS_KEY unset AWS_SESSION_TOKEN export AWS_EC2_METADATA_DISABLED=true USERNAME=$1 PASSWORD=$2 ROLE=$3 SESSION=${4:-"test"} RGW=rgw.example.com source token ${USERNAME} ${PASSWORD} ${ROLE} IDM_ASSUME_ROLE_CREDS=$(aws sts assume-role-with-web-identity \ --role-arn "arn:aws:iam:::role/${ROLE}" \ --role-session-name ${SESSION} \ --endpoint=https://${RGW} \ --web-identity-token="${KC_ACCESS_TOKEN}") export AWS_ACCESS_KEY_ID=$(echo $IDM_ASSUME_ROLE_CREDS | jq -r .Credentials.AccessKeyId) export AWS_SECRET_ACCESS_KEY=$(echo $IDM_ASSUME_ROLE_CREDS | jq -r .Credentials.SecretAccessKey) export AWS_SESSION_TOKEN=$(echo $IDM_ASSUME_ROLE_CREDS | jq -r .Credentials.SessionToken)
- type EOF manually please to avoid resolution during cat'ing
-
NOTE the environment setting of
AWS_EC2_METADATA_DISABLED=true
disabled AWS default metadata resolution and speeds up the process by not asking forhttp://169.254.169.254/latest/meta-data/...
-
Source the
sts-role
with parametersusername
password
STSrole
and optional a session identifier# ensure to export your Client_secret export CLIENT_SECRET=.... # STS authentication with a specific session name "admin" source sts-role lisa changeme rgwadmins admin # STS authentication with a default session name "test" source sts-role lisa changeme rgwadmins
-
Access your S3 API to create your bucket with the STS authentiation workflow
aws --endpoint https://${RGW} s3api create-bucket --bucket=testing
-
Upload some sample content into your bucket
aws --endpoint https://${RGW} s3 cp ca.crt s3://testing/
-
List all content from your bucket
aws --endpoint https://${RGW} s3 ls s3://testing/
-
Congratulations, you successfully enabled STS on your RGW
Even though, STS isn't hard to implement the depending infrastructure is making the setup complicated and "human" error prone due to various small settings not being aligned between all the layers for authentication.
I do definetly recommend to use a separate Realm and Client for integrating STS in an existing OIDC/SSO infrastructure as some changes might otherwise conflict and or break existing working flows.
One of those examples is preferred_username
vs username
.
Another one is that Role mappings might invalidate existing authentication flows.
It is also quite annoying that Client side wise the error is always the same no matter what is broken in the workflow.
Lack of human-reabable debug on the RGW side makes it even more difficult to understand which checkbox is missing.
STS is not hard to be setup but tricky to debug due to the lack of none Engineering responses in the error logs. We have a Bugzilla opened to improve that.
-
Always turn on rgw debug logs with
cephadm shell ceph config set client.rgw debug_rgw 20/5 cephadm shell ceph config set client.rgw debug_ms 1/5
-
disable debugging after you have finished
ceph config rm client.rgw debug_rgw ceph config rm client.rgw debug_ms
-
-
Watch the journal output for and reduce noise of none related content
journalctl --no-pager -f | \ grep radosgw | \ egrep -ve '(ping cookie|osd_op|ping magic|lua background)'
-
Double check the Signature Certificate Fingerprint
curl --cacert ca.crt -s \ $(curl --cacert ca.crt -s \ https://${SSO}/realms/${SSOREALM}/.well-known/openid-configuration | \ jq -r .jwks_uri ) | \ jq -r ' .keys[] | select(.use=="sig") | .x5c[] | ["-----BEGIN CERTIFICATE-----", ., "-----END CERTIFICATE-----"] | join("\n")' | \ openssl x509 -noout -fingerprint | \ cut -f2 -d'=' | sed -e 's#:##g; '
Check the response with the configured
open-id-connect
FingerprintAWS_REGION=${REGION} \ AWS_ACCESS_KEY_ID=${OIDCUID} \ AWS_SECRET_ACCESS_KEY=${OIDCPWD} \ aws --endpoint-url https://${RGW} \ iam get-open-id-connect-provider \ --open-id-connect-provider-arn \ arn:aws:iam:::oidc-provider/${SSO}/realms/${SSOREALM} \ jq -r '.ThumbprintList'
-
Double check that your System have a valid SSL Chain of trust
podman exec -ti $(podman ps | \ awk ' /rgw-default/ { print $1 } ') \ curl https://sso.example.com -I
You should not see any SSL issue here ...
-
Double check our token and introspect token from OIDC
cat <<'EOF'> jwt decode_base64_url() { local len=$((${#1} % 4)) local result="$1" if [ $len -eq 2 ]; then result="$1"'==' elif [ $len -eq 3 ]; then result="$1"'=' fi echo "$result" | tr '_-' '/+' | openssl enc -d -base64 } decode_jwt(){ decode_base64_url $(echo -n $1 | cut -d "." -f 2) | jq . } EOF source jwt source token lisa changeme decode_jwt ${KC_ACCESS_TOKEN}
- Ceph and STS requires following attributes to be presented
-
aud, shall be ceph for the Lab and represenst the Realm/Client
-
sub, represents the username from LDAP
uid
-
groups, represents the mapper for the roles, check on prefixing chars like
/
-
scope, shall contain
openid
to ensure it will be considered in scoping -
iss, shall be the SSO base URI including the realm NOTE RHSSO 7 will have a path including
/auth
where RHBK will notdecode_jwt ${KC_ACCESS_TOKEN} { "exp": 1715163080, "iat": 1715162780, "jti": "7be09445-d59d-4cc3-bb38-31ec6ddc9863", "iss": "https://sso.example.com/realms/ceph", "aud": "ceph", "sub": "lisa", "typ": "Bearer", "azp": "ceph", "session_state": "e92becf1-9390-4f63-b07d-fdb88dff4c03", "acr": "1", "allowed-origins": [ "*" ], "realm_access": { "roles": [ "rgwadmins" ] }, "scope": "openid profile email", "sid": "e92becf1-9390-4f63-b07d-fdb88dff4c03", "email_verified": false, "name": "Lisa Collins", "groups": [ "rgwadmins" ], "preferred_username": "lisa", "given_name": "Lisa Collins" }
-
Ensure the Token is valid
# access token attribute "exp": 1715163080 date --date='@1715163080'
-
- Ceph and STS requires following attributes to be presented