Keycloak Doesn't work with certain passwords (postgres)

[gerethd]

Describe the bug

using the password y3xc8%VpZ8W*RRSF7X$$zA5d keycloak was unable to connect to my postgres (specifically postgres 14) database. After verifying the password was indeed correct via cmdline pgsql, I believe this may have to do with there being a $ or two in the string. Changing the password on the database and in keycloak to one lacking the $$ caused it to work first try.

Version

latest tag as of writing

Expected behavior

I expect keycloak to connect.

Actual behavior

For an optional runtime and bypass this step, please run the 'build' command prior to starting the server:

	kc.sh build --db-url=jdbc:postgresql://zevrant-keycloak-service-db:5432/keycloak --db-username=zevrant


Updating the configuration and installing your custom providers, if any. Please wait.
2022-03-14 13:58:30,128 INFO  [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 6714ms
Server configuration updated and persisted. Run the following command to review the configuration:

	kc.sh show-config

Next time you run the server, just run:

	kc.sh start

2022-03-14 13:58:32,440 INFO  [org.key.qua.run.hos.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: zevrant-keycloak-service, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>
2022-03-14 13:58:32,879 WARN  [io.agr.pool] (agroal-11) Datasource '<default>': FATAL: password authentication failed for user "zevrant"
2022-03-14 13:58:32,880 WARN  [org.hib.eng.jdb.env.int.JdbcEnvironmentInitiator] (JPA Startup Thread: keycloak-default) HHH000342: Could not obtain connection to query metadata: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "zevrant"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:623)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:163)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:215)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
	at org.postgresql.Driver.makeConnection(Driver.java:466)
	at org.postgresql.Driver.connect(Driver.java:265)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:103)
	at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:49)
	at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:35)
	at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:210)
	at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:509)
	at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:490)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:75)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

How to Reproduce?

Docker image

FROM quay.io/keycloak/keycloak-x:latest

RUN whoami && groups

USER root

RUN microdnf install -y jq python3-pip openssl \
    && mkdir -p /etc/x509/https/ \
#    && chown -R root:jboss /etc/x509/https/ \
    && chmod -R 0770 /etc/x509/https/

USER keycloak

ENTRYPOINT KC_HOSTNAME=$SERVICE_NAME KC_DB=postgres KC_METRICS_ENABLED=true /opt/keycloak/bin/kc.sh start --db-password 'vFEc84zDEz7AvyN8M7jCm%Q8' --auto-build --db-url $DB_ADDR --db-username $DB_USER \
    && /opt/keycloak/bin/kc.sh start

CMD ["-b", "0.0.0.0"]

Anything else?

This isn't a game breaking bug but it is rather annoying to troubleshoot, Dockerfile given is as close as i can gicew without providing access to proprietary resources, in this case access to my keyvault and private CA Authority.

[dark-enstein]

I am also having this issue. It's crazy annoying

[vmuzikar]

@gerethd @Ayobama Thanks for the report! Can you please try it with the latest 19.0.1 release? In any case, please consider using another config method (like env variable or config file) as providing password through CLI in plain text is considered insecure.

[DGuhr]

FTR: this is not a general bug in newer versions.
I just tried it using the following compose.yml that sets the password using the CLI in the command section:

services:
  keycloak:
    container_name: keycloak
    image: quay.io/keycloak/keycloak:19.0.1
    command: ["start-dev","--db-password='y3xc8%VpZ8W*RRSF7X$$zA5d'"]
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin
      - KC_DB=postgres #build option
      - KC_FEATURES=admin2 #build option
      - KC_DB_URL_HOST=postgres
      - KC_DB_URL_DATABASE=keycloak 
      - KC_DB_USERNAME=keycloak 
    ports:
      - 8080:8080 
      - 8443:8443
    depends_on:
      - postgres
  postgres:
    image: postgres:13-alpine
    environment:
      - POSTGRES_DB=keycloak
      - POSTGRES_USER=keycloak
      - POSTGRES_PASSWORD=y3xc8%VpZ8W*RRSF7X$$zA5d
      - POSTGRES_HOST_AUTH_METHOD=scram-sha-256
      - POSTGRES_INITDB_ARGS=--auth-host=scram-sha-256
    ports:
      - 5432:5432
    volumes:
      - pgtest-volume:/var/lib/postgresql/data
volumes:
  pgtest-volume:
    driver: local

So @Ayobama @gerethd I strongly suggest you to update Keycloak to use the quay.io/keycloak/keycloak image, currently at 19.0.1 - the keycloak-x image is considered alpha and deprecated since v17, where we made quarkus based Keycloak the default.

[gerethd]

@gerethd @Ayobama Thanks for the report! Can you please try it with the latest 19.0.1 release? In any case, please consider using another config method (like env variable or config file) as providing password through CLI in plain text is considered insecure.

For starters no this was not hardcoded/passed in via the cli, this was a kubernetes secret/docker secret mounted as a file within the container, and passed in at runtime, and was simplified to the above to make it easier for your team to address/see what was occuring.

I have updated multiple times since then as there have been multiple releases since then. When this was initially posted this was the latest version available or close to it.

As a user of this product I find it quite worrisome there wasn't so much as an acknowledgement from the team for more than 6 months of this issue. I would hope this is more a 'slipped through the cracks' kind of scenario and not the norm

[augurpl]

omfg, I've wasted few hours trying to figure why connection to cloud sql is not working. This was main reason... I'm using latest version 23.0.6.

[shawkins]

@augurpl we added a doc note about options values containing $$ with #19831 - for now they have to be escaped.

[nehalandrew]

Hello @shawkins

I had the same problem with the special characters { }. It took me a long time to solve this trivial problem. Thanks to this problem at stackoverflow.

It is quite strange that the log says that the problem is caused by Hibernate and not by password check.

Maybe it would be better to add password validation logging before the Hibernate?

P.S. I would love to make my first contribution to an open source project, but due to time limitations I can't.

[shawkins]

@nehalandrew please start new issue(s) for this. There are a couple of things being brought up here:

• clear error message when database authentication fails
• Unexpected behavior with { } in a password - was your case not covered by the existing docs?

Also adding a link here to the latest quarkus issue to be captured on $$: quarkusio/quarkus#41883 - there is a further upstream fix proposed: smallrye/smallrye-common#344