"Could not fetch signing key for OpenID Connect"

[erstaples]

I brought up a Kubernetes pod with Hydra and a fresh mysql database and it spins up fine the first time. If I kill the pod and try to bring it back up I get the following error:

time="2017-04-28T22:50:51Z" level=info msg="Connecting with mysql://*:*@tcp(hydra-mysql:3306)/hydra?parseTime=true"
time="2017-04-28T22:50:51Z" level=info msg="Connected to SQL!"
time="2017-04-28T22:50:51Z" level=warning msg="Expected system secret to be at least 32 characters long, got 6 characters."
time="2017-04-28T22:50:51Z" level=info msg="Generating a random system secret..."
time="2017-04-28T22:50:51Z" level=info msg="Generated system secret: fS$XqAC2/82B%94pIysT_(RAp%3>kc=<"
time="2017-04-28T22:50:51Z" level=warning msg="WARNING: DO NOT generate system secrets in production. The secret will be leaked to the logs."
time="2017-04-28T22:50:51Z" level=info msg="Applied 0 migrations mysql!"
Could not fetch signing key for OpenID Connect

It only seems to happen when Hydra is connecting to an existing mysql db, i.e. on a db where the migrations have already ran.

I do nothing, run no hydra commands, or do anything with the database between starting up the pod for the first time, killing it, and bringing it back up again, which leads me to believe this is a bug and not something I messed up on my end (still very possible it's me; I'm rather new to OAuth2 and OpenID Connect).

I'm using oryd/hydra:0.7.10 and mysql:5.7 images. My startup command in hydra is /go/bin/hydra host --dangerous-auto-logon --dangerous-force-http

[aeneasr]

You either forgot to set SYSTEM_SECRET or used a wrong SYSTEM_SECRET. This is the error you get when that's wrong. Could use a better one of course...

[erstaples]

Ah, great. That was it! I had thought that leaving SYSTEM_SECRET empty and allowing the system to generate a system secret was good enough, but yeah of course it's going to try to generate a new system secret on pod recreation. So, yeah, totally my fault; setting SYSTEM_SECRET fixed this.

Thanks for the quick response.

[aeneasr]

I will add a note to the error message to make this more obvious :)

…

Am 29.04.2017 um 04:14 schrieb Eric Staples ***@***.***>: Ah, great. That was it! I had thought that leaving SYSTEM_SECRET empty and allowing the system to generate a system secret was good enough, but yeah of course it's going to try to generate a new system secret on pod recreation. So, yeah, totally my fault; setting SYSTEM_SECRET fixed this. Thanks for the quick response. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[dkushner]

I am experiencing this same issue when attempting to start up hydra in a local docker-compose cluster running against a stock postgres image in the same cluster. This behaviour should be impossible in that case since the postgres container does not save state between runs, yeah? How might this be happening?

[aeneasr]

did you forget to run hydra migrate before?

…

Am 25.05.2017 um 02:09 schrieb Dave Kushner ***@***.***>: I am experiencing this same issue when attempting to start up hydra in a local docker-compose cluster running against a stock postgres image in the same cluster. This behaviour should be impossible in that case since the postgres container does not save state between runs, yeah? How might this be happening? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[dkushner]

@arekkas, I did not but had a misunderstanding about how docker-compose was handling container state which you actually helped me address in the Gitter. Appreciated!

[grega]

@dkushner could you share your findings re. docker-compose + postgres + Hydra's migrations?

[nicerobot]

I opened #816 since it seems I can't reopen this issue.

I'm having the same problem. I shut down the container and trying to rerun fails. The problem does indeed appear to be the SYSTEM_SECRET but it's because it's not being respected from the command-line.

$ echo $SYSTEM_SECRET 
27277

$ docker run -d \
  --name ory-hydra-example--hydra \
  --network hydraguide \
  -p 9000:4444 \
  -e SYSTEM_SECRET=${SYSTEM_SECRET} \
  -e DATABASE_URL=${DATABASE_URL} \
  -e ISSUER=https://localhost:9000/ \
  -e CONSENT_URL=http://localhost:9020/consent \
  -e FORCE_ROOT_CLIENT_CREDENTIALS=admin:demo-password \
  oryd/hydra:v0.11.6

Notice that the logs show "Generating a random system secret..." and this happened during the first run so it's actually impossible to restart the server.

$ docker logs ory-hydra-example--hydra
time="2018-03-19T04:25:55Z" level=info msg="Connecting with postgres://*:*@localhost:5432/hydra?sslmode=disable"
time="2018-03-19T04:25:55Z" level=info msg="Connected to SQL!"
time="2018-03-19T04:25:55Z" level=warning msg="Expected system secret to be at least 32 characters long, got 4 characters."
time="2018-03-19T04:25:55Z" level=info msg="Generating a random system secret..."
time="2018-03-19T04:25:55Z" level=info msg="Generated system secret: 5CedSNF4Rdh9pUs6ZeI1kzJAZcug-mYj"
time="2018-03-19T04:25:55Z" level=warning msg="WARNING: DO NOT generate system secrets in production. The secret will be leaked to the logs."
Could not fetch signing key for OpenID Connect - did you forget to run "hydra migrate sql" or forget to set the SYSTEM_SECRET? Got error: unexpected end of JSON input

If i delete the tables and run migrate again, it works again though the secret is still randomly generated.

[aeneasr]

See #816 Please avoid dupes in the future, thank you.

…

Am 19.03.2018 um 05:40 schrieb nicerobot ***@***.***>: I'm having the same problem. I shut down the container and trying to rerun fails. The problem does indeed appear to be the SYSTEM_SECRET but it's because it's not being respected from the command-line. $ echo $SYSTEM_SECRET 27277 $ docker run -d \ --name ory-hydra-example--hydra \ --network hydraguide \ -p 9000:4444 \ -e SYSTEM_SECRET=${SYSTEM_SECRET} \ -e DATABASE_URL=${DATABASE_URL} \ -e ISSUER=https://localhost:9000/ \ -e CONSENT_URL=http://localhost:9020/consent \ -e FORCE_ROOT_CLIENT_CREDENTIALS=admin:demo-password \ oryd/hydra:v0.11.6 Notice that the logs show "Generating a random system secret..." and this happened during the first run so it's actually impossible to restart the server. $ docker logs ory-hydra-example--hydra time="2018-03-19T04:25:55Z" level=info msg="Connecting with ***@***.***:5432/hydra?sslmode=disable" time="2018-03-19T04:25:55Z" level=info msg="Connected to SQL!" time="2018-03-19T04:25:55Z" level=warning msg="Expected system secret to be at least 32 characters long, got 4 characters." time="2018-03-19T04:25:55Z" level=info msg="Generating a random system secret..." time="2018-03-19T04:25:55Z" level=info msg="Generated system secret: 5CedSNF4Rdh9pUs6ZeI1kzJAZcug-mYj" time="2018-03-19T04:25:55Z" level=warning msg="WARNING: DO NOT generate system secrets in production. The secret will be leaked to the logs." Could not fetch signing key for OpenID Connect - did you forget to run "hydra migrate sql" or forget to set the SYSTEM_SECRET? Got error: unexpected end of JSON input If i delete the tables and run migrate again, it works again though the secret is still randomly generated. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nicerobot]

@arekkas I opened #816 when I realized posting to this one didn't reopen it and I wasn't confident that anyone would address a closed bug. I decided to keep the comment above and just amend the comment with the newly opened bug to point that out.

[aeneasr]

Don't worry, just want to avoid people searching for similar issues that are unanswered because of duplication.