Key rotation leads to "Could not fetch private signing key for OpenID Connect"

[anon6789]

Hi,

I tested the new system key rotation feature of hydra v1.0.0-beta.9-alpine and it sees there is a bug.

First I start the hydra migration:

hydra migrate sql -e

Starting hydra:

SYSTEM_SECRET=system_secret_1 COOKIE_SECRET=cookie_secret_1 hydra serve all

Hydra is running normally.

Now I try to rotate the SYSTEM_SECRET. I use the new migrate secret command:

OLD_SYSTEM_SECRET=system_secret_1 NEW_SYSTEM_SECRET=system_secret_2 hydra migrate secret $DATABASE_URL

This generates the following log message:

Rotating encryption keys for JSON Web Key storage...
Error "unexpected end of JSON input" occurred while trying to rotate the JSON Web Key storage. All changes have been reverted.Rotating encryption keys for JSON Web Key storage completed successfully!
You may now run ORY Hydra with the new system secret. [...]

Then i run Hydra with the new secret:

SYSTEM_SECRET=system_secret_2 COOKIE_SECRET=cookie_secret_1 hydra serve all

This generates an error:

{"level":"info","msg":"Connecting with postgres://*:*@127.0.0.1:5432/hydra?sslmode=disable","time":"2018-09-10T06:59:54Z"}
{"level":"info","msg":"Connected to SQL!","time":"2018-09-10T06:59:54Z"}
{"level":"warning","msg":"Expected system secret to be at least 32 characters long, got 0 characters.","time":"2018-09-10T06:59:54Z"}
{"level":"info","msg":"Generating a random system secret...","time":"2018-09-10T06:59:54Z"}
{"level":"info","msg":"Generated system secret: _7~QDXr6u0lqmJ1KjFGxtn47udToKXFo","time":"2018-09-10T06:59:54Z"}
{"level":"warning","msg":"WARNING: DO NOT generate system secrets in production. The secret will be leaked to the logs.","time":"2018-09-10T06:59:54Z"}
{"error":"unexpected end of JSON input","level":"fatal","msg":"Could not fetch private signing key for OpenID Connect - did you forget to run \"hydra migrate sql\" or forget to set the SYSTEM_SECRET?","time":"2018-09-10T06:59:54Z"}

When I try to start hydra with the old SYSTEM_SECRET the same error appears.
Running hydra with the environment variable ROTATED_SYSTEM_SECRET gives the same error message as well:

SYSTEM_SECRET=system_secret_2 COOKIE_SECRET=cookie_secret_1 ROTATED_SYSTEM_SECRET=system_secret_2 hydra serve all

I would expect hydra to start normal with the new system secret.

All used secrets are 32 alphanumerics.

[aeneasr]

Thanks! I’ll investigate this over the weekend!

[aeneasr]

Ok so this is actually a mixture of bad CLI error messages and mistaken assumptions:

All used secrets are 32 alphanumerics.

No they're not, SYSTEM_SECRET=system_secret_2 and SYSTEM_SECRET=system_secret_1 are each only 16 characters long.

I'll fix the broken error messages but the root cause of this issue is misconfiguration. Thank you for reporting!

[aeneasr]

Reopening for investigation following discussion on discord