Signature support in OPA bundles

[fams]

OPA allows to store and download bundles from an https endpoint. To guarantee to avoid errors or malicious injections of rules or even data, to implement a form of signing of OPA bundles will be very likely.
It can be an extension of the dotManifest file with bundled files hash and an RSA signature of the object like in JOSE.

ex:
{
"revision": "7864d60dd78d748dbce54b569e939f5b0dc07486",
"roots": ["permissions", "istio/auth", "certificates"],
"validation": {
"files": [{
"filename": "/perms/data.json",
"sha-1": "974313a39889405a54d53d4f5083c541d1e101e4"
}, {
"filename": "/rules/rule.rego",
"sha-1": "5fa778fadccb8a48bb0bbc904d63179f24b031e7"
}],
"rs256": "UHNkNvv6ug-ZTzw8gPbVN3npmCmTH7wh7z4dfWMXUY67A0DQDRqaErURm1F7dgEk_vaekWfUTI1WgiTPgwu8guDRNz_eRluQYUh5cbfFhGCQraMC5Y5jQRqCtNd_Df4KTvuoy3zdmJPaK4QSSiaUwK1Lxf74Ek1SRGkyrGDiGp7dch_gwTGqhEGY2dtiM9TFO9ar1kK6nt9w1HT80_TLMM0Js2x7Kywgq4ZP8ARuu6j7nljJr6IBBAx_fooypEww71ucbXUk8AbiUBw-8eP_cmFnYjR-rUO9dYOH2smfhM1ZbQDRxIxL4QkxscHsEhdxHasKlRc0wmcpixLSDa7NrA"
}
}

[benjlai]

Great idea fams. I would probably like to suggest a couple of tweaks:

1. Align the format to the JWS standard with a payload, projected and header parameters

2. Each file in the manifest is signed. This accommodates different files being generated from different sources e.g rego files will come from developers (makers) and will be validated by checkers who will then approve the change and sign the rego file. Where staff2groups will be synced from an LDAP sync service by an already trusted service.

3. Capture the revision id for each file to indicate the GIT commit hash

4. A release tag so that the bundle can be identified in the decision logs

{
  "payload": {
    "bundle": {
      "release": "release_19.232.212",
      "files": [
        {
          "filename": "uam/rego/uam2-resources.rego",
          "signature": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
          "revision": "2e162430f7bc93f35164a4ffe8ad0cb5c00c0a9a"
        },
        {
          "filename": "uam/entitlements/uam-entitlements.json",
          "signature": "cc8567d70002e957612902a8e985ea129d831ebe04057d88fb644857caa45d11",
          "revision": "2e162430f7bc93f35164a4ffe8ad0cb5c00c0a9a"
        },
        {
          "filename": "ldap/staff2groups.json",
          "signature": "03f329983b86f7d9a9f5fef85305880101d5e302afafa20154d094b229f75773",
          "revision": "2e162430f7bc93f35164a4ffe8ad0cb5c00c0a9a"
        }
      ]
    }
  },
  "protected": {
    "alg": "ES256"
  },
  "header": {
    "signature": "DtEhU3ljbEg8L38VWAfUAqOyKAM6-Xx-F4GawxaepmXFCgfTjDxw5djxLa8ISlSApmWQxfKTUJqPP3-Kg6NU01Q"
  }
}

[koponen-styra]

A few quick thoughts around this.

1. I quite like the idea of having this sort of "signature manifest" declaring what is signed and how. However, requiring a signature for each file seems wasteful. I think a single signature should apply to a set of files (which could be a set of one in special case, of course).

2. I find the embedding of specific metadata into signature envelope/manifest itself non-ideal. Metadata seems better placed elsewhere, but again, the signatures should obviously cover the metadata just like they cover content.

3. Since this will be implemented not just by OPA but potentially any system involved in serving/managing bundles, it would be good to consider the implementation implications of the proposed scheme. In particular, I think we should try to use standard JWS signatures as building blocks in the design. It is likely be simplify the implementations considerably. In addition, we have less to worry about the correct, secure use of crypto primitives...

This definitely needs more thought.

[timothyhinrichs]

Typically we see people build each bundle so all the files come from a single git hash (say) on master. The presence of the file in master ensures it has been approved by the correct people (since they merged it). The signature verification ensures that the bundle as a whole is coming from a trusted source.

Are you imagining a bundle being built out of files from different repositories or different branches within the same repository?

[benjlai]

Thanks for the great feedback which I hope I've incorporated in this update.

1. I think having 2 separate manifests, (i.e. existing OPA manifest and a signature manifest) could lead to inconsistencies where the manifest list different files to the signature manifest. To eliminate this I have combined the two

2. I've added the payload, protected and header entries so that any standard JWS implementation can easily validate the signature

3. Each file has both a revision so it can be easily identified in GIT and a SHA-1 which can be easily calculated.

{
  "revision" : "7864d60dd78d748dbce54b569e939f5b0dc07486",
  "roots": ["roles", "http/example/authz"],
  "payload": {
      "files": [
        {
          "filename": "uam/rego/uam2-resources.rego",
          "sha-1": "5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
          "revision": "2e162430f7bc93f35164a4ffe8ad0cb5c00c0a9a"
        },
        {
          "filename": "uam/entitlements/uam-entitlements.json",
          "sha-1": "cc8567d70002e957612902a8e985ea129d831ebe04057d88fb644857caa45d11",
          "revision": "3e162430f7bc93f35164a4ffe8ad0cb5c00c0a9b"
        },
        {
          "filename": "ldap/staff2groups.json",
          "sha-1": "03f329983b86f7d9a9f5fef85305880101d5e302afafa20154d094b229f75773",
          "revision": "4e162430f7bc93f35164a4ffe8ad0cb5c00c0a9c"
        }
      ]
  },
  "protected": {
    "alg": "ES256"
  },
  "header": {
    "signature": "DtEhU3ljbEg8L38VWAfUAqOyKAM6-Xx-F4GawxaepmXFCgfTjDxw5djxLa8ISlSApmWQxfKTUJqPP3-Kg6NU01Q"
  }

}

Once everyone has fed back I put together some examples to encode and decode...

[koponen-styra]

I think merging the manifests opens a problem of not being able to sign the .manifest itself. Hence, I'm still in favour of having the signatures outside of it. This keeps the signature envelope and the signed content clearly separated. I tend to think the inconsistency issues are more than resolvable within a single tar file.

Also, per the JWS general syntax, most of the fields in the signature are base64 url encoded. This does include the payload field. Therefore, I would not place the file list under the "payload" field. This is to avoid confusion as that field could not be used as such for the validation anyway but it has to be base64 url encoded first. Fortunately, we don't have to include the base64 encoded version to the signature itself (per JOSE detached content mode).

All and all, here's the ".signature" file the above results in:

{
    "signatures": [
        {
            "files": [
                {
                    "name": ".manifest",
                    "sha-256": "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2",
                },
                {
                    "name": "uam/rego/uam2-resources.rego",
                    "sha-256": "88ecde925da3c6f8ec3d140683da9d2a422f26c1ae1d9212da1e5a53416dcc88",
                },
                {
                    "name": "uam/entitlements/uam-entitlements.json",
                    "sha-256": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9",
                },
                {
                    "name": "ldap/staff2groups.json",
                    "sha-256": "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae",
                }
            ],
            "protected": "eyJhbGciOiJFUzI1NiJ9",
            "header": {"kid":"e9bc097a-ce51-4036-9562-d2ade882db0d"},
            "signature": "DtEhU3ljbEg8L38VWAfUAqOyKAM6-Xx-F4GawxaepmXFCgfTjDxw5djxLa8ISlSApmWQxfKTUJqPP3-Kg6NU1Q",
        },
    ]
}

I did consider also removing the digests from the file listing, because they don't really add anything to the security of the signature. Then would merely need a single top level "digest" property to indicate the cryptographic digest algorithm to use to compute digests for files. However, in the end, I convinced myself that having the digests with the files is nice since then the structure is exactly the payload to construct, base64 encode and pass for the signature validation. Sort of self-documenting.

Note the support for multiple signatures. It is the validator's responsibility to check all the files are covered and no additional files are present.

[ashish246]

Thanks for the feedback guys. Following JOSE's/JWT's spec and JWS's compact serialization approach, one of the ways we discussed is OPA to support the verification of signed bundles (files/data) where signatures will essentially be compact, URL-safe, Base64 encodedJWT token.
The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signatures (JWS) compact serialization. The claims may contain the private claims (e.g. list of files with their SHA hash within that bundle, scope, etc. in this case). Also, the signature/token will be in a file separate to .manifest file.

A sample JWT token is mentioned below.

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmaWxlcyI6Ilt7XCJuYW1lXCI6XCJkYi91YW0yL2VudGl0bGVtZW50cy9kYXRhLmpzb25cIixcInNoYS0yNTZcIjpcImMyMTMxNTQ0YzcxNmEyNWE1ZTMxZjUwNDMwMGY1MjQwZTgyMzVjYWRiOWE1N2YwYmQxYjZmNGJkNzRiMjY2MTJcIn0se1wibmFtZVwiOlwiZGIvdWFtMi9wb2xpY3kvb3BhLXBvbGljeS5yZWdvXCIsXCJzaGEtMjU2XCI6XCI0MmNmZTY3NjhiNTdiYjVmNzUwM2MxNjVjMjhkZDA3YWM1YjgxMzU1NGViYzg1MGYyY2MzNTg0M2U3MTM3YjFkXCJ9XSIsImlhdCI6MTU4OTE3MDIzMSwiaXNzIjoiSldUU2VydmljZSIsImp3a3MtdXJsIjoiaHR0cHM6Ly9naXRodWIuc2VydmljZS5hbnovY3NwL29wYS1zb3VyY2VkYXRhLWJ1bmRsZXMiLCJzY29wZSI6IkFVLkdMT0JBTC5PUEEuV1JJVEUiLCJqdGkiOiJmMTAzODVhMS0yMDgzLTQ5M2YtODgxZC03Yjk4NmQ5Y2NlN2YiLCJleHAiOjE1ODk0MjA4NjF9.y0hQug8M4eccJ9LpcLQKlKb4s3zGm1RI1IsKh6TAVik

Upon decoding, the sample header and claims would look like -

{
 "typ": "JWT",
 "alg": "RS256",
 "kid": "anz-uam2-opa-poc"
}

Claims:

{
  "files": "[{\"name\":\"db/uam2/entitlements/data.json\",\"sha-256\":\"c2131544c716a25a5e31f504300f5240e8235cadb9a57f0bd1b6f4bd74b26612\"},{\"name\":\"db/uam2/policy/opa-policy.rego\",\"sha-256\":\"42cfe6768b57bb5f7503c165c28dd07ac5b813554ebc850f2cc35843e7137b1d\"}]",
  "iat": 1589170231,
  "iss": "JWTService",
  "scope": "AU.GLOBAL.OPA.WRITE"
}

OPA could have kid, scope and JWKS URL (to fetch the public key for the given kid) configured either through boot configuration or through discovery bundles through DAS. Eventually, OPA will validate the kid, scope and JWT token it receives in the .signature file to verify the integrity of bundles, as well as the integrity of each file within it through their SHA hash, before consuming them.

Here is the repository with POC code which has a simple Go and Java implementations to generate the JWT token and cross verify it -
opa-signature-poc

Please review it and share your thoughts.

[koponen-styra]

Two observations:

1. It would be good to have the signature file itself to be pure JSON (object). That way, we can evolve the signatures more easily in the future, by adding fields and features, or even moving away from JWS if we were so to decide. That is, I'd make the file as follows: {"signature": <compact serialization of JWS>}.

2. Identifying the files requires now first parsing the signature and then doing a second JSON decoding for the claim contents, before the file list is available. This could potentially also cause issues with JWS libraries that may be geared towards not having very large tokens, as compact serialization is geared towards transmissions requiring compact representation -- just being pessimistic here with this assumption without hard data. I'd move the files to the outer object, next to signature {"signature": <JWS>, "files": [...]}. Signature verification and creation can use the files content as the payload they sign.

What do you think?

[ashish246]

Both the suggestions sound logical, thoughtful, and certainly feasible - let me have a quick discussion here and confirm you on that.

A followup question on the point - 2 though:

When you say signature in the .signature.json file, are you referring to JWT token which is header.payload.signature (either attached or detached content mode) or just the signature part of it (I am guessing you would still need header part to determine alg and kid attributes)?

[koponen-styra]

With the second point, I was referring to the requirement to take the whole signature (all of its parts), and then needing to start unpacking it (whatever it entails) to get the claims out, only after which you can identify the files the signature is about.

[ashish246]

Alright, so per your suggestion, I've modified the POC a bit to produce .signature file in JSON format. I also added the files property in that JSON which will be the payload used for signing. POC code is updated in the Github repo. Hers is the sample content of the .signature file -

{
  "signature": "eyJraWQiOiJ1YW0yLW9wYS1wb2MiLCJhbGciOiJSUzI1NiJ9.eyJqd2tzLXVybCI6Imh0dHBzOlwvXC9naXRodWIuY29tXC9jc3BcL29wYS1zb3VyY2VkYXRhLWJ1bmRsZXMiLCJzY29wZSI6IkFVLkdMT0JBTC5PUEEuV1JJVEUiLCJpc3MiOiJKV1RTZXJ2aWNlIiwiZmlsZXMiOiJbe1wibmFtZVwiOlwiZGJcL3VhbTJcL2VudGl0bGVtZW50c1wvZGF0YS5qc29uXCIsXCJzaGEtMjU2XCI6XCJjMjEzMTU0NGM3MTZhMjVhNWUzMWY1MDQzMDBmNTI0MGU4MjM1Y2FkYjlhNTdmMGJkMWI2ZjRiZDc0YjI2NjEyXCJ9LHtcIm5hbWVcIjpcImRiXC91YW0yXC9wb2xpY3lcL29wYS1wb2xpY3kucmVnb1wiLFwic2hhLTI1NlwiOlwiNDJjZmU2NzY4YjU3YmI1Zjc1MDNjMTY1YzI4ZGQwN2FjNWI4MTM1NTRlYmM4NTBmMmNjMzU4NDNlNzEzN2IxZFwifV0iLCJpYXQiOjE1ODk1Mzg4ODh9.AbyF9mr5VCTDwN3maT-Lm_CwtpWkxWS2sqPAujnFZdID9Wh0W6Jvt2JLvYvrmAPcOtYc4KPPesz6fbASnOicGZYKKK6IthK8bvOK119FcbAjLfKG__5fRT5ejROyWbsejc92f1Ij1OrwSwn6WNjXoLPbpeJLWOyvb_Qw48cI83_R5V8oS1IDYWdVdxaJojZQBfXysV7AIt5ALIxtJIsve28ILMpclCRJXKnomjccN6qskJdbSx0101FtUumx0G70B81uZ9EI80TGM7WNI7maXwCQVWKs9b7wiiuC1bxaNB0rkY5NIJ17LKr-AJRf43QlxzgwYVO8bvQVl6D_lVDbZvs",
  "files": [
    {
      "name": "db/uam2/entitlements/data.json",
      "sha-256": "c2131544c716a25a5e31f504300f5240e8235cadb9a57f0bd1b6f4bd74b26612"
    },
    {
      "name": "db/uam2/policy/opa-policy.rego",
      "sha-256": "42cfe6768b57bb5f7503c165c28dd07ac5b813554ebc850f2cc35843e7137b1d"
    }
  ]
}

Upon decoding the signature, claims would look like something like this -

{
 "jwks-url": "https://github.com/csp/opa-sourcedata-bundles",
 "scope": "AU.GLOBAL.OPA.WRITE",
 "iss": "JWTService",
 "files": "[{\"name\":\"db/uam2/entitlements/data.json\",\"sha-256\":\"c2131544c716a25a5e31f504300f5240e8235cadb9a57f0bd1b6f4bd74b26612\"},{\"name\":\"db/uam2/policy/opa-policy.rego\",\"sha-256\":\"42cfe6768b57bb5f7503c165c28dd07ac5b813554ebc850f2cc35843e7137b1d\"}]",
 "iat": 1589538888
}

Does that aligns with your thoughts?

[koponen-styra]

Looking pretty much what I thought. One more thing: I think we can drop the "files" from the claims. Right?

[benjlai]

Looking pretty much what I thought. One more thing: I think we can drop the "files" from the claims. Right?

The files need to be inside the claim e.g.

{
 "jwks-url": "https://github.com/csp/opa-sourcedata-bundles",
 "scope": "AU.GLOBAL.OPA.WRITE",
 "iss": "JWTService",
 "files": "[{\"name\":\"db/uam2/entitlements/data.json\",\"sha-256\":\"c2131544c716a25a5e31f504300f5240e8235cadb9a57f0bd1b6f4bd74b26612\"},{\"name\":\"db/uam2/policy/opa-policy.rego\",\"sha-256\":\"42cfe6768b57bb5f7503c165c28dd07ac5b813554ebc850f2cc35843e7137b1d\"}]",
 "iat": 1589538888
}

If they are outside of the claim then anyone can change the files contents and update the SHA-256.

So we only just need the compact serialized signature:

{
  "signature": "eyJraWQiOiJ1YW0yLW9wYS1wb2MiLCJhbGciOiJSUzI1NiJ9.eyJqd2tzLXVybCI6Imh0dHBzOlwvXC9naXRodWIuY29tXC9jc3BcL29wYS1zb3VyY2VkYXRhLWJ1bmRsZXMiLCJzY29wZSI6IkFVLkdMT0JBTC5PUEEuV1JJVEUiLCJpc3MiOiJKV1RTZXJ2aWNlIiwiZmlsZXMiOiJbe1wibmFtZVwiOlwiZGJcL3VhbTJcL2VudGl0bGVtZW50c1wvZGF0YS5qc29uXCIsXCJzaGEtMjU2XCI6XCJjMjEzMTU0NGM3MTZhMjVhNWUzMWY1MDQzMDBmNTI0MGU4MjM1Y2FkYjlhNTdmMGJkMWI2ZjRiZDc0YjI2NjEyXCJ9LHtcIm5hbWVcIjpcImRiXC91YW0yXC9wb2xpY3lcL29wYS1wb2xpY3kucmVnb1wiLFwic2hhLTI1NlwiOlwiNDJjZmU2NzY4YjU3YmI1Zjc1MDNjMTY1YzI4ZGQwN2FjNWI4MTM1NTRlYmM4NTBmMmNjMzU4NDNlNzEzN2IxZFwifV0iLCJpYXQiOjE1ODk1Mzg4ODh9.AbyF9mr5VCTDwN3maT-Lm_CwtpWkxWS2sqPAujnFZdID9Wh0W6Jvt2JLvYvrmAPcOtYc4KPPesz6fbASnOicGZYKKK6IthK8bvOK119FcbAjLfKG__5fRT5ejROyWbsejc92f1Ij1OrwSwn6WNjXoLPbpeJLWOyvb_Qw48cI83_R5V8oS1IDYWdVdxaJojZQBfXysV7AIt5ALIxtJIsve28ILMpclCRJXKnomjccN6qskJdbSx0101FtUumx0G70B81uZ9EI80TGM7WNI7maXwCQVWKs9b7wiiuC1bxaNB0rkY5NIJ17LKr-AJRf43QlxzgwYVO8bvQVl6D_lVDbZvs",
}

[koponen-styra]

There's no security impact for the following reason. First, the files can be the payload as such and this would guarantee its integrity. But to avoid the duplication of it and keep the files outside for convenient access, we can use the detached signature approach. The payload integrity is still protected by the signature, even though the payload is transmitted outside of the signature for convenience.

[ashish246]

Guys, as discussed previously, I've added CLI options to both, Go & Java, implementations of the POC code at opa-signature-poc. It currently has signing/verification approach for both asymmetric(RSA) and symmetric(HMAC) keys, however, it only uses the compact serialization approach.

Signatures can be cross verified across Go and Java commands/implementations. README.md contains the list of commands and flags and rest is descriptive enough.

Please take a look and try it out. We can discuss it in our next catchup.

[ashish246]

Just to add a bit more around verify command. It is currently doing the following:

• Verify the signature for the supplied (public)key
• Verify the number (& name) of the files in the payload/claims against the supplied target directory. Fails if the target directory doesn't have the exact number (& name) files as in the payload/claims
• Verify the content of each file by verifying the SHA hash in the payload/claims against the SHA hash generated for each file in the target directory