EncryptedPaginators are not decrypting items when using KMS crypto materials provider

[mattsb42-aws]

A customer reported issues with an EncryptedPaginator failing to decrypt when using the AWS KMS crypto materials manager.

In digging into this, I realized that we also do not test for this at all...

I have duplicated this issue by adding paginator tests to the integration test suite.

The issue appears to be that the KMS encryption context is correctly including the hash and sort attribute names on ecrypt, but not on decrypt.

This appears to affect both the scan and query paginators.

...
DEBUG    botocore.endpoint:endpoint.py:101 Making request for OperationModel(name=GenerateDataKey) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'TrentService.GenerateDataKey', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.216 Python/3.7.0 Darwin/16.7.0 Botocore/1.12.216 DynamodbEncryptionSdkPython/1.1.0'}, 'body': b'{"KeyId": "arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2f7f", "NumberOfBytes": 32, "EncryptionContext": {"*amzn-ddb-env-alg*": "AES/256", "*amzn-ddb-sig-alg*": "HmacSHA256/256", "partition_attribute": "test_value", "sort_attribute": "2231.0001", "*aws-kms-table*": "DDBEC-test-resources-TestTable-HS6VNXM82B6J"}}', 'url': 'https://kms.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1111ffb38>, 'has_streaming_input': False, 'auth_type': None}}

...

DEBUG    botocore.endpoint:endpoint.py:101 Making request for OperationModel(name=Decrypt) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'TrentService.Decrypt', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.9.216 Python/3.7.0 Darwin/16.7.0 Botocore/1.12.216 DynamodbEncryptionSdkPython/1.1.0'}, 'body': b'{"CiphertextBlob": "AQEBAHhA84wnXjEJdBbBBylRUFcZZK2j7xwh6UyLoL28nQ+0FAAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDWTPzuaYLFaMum2QQIBEIA7FsxQ6CVMg66d3RKRXgfSvDOpLnLWZc/6qaPhLYhs7nU9l8Z7ROgvqCNg3lGEYc4VYegUKv285/y0xFU=", "EncryptionContext": {"*amzn-ddb-env-alg*": "AES/256", "*amzn-ddb-sig-alg*": "HmacSHA256/256", "*aws-kms-table*": "DDBEC-test-resources-TestTable-HS6VNXM82B6J"}}', 'url': 'https://kms.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1111ffb38>, 'has_streaming_input': False, 'auth_type': None}}

[mattsb42-aws]

Verified that the DDB encryption context passed to the _decrypt_initial_materials call does contain the partition and sort key name values, but the KMS encryption context does not.

ddb_table_name = 'DDBEC-test-resources-TestTable-HS6VNXM82B6J'
aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(_key_id='arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2...rant_tokens=(), _material_description={}, _regional_clients={'us-west-2': <botocore.client.KMS object at 0x11053ca58>})
parametrized_actions = AttributeActions(default_action=<CryptoAction.SIGN_ONLY: 1>, attribute_actions={'number_set': <CryptoAction.DO_NOTHING...: <CryptoAction.DO_NOTHING: 0>, 'binary_set': <CryptoAction.DO_NOTHING: 0>, 'map': <CryptoAction.ENCRYPT_AND_SIGN: 2>})
parametrized_item = {'binary': b'this is a bytestring! \x01', 'binary_set': {b'\x00\x00\x00', b'\x00\x00\x02', b'\x00\x01\x00'}, 'decimal': Decimal('123.456'), 'int': 5, ...}

    def test_ephemeral_batch_item_cycle_scan_paginator_kms(ddb_table_name, aws_kms_cmp, parametrized_actions, parametrized_item):
        """Test a the AWS KMS CMP against a small number of curated items using the scan paginator."""
        functional_test_utils.client_cycle_batch_items_check_scan_paginator(
>           aws_kms_cmp, parametrized_actions, parametrized_item, ddb_table_name
        )

aws_kms_cmp = AwsKmsCryptographicMaterialsProvider(_key_id='arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2f7f', _botocore_session=<botocore.session.Session object at 0x110037f60>, _grant_tokens=(), _material_description={}, _regional_clients={'us-west-2': <botocore.client.KMS object at 0x11053ca58>})
ddb_table_name = 'DDBEC-test-resources-TestTable-HS6VNXM82B6J'
parametrized_actions = AttributeActions(default_action=<CryptoAction.SIGN_ONLY: 1>, attribute_actions={'number_set': <CryptoAction.DO_NOTHING: 0>, 'string_set': <CryptoAction.DO_NOTHING: 0>, 'binary_set': <CryptoAction.DO_NOTHING: 0>, 'map': <CryptoAction.ENCRYPT_AND_SIGN: 2>})
parametrized_item = {'binary': b'this is a bytestring! \x01',
 'binary_set': {b'\x00\x00\x00', b'\x00\x01\x00', b'\x00\x00\x02'},
 'decimal': Decimal('123.456'),
 'int': 5,
 'list': [5,
          Decimal('123.456'),
          'this is a string',
          b'this is a bytestring! \x01',
          {3, 4, 5},
          {'abc', 'def', 'geh'},
          {b'\x00\x00\x00', b'\x00\x01\x00', b'\x00\x00\x02'}],
 'map': {'binary': b'this is a bytestring! \x01',
         'binary_set': {b'\x00\x00\x00', b'\x00\x01\x00', b'\x00\x00\x02'},
         'decimal': Decimal('123.456'),
         'int': 5,
         'list': [5,
                  Decimal('123.456'),
                  'this is a string',
                  b'this is a bytestring! \x01',
                  {3, 4, 5},
                  {'abc', 'def', 'geh'},
                  {b'\x00\x00\x00', b'\x00\x01\x00', b'\x00\x00\x02'}],
         'number_set': {3, 4, 5},
         'string': 'this is a string',
         'string_set': {'abc', 'def', 'geh'}},
 'number_set': {3, 4, 5},
 'string': 'this is a string',
 'string_set': {'abc', 'def', 'geh'}}

test/integration/encrypted/test_client.py:69: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test/functional/functional_test_utils.py:734: in client_cycle_batch_items_check_scan_paginator
    for page in encrypted_paginator.paginate(TableName=table_name, ConsistentRead=True):
.tox/py37-manual/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/client.py:108: in paginate
    page["Items"][pos] = self._decrypt_method(item=value, crypto_config=crypto_config)
.tox/py37-manual/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/item.py:199: in decrypt_dynamodb_item
    decryption_materials = inner_crypto_config.decryption_materials()
.tox/py37-manual/lib/python3.7/site-packages/dynamodb_encryption_sdk/encrypted/__init__.py:88: in decryption_materials
    return self.materials_provider.decryption_materials(self.encryption_context)
.tox/py37-manual/lib/python3.7/site-packages/dynamodb_encryption_sdk/material_providers/aws_kms.py:461: in decryption_materials
    initial_material = self._decrypt_initial_material(encryption_context)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = AwsKmsCryptographicMaterialsProvider(_key_id='arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2...rant_tokens=(), _material_description={}, _regional_clients={'us-west-2': <botocore.client.KMS object at 0x11053ca58>})
encryption_context = EncryptionContext(table_name='DDBEC-test-resources-TestTable-HS6VNXM82B6J', partition_key_name='partition_attribute', ...': '/CBC/PKCS5Padding', 'amzn-ddb-sig-alg': 'HmacSHA256/256', 'amzn-ddb-wrap-alg': 'kms', 'aws-kms-ec-attr': '*keys*'})

    def _decrypt_initial_material(self, encryption_context):
        # type: (EncryptionContext) -> bytes
        """Decrypt an encrypted initial cryptographic material value.
    
        :param encryption_context: Encryption context providing information about request
        :type encryption_context: EncryptionContext
        :returns: Plaintext of initial cryptographic material
        :rtype: bytes
        """
        key_id = self._select_key_id(encryption_context)
        self._validate_key_id(key_id, encryption_context)
        kms_encryption_context = self._kms_encryption_context(
            encryption_context=encryption_context,
            encryption_description=encryption_context.material_description.get(
                MaterialDescriptionKeys.CONTENT_ENCRYPTION_ALGORITHM.value
            ),
            signing_description=encryption_context.material_description.get(
                MaterialDescriptionKeys.ITEM_SIGNATURE_ALGORITHM.value
            ),
        )
        encrypted_initial_material = base64.b64decode(
            to_bytes(encryption_context.material_description.get(MaterialDescriptionKeys.WRAPPED_DATA_KEY.value))
        )
        kms_params = dict(CiphertextBlob=encrypted_initial_material, EncryptionContext=kms_encryption_context)
        if self._grant_tokens:
            kms_params["GrantTokens"] = self._grant_tokens
        # Catch any boto3 errors and normalize to expected UnwrappingError
        try:
            response = self._client(key_id).decrypt(**kms_params)
            return response["Plaintext"]
        except (botocore.exceptions.ClientError, KeyError):
            message = "Failed to unwrap AWS KMS protected materials"
            _LOGGER.exception(message)
>           raise UnwrappingError(message)

E           dynamodb_encryption_sdk.exceptions.UnwrappingError: Failed to unwrap AWS KMS protected materials

encrypted_initial_material = (b"\x01\x01\x01\x00x@\xf3\x8c'^1\tt\x16\xc1\x07)QPW\x19d\xad\xa3\xef\x1c!\xe9"
 b'L\x8b\xa0\xbd\xbc\x9d\x0f\xb4\x14\x00\x00\x00~0|\x06\t*\x86H\x86\xf7\r\x01'
 b'\x07\x06\xa0o0m\x02\x01\x000h\x06\t*\x86H\x86\xf7\r\x01\x07\x010\x1e'
 b'\x06\t`\x86H\x01e\x03\x04\x01.0\x11\x04\x0c\xa0\xcb,\xfe\x9d\xf1\xfa\xc9<'
 b'k\x0e\x1f\x02\x01\x10\x80;\x8a\x97\x99\x7f\xb5\xa9\xe3\xb5\n\x12\x90\xce'
 b'Z\xf7\xe9\xa2\x8bH\x08\xfc\xd0I;E\r-?w\xdbs\xcd\xb4"\x11\x03\xd9'
 b':\x8f\xf8\x85\x87\xd0\x7fd\x0e\xe3\x0c\xe9p\x084\x88w\xd8\x93\x81\x1d\xc1c')
encryption_context = EncryptionContext(table_name='DDBEC-test-resources-TestTable-HS6VNXM82B6J', partition_key_name='partition_attribute', sort_key_name='sort_attribute', attributes={}, material_description={'amzn-ddb-env-alg': 'AES/256', 'amzn-ddb-env-key': 'AQEBAHhA84wnXjEJdBbBBylRUFcZZK2j7xwh6UyLoL28nQ+0FAAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDKDLLP6d8frJPGsOHwIBEIA7ipeZf7Wp47UKEpDOWvfpootICPzQSTtFDS0/d9tzzbQiEQPZOo/4hYfQf2QO4wzpcAg0iHfYk4EdwWM=', 'amzn-ddb-map-signingAlg': 'HmacSHA256', 'amzn-ddb-map-sym-mode': '/CBC/PKCS5Padding', 'amzn-ddb-sig-alg': 'HmacSHA256/256', 'amzn-ddb-wrap-alg': 'kms', 'aws-kms-ec-attr': '*keys*'})
key_id     = 'arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2f7f'
kms_encryption_context = {'*amzn-ddb-env-alg*': 'AES/256',
 '*amzn-ddb-sig-alg*': 'HmacSHA256/256',
 '*aws-kms-table*': 'DDBEC-test-resources-TestTable-HS6VNXM82B6J'}
kms_params = {'CiphertextBlob': b"\x01\x01\x01\x00x@\xf3\x8c'^1\tt\x16\xc1\x07)QPW"
                   b'\x19d\xad\xa3\xef\x1c!\xe9L\x8b\xa0\xbd\xbc\x9d\x0f\xb4'
                   b'\x14\x00\x00\x00~0|\x06\t*\x86H\x86\xf7\r\x01\x07\x06\xa0o'
                   b'0m\x02\x01\x000h\x06\t*\x86H\x86\xf7\r\x01\x07\x010\x1e'
                   b'\x06\t`\x86H\x01e\x03\x04\x01.0\x11\x04\x0c\xa0'
                   b'\xcb,\xfe\x9d\xf1\xfa\xc9<k\x0e\x1f\x02\x01\x10\x80;'
                   b'\x8a\x97\x99\x7f\xb5\xa9\xe3\xb5\n\x12\x90\xce'
                   b'Z\xf7\xe9\xa2\x8bH\x08\xfc\xd0I;E\r-?w\xdbs\xcd\xb4'
                   b'"\x11\x03\xd9:\x8f\xf8\x85\x87\xd0\x7fd\x0e\xe3\x0c\xe9'
                   b'p\x084\x88w\xd8\x93\x81\x1d\xc1c',
 'EncryptionContext': {'*amzn-ddb-env-alg*': 'AES/256',
                       '*amzn-ddb-sig-alg*': 'HmacSHA256/256',
                       '*aws-kms-table*': 'DDBEC-test-resources-TestTable-HS6VNXM82B6J'}}
message    = 'Failed to unwrap AWS KMS protected materials'
self       = AwsKmsCryptographicMaterialsProvider(_key_id='arn:aws:kms:us-west-2:658956600833:key/b3537ef1-d8dc-4780-9f5a-55776cbb2f7f', _botocore_session=<botocore.session.Session object at 0x110037f60>, _grant_tokens=(), _material_description={}, _regional_clients={'us-west-2': <botocore.client.KMS object at 0x11053ca58>})

[mattsb42-aws]

However, the attributes are not set in the DDB encryption context.

This results in the KMS encryption context not containing the values because when we try to retrieve them from the DDB encryption context item attributes, they are not present.

[mattsb42-aws]

x_x ...looks like the paginator decrypt logic missed out on an update that called the decrypt method with crypto_config.with_item rather than the bare crypto_config.

[mattsb42-aws]

This shipped in 1.1.1