Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a0e47a9
do not create or drop the eccCollection
kevinAlbs Mar 30, 2023
0c882b1
DRIVERS-2586 do not document `escCollection` and `ecocCollection` opt…
kevinAlbs Mar 30, 2023
b7d37e8
remove eccCollection from fle2v2-CreateCollection
kevinAlbs Mar 31, 2023
c86aa74
remove `eccCollection` from `encryptedFields` data files
kevinAlbs Mar 31, 2023
e455139
regenerate fle2v2 tests
kevinAlbs Mar 31, 2023
7c06b67
remove eccCollection from fle2v2-EncryptedFields-vs-EncryptedFieldsMap
kevinAlbs Mar 31, 2023
593933c
remove eccCollection from fle2v2-Range-WrongType
kevinAlbs Mar 31, 2023
229862f
assert eccCollection is not created
kevinAlbs Apr 4, 2023
8a600f4
remove incorrect comment
kevinAlbs Apr 4, 2023
e30e83f
add wire version check and test
kevinAlbs Apr 5, 2023
f7c028a
remove `escCollection` and `ecocCollection` from test data
kevinAlbs Apr 6, 2023
48cb61c
regenerate tests
kevinAlbs Apr 6, 2023
72280f9
add $$exists to legacy format
kevinAlbs Apr 6, 2023
426533f
add $$exists to tests
kevinAlbs Apr 6, 2023
fbc7b17
test encryptedFields is consulted for metadata collection names
kevinAlbs Apr 7, 2023
234a07c
change SHOULD to MUST
kevinAlbs Apr 7, 2023
1996d63
use YAML anchors for collection names
kevinAlbs Apr 7, 2023
4031787
add comment that ecc collection is no longer created for QEv2
kevinAlbs Apr 7, 2023
1325094
remove `encryptedFieldsMap with cyclic entries does not loop`
kevinAlbs Apr 7, 2023
fcbd905
use YAML anchors for encryptedFields
kevinAlbs Apr 7, 2023
c108dbb
use `null`, not $$exists
kevinAlbs Apr 10, 2023
b5adfda
Revert "add $$exists to legacy format"
kevinAlbs Apr 10, 2023
09755f0
swap order of `base64` and `subType`
kevinAlbs Apr 10, 2023
3fc008e
remove unnecessary anchor
kevinAlbs Apr 10, 2023
d202bb6
regenerate fle2v2-CreateCollection-OldServer
kevinAlbs Apr 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 27 additions & 10 deletions source/client-side-encryption/client-side-encryption.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,6 @@ encryptedFields

{
"escCollection": "enxcol_.CollectionName.esc",
"eccCollection": "enxcol_.CollectionName.ecc",
"ecocCollection": "enxcol_.CollectionName.ecoc",
"fields": [
{
Expand All @@ -134,7 +133,6 @@ encryptedFields

The acronyms within ``encryptedFields`` are defined as follows:

* ECC: Encrypted Cache Collection
* ECOC: Encrypted Compaction Collection
* ESC: Encrypted State Collection

Expand Down Expand Up @@ -914,6 +912,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
will be interpreted by the helper method and MUST be passed to the `create`_
command.

.. note::
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
``encryptedFields``. SERVER-74069 added server-side validation for those fields
and no longer allows names to deviate from the following:

- ``enxcol_.<collectionName>.esc``
- ``enxcol_.<collectionName>.ecoc`

Drivers MUST NOT document the ``escCollection`` and ``ecocCollection``
options.

For a helper function, ``CreateCollection(collectionName, collectionOptions)``
with the name of the database associated as `dbName`, look up the encrypted
fields ``encryptedFields`` for the collection as
Expand All @@ -924,17 +933,16 @@ If a set of ``encryptedFields`` was found, then do the following operations. If
any of the following operations error, the remaining operations are not
attempted:

- Check the wire version of the writable server. If the wire version is less
than 21 (for server 7.0.0), return an error containing the error message:
"Driver support of Queryable Encryption is incompatible with server. Upgrade
server to use Queryable Encryption."
- Create the collection with name ``encryptedFields["escCollection"]`` as a
clustered collection using the options
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
``encryptedFields["escCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.esc``. Creating this collection MUST NOT check if
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
- Create the collection with name ``encryptedFields["eccCollection"]`` as a
clustered collection using the options
``{clusteredIndex: {key: {_id: 1}, unique: true}}``. If
``encryptedFields["eccCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecc``. Creating this collection MUST NOT check if
the collection namespace is in the ``AutoEncryptionOpts.encryptedFieldsMap``.
- Create the collection with name ``encryptedFields["ecocCollection"]`` as a
clustered collection using the options
Expand Down Expand Up @@ -1002,6 +1010,17 @@ Drivers MUST support a BSON document option named ``encryptedFields`` for any
``Collection.drop()``). This option will only be interpreted by the helper
method and MUST NOT be passed to the `drop`_ command.

.. note::
Users are not expected to set the ``escCollection`` and ``ecocCollection`` in
``encryptedFields``. SERVER-74069 added server-side validation for those fields
and no longer allows names to deviate from the following:

- ``enxcol_.<collectionName>.esc``
- ``enxcol_.<collectionName>.ecoc`

Drivers SHOULD NOT document the ``escCollection`` and ``ecocCollection``
options.

For a helper function ``DropCollection(dropOptions)`` with associated collection
named `collName` and database named `dbName`, look up the encrypted fields
``encryptedFields`` as `GetEncryptedFields(dropOptions, collName, dbname, true)`
Expand All @@ -1015,9 +1034,6 @@ are not attempted. A ``namespace not found`` error returned from the server
- Drop the collection with name ``encryptedFields["escCollection"]``. If
``encryptedFields["escCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.esc``.
- Drop the collection with name ``encryptedFields["eccCollection"]``. If
``encryptedFields["eccCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecc``.
- Drop the collection with name ``encryptedFields["ecocCollection"]``. If
``encryptedFields["ecocCollection"]`` is not set, use the collection name
``enxcol_.<collectionName>.ecoc``.
Expand Down Expand Up @@ -2714,6 +2730,7 @@ explicit session parameter as described in the
Changelog
=========

:2023-03-30: Remove ECC collection
:2023-02-01: Replace ``DataKeyOpts`` with ``masterKey`` in ``createEncryptedCollection``.
:2023-01-31: ``createEncryptedCollection`` does not check AutoEncryptionOptions for the encryptedFieldsMap.
:2023-01-30: Return ``encryptedFields`` on ``CreateCollection`` error.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
Copy link
Member

@jmikola jmikola Apr 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevinAlbs: I just noticed this, but if you're removing the metadata collections here, doesn't that make this file identical to its range-encryptedFields-<type>.json counterpart used by Prose Test 22? The only difference I see is that the range- files also specify contention: 0, which is the default and may be irrelevant.

If so, would it make sense to just stick with a single file and update the yamlfile() calls in the test templates?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related PR where I encountered this: #1399

Copy link
Contributor Author

@kevinAlbs kevinAlbs Apr 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only difference I see is that the range- files also specify contention: 0, which is the default and may be irrelevant.

Another difference: the field names for encryptedDouble / encryptedDecimal in encryptedFields-Range-<type>.json are encryptedDoubleNoPrecision / encryptedDecimalNoPrecision in range-encryptedFields-<type>.json

I suggest filing a DRIVERS ticket for the improvement. This change can be made in a separate PR to avoid adding unrelated changes to this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"fields": [
{
"keyId": {
Expand Down Expand Up @@ -33,4 +30,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand All @@ -23,4 +20,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -32,4 +29,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand All @@ -23,4 +20,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -32,4 +29,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -29,4 +26,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -29,4 +26,4 @@
}
}
]
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down Expand Up @@ -30,4 +27,4 @@
"bsonType": "string"
}
]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@ tests:
local: {{ local_provider() }}
encryptedFieldsMap: {
"default.default": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": []
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,7 @@ tests:
},
"schema": {
"default.default": {
"fields": [],
"escCollection": "enxcol_.default.esc",
"ecocCollection": "enxcol_.default.ecoc"
"fields": []
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@
"collection_name": "default",
"data": [],
"encrypted_fields": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ runOn:
database_name: &database_name "default"
collection_name: &collection_name "default"
data: []
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
key_vault_data: [{'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]

tests:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@
"collection_name": "default",
"data": [],
"encrypted_fields": {
"escCollection": "enxcol_.default.esc",
"eccCollection": "enxcol_.default.ecc",
"ecocCollection": "enxcol_.default.ecoc",
"fields": [
{
"keyId": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ runOn:
database_name: &database_name "default"
collection_name: &collection_name "default"
data: []
encrypted_fields: &encrypted_fields {'escCollection': 'enxcol_.default.esc', 'eccCollection': 'enxcol_.default.ecc', 'ecocCollection': 'enxcol_.default.ecoc', 'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
encrypted_fields: &encrypted_fields {'fields': [{'keyId': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedIndexed', 'bsonType': 'string', 'queries': {'queryType': 'equality', 'contention': {'$numberLong': '0'}}}, {'keyId': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'path': 'encryptedUnindexed', 'bsonType': 'string'}]}
key_vault_data: [ {'_id': {'$binary': {'base64': 'EjRWeBI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'sHe0kz57YW7v8g9VP9sf/+K1ex4JqKc5rf/URX3n3p8XdZ6+15uXPaSayC6adWbNxkFskuMCOifDoTT+rkqMtFkDclOy884RuGGtUysq3X7zkAWYTKi8QAfKkajvVbZl2y23UqgVasdQu3OVBQCrH/xY00nNAs/52e958nVjBuzQkSb1T8pKJAyjZsHJ60+FtnfafDZSTAIBJYn7UWBCwQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}}, {'_id': {'$binary': {'base64': 'q83vqxI0mHYSNBI0VniQEg==', 'subType': '04'}}, 'keyMaterial': {'$binary': {'base64': 'HBk9BWihXExNDvTp1lUxOuxuZK2Pe2ZdVdlsxPEBkiO1bS4mG5NNDsQ7zVxJAH8BtdOYp72Ku4Y3nwc0BUpIKsvAKX4eYXtlhv5zUQxWdeNFhg9qK7qb8nqhnnLeT0f25jFSqzWJoT379hfwDeu0bebJHr35QrJ8myZdPMTEDYF08QYQ48ShRBli0S+QzBHHAQiM2iJNr4svg2WR8JSeWQ==', 'subType': '00'}}, 'creationDate': {'$date': {'$numberLong': '1648914851981'}}, 'updateDate': {'$date': {'$numberLong': '1648914851981'}}, 'status': {'$numberInt': '0'}, 'masterKey': {'provider': 'local'}} ]
tests:
- description: "Compact works"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"runOn": [
{
"minServerVersion": "6.0.0",
"maxServerVersion": "6.3.99",
"topology": [
"replicaset",
"sharded",
"load-balanced"
]
}
],
"database_name": "default",
"collection_name": "default",
"tests": [
{
"description": "driver returns an error if creating a QEv2 collection on unsupported server",
"clientOptions": {
"autoEncryptOpts": {
"kmsProviders": {
"aws": {}
},
"encryptedFieldsMap": {
"default.encryptedCollection": {
"fields": [
{
"path": "firstName",
"bsonType": "string",
"keyId": {
"$binary": {
"base64": "AAAAAAAAAAAAAAAAAAAAAA==",
"subType": "04"
}
}
}
]
}
}
}
},
"operations": [
{
"name": "dropCollection",
"object": "database",
"arguments": {
"collection": "encryptedCollection"
}
},
{
"name": "createCollection",
"object": "database",
"arguments": {
"collection": "encryptedCollection"
},
"result": {
"errorContains": "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."
}
}
]
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Requires libmongocrypt 1.8.0.
runOn:
- minServerVersion: "6.0.0"
maxServerVersion: "6.3.99"
# FLE 2 Encrypted collections are not supported on standalone.
topology: [ "replicaset", "sharded", "load-balanced" ]

database_name: &database_name "default"
collection_name: &collection_name "default"

tests:
- description: "driver returns an error if creating a QEv2 collection on unsupported server"
clientOptions:
autoEncryptOpts:
kmsProviders:
aws: {} # Credentials filled in from environment.
encryptedFieldsMap:
default.encryptedCollection: {
"fields": [
{
"path": "firstName",
"bsonType": "string",
"keyId": { "$binary": { "base64": "AAAAAAAAAAAAAAAAAAAAAA==", "subType": "04" }}
}
]
}

operations:
# Do an initial drop to remove collections that may exist from previous test runs.
- name: dropCollection
object: database
arguments:
collection: "encryptedCollection"
- name: createCollection
object: database
arguments:
collection: "encryptedCollection"
result:
errorContains: "Driver support of Queryable Encryption is incompatible with server. Upgrade server to use Queryable Encryption."
Loading