INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

(see previous attempts in #318, #319 and #323 for additional context)

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

aclark4life · 2025-06-27T18:08:35Z

Wrong commit message for 65bd15a and I don't want to force push yet. It should have said:

"Only create an encrypted connection once then reuse it."

I'm aware that _nodb_cursor is slated for removal but in the meantime I can keep going with other fixes with this approach, and it does satisfy the design we all agree on (I think) of maintaining two simultaneous connections:

Unencrypted connection unless we need it
Encrypted connection when we need that can be used.

timgraham · 2025-06-27T20:05:16Z

I'm aware that _nodb_cursor is slated for removal but in the meantime I can keep going with other fixes with this approach, and it does satisfy the design we all agree on (I think) of maintaining two simultaneous connections:

It's not working as you think it is. As I said elsewhere, _nodb_cursor is not used by this backend.

Does this fix the "command not supported for auto encryption: buildinfo" error? If so, it's perhaps because self.settings_dict["OPTIONS"].pop("auto_encryption_opts") is having the side effect of altering settings_dict before DatabaseWrapper.connection is initialized.

I'd suggest to use my patch is as a starting point for maintaining two connections. self.connection should be the encrypted version (secure by default) with a fallback to a non-encrypted connection only as needed (e.g. for commands like buildInfo). At least it will help us understand whether that's a viable approach. As I mentioned in the design doc, I'm not sure if using an encrypted connection for non-encrypted collections is problematic. If so, we'll have to go back to the drawing board on the design.

aclark4life · 2025-06-27T21:51:58Z

It's not working as you think it is. As I said elsewhere, _nodb_cursor is not used by this backend.

I don't disagree, but it feels a lot like _start_transaction_under_autocommit which gets called by start_transaction_under_autocommit because autocommit is False. Django appears to stumble into _nodb_cursor when the encrypted connection fails to get the database version and while we don't use a cursor in this backend, we do have a "nosql" cursor that has __enter__ and __exit__ (I assume) to meet Django's expectations and we get an opportunity to modify the connection. @Jibola mentioned this design is suspect yesterday and I agree with both of you, particularly with regard to the desire to start with and maintain an encrypted connection first.

Does this fix the "command not supported for auto encryption: buildinfo" error? If so, it's perhaps because self.settings_dict["OPTIONS"].pop("auto_encryption_opts") is having the side effect of altering settings_dict before DatabaseWrapper.connection is initialized.

Yes it works by design, not a side effect. I'm deep.copying settings_dict when DatabaseWrapper is initialized and so when DatabaseWrapper.connection is initialized it's unencrypted. When the schema needs encryption later, it's retrieved from _settings_dict.

I'd suggest to use my patch is as a starting point for maintaining two connections. self.connection should be the encrypted version (secure by default) with a fallback to a non-encrypted connection only as needed (e.g. for commands like buildInfo). At least it will help us understand whether that's a viable approach. As I mentioned in the design doc, I'm not sure if using an encrypted connection for non-encrypted collections is problematic. If so, we'll have to go back to the drawing board on the design.

I'd make a few passes at it but did not get anywhere, I'll try again though.

timgraham · 2025-06-27T22:12:08Z

Your "stumble" theory of how it's working isn't correct. _nodb_cursor is only used on one place: to create the test database. As I said, I could imagine that perhaps this method causes connection to later be initialized without auto_encryption_opts because of self.settings_dict["OPTIONS"].pop("auto_encryption_opts"). The connection that's created in your _nodb_cursor is never used.

aclark4life · 2025-06-28T00:10:12Z

Your "stumble" theory of how it's working isn't correct. _nodb_cursor is only used on one place: to create the test database. As I said, I could imagine that perhaps this method causes connection to later be initialized without auto_encryption_opts because of self.settings_dict["OPTIONS"].pop("auto_encryption_opts"). The connection that's created in your _nodb_cursor is never used.

Copy that, thanks!

I've removed _nodb_cursor in 8e83ada and discovered the version check is the only time that error occurs. I now get errors like:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt
    encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt
    return run_state_machine(ctx, self.callback)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine
    result = callback.mark_command(ctx.database, mongocryptd_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
        inflated_cmd, codec_options=DEFAULT_RAW_BSON_OPTIONS
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command
    return self._command(
           ~~~~~~~~~~~~~^
        connection,
        ^^^^^^^^^^^
    ...<7 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command
    return conn.command(
           ~~~~~~~~~~~~^
        self._name,
        ^^^^^^^^^^^
    ...<8 lines>...
        client=self._client,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner
    return func(*args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command
    helpers_shared._check_command_response(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        response_doc,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        parse_write_concern_error=parse_write_concern_error,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection., full error: RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Still working on an unencrypted connection, but perhaps the only time we need it is for the version check.

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

- Refactor tests

django_mongodb_backend/schema.py

aclark4life · 2025-07-02T02:49:19Z

@ShaneHarvey @Jibola @timgraham FYI here is the pipeline that causes the let error:

(Pdb) pprint.pprint(pipeline)
[{'$lookup': {'as': 'django_content_type',
              'from': 'django_content_type',
              'let': {'parent__field__0': '$content_type_id'},
              'pipeline': [{'$match': {'$expr': {'$and': [{'$eq': ['$$parent__field__0',
                                                                   '$_id']}]}}}]}},
 {'$unwind': '$django_content_type'},
 {'$match': {'$expr': {'$in': ['$content_type_id',
                               (ObjectId('6864933ec7cf8179e3ef1f8d'),)]}}},
 {'$project': {'codename': 1,
               'content_type_id': 1,
               'django_content_type': {'app_label': 1, 'model': 1}}},
 {'$sort': SON([('django_content_type.app_label', 1), ('django_content_type.model', 1), ('codename', 1)])}]

And here is the error again with some additional debug:

(Pdb) errmsg
"Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection."
(Pdb) code
51208
(Pdb) response
RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))
(Pdb) max_wire_version
26

And the full traceback:


Running post-migrate handlers for application contenttypes
Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt
    encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt
    return run_state_machine(ctx, self.callback)
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine
    result = callback.mark_command(ctx.database, mongocryptd_cmd)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
        inflated_cmd, codec_options=DEFAULT_RAW_BSON_OPTIONS
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command
    return self._command(
           ~~~~~~~~~~~~~^
        connection,
        ^^^^^^^^^^^
    ...<7 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command
    return conn.command(
           ~~~~~~~~~~~~^
        self._name,
        ^^^^^^^^^^^
    ...<8 lines>...
        client=self._client,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner
    return func(*args, **kwargs)
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command
    helpers_shared._check_command_response(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        response_doc,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        parse_write_concern_error=parse_write_concern_error,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection., full error: RawBSONDocument(b"\xa7\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00d\x00\x00\x00Non-empty 'let' field is not allowed in the $lookup aggregation stage over an encrypted collection.\x00\x10code\x00\x08\xc8\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location51208\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Test settings:

import os

from django_mongodb_backend import encryption, parse_uri

kms_providers = encryption.get_kms_providers()

auto_encryption_opts = encryption.get_auto_encryption_opts(
    kms_providers=kms_providers,
)

DATABASE_URL = os.environ.get("MONGODB_URI", "mongodb://localhost:27017")
DATABASES = {
    "default": parse_uri(
        DATABASE_URL, db_name="djangotests",
    ),
    "encrypted": parse_uri(
        DATABASE_URL, options={"auto_encryption_opts": auto_encryption_opts},
            db_name="encrypted_djangotests",
    ),
}

DEFAULT_AUTO_FIELD = "django_mongodb_backend.fields.ObjectIdAutoField"
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
SECRET_KEY = "django_tests_secret_key"
USE_TZ = False

This is happening in the encryption_ tests with a database router configured to use the encrypted database, but it happens before any tests are run or any routing occurs. I've confirmed that the encrypted database is created, so it appears that something needs to be done to address this issue in either our backend or PyMongo with the ideal candidate, perhaps, being a change to the MQL in the pipeline if possible.

tests/encryption_/tests.py

timgraham · 2025-07-14T22:27:02Z

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

+    def get_encrypted_fields_map(self, connection):
+        return {
+            "fields": [
+                field
+                for app_config in apps.get_app_configs()
+                for model in router.get_migratable_models(
+                    app_config, connection.alias, include_auto_created=False
+                )
+                if getattr(model, "encrypted", False)
+                for field in connection.schema_editor()._get_encrypted_fields_map(model)
+            ]
+        }


This format doesn't look correct. Doesn't it have to include the database and collection information? Look at the pymongo example.

Maybe. If I pass the results as schema_map to AutoEncryptionOpts I don't get an error, but I'll try with namespace.

Please add the database name too. Quoting the linked example:

# The MongoDB namespace (db.collection) used to store the # encrypted documents in this example. encrypted_namespace = "test.coll"

django_mongodb_backend/encryption.py

docs/source/howto/encryption.rst

django_mongodb_backend/encryption.py

timgraham · 2025-07-14T22:55:16Z

tests/backend_/test_features.py

+        connection.features.__dict__.pop("supports_queryable_encryption", None)
+
+    def tearDown(self):
+        connection.features.__dict__.pop("supports_queryable_encryption", None)


This can use the del ... version since it will exist by now.

This can use the del ... version since it will exist by now.

By now? I still get an attribute error …

I mean in tearDown() since each test will (presumably) initialize the attribute.

WaVEV · 2025-07-15T03:55:01Z

django_mongodb_backend/encryption.py

+
+class EncryptedRouter:
+    def _get_db_for_model(self, model):
+        if getattr(model, "encrypted", False):


Sorry If I ask something that was previously discussed, but how does it know if a model was an encrypted model or a model that has a field encrypted = models.BooleanField(). Maybe with getattr(model, "encrypted", False) is True could save for some false positives.

Good catch, thanks. I don't think we've finalized using that conditional yet and more importantly there is one in schema._create_collection.

WaVEV · 2025-07-15T04:08:39Z

django_mongodb_backend/fields/encryption.py

+        return name, path, args, kwargs
+
+
+class EncryptedCharField(EncryptedFieldMixin, models.CharField):


I didn't find in the docs if an encrypted collection could have an aggregate query. So my question is:
does it support all the lookups from CharField ?

No aggregation stages are supported and two tests from Django's test_charfield are failing, though only one is an aggregation stage failure:

====================================================================== ERROR: test_assignment_from_choice_enum (encryption_.test_charfield.TestEncryptedCharField.test_assignment_from_choice_enum) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors yield File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt return run_state_machine(ctx, self.callback) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine result = callback.mark_command(ctx.database, mongocryptd_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command res = self.mongocryptd_client[database].command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command return self._command( ^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command return conn.command( ^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command return command( ^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command helpers_shared._check_command_response( File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Comparison disallowed between fields where one is randomly encrypted; field 'title' is randomly encrypted., full error: RawBSONDocument(b"\xae\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00k\x00\x00\x00Comparison disallowed between fields where one is randomly encrypted; field 'title' is randomly encrypted.\x00\x10code\x00\xb6y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31158\x00\x00", codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME)) ERROR: test_lookup_integer_in_charfield (encryption_.test_charfield.TestEncryptedCharField.test_lookup_integer_in_charfield) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors yield File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 466, in encrypt encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt return run_state_machine(ctx, self.callback) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.12/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine result = callback.mark_command(ctx.database, mongocryptd_cmd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 286, in mark_command res = self.mongocryptd_client[database].command( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 930, in command return self._command( ^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 770, in _command return conn.command( ^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 47, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 414, in command return command( ^^^^^^^^ File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command helpers_shared._check_command_response( File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 250, in _check_command_response raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption., full error: RawBSONDocument(b'\xa6\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00c\x00\x00\x00Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption.\x00\x10code\x00#y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31011\x00\x00', codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

Cool, Are lookups needed? We could try to add something. But it won't be so easy given we cannot use aggregate.

Only testing EncryptedIntegerField

Both checks cannot exist in the same class else one may be interrupted by the other and fail as a result. Instead, check the version once and cache the results so subsequent checks can check the cache instead of the connection.

Regression in daa9a8e

We need a test that verifies the fields are: - In the collection - Encrypted Possibly via comparing type to bson

Let's leave `EncryptedRouter` in for now and bikeshed at the end. It seems to me as appropriate as any other helper to include, but I'm open to discussion. Also `KMS_CREDENTIALS` are not in use yet. I will test with Azure and AWS prior to the end of this month.

Regression in 8a05af8

Also, - Use db_table in management command - Move feature check to base class

timgraham · 2025-07-17T01:14:17Z

django_mongodb_backend/models.py

+
+    class Meta:
+        abstract = True
+        required_db_features = {"supports_queryable_encryption"}


I don't think required_db_features is appropriate for EncryptedModel since that will silently cause encrypted models not to be created in user projects.

timgraham · 2025-07-17T01:17:17Z

tests/backend_/test_features.py

+        connection.features.__dict__.pop("supports_queryable_encryption", None)
+
+    def tearDown(self):
+        connection.features.__dict__.pop("supports_queryable_encryption", None)


I mean in tearDown() since each test will (presumably) initialize the attribute.

timgraham · 2025-07-17T01:19:53Z

tests/encryption_/models.py

+    class Meta:
+        db_table = "billing"


See https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/#model-style on positioning of Meta.

timgraham · 2025-07-17T01:21:11Z

tests/encryption_/models.py

+
+class Patient(EncryptedModel):
+    class Meta:
+        db_table = "patient"


If the custom table names are only temporary for debugging or something that's fine, but it's not really appropriate to use an unprefixed table name that could collide with other test apps.

timgraham · 2025-07-17T01:23:08Z

tests/encryption_/models.py

+    class Meta:
+        db_table = "billing"
+
+    cc_type = EncryptedCharField("cc_type", max_length=20, queries=QueryType.equality())


Why do you add an explicit verbose name to every field (e.g. "cc_type")?

timgraham · 2025-07-17T01:54:03Z

docs/source/howto/encryption.rst

+:class:`~pymongo.encryption_options.AutoEncryptionOpts` requires a key vault
+namespace to store encryption keys. The key vault namespace is typically a
+combination of a database and collection name. ``KEY_VAULT_COLLECTION_NAME``
+and ``KEY_VAULT_DATABASE_NAME`` are defined in :mod:`~django_mongodb_backend.encryption`
+and used to create the key vault namespace with can be imported and used as follows.
+
+``KEY_VAULT_NAMESPACE``
+~~~~~~~~~~~~~~~~~~~~~~~
+
+E.g.::
+
+    AutoEncryptionOpts(
+        key_vault_namespace=encryption.KEY_VAULT_NAMESPACE,
+        ...
+    )


I think it would be simpler and more transparent to document an example:

AutoEncryptionOpts( key_vault_namespace="keyvault.__keyvault",

rather than to provide and use a constant.

timgraham · 2025-07-17T02:02:50Z

django_mongodb_backend/encryption.py

+KMS_CREDENTIALS = {
+    "aws": {
+        "key": os.getenv("AWS_KEY_ARN", ""),
+        "region": os.getenv("AWS_KEY_REGION", ""),
+    },
+    "azure": {
+        "keyName": os.getenv("AZURE_KEY_NAME", ""),
+        "keyVaultEndpoint": os.getenv("AZURE_KEY_VAULT_ENDPOINT", ""),
+    },
+    "gcp": {
+        "projectId": os.getenv("GCP_PROJECT_ID", ""),
+        "location": os.getenv("GCP_LOCATION", ""),
+        "keyRing": os.getenv("GCP_KEY_RING", ""),
+        "keyName": os.getenv("GCP_KEY_NAME", ""),
+    },
+    "kmip": {},
+    "local": {},
+}


Is there some documentation we can link to help users know how to configure credentials, providers, etc? It doesn't feel like Django's job to document and maintain this sort of mapping.

I also read:

To enable the driver’s behavior to obtain credentials from the environment, add the appropriate key (“aws”, “gcp”, or “azure”) with an empty map to “kms_providers” in either AutoEncryptionOpts or ClientEncryption options.

so this won't work for that use case (I think).

I'd suggest trying to minimize the amount of "helpers" in this PR. We can always add things later if there are user pain points, but I feel these thing shouldn't be our focus for v1. Really, we should enhance MongoDB/pymongo docs if it's unclear how to construct the providers dictionary. I don't think a solution of "set these environment variables instead" is making things simpler.

timgraham · 2025-07-17T02:06:45Z

django_mongodb_backend/schema.py

+            kms_providers = options._kms_providers
+            codec_options = CodecOptions()
+
+            ce = ClientEncryption(kms_providers, key_vault_namespace, client, codec_options)


There is some example code that uses:

codec_options=client.codec_options

which might be more appropriate (though you wonder why codec_options and is a required argument if options can be retrieved from the also passed client).

timgraham · 2025-07-17T02:07:14Z

django_mongodb_backend/schema.py

+
+            ce = ClientEncryption(kms_providers, key_vault_namespace, client, codec_options)
+
+            # TODO: Validate schema! `create_encrypted_collection` appears to


What would validating it involve?

timgraham · 2025-07-17T02:15:03Z

tests/encryption_/tests.py

+        class Tee(StringIO):
+            """Print the output of management commands to stdout."""
+
+            def write(self, txt):
+                sys.stdout.write(txt)
+                super().write(txt)
+
+        out = Tee()


Tee is for temporary debugging?

INTPYTHON-527 Add Queryable Encryption support

bc52c8e

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

Fix test for unencrypted field not in field map

38fb110

timgraham reviewed Jun 27, 2025

View reviewed changes

Fix test for unencrypted field not in field map

65bd15a

Add comment about suppressing EncryptedCollectionError

e08945b

aclark4life added 2 commits June 27, 2025 18:12

Don't rely on features to fall back to unencrypted

7b34b44

Remove _nodb_cursor and disable version check

8e83ada

aclark4life added 2 commits June 27, 2025 20:15

Don't surpress encrypted error

4da895c

Rename get_encrypted_client -> get_client_encryption

ed54a9b

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

aclark4life added 2 commits June 30, 2025 18:05

Add encryption router

8a7766c

Add "encryption" database to encryption tests

eab2f2e

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

aclark4life added 2 commits July 1, 2025 15:02

Move encrypted_fields_map to schema (1/2)

10a361e

Move encrypted_fields_map to schema (2/x)

01d5485

- Refactor tests

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

aclark4life added 4 commits July 1, 2025 20:14

Refactor helpers

db32487

Restore get_database_version functionality

b2be223

Move encrypted router to tests

27d4b8e

Fix router tests

c4d1c66

Test feature supports_queryable_encryption

2772aff

Update docs

75c3cd1

timgraham reviewed Jul 14, 2025

View reviewed changes

aclark4life added 9 commits July 14, 2025 20:28

Move kms_credentials to settings

a81d2ae

Remove get_auto_encryption_opts

e562718

Remove get_client_encryption

3dca177

Define public helpers API

3432818

Refactor test_auto_encryption_opts

cb7f153

Assert the entire expected output via json.dumps

e0ef5b3

Update docs

ba4a6c8

Rename test methods

1b9a714

Fix doc string

3340ae7

WaVEV reviewed Jul 15, 2025

View reviewed changes

aclark4life and others added 16 commits July 15, 2025 07:56

Rename tests -> test_schema & add charfield tests

c90406b

Add test_integerfield from django

8a1f381

Only testing EncryptedIntegerField

Avoid reentrancy issue checking mongodb version

edb2fa6

Both checks cannot exist in the same class else one may be interrupted by the other and fail as a result. Instead, check the version once and cache the results so subsequent checks can check the cache instead of the connection.

Add encrypted Post model schema to expected schema

0701160

Re-add namespace to schema_map

08f7934

Add a note about copying field tests from Django

2c4d53b

Add query type tests, remove django field tests

9919ce9

Restored documentation navigation

45ea5b5

Regression in daa9a8e

Test negative query type cases

3e468e7

Refactor and start watching for bad schema maps

8869bc1

We need a test that verifies the fields are: - In the collection - Encrypted Possibly via comparing type to bson

Refactor and update helpers

8a05af8

Update docs

3353fd0

Let's leave `EncryptedRouter` in for now and bikeshed at the end. It seems to me as appropriate as any other helper to include, but I'm open to discussion. Also `KMS_CREDENTIALS` are not in use yet. I will test with Azure and AWS prior to the end of this month.

Add billing model fields & tee command output

948d21c

Fix router

aae8df9

Regression in 8a05af8

Add a fixme to test router for kms_provider

9c7c82f

Use custom db table for tests

cec0289

Also, - Use db_table in management command - Move feature check to base class

timgraham reviewed Jul 17, 2025

View reviewed changes

		return name, path, args, kwargs


		class EncryptedCharField(EncryptedFieldMixin, models.CharField):


		ce = ClientEncryption(kms_providers, key_vault_namespace, client, codec_options)

		# TODO: Validate schema! `create_encrypted_collection` appears to

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025

Uh oh!

timgraham commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025

Uh oh!

timgraham commented Jun 27, 2025

Uh oh!

aclark4life commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclark4life Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

aclark4life commented Jun 27, 2025 •

edited

Loading

timgraham commented Jun 27, 2025 •

edited

Loading

aclark4life commented Jun 28, 2025 •

edited

Loading

aclark4life commented Jul 2, 2025 •

edited

Loading

aclark4life Jul 15, 2025 •

edited

Loading