Serialized values should include key and collection

[strawgate]

In order to support collection and key enumeration we need to be able to "undo" the hashing/sanitization that some stores do which means we probably need to include the unsanitized collection and key in the document itself.

We don't need this info for get and put because we can just use the values for collection and key that we were already given as arguments

[claude]

Claude finished @strawgate's task —— View job

---

Investigation Complete ✓

I've completed my investigation of issue #203 regarding storing unsanitized collection and key values in serialized documents to support enumeration.

---

Recommendation

Add key and collection fields to the serialized document structure in stores that perform sanitization/hashing. This will enable proper enumeration support when the original keys/collections differ from their sanitized/hashed storage identifiers.

The implementation should:

1. Update SerializationAdapter.dump_dict() to accept optional key and collection parameters
2. Modify _put_managed_entry() methods to pass these values to the adapter
3. Update _get_collection_keys() to return the original (unsanitized) key values from documents
4. Ensure backward compatibility with existing documents that lack these fields

Findings

Current Architecture

Serialization Flow:

• Values are wrapped in ManagedEntry objects containing value, created_at, and expires_at
• Stores use SerializationAdapter to convert ManagedEntry to store-specific formats
• Adapters support different date formats (isoformat/datetime) and value formats (string/dict)

Current Document Structure (Elasticsearch example):

{
  "value": {"flattened": {"name": "Alice", "age": 30}},
  "created_at": "2025-11-02T16:30:00.000000+00:00",
  "expires_at": "2025-11-02T16:40:00.000000+00:00"
}

Stores That Perform Sanitization:

1. ElasticsearchStore - Sanitizes both index names (collections) and document IDs (keys)

   • Collection: sanitize_string() with max 240 chars, lowercase+numbers+_-.
   • Key: sanitize_string() with max 256 chars, alphanumeric only

2. MongoDBStore - Sanitizes collection names

   • Collection: sanitize_string() with max 200 chars, alphanumeric+_

3. KeyringStore - Sanitizes both collections and keys

   • Uses sanitize_string() with hashing for both

4. WindowsRegistryStore - Sanitizes both collections and keys

   • Uses sanitize_string() with character restrictions

The Problem

Current Enumeration Limitation:
The _get_collection_keys() method in Elasticsearch (lines 425-453 in elasticsearch/store.py) queries for keys but retrieves them from a "key" field in the document. However, this field doesn't currently exist in the stored documents. The current implementation relies on Elasticsearch's built-in _id field, which contains the sanitized key.

Looking at the code at line 432-442:

result: ObjectApiResponse[Any] = await self._client.options(ignore_status=404).search(
    index=self._sanitize_index_name(collection=collection),
    fields=[{"key": None}],  # Requests "key" field
    body={
        "query": {
            "term": {
                "collection": collection,  # Queries for "collection" field that also doesn't exist
            },
        },
    },

The query attempts to:

1. Filter by a "collection" field that doesn't exist in documents
2. Retrieve a "key" field that doesn't exist in documents

This means:

• For get/put operations: No issue - the caller provides the original key/collection
• For enumeration: Cannot recover original keys/collections from sanitized storage identifiers

Why This Matters

When sanitization changes a key/collection name (e.g., "my-special-key!" becomes "my_special_key-a1b2c3d4"), enumeration would return the sanitized version instead of the original, breaking the contract.

Detailed Action Plan

Phase 1: Update Core Serialization Infrastructure

1. Modify SerializationAdapter base class (key-value-shared/src/key_value/shared/utils/serialization.py)

   • Update dump_dict() signature to accept optional key: str | None = None, collection: str | None = None
   • Include these fields in the output dictionary when provided
   • Update load_dict() to handle these fields (but don't require them for backward compatibility)

2. Update store-specific adapters to handle key/collection fields:

   • ElasticsearchSerializationAdapter.prepare_dump() - Add key and collection to top level
   • MongoDBSerializationAdapter.prepare_dump() - Add key and collection to top level
   • Other adapters as needed

Phase 2: Update Store Implementations

3. Modify _put_managed_entry() in stores that sanitize:

   • ElasticsearchStore (line 320-343)
   • MongoDBStore
   • KeyringStore
   • WindowsRegistryStore

   Change from:

   document: dict[str, Any] = self._adapter.dump_dict(entry=managed_entry)

   To:

   document: dict[str, Any] = self._adapter.dump_dict(
       entry=managed_entry,
       key=key,
       collection=collection
   )

4. Update _put_managed_entries() for bulk operations in the same stores

Phase 3: Update Enumeration Methods

5. Modify _get_collection_keys() implementations:

   • Update queries to use the new key field instead of document ID
   • Add fallback logic for documents without the key field (backward compatibility)
   • For Elasticsearch: Update the search query at line 430-442 to retrieve the key field

6. Update _get_collection_names() if needed:

   • Similar changes to use the collection field from documents

Phase 4: Update Schema Mappings

7. Update Elasticsearch index mappings (line 49-77 in elasticsearch/store.py):

   DEFAULT_MAPPING = {
       "properties": {
           "created_at": {"type": "date"},
           "expires_at": {"type": "date"},
           "collection": {"type": "keyword"},  # ADD THIS
           "key": {"type": "keyword"},          # ADD THIS
           "value": {
               "properties": {
                   "string": {"type": "object", "enabled": False},
                   "flattened": {"type": "flattened"},
               },
           },
       },
   }

Phase 5: Testing & Migration

8. Add tests for enumeration with sanitized keys:

   • Test that keys() returns original keys, not sanitized versions
   • Test that collections() returns original collection names
   • Test backward compatibility with documents lacking key/collection fields

9. Update snapshot tests in test_elasticsearch.py and test_mongodb.py:

   • Update expected document structure to include key and collection fields
   • Lines 170-186 in test_elasticsearch.py would expect:

     {
         "key": "test_key",
         "collection": "test",
         "value": {"flattened": {"name": "Alice", "age": 30}},
         "created_at": IsStr(min_length=20, max_length=40),
     }

10. Run codegen: make codegen to generate sync versions

11. Documentation updates:

    • Update migration guide to note the schema change
    • Document that old documents will still work (backward compatible)

Related Issues and Pull Requests

Repository	Issue/PR	Relevance
strawgate/py-key-value	#171 - Adding key enumeration to the protocol	Discusses adding key enumeration to the core protocol. This issue (#203) is a prerequisite for proper enumeration support in stores that sanitize keys.
strawgate/py-key-value	#165 - Review ManagedEntry	Recent review and refactor of ManagedEntry structure. The serialization adapter pattern introduced here is the foundation for adding key/collection fields.
strawgate/py-key-value	#180 - Add tests for raw document format	Adds snapshot tests for document formats across stores. These tests will need updating to reflect the new key/collection fields.
strawgate/py-key-value	#174 - Refactor ManagedEntry serialization with adapter pattern	Introduced the SerializationAdapter pattern that makes this enhancement possible without breaking existing stores.

Related Files

File	Relevance	Key Lines
key-value-shared/src/key_value/shared/utils/serialization.py	Core serialization adapter that needs to be extended to support key/collection fields	114-132
key-value-shared/src/key_value/shared/utils/managed_entry.py	Defines ManagedEntry dataclass that wraps values	14-64
key-value-shared/src/key_value/shared/utils/sanitize.py	Contains sanitization logic that creates the mismatch between original and stored keys	64-136
key-value-aio/src/key_value/aio/stores/elasticsearch/store.py	Elasticsearch store implementation that sanitizes both collections and keys	232-246, 320-343, 425-453
key-value-aio/src/key_value/aio/stores/mongodb/store.py	MongoDB store implementation that sanitizes collection names	190-199
key-value-aio/src/key_value/aio/protocols/key_value.py	Defines enumeration protocols (AsyncEnumerateKeysProtocol, AsyncEnumerateCollectionsProtocol)	138-159
key-value-aio/tests/stores/elasticsearch/test_elasticsearch.py	Contains snapshot tests that validate document structure	160-186

---

Problems Encountered During Investigation

None - all investigation tools and repository resources were accessible and provided comprehensive information about the codebase architecture, serialization patterns, and store implementations.

---

[strawgate]

@claude let's make a pull request for this, I'm starting to think our managed entries need to include a version number as well

[claude]

Claude finished @strawgate's task —— View job • claude/issue-203-20251102-1638 • Create PR ➔

---

Implementation Complete ✓

I've successfully implemented the changes to store unsanitized key and collection values in serialized documents, plus added a version field to managed entries.

Changes Made

Core Infrastructure:

• Updated SerializationAdapter.dump_dict() to accept optional key, collection, and version parameters
• Updated SerializationAdapter.dump_json() to pass through these parameters
• All serialized documents now include a version field (default: 1)

Store Updates:

• Elasticsearch: Updated mapping to include key, collection, and version fields; added backward compatibility in _get_collection_keys()
• MongoDB: Updated to pass key/collection to serializer
• Keyring, WindowsRegistry, Memcached: Updated stores that sanitize keys/collections
• DynamoDB, Redis, RocksDB, Simple, Valkey, Vault, Disk: Updated for consistency

Backward Compatibility:

• Elasticsearch's _get_collection_keys() now falls back to document ID for old documents without the key field
• Existing documents will continue to work without modification

Task Summary

• Gather context and review relevant code
• Update SerializationAdapter to support key, collection, and version fields
• Update all 12 store implementations
• Run codegen to generate sync versions
• Run lint (passed)
• Commit and push changes
• Provide PR creation link

Create Pull Request

Note: Type check shows 8 pre-existing errors in the Redis store (sync version) that are unrelated to these changes.

---

| Branch: claude/issue-203-20251102-1638