Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(query): Make query serialization thread-safe #198

Merged
merged 1 commit into from
Oct 9, 2024

Conversation

snigdhas
Copy link
Member

@snigdhas snigdhas commented Oct 9, 2024

Serializing multiple queries in multiple threads is not thread safe. The PRINTER global is reused by the Query serializer across these requests and maintains a local Translator in the translator classvar.

This would be fine if the translator were never mutated. However, the visit function temporarily mutates the translator using entity_aliases().

If multiple queries are being serialized concurrently, there are bad inverleavings of this code that result in invalid queries.

This can be reproduced with this diff and some long running queries issued in a batch (example):

diff --git a/snuba_sdk/query.py b/snuba_sdk/query.py
index 35d349a..eea1835 100644
--- a/snuba_sdk/query.py
+++ b/snuba_sdk/query.py
@@ -175,7 +175,9 @@ class Query(BaseQuery):
     def serialize(self) -> str:
         self.validate()
         optimized = self._optimize()
+        print(f"query visiting #{id(self)=}", flush=True)
         return PRINTER.visit(optimized)
+        print(f"query done visiting #{id(self)=}", flush=True)
 
     def print(self) -> str:
         self.validate()
diff --git a/snuba_sdk/query_visitors.py b/snuba_sdk/query_visitors.py
index 3bd8a26..e5f9922 100644
--- a/snuba_sdk/query_visitors.py
+++ b/snuba_sdk/query_visitors.py
@@ -138,7 +138,7 @@ class Printer(QueryVisitor[str]):
 
     def visit(self, query: main.Query) -> str:
         if isinstance(query.match, Join):
-            with entity_aliases(self.translator):
+            with entity_aliases(self.translator, id(query)):
                 return super().visit(query)
 
         return super().visit(query)
diff --git a/snuba_sdk/visitors.py b/snuba_sdk/visitors.py
index c787731..0035e54 100644
--- a/snuba_sdk/visitors.py
+++ b/snuba_sdk/visitors.py
@@ -329,10 +329,12 @@ class Translation(ExpressionVisitor[str]):
 
 
 @contextmanager
-def entity_aliases(translator: Translation) -> Generator[None, None, None]:
+def entity_aliases(translator: Translation, id) -> Generator[None, None, None]:
+    print(f"translator visiting #{id=}", flush=True)
     translator.use_entity_aliases = True
     yield
     translator.use_entity_aliases = False
+    print(f"done visiting #{id=}", flush=True)
 
 
 class ExpressionFinder(ExpressionVisitor[Set[Expression]]):

When we're lucky, the output shows that the translator's use_entity_aliases field is not mutated by multiple threads. When we're unlucky, it is.

Good

query visiting #id(self)=6155267360
translator visiting #id=6155267360
done visiting #id=6155267360
query visiting #id(self)=6155270864
translator visiting #id=6155270864
done visiting #id=6155270864

Bad

query visiting #id(self)=6155296096
translator visiting #id=6155296096
query visiting #id(self)=6155292592
translator visiting #id=6155292592
done visiting #id=6155296096
done visiting #id=6155292592

@snigdhas snigdhas marked this pull request as ready for review October 9, 2024 07:10
@snigdhas snigdhas requested a review from a team as a code owner October 9, 2024 07:10
@snigdhas snigdhas requested a review from a team October 9, 2024 07:10
Copy link
Member

@armenzg armenzg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@snigdhas snigdhas merged commit fadfb50 into main Oct 9, 2024
11 checks passed
@snigdhas snigdhas deleted the snigdha/thread-safe branch October 9, 2024 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants