Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GafferPop should 'squash' GetElements Operations #3277

Closed
p29876 opened this issue Sep 3, 2024 · 0 comments · Fixed by #3278
Closed

GafferPop should 'squash' GetElements Operations #3277

p29876 opened this issue Sep 3, 2024 · 0 comments · Fixed by #3278
Assignees
@p29876
Labels
bug Confirmed or suspected bug tinkerpop Specific to/touches the tinkerpop module
Milestone
2.3.1

Comments

@p29876
Copy link
Member

p29876 commented Sep 3, 2024

At the moment if you run a query like g.V('1').outE(), for every edge on the vertex '1' GafferPop will run a GetElements operation with only one seed. This can be very problematic for queries that touch a lot of data as each operation has overhead and gets logged so could end up with thousands of operations being called at once.

To fix this we might need to implement custom vertex/edge traversal strategies so that all the seeds are acquired first then added to one big operation rather than lots of smaller ones.
@p29876 p29876 added bug Confirmed or suspected bug tinkerpop Specific to/touches the tinkerpop module labels Sep 3, 2024
@p29876 p29876 self-assigned this Sep 3, 2024
p29876 added a commit that referenced this issue Sep 3, 2024
@p29876
Gh-3277 Add custom GafferPop Vertex Step
d168238 
Add a fold step before each vertex step to collect all vertex ids
so the input to the new VertexStep is a list rather than single ids.
This means we can do a single Gaffer query for all ids.

Note: input seeds are essentially deduped so expected output may change
@github-actions github-actions bot mentioned this issue Sep 3, 2024
Merged
p29876 added a commit that referenced this issue Sep 5, 2024
@p29876
Merge branch 'develop' into gh-3277-gafferpop-custom-vertex-step
df1245a
p29876 added a commit that referenced this issue Sep 5, 2024
@p29876
Merge branch 'develop' into gh-3277-gafferpop-custom-vertex-step
b7f0932
@p29876 p29876 added this to the 2.3.1 milestone Sep 5, 2024
wb36499 added a commit that referenced this issue Sep 12, 2024
@wb36499
Merge branch 'develop' into gh-3277-gafferpop-custom-vertex-step
1689eab
wb36499 added a commit that referenced this issue Sep 12, 2024
@p29876 @wb36499
Gh-3277: Add custom GafferPop Vertex Step (#3278)
58ad080 
* Gh-3277 Add custom GafferPop Vertex Step

Add a fold step before each vertex step to collect all vertex ids
so the input to the new VertexStep is a list rather than single ids.
This means we can do a single Gaffer query for all ids.

Note: input seeds are essentially deduped so expected output may change

* Add ListVertexStep

change tab width

* add unit tests

* smells

* code cov

* code cov

* code cov

* iterable

* rename vertex step

* try again

---------

Co-authored-by: wb36499 <166839644+wb36499@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@p29876 p29876

Labels
bug Confirmed or suspected bug tinkerpop Specific to/touches the tinkerpop module
Projects
None yet
Milestone
2.3.1
Development

Successfully merging a pull request may close this issue.

Gh-3277: Add custom GafferPop Vertex Step gchq/Gaffer
1 participant
@p29876