Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support DocumentDB by replacing certain MongoDB operations #4886

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open

Support DocumentDB by replacing certain MongoDB operations #4886

wants to merge 28 commits into from

Conversation

kwx4957
Copy link
Contributor

@kwx4957 kwx4957 commented Sep 15, 2024 •
edited
Loading

What is changed

ChaosCenter

  • Add pre-commit check for aws documentDB

Auth Server

  • Refactor CreatePaginationStage and add int return values
  • $facet to $group and $project
    • Calculate total and retrieve data by creating an array using $push with $$ROOT, then slice the data with $slice
  • The result struct changed no longer returns an array

GraphQL Server

  • DB schema changed from array to int
  • Export and add common function for CreatePaginationStage
  • $lookup multiple join seperate each stage $lookup and $project etc

5 Types

  • $facet to $count
  • $facet to $group
    • Use $switch to count data based on specific cases in groupbyTotal
  • $facet to $group and $project
    • Calculate total and retrieve data by creating an array using $push with $$ROOT, then slice the data with $slice
  • $facet to $group, $project, and $unwind
  • $bucket to $switch
    • Calculate avg_resiliency_score

Test Code

  • chaos_experiment_run
    • Changed data type from array to integer

AWS documentDB e2e test result

I ran the e2e test on aws documnetdb instead of mongodb, and got the following results

image

The Google Drive link below is videos of the e2e test.
https://drive.google.com/drive/folders/1_Rs8TPVlOg7KKDCRqzlgtePDO9Kp4I0O?usp=drive_link

Run simple Experiment

i ran simple chaosexperiment using k6. this is the result.
image

Related proposal: #4812
Related issue: #4459

kwx4957 added 23 commits September 8, 2024 21:03
@kwx4957
Refactor using default page values
1bf1fdf 
Add return int values for paging

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
6f6141f 
$facet to $group and $project
Change the result type. return type has changed

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Refactor extract common pagination feature
f1ea9a8 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
93d6a6c 
Replace $facet to $group and $project
Change the result type. return type has changed
Extract common pagination feature

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
342d0ad 
Replace $facet to $count
Change the result type. return type has changed

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
4bdaad1 
Replace $facet to $group and $project
Change the result type. return type has changed
Extract common pagination feature

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
05bb505 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
82e28c7 
Replace $facet to $group and $project
Change the result type. return type has changed
Extract common pagination feature

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix change reposne struect type
890f9de 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
b91f7d5 
Replace $facet to $group and $project
Change the result type. return type has changed
Extract common pagination feature

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
858f03c 
Replace $facet to $group and $project

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix return type String to int in groupByResScoreStage
289b6bc 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix ListExperimentRun test code expect array to int
ee27c4a 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
15df6b2 
Replace $facet to $group and $project

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat replace mongodb operation $facet for support aws documentDB
73607c9 
Replace $facet to $group, $project and $unwind

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix null pointer error
d23954a 
Add a condition to check for null pointer error if the res value is no present

Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Merge remote-tracking branch 'upstream/master' into feat/documentdb
4bd1652
@kwx4957
Feat separte pipeline in $lookup
731e22b 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat separte pipeline to match and addField in $lookup
42894de 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat separte pipeline to match and project in $lookup
7639c87 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat separte pipeline to project in $lookup
19d9dfd 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Feat separte pipeline to addField in $lookup
d232159 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Add pre-commit check for aws documentDB
526cbf0 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@namkyu1999
Copy link
Member

Can you check a backend-checks?

@kwx4957
Fix goimports
dc74d1b 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Copy link
Contributor Author

kwx4957 commented Sep 16, 2024

yes, i will check

@namkyu1999 namkyu1999 assigned amityt and namkyu1999 and unassigned amityt and namkyu1999 Sep 16, 2024
@kwx4957
Add pre-commit check $lookup multiple join
2c0848c 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Copy link
Contributor Author

kwx4957 commented Sep 17, 2024

i find a error when get ListExperimentRun, i will fix it

@kwx4957
Fix missing execution_data field added
9cb3bc2 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix missing revision.execution_manifest field added
839c3a3 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@kwx4957
Fix filtering condition for AWS DocumentDB compatibility
1169a34 
Signed-off-by: DongYoung Kim <kwx4957@gmail.com>
@namkyu1999 namkyu1999 added the need-approvers-review Reminder label to the codeowners/maintainers for stale PRs that are more that 2 weeks old label Nov 15, 2024
@Saranya-jena
Copy link
Contributor

Hi @kwx4957 can you perform a benchmarking test to check how these changes have affected the API latency? You can use a mongo compass to run the mongo pipelines and check the latencie==y.

@Saranya-jena
Copy link
Contributor

Otherwise the changes look good to me! 🚀

@kwx4957
Copy link
Contributor Author

kwx4957 commented Nov 23, 2024

@Saranya-jena Thank you for informing me about the tool I wasn't aware of. yes I will run a benchmarking test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SarthakJain26 SarthakJain26 Awaiting requested review from SarthakJain26

@Saranya-jena Saranya-jena Awaiting requested review from Saranya-jena

@amityt amityt Awaiting requested review from amityt

@Jonsy13 Jonsy13 Awaiting requested review from Jonsy13

@namkyu1999 namkyu1999 Awaiting requested review from namkyu1999

@neelanjan00 neelanjan00 Awaiting requested review from neelanjan00

Assignees

@kwx4957 kwx4957

Labels
mentorship-kr need-approvers-review Reminder label to the codeowners/maintainers for stale PRs that are more that 2 weeks old
Projects
Open Source Contribution Academy 2024
Status: In Review
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kwx4957 @namkyu1999 @Saranya-jena @amityt