fix: transaction ...: in use: for query #2278

Flo4604 · 2024-10-08T13:44:53Z

What does this PR do?

This fixes a transaction error that sometimes occurs when re-validating the AuditlogBucket.
In this case, we were querying the database with the transaction in the background, and at the same time, the transaction was used to insert into the auditLogTarget or auditLog tables.

This would lead to a transaction ...: in use: for query query and a internal server error code.

Fixes #2197

Type of change

Bug fix (non-breaking change which fixes an issue)
Chore (refactoring code, technical debt, workflow improvements)
Enhancement (small improvements)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How should this be tested?

This can be tested by running the API locally and then:

Get the following script: https://gist.github.com/Flo4604/ab153a85fd86ee09adbe185976e5e837
Run it without the changes
Wait a minute and a few seconds and rerun it still with the old version; this can be done once a minute as this is when the cache re-validation happens; this re-produces it quite reliably at least on my machine.
Add the changes to your code
Re-do 2 + 3; don't see the same error again.

Checklist

Required

Appreciated

If a UI change was made: Added a screen recording or screenshots to this PR
Updated the Unkey Docs if changes were necessary

Summary by CodeRabbit

Chores
- Improved the audit log querying method to enhance performance and prevent potential race conditions.
- Enhanced the efficiency of audit log insertions by batching operations.
- Minor formatting adjustments for better readability.

- Fixes unkeyed#2197 - The error is caused by the cache revalidation happening in the background, so we have a racecondition with the other queries we are running in the insertGenericAuditLogs function

changeset-bot · 2024-10-08T13:44:56Z

⚠️ No Changeset found

Latest commit: 2539db5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

vercel · 2024-10-08T13:45:00Z

@Flo4604 is attempting to deploy a commit to the Unkey Team on Vercel.

A member of the Team first needs to authorize it.

coderabbitai · 2024-10-08T13:45:01Z

📝 Walkthrough

Walkthrough

The changes in this pull request focus on the insertGenericAuditLogs function within the apps/api/src/pkg/audit.ts file. The primary modification involves switching the database instance used for querying the audit log bucket from a transaction-based instance to a read-only instance. This adjustment ensures that read operations do not interfere with ongoing transactions. Additionally, the function now accumulates insertions into separate arrays for batch processing, enhancing efficiency while maintaining the overall structure and logic of the function.

Changes

File	Change Summary
apps/api/src/pkg/audit.ts	Modified `insertGenericAuditLogs` to use `db.readonly` for querying audit logs; optimized insertions by batching them.

Assessment against linked issues

Objective	Addressed	Explanation
Resolve transaction in use error when creating role (#2197)	✅

Possibly related PRs

refactor: query audit logs from planetscale #2181: The changes in this PR involve querying audit logs, which is closely related to the modifications made to the insertGenericAuditLogs function in the main PR, as both deal with the handling and retrieval of audit logs.
perf: add database indices #2192: This PR adds indices to the audit_log and audit_log_target tables, which could enhance the performance of queries related to audit logs, aligning with the focus on audit log management in the main PR.

Suggested reviewers

mcstepp

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

vercel · 2024-10-08T13:45:01Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
dashboard	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 11, 2024 1:31pm
engineering	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 11, 2024 1:31pm
play	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 11, 2024 1:31pm
workflows	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 11, 2024 1:31pm
www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 11, 2024 1:31pm

github-actions · 2024-10-08T13:45:06Z

Thank you for following the naming conventions for pull request titles! 🙏

coderabbitai

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (3)

apps/api/src/pkg/audit.ts (3)

57-63: Improved code readability

The addition of empty lines enhances the code's readability without altering its functionality. This change aligns with good coding practices and makes the code easier to understand and maintain.

102-102: Consistent formatting improvement

The addition of this empty line is consistent with the previous formatting changes. It effectively separates the audit log insertion from the audit log target insertion, further enhancing the code's readability.

Line range hint 1-115: Summary of changes and suggestion for further optimization

The changes in this file effectively address the reported issue of transaction conflicts during the re-validation of AuditlogBucket. The main functional change, using db.readonly for querying the audit log bucket, prevents interference with ongoing transactions. The additional formatting improvements enhance code readability.

While these changes are excellent, there might be an opportunity for further optimization:

Consider implementing a more robust error handling mechanism for the cache retrieval and database operations. This could include retrying failed operations or implementing a circuit breaker pattern to prevent cascading failures.

The caching logic could potentially be optimized by implementing a write-through or write-behind caching strategy, which could reduce the number of database queries in high-traffic scenarios.

Would you like me to provide more detailed suggestions on implementing these optimizations?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between edbeb2d and 518f77a.

📒 Files selected for processing (1)

apps/api/src/pkg/audit.ts (2 hunks)

🧰 Additional context used

🔇 Additional comments (1)

apps/api/src/pkg/audit.ts (1)

49-53: Excellent change to prevent transaction conflicts

The switch from tx ?? db.primary to db.readonly for querying the audit log bucket is a great solution. This change addresses the reported issue of transaction conflicts during role creation and other operations. By using a read-only connection, we ensure that the query doesn't interfere with ongoing transactions, thus preventing the "Transaction in use" error.

chronark · 2024-10-08T13:57:56Z

well now that I see this, using the transaction is obviously a bad idea lol

also bulk insert auditLogs and auditLogTargets

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between a61f1ad and e4ae521.

📒 Files selected for processing (1)

apps/api/src/pkg/audit.ts (5 hunks)

🧰 Additional context used

🔇 Additional comments (3)

apps/api/src/pkg/audit.ts (3)

44-46: Good Practice: Implementing Batch Inserts for Efficiency

Introducing auditLogsInserts and auditLogTargetInserts arrays to accumulate audit log entries for batch insertion enhances performance by reducing the number of database operations. This change is efficient and aligns with best practices for handling bulk data inserts.

52-54: Appropriate Use of Read-Only Database Instance

Using db.readonly outside of the transaction to fetch the audit log bucket is a sensible approach to avoid race conditions, as noted in the comments. This ensures that read operations do not interfere with transaction scopes, which can help maintain data integrity.

124-126: Verify Intentional Fallback to Non-Transactional Inserts

The batch insert operations use (tx ?? db.primary), which means they will proceed without a transaction if tx is undefined. Verify that this behavior is intentional. If atomicity is required across these insert operations, it would be advisable to ensure that a transaction is always used.

If necessary, you might want to enforce the use of a transaction or handle the absence of a transaction explicitly.

apps/api/src/pkg/audit.ts

chronark · 2024-10-08T15:55:20Z

we're working this out with flo in discord btw

oss-gg · 2024-10-11T13:34:57Z

Awarding Flo4604: 300 points 🕹️ Well done! Check out your new contribution on oss.gg/Flo4604

* fix: transaction ...: in use: for query - Fixes unkeyed#2197 - The error is caused by the cache revalidation happening in the background, so we have a racecondition with the other queries we are running in the insertGenericAuditLogs function * docs: explain for future self * fix: another racecondition also bulk insert auditLogs and auditLogTargets * [autofix.ci] apply automated fixes --------- Co-authored-by: chronark <dev@chronark.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

* fix:breadcrum-cache-issue * chore:rename-revalidateMytag * refactor: move-revalidateTag * refactor: extract tags into reusable utility * feat: add template * Update 3_follow_the_unkey_x_account.md (#2323) * fix: capture uniqueness error and return it correctly (#2337) * docs: getVerifications instead of get (#2329) * Update 3_follow_the_unkey_x_account.md (#2324) Followed unkey on Twitter Co-authored-by: Andreas Thomas <dev@chronark.com> * fix: transaction ...: in use: for query (#2278) * fix: transaction ...: in use: for query - Fixes #2197 - The error is caused by the cache revalidation happening in the background, so we have a racecondition with the other queries we are running in the insertGenericAuditLogs function * docs: explain for future self * fix: another racecondition also bulk insert auditLogs and auditLogTargets * [autofix.ci] apply automated fixes --------- Co-authored-by: chronark <dev@chronark.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: handle undefined ratelimits * Update 3_follow_the_unkey_x_account.md (#2338) * fix: handle case when blogs array is empty * [oss.gg] side quest : added another framework; hono on cloudflare workers. (#2345) * fix: weird spacing in changelog by removing the extra gap (#2340) * fix: weird spacing in changelog * feat: formatted * oss.gg: create a template for ratelimiting a oak server in deno with unkey (#2308) Co-authored-by: Andreas Thomas <dev@chronark.com> * follow unkey on X #2252 (#2315) Co-authored-by: Your Name <you@example.com> * Follow the Unkey X account: @unkeydev Complete! (#2332) * Update 3_follow_the_unkey_x_account.md * Update 3_follow_the_unkey_x_account.md --------- Co-authored-by: Andreas Thomas <dev@chronark.com> * fix(www): analytics bento code snippet is not readable (#2311) - add hover effect to analytics bento removing gradient background - make copy code button sticky to top-right corner - impl independent vertical scrolls for language switcher and code editor * Update 7_create_a_template.md * Update 7_create_a_template.md * chore(deps-dev): bump @content-collections/core from 0.6.2 to 0.7.2 (#2255) Bumps @content-collections/core from 0.6.2 to 0.7.2. --- updated-dependencies: - dependency-name: "@content-collections/core" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: order audit logs by time, show latest on top (#2295) * feat: add template * feat: Unkey FastAPI boilerplate (#2307) * feat: add template * feat: follow unkey on X (#2357) * Update 6_record_onboarding.md (#2301) * fix: reset input fields in API delete dialog after reopening (#2316) * fix: reset input fields in API delete dialog after reopening * refactor: remove unnecessary useEffect and utilize existing onOpenChange function for dialog * refactor: removed unused imports * Update 3_follow_the_unkey_x_account.md (#2364) Added name to the followed list. Co-authored-by: Andreas Thomas <dev@chronark.com> * ci: add label * feat: add hover to input fields * Added gaps and width for md (#2371) * docs: update overview.mdx (#2384) avaliable -> available * oss.gg side quest 3_follow_the_unkey_x_account.md (#2399) Followed the unkey x account * feat: Following unkey acc on twitter #2407 (#2408) * Feat: Followed Unkey on X * Feat: Followed Unkey on Twitter --------- Co-authored-by: Andreas Thomas <dev@chronark.com> * resolve merge conflicts * resolve merge conflicts --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: chronark <dev@chronark.com> Co-authored-by: Emily Marie Ahtúnan <86572370+Emmarie-Ahtunan@users.noreply.github.com> Co-authored-by: Harsh Shrikant Bhat <90265455+harshsbhat@users.noreply.github.com> Co-authored-by: Aritra Sadhukhan <60315087+aritradevelops@users.noreply.github.com> Co-authored-by: Flo <53355483+Flo4604@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Devang Rakholiya <116096508+Devang0907@users.noreply.github.com> Co-authored-by: Prabin <42871240+prabincankod@users.noreply.github.com> Co-authored-by: ZiaCodes <72739794+Khaan25@users.noreply.github.com> Co-authored-by: Devansh Baghel <77718741+Devansh-Baghel@users.noreply.github.com> Co-authored-by: Shreenarayan jaiswal <127131074+shreenarayan123@users.noreply.github.com> Co-authored-by: Your Name <you@example.com> Co-authored-by: Chirag Arora <chirag8023@gmail.com> Co-authored-by: Nazar Poshtarenko <32395926+unrenamed@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Saketh Pavan Goti <44170829+Sakethpavan@users.noreply.github.com> Co-authored-by: Teddy ASSIH <110066080+Ionfinisher@users.noreply.github.com> Co-authored-by: Vardhaman Bhandari <97441447+Vardhaman619@users.noreply.github.com> Co-authored-by: Atharva_404 <72994819+Atharva-3000@users.noreply.github.com> Co-authored-by: Rishi <rishipatel0826@gmail.com> Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com> Co-authored-by: Ayan Mondal <ayanmondal1805@gmail.com> Co-authored-by: Kunal Arya <105645118+kunalarya873@users.noreply.github.com>

fix: transaction ...: in use: for query

13371ce

- Fixes unkeyed#2197 - The error is caused by the cache revalidation happening in the background, so we have a racecondition with the other queries we are running in the insertGenericAuditLogs function

Flo4604 requested review from perkinsjr, chronark, mcstepp and MichaelUnkey as code owners October 8, 2024 13:44

github-actions bot added 🕹️ 300 points Bug Something isn't working hacktoberfest 🕹️ oss.gg labels Oct 8, 2024

Merge branch 'main' into fix/v1_permissions_createRole_transaction_error

518f77a

vercel bot deployed to Preview – engineering October 8, 2024 13:51 View deployment

coderabbitai bot reviewed Oct 8, 2024

View reviewed changes

chronark added 2 commits October 8, 2024 16:00

docs: explain for future self

479043a

Merge branch 'main' into fix/v1_permissions_createRole_transaction_error

a61f1ad

chronark approved these changes Oct 8, 2024

View reviewed changes

chronark self-assigned this Oct 8, 2024

vercel bot deployed to Preview – engineering October 8, 2024 14:03 View deployment

fix: another racecondition

e4ae521

also bulk insert auditLogs and auditLogTargets

vercel bot deployed to Preview – engineering October 8, 2024 15:34 View deployment

coderabbitai bot reviewed Oct 8, 2024

View reviewed changes

apps/api/src/pkg/audit.ts Show resolved Hide resolved

Merge branch 'main' into fix/v1_permissions_createRole_transaction_error

2beb1e6

vercel bot deployed to Preview – engineering October 8, 2024 16:38 View deployment

Merge branch 'main' into fix/v1_permissions_createRole_transaction_error

cb025c9

vercel bot deployed to Preview – engineering October 9, 2024 09:10 View deployment

chronark approved these changes Oct 10, 2024

View reviewed changes

chronark enabled auto-merge October 10, 2024 10:01

[autofix.ci] apply automated fixes

41ca6b9

vercel bot deployed to Preview – www October 10, 2024 10:08 View deployment

vercel bot deployed to Preview – play October 10, 2024 10:10 View deployment

vercel bot deployed to Preview – dashboard October 10, 2024 10:11 View deployment

vercel bot deployed to Preview – workflows October 10, 2024 10:11 View deployment

vercel bot deployed to Preview – engineering October 10, 2024 10:12 View deployment

Merge branch 'main' into fix/v1_permissions_createRole_transaction_error

2539db5

vercel bot deployed to Preview – engineering October 10, 2024 11:09 View deployment

chronark approved these changes Oct 11, 2024

View reviewed changes

vercel bot deployed to Preview – www October 11, 2024 13:29 View deployment

vercel bot deployed to Preview – play October 11, 2024 13:30 View deployment

vercel bot deployed to Preview – workflows October 11, 2024 13:30 View deployment

vercel bot deployed to Preview – dashboard October 11, 2024 13:31 View deployment

chronark added this pull request to the merge queue Oct 11, 2024

Merged via the queue into unkeyed:main with commit 534908e Oct 11, 2024
24 checks passed

Flo4604 deleted the fix/v1_permissions_createRole_transaction_error branch October 11, 2024 18:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: transaction ...: in use: for query #2278

fix: transaction ...: in use: for query #2278

Flo4604 commented Oct 8, 2024 •

edited by coderabbitai bot

Loading

changeset-bot bot commented Oct 8, 2024 •

edited

Loading

vercel bot commented Oct 8, 2024

coderabbitai bot commented Oct 8, 2024 •

edited

Loading

Walkthrough

Changes

Assessment against linked issues

Possibly related PRs

Suggested reviewers

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

vercel bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading

coderabbitai bot left a comment

chronark commented Oct 8, 2024

coderabbitai bot left a comment

chronark commented Oct 8, 2024

oss-gg bot commented Oct 11, 2024

fix: transaction ...: in use: for query #2278

fix: transaction ...: in use: for query #2278

Conversation

Flo4604 commented Oct 8, 2024 • edited by coderabbitai bot Loading

What does this PR do?

Type of change

How should this be tested?

Checklist

Required

Appreciated

Summary by CodeRabbit

changeset-bot bot commented Oct 8, 2024 • edited Loading

⚠️ No Changeset found

vercel bot commented Oct 8, 2024

coderabbitai bot commented Oct 8, 2024 • edited Loading

Walkthrough

Changes

Assessment against linked issues

Possibly related PRs

Suggested reviewers

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

vercel bot commented Oct 8, 2024 • edited Loading

github-actions bot commented Oct 8, 2024 • edited Loading

coderabbitai bot left a comment

Choose a reason for hiding this comment

chronark commented Oct 8, 2024

coderabbitai bot left a comment

Choose a reason for hiding this comment

chronark commented Oct 8, 2024

oss-gg bot commented Oct 11, 2024

Flo4604 commented Oct 8, 2024 •

edited by coderabbitai bot

Loading

changeset-bot bot commented Oct 8, 2024 •

edited

Loading

coderabbitai bot commented Oct 8, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

vercel bot commented Oct 8, 2024 •

edited

Loading

github-actions bot commented Oct 8, 2024 •

edited

Loading