Skip to content

Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix #1698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 12, 2025

Conversation

matt-aitken
Copy link
Member

@matt-aitken matt-aitken commented Feb 12, 2025

This fixes two problems

  1. It was possible to create TaskRunAttempt when the run was final. This was a rare edge case where a run container became unresponsive for a long period and then tried to create an attempt. It was creating an attempt when the run was already finished.
  2. When resuming it wasn't filtering for finished attempts, it was just using the first one. Now it gets the most recent finished one.

Summary by CodeRabbit

  • Bug Fixes

    • Prevented new task attempts from being initiated on already completed tasks, resulting in clearer error feedback.
    • Refined the resumption process to accurately select and validate eligible task attempts and dependencies.
  • Chores

    • Added enhanced logging for batch operations to improve monitoring and diagnostics.

Copy link

changeset-bot bot commented Feb 12, 2025

⚠️ No Changeset found

Latest commit: 3b2b2be

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

coderabbitai bot commented Feb 12, 2025

Walkthrough

This PR introduces modifications to two main services and a logging update. In the CreateTaskRunAttemptService, a new check prevents further attempts when a task run is already finished, and additional tracing information is captured. In the ResumeAttemptService, the selection of task run attempt attributes and batch dependencies retrieval has been refined, with enhanced error logging. Additionally, a new log statement is added to a batch task function to record task completion details.

Changes

File(s) Change Summary
apps/webapp/.../createTaskRunAttempt.server.ts Added an import for FINAL_RUN_STATUSES and a check in the call method to throw a ServiceValidationError if the task run status is final. Also, updated the tracing span to include taskRunStatus.
apps/webapp/.../resumeAttempt.server.ts Updated latestAttemptSelect to include the status field. Modified the query from include to select for dependencies and batch dependencies, with enhanced filtering and error logging for batch attempts.
references/.../trigger/example.ts Added a new logging statement after retrieving batch task results to log the completion of the batch task.

Possibly related PRs

Suggested reviewers

  • ericallam

Poem

I'm a hopping rabbit with a code-lit gleam,
Guarding task runs like a vigilant dream.
Finished attempts face a strict "no-go" sign,
And logs sparkle with every refined line.
With keen eyes and swift hops, our code's prime—
Celebrating changes in a rhythmic rhyme!
🐇✨

Warning

There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

apps/webapp/app/v3/services/resumeAttempt.server.ts

Oops! Something went wrong! :(

ESLint: 8.45.0

ESLint couldn't find the config "custom" to extend from. Please check that the name of the config is correct.

The config "custom" was referenced from the config file in "/.eslintrc.js".

If you still have problems, please stop by https://eslint.org/chat/help to chat with the team.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between eacc3b1 and 3b2b2be.

📒 Files selected for processing (1)
  • apps/webapp/app/v3/services/resumeAttempt.server.ts (4 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/webapp/app/v3/services/resumeAttempt.server.ts
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: units / 🧪 Unit Tests
  • GitHub Check: Analyze (javascript-typescript)

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
references/hello-world/src/trigger/example.ts (1)

83-83: LGTM! Consider enhancing the log message with additional context.

The added log statement improves visibility into batch task completion.

Consider including the batch task ID and count in the log message for better traceability:

-    logger.info("Batch task complete", { results });
+    logger.info("Batch task complete", { id: "batch", count: payload.count, results });
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between bd0cc54 and eacc3b1.

📒 Files selected for processing (3)
  • apps/webapp/app/v3/services/createTaskRunAttempt.server.ts (2 hunks)
  • apps/webapp/app/v3/services/resumeAttempt.server.ts (4 hunks)
  • references/hello-world/src/trigger/example.ts (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: typecheck / typecheck
  • GitHub Check: units / 🧪 Unit Tests
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (4)
apps/webapp/app/v3/services/resumeAttempt.server.ts (2)

29-31: LGTM! Status field addition improves attempt tracking.

Including the status field in latestAttemptSelect enables better filtering and validation of attempts.


135-154: Excellent enhancement to batch dependency handling!

The new implementation correctly:

  1. Filters for attempts in final states
  2. Sorts by attempt number to get the most recent
  3. Adds comprehensive error logging when not all batch items have attempts
apps/webapp/app/v3/services/createTaskRunAttempt.server.ts (2)

95-95: LGTM! Enhanced tracing with task run status.

Adding taskRunStatus to span attributes improves observability.


101-104: Excellent addition of final state validation!

This check directly addresses the PR objective by preventing TaskRunAttempt creation for finished runs.

@matt-aitken matt-aitken merged commit d214aa9 into main Feb 12, 2025
11 checks passed
@matt-aitken matt-aitken deleted the fix-stuck-batches-no-attempts branch February 12, 2025 14:02
samejr pushed a commit that referenced this pull request Feb 13, 2025
…continue fix (#1698)

* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)

* Don’t create an attempt if the run is already in a final status

* Don’t get all the columns for the query. Improved the logging.

* Added a log to the batch example

* Filter out the undefined values
samejr added a commit that referenced this pull request Feb 13, 2025
* Uses image cards for the frameworks

* Removes old snippets

* New AI agents side menu section

* WIP adding new ai agent pages

* Better overview page

* More copy added to the agent example pages

* Copy improvements

* Removes “Creating a project” page and side menu section

* Fixes broken links

* Updates to the latest Mintlify version, fixes issues, changes theme

* Adds descriptions to the main dropdown menu items

* Reformatted Introduction docs ‘landing page’

* Retry heartbeat timeouts by putting back in the queue (#1689)

* If there’s a heartbeat error and no attempts we put it back in the queue to try again

* When nacking, return whether it was put back in the queue or not

* Try and nack, if it fails then fail the run

* Consolidated switch statement

* Fail executing/retrying runs

* OOM retrying on larger machines (#1691)

* OOM retrying on larger machines

* Create forty-windows-shop.md

* Update forty-windows-shop.md

* Only retry again if the machine is different from the original

* Kubernetes OOMs appear as non-zero sigkills, adding support for treating these as OOMs

* Complete the original attempt span if retrying due to an OOM

* Revert "Complete the original attempt span if retrying due to an OOM"

This reverts commit 5f652c6.

* chore: Update version for release (#1666)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.3.14

* Set machine when triggering docs

* Batch queue runs that are waiting for deploy (#1693)

* Detect ffmpeg OOM errors, added manual OutOfMemoryError (#1694)

* Detect ffmpeg OOM errors, added manual OutOfMemoryError

* Create eighty-spies-knock.md

* Improved the machines docs, including the new OutOfMemoryError

* chore: Update version for release (#1695)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Release 3.3.15

* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created (#1696)

* Create new partitioned TaskEvent table, and switch to it gradually as new runs are created

* Add env var for partition window in seconds

* Make startCreatedAt required in task event store

* Don't create an attempt if the run is final, batchTriggerAndWait bad continue fix (#1698)

* WIP fix for ResumeAttemptService selecting the wrong attempt (which has no error or output)

* Don’t create an attempt if the run is already in a final status

* Don’t get all the columns for the query. Improved the logging.

* Added a log to the batch example

* Filter out the undefined values

* Fix missing logs on child runs by using the root task run createdAt if it exists (#1697)

* Provider changes to support image cache (#1700)

* add env var for additional pull secrets

* make static images configurable

* optional image prefixes

* optional labels with sample rates

* add missing core paths

* remove excessive logs

* Fix run container exits after OOM retries (#1701)

* remove unused imports

* tell run to exit before force requeue

* handle exit for case where we already retried after oom

* improve retry span and add machine props

* don't try to exit run in dev

* Upgrade local dev to use electric beta.15 (#1699)

* Text fixes

* Removed pnpm files

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant