docs: Max duration #1379
Merged
docs: Max duration #1379
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
Caution Review failedThe pull request is closed. WalkthroughThe pull request introduces several updates to the documentation and configuration files related to the Changes
Possibly related PRs
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (11)
packages/cli-v3/templates/examples/simple.mjs.template (1)
5-5: Approved: Good addition ofmaxDurationpropertyThe addition of the
maxDurationproperty is a good improvement to the task configuration. It sets a clear limit for the task execution time, which can help prevent long-running tasks from consuming excessive resources.Some suggestions for further improvement:
- Consider adding a brief comment explaining what happens when the max duration is reached (e.g., task termination, error thrown).
- If applicable, consider making the duration configurable, perhaps through an environment variable, to allow for easier adjustments in different environments.
Would you like me to provide an example of how to implement these suggestions?
packages/cli-v3/templates/examples/schedule.ts.template (1)
7-7: LGTM! Consider enhancing the comment for clarity.The addition of the
maxDurationproperty is a good practice to limit the execution time of the scheduled task. The 5-minute duration provides ample time for the current task logic and allows for future expansion.Consider updating the comment to provide more context:
- maxDuration: 300, // 5 minutes + maxDuration: 300, // 5 minutes maximum execution timeThis change clarifies that the duration refers to the maximum execution time of the task.
references/v3-catalog/trigger.config.ts (1)
20-21: Clarify the intention behind commenting outmaxDurationThank you for adding the explanatory comment with a link to the documentation. This improves the code's maintainability and helps developers understand the purpose of the
maxDurationsetting.However, there seems to be a discrepancy between the comment and the implementation:
- The comment suggests setting the
maxDurationto 300s for all tasks.- The actual
maxDurationproperty is commented out, which means it's not active in the configuration.Could you clarify the intention here? If the goal is to set a 300s max duration for all tasks, we should uncomment the
maxDurationproperty. If not, we might want to adjust the comment to reflect the current state.Consider one of these options:
- If you want to apply the 300s max duration:
// Set the maxDuration to 300s for all tasks. See https://trigger.dev/docs/runs/max-duration maxDuration: 300,
- If you want to keep it commented out but document the option:
// Optionally set the maxDuration for all tasks. Uncomment and adjust as needed. // See https://trigger.dev/docs/runs/max-duration // maxDuration: 300,docs/runs/max-duration.mdx (2)
1-12: Minor grammatical correction needed in the introduction.The introduction provides a clear overview of the
maxDurationfeature. However, there's a small grammatical issue that should be addressed:-By default tasks can execute indefinitely, which can be great! But you also might want to set a `maxDuration` to prevent a task from running too long. +By default, tasks can execute indefinitely, which can be great! But you also might want to set a `maxDuration` to prevent a task from running too long.Adding a comma after "By default" improves readability and adheres to proper punctuation rules.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~6-~6: Did you mean: “By default,”?
Context: ...ximum duration for a task to run." --- By default tasks can execute indefinitely, which c...(BY_DEFAULT_COMMA)
43-55: Minor punctuation improvement suggested.This section effectively explains how to configure a default
maxDurationin the config file. The code example is clear and relevant. However, there's a small punctuation improvement that can be made:-You can set a default `maxDuration` for all tasks in your [config file](/config/config-file#default-machine). This will apply to all tasks unless you override it on a specific task or run. +You can set a default `maxDuration` for all tasks in your [config file](/config/config-file#default-machine). This will apply to all tasks, unless you override it on a specific task or run.Adding a comma before "unless" improves readability and adheres to proper punctuation rules for complex sentences.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~45-~45: Possible missing comma found.
Context: ...efault-machine). This will apply to all tasks unless you override it on a specific ta...(AI_HYDRA_LEO_MISSING_COMMA)
docs/tasks/overview.mdx (2)
124-139: LGTM! Clear explanation of themaxDurationoption.The new section effectively introduces the
maxDurationoption, providing a clear explanation of its purpose and a concise code example. The link to the detailed guide is helpful for users seeking more information.Consider adding a brief note about the default behavior when
maxDurationis not specified, to provide a complete picture of this option's functionality.🧰 Tools
🪛 LanguageTool
[uncategorized] ~125-~125: Did you mean: “By default,”?
Context: ... }, }); ``` ###maxDurationoption By default tasks can execute indefinitely, which c...(BY_DEFAULT_COMMA)
126-126: Minor grammatical improvement suggested.Consider revising the first sentence of the
maxDurationsection for improved clarity:- By default tasks can execute indefinitely, which can be great! + By default, tasks can execute indefinitely, which can be great!This small change adds a comma after "By default," improving the sentence structure.
docs/config/config-file.mdx (3)
155-168: Consider adding a brief explanation ofmaxDuration.The new "Max duration" section is well-structured and provides a clear example of how to set the
maxDurationin the configuration. However, it would be helpful to briefly explain whatmaxDurationrepresents (e.g., the maximum time a task is allowed to run before it's terminated) to provide more context for users.
Line range hint
470-481: Consider adding a brief description of audioWaveform.The section provides clear instructions on how to use the audioWaveform build extension. To improve user understanding, consider adding a brief description of what audioWaveform is and its primary use cases. This context will help users decide whether they need this extension in their project.
Line range hint
622-644: Consider adding a brief explanation of aptGet.The section provides clear instructions on how to use the aptGet build extension, including how to specify package versions. To improve user understanding, consider adding a brief explanation of what aptGet is and why users might need it in their Trigger.dev projects. This context will help users better understand the purpose and benefits of this extension.
docs/triggering.mdx (1)
745-748: NewmaxDurationoption addedThe addition of the
maxDurationoption is a valuable enhancement to the task configuration capabilities. This new section provides users with a quick reference and directs them to more detailed documentation.To improve readability and address the style suggestion, consider rewording the second sentence:
### `maxDuration` - View our [maxDuration doc](/runs/max-duration) for more information. + For detailed information, please refer to our [maxDuration documentation](/runs/max-duration).This change maintains clarity while avoiding repetitive sentence beginnings.
🧰 Tools
🪛 LanguageTool
[style] ~746-~746: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...r more information. ###maxDurationView our [maxDuration doc](/runs/max-duratio...(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
docs/runs/max-duration-error.pngis excluded by!**/*.png
📒 Files selected for processing (12)
- docs/config/config-file.mdx (1 hunks)
- docs/mint.json (9 hunks)
- docs/runs/max-duration.mdx (1 hunks)
- docs/tasks/overview.mdx (6 hunks)
- docs/triggering.mdx (1 hunks)
- packages/cli-v3/templates/examples/schedule.mjs.template (1 hunks)
- packages/cli-v3/templates/examples/schedule.ts.template (1 hunks)
- packages/cli-v3/templates/examples/simple.mjs.template (1 hunks)
- packages/cli-v3/templates/examples/simple.ts.template (1 hunks)
- packages/cli-v3/templates/trigger.config.mjs.template (1 hunks)
- packages/cli-v3/templates/trigger.config.ts.template (1 hunks)
- references/v3-catalog/trigger.config.ts (1 hunks)
✅ Files skipped from review due to trivial changes (3)
- docs/mint.json
- packages/cli-v3/templates/trigger.config.mjs.template
- packages/cli-v3/templates/trigger.config.ts.template
🧰 Additional context used
🪛 LanguageTool
docs/runs/max-duration.mdx
[uncategorized] ~6-~6: Did you mean: “By default,”?
Context: ...ximum duration for a task to run." --- By default tasks can execute indefinitely, which c...(BY_DEFAULT_COMMA)
[uncategorized] ~45-~45: Possible missing comma found.
Context: ...efault-machine). This will apply to all tasks unless you override it on a specific ta...(AI_HYDRA_LEO_MISSING_COMMA)
docs/tasks/overview.mdx
[uncategorized] ~125-~125: Did you mean: “By default,”?
Context: ... }, }); ``` ###maxDurationoption By default tasks can execute indefinitely, which c...(BY_DEFAULT_COMMA)
docs/triggering.mdx
[style] ~746-~746: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...r more information. ###maxDurationView our [maxDuration doc](/runs/max-duratio...(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🔇 Additional comments (12)
packages/cli-v3/templates/examples/simple.mjs.template (1)
Line range hint
1-14: Overall: Approved - Positive impact on task configurationThis change successfully implements the PR objective of documenting the maximum duration for the task. The addition of the
maxDurationproperty improves the task configuration by setting a clear execution time limit, which is a good practice for managing resources and preventing indefinitely running tasks.The change is minimal and focused, not affecting the core logic of the task, which reduces the risk of introducing bugs. It also serves as a good example for other tasks that might benefit from similar timeout configurations.
packages/cli-v3/templates/examples/simple.ts.template (1)
5-5: Verify SDK support formaxDurationand update documentation if necessary.The addition of the
maxDurationproperty is a good practice for preventing long-running tasks. However, please ensure the following:
- Verify that the
@trigger.dev/sdk/v3supports this property. If it's a new feature, make sure you're using a compatible version of the SDK.- Consider updating the documentation to reflect this new property and its usage in task definitions.
- Evaluate if other tasks in the project would benefit from having a
maxDurationset, and apply it consistently if appropriate.To verify the SDK support and find potential documentation updates, run the following script:
packages/cli-v3/templates/examples/schedule.mjs.template (1)
7-7: Approve the addition ofmaxDurationwith suggestions for improvementThe addition of the
maxDurationproperty is a good safeguard against long-running tasks. However, I have a few suggestions to enhance this change:
Consider adding a comment explaining what happens when the max duration is reached. This will help developers understand the implications of this limit.
The PR title "docs: Max duration" doesn't fully reflect this functional change. Consider updating it to something like "feat: Add max duration to scheduled tasks".
It's important to verify if any existing tasks might be affected by this 5-minute limit. If there are tasks that legitimately need more time, you might need to adjust this value or make it configurable.
Here's a suggested improvement to the comment:
- maxDuration: 300, // 5 minutes + maxDuration: 300, // 5 minutes - Task will be terminated if it exceeds this durationTo check for potentially affected tasks, you can run:
This script searches for scheduled tasks with wait times exceeding 5 minutes or potential infinite loops, which might be affected by the new limit.
✅ Verification successful
Verified that no existing scheduled tasks are affected by the new
maxDurationlimitNo existing scheduled tasks were found that might exceed the 5-minute limit. Therefore, the addition of
maxDurationcan be safely integrated without impacting current tasks.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Search for existing scheduled tasks that might be affected by the new maxDuration limit # Look for tasks with wait times or loops that might exceed 5 minutes rg --type javascript -e "schedules.task" -A 20 | rg -e "wait.for.*minutes.*[6-9]|minutes.*[1-9][0-9]" -e "while.*true" -e "for.*;" -A 10Length of output: 171
Script:
#!/bin/bash # Search for existing scheduled tasks that might be affected by the new maxDuration limit # Look for tasks with wait times or loops that might exceed 5 minutes rg --type js -e "schedules.task" -A 20 | rg -e "wait.for.*minutes.*[6-9]|minutes.*[1-9][0-9]" -e "while.*true" -e "for.*;" -A 10Length of output: 128
Script:
#!/bin/bash # Search for existing scheduled tasks that might be affected by the new maxDuration limit # Look for tasks with wait times or loops that might exceed 5 minutes rg -g "*.mjs" -e "schedules.task" -A 20 | rg -e "wait\.for.*minutes.*[6-9]" -e "minutes.*\d{2,}" -e "while\s*\(.*\)\s*\{" -e "for\s*\(.*\)\s*\{" -A 10Length of output: 150
docs/runs/max-duration.mdx (5)
13-41: Excellent explanation of themaxDurationfeature.This section provides a comprehensive explanation of how the
maxDurationfeature works. The content is well-structured and includes:
- A clear description of how
maxDurationis measured.- A relevant code example demonstrating CPU time inspection.
- A visual representation of the error message.
- Important information about minimum and maximum duration limits.
The explanation is thorough and will help users understand and implement the feature effectively.
57-87: Clear explanation of task-specificmaxDurationconfiguration.This section effectively covers how to configure
maxDurationfor specific tasks. It includes:
- A clear explanation of setting
maxDurationfor a task.- An example demonstrating how to set a specific duration.
- An example showing how to disable
maxDurationfor a task.- Clarification on how task-specific settings override the default configuration.
The content is well-structured and provides users with the necessary information to configure
maxDurationat the task level.
89-118: Comprehensive guide on run-specificmaxDurationconfiguration.This section effectively explains how to configure
maxDurationfor specific runs. It includes:
- A clear explanation of setting
maxDurationwhen triggering a task.- An example demonstrating how to set a specific duration for a run.
- An example showing how to disable
maxDurationfor a specific run.The content provides users with a clear understanding of how to manage
maxDurationat the run level, complementing the previous sections on global and task-level configurations.
120-134: Clear explanation of accessingmaxDurationin run context.This section effectively explains how to access the
maxDurationvalue within the run context. The code example provided is concise and clearly demonstrates how to log themaxDurationvalue.This information is valuable for developers who need to access or use the
maxDurationvalue programmatically within their tasks.
1-138: Overall, excellent documentation for themaxDurationfeature.This documentation provides a comprehensive and well-structured guide to the
maxDurationfeature. It covers all aspects of the feature, including:
- How it works
- Configuration at different levels (global, task-specific, and run-specific)
- Accessing
maxDurationin the run context- Interaction with lifecycle functions
The explanations are clear, and the code examples are relevant and helpful. The minor grammatical and punctuation improvements suggested will further enhance the readability of the document.
Great job on creating this thorough and user-friendly documentation! The suggested expansions, particularly for the section on lifecycle functions, will make this an even more valuable resource for developers using the Trigger framework.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~6-~6: Did you mean: “By default,”?
Context: ...ximum duration for a task to run." --- By default tasks can execute indefinitely, which c...(BY_DEFAULT_COMMA)
[uncategorized] ~45-~45: Possible missing comma found.
Context: ...efault-machine). This will apply to all tasks unless you override it on a specific ta...(AI_HYDRA_LEO_MISSING_COMMA)
docs/tasks/overview.mdx (1)
174-174: Formatting improvements in<Info>tags.The removal of hyphens at the beginning of lines within
<Info>tags improves consistency and readability throughout the document. These changes maintain the original content while enhancing the overall presentation.Also applies to: 192-192, 199-200, 232-232, 263-263, 294-294, 302-302
docs/config/config-file.mdx (3)
Line range hint
483-507: Well-structured and informative puppeteer section.The puppeteer build extension section is comprehensive and well-organized. It includes important elements such as:
- A warning about web scraping.
- Clear instructions for adding the extension to the configuration.
- Guidance on setting up the necessary environment variable.
- A link to a detailed example for further reference.
This structure provides users with all the necessary information to get started with puppeteer in their Trigger.dev projects.
Line range hint
509-542: Comprehensive and well-structured ffmpeg section.The ffmpeg build extension section is thorough and well-organized. It includes:
- Clear instructions for adding the extension to the configuration.
- Information on how to specify a particular version of ffmpeg.
- Explanation of automatically added environment variables.
- A link to a detailed example for further reference.
This structure provides users with all the necessary information to effectively use ffmpeg in their Trigger.dev projects.
Line range hint
1-644: Excellent additions to the configuration documentation.The new sections on max duration and build extensions (audioWaveform, puppeteer, ffmpeg, and aptGet) are valuable additions to the
trigger.config.tsdocumentation. They are well-integrated into the existing document structure and provide clear, comprehensive information for users. The overall flow and readability of the document have been maintained, making it easier for users to understand and implement these new features in their Trigger.dev projects.
…d out maxDuration in the config file
ericallam
force-pushed
the
docs/max-duration
branch
from
6018524 to
553c58e
Compare
samejr
added a commit
that referenced
this pull request
Oct 8, 2024
commit 886429b Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <8297864+D-K-P@users.noreply.github.com> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <james@trigger.dev> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <eric@trigger.dev> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <eallam@icloud.com> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <eallam@icloud.com> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <eric@trigger.dev> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <james@trigger.dev> commit 665ccf8 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <matt@mattaitken.com> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint
samejr
added a commit
that referenced
this pull request
Oct 8, 2024
commit 886429b Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <8297864+D-K-P@users.noreply.github.com> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <james@trigger.dev> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <eric@trigger.dev> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <eallam@icloud.com> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <eallam@icloud.com> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <eric@trigger.dev> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <james@trigger.dev> commit 665ccf8 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <matt@mattaitken.com> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint
samejr
added a commit
that referenced
this pull request
Oct 10, 2024
* Checkbox component can have its label styles * Improved the dialog footer * Handle sending feedback to Slack using Plain * WIP making the modal conditional * Moved the Plain form action into the select plan file * removed comment * Show a confirmation diaglog if you’re downgrading from Pro to Hobby * Downgrading to Hobby works * Use redirectWithErrorMessage instead of throw error * The cancel form now submits the data correctly * Modals don’t trigger when you upgrade * Copy improvements * Added a tooltip to explain the link to the pricing page * Squashed commit of the following: commit 886429b Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <8297864+D-K-P@users.noreply.github.com> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <james@trigger.dev> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <eric@trigger.dev> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <eallam@icloud.com> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <eallam@icloud.com> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <eric@trigger.dev> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <james@trigger.dev> commit 665ccf8 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <matt@mattaitken.com> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint * Squashed commit of the following: commit 886429b Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:36:46 2024 +0100 Removed emails, @trigger.dev/database and @trigger.dev/otlp-importer from changesets config commit f65157a Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:31:27 2024 +0100 Lockfile with run-engine removed commit 3d67bb8 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:24:31 2024 +0100 Removed run-engine from the webapp package.json/tsconfig commit d30e971 Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 18:06:04 2024 +0100 Dockerfile fix because the database package has been moved commit f2babbf Author: Matt Aitken <matt@mattaitken.com> Date: Tue Oct 8 09:41:22 2024 -0700 Internal packages (testcontainers, redis-worker and zod-worker) (#1392) * Some notes on the new run engine * lockfile with setup for the run engine * Documenting where TaskRun is currently mutated, to try figure out the shape of the new system * Added notes about how triggering currently works * Details about when triggering happens * Lots of notes about waitpoints * Started scaffolding the RunEngine * Sketch of Prisma waitpoint schema while it’s fresh in my mind * Got Prisma working with testcontainers * Use beforeEach/afterEach * Simple Prisma and Redis test * Return Redis options instead of a client * Simplified things * A very simple FIFO pull-based queue to check the tests working properly * Use vitest extend * Separate redis, postgres and combined tests for faster testing * Some fixes and test improvements * Pass a logger into the queue * A queue processor that processes items from the given queue as fast as it can * Test for retrying an item that wasn’t processed * First draft of waitpoints in the Prisma schema * Remove the custom logger from the test * Added a completedAt to Waitpoint * Notes on the flow for an execution starting * Added redlock, moved some files around * Starting point for the TaskRunExecutionSnapshot table * Added relationships to TaskRunExecutionSnapshot * Change some tsconfig * Moved some things around * Added some packages * WIP on the RunQueue * Fix for some imports * Key producer with some tests * Removed the nv type from the keys… it’s not useful to do global queries * Passing unit tests for all the public key producer functions * Some basic tests passing for the RunQueue * Simple enqueue test working * Enqueue and dequeue for dev is working * Don’t log everything during the tests * Enqueuing/dequeuing from the shared queue is working * Tests for getting a shared queue * The key producer sharedQueue can now be named, to allow multiple separate queues * The key producer uses the name of the queue as the input * Extra info in the Prisma schema * Dequeuing a message gets the payload and sets the task concurrency all in one Lua script * Adding more keys so we can read the concurrency from the queue * Setting the concurrency with dequeue and enquque is working * Improved the tests and fixed some bugs * Acking is resetting the concurrencies * Check the key has been removed after acking * Nacking is working * Changed the package to CommonJS + Node10 so it works with Redlock * Moved the database, otel and emails packages to be in internal-packages * Moved some Prisma code to the database package * Started using the RunEngine for triggering * Progress on run engine triggering, first waitpoint code * Create a delay waitpoint * Moved ZodWorker to an internal package so it can be used in the run engine as well as the webapp * Web app now uses the zod worker package * Added parseNaturalLanguageDuration to core/apps * internal-packages/zod-worker in the lockfile * Pass in the master queue, remove old rebalance workers code * Add masterQueue to TaskRun * Fixed the tests * Moved waitpoint code into the run engine, also the zod worker * Completing waitpoints * An experiment to create a new test container with environment * More changes to triggering * Started testing triggering * Test for a run getting triggered and being enqueued * Removed dequeueMessageInEnv * Update dev queue tests to use the shared queue function * Schema changes for TaskRunExecutionSnapshot * First execution snapshot when the run is created. Dequeue run function added to the engine * Separate internal package for testcontainers so they can be used elsewhere * Remove the simple queue and testcontainers from the run-engine. They’re going to be separate * Fix for the wrong path to the Prisma schem,a * Added the testcontainers package to the run-engine * redis-worker package, just a copy of the simple queue for now * The queue now uses Lua to enqueue dequeue * The queue now has a catalog and an invisible period after dequeuing * Added a visibility timeout and acking, with tests * Added more Redis connection logging, deleted todos * Visibility timeouts are now defined on the catalog and can be overridden when enqueuing * Dequeue multiple items at once * Test for dequeuing multiple items * Export some types to be used elsewhere * Partial refactor of the processor * First stab at a worker with concurrency and NodeWorkers * Don’t have a default visibility timeout in the queue * Worker setup and processing items in a simple test * Process jobs in parallel with retrying * Get the attempt when dequeuing * Workers do exponential backoff * Moved todos * DLQ functionality * DLQ tests * Same cluster for all keys in the same queue * Added DLQ tests * Whitespace * Redis pubsub to redrive from the worker * Fixed database paths * Fix for path to zod-worker * Fixes for typecheck errors, mostly with TS versions and module resolution * Redlock required a patch * Moved the new DB migrations to the new database package folder * Remove the run-engine package * Remove the RunEngine prisma schema changes * Delete triggerTaskV2 * Remove zodworker test script (no tests) * Update test-containers readme * Generate the client first * Use a specific version of the prisma package * Generate the prisma client before running the unit tests commit fc60947 Author: Dan <8297864+D-K-P@users.noreply.github.com> Date: Tue Oct 8 14:36:03 2024 +0100 Supabase database webhook example upgrade (#1386) * Added overview for guides and examples section and split them all out * New supabase guide wip * Updated images and improved docs * Trimmed the supabase prereqs * Supabase guide wip * more updates * Replaced old database webhook guide * Created one intro page and removed snippets * Updated guide sidebar titles * Code updates * More improvements * Updates and added images * Compressed image * Updated guides descriptions and edge function basic * Removed bold * Updated redirects * Fixed broken links * Updated intro commit 07f82ea Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 13:28:54 2024 +0100 Release 3.0.11 commit 13ebfcc Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue Oct 8 13:24:38 2024 +0100 chore: Update version for release (#1381) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 2a04d17 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:24:23 2024 +0100 Simplify showLogs expression commit 002ae4b Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:29 2024 +0100 Fix dotenv overrides for dev runs (#1388) * override dashboard dev env vars with local .env * add changeset * add simple task for testing env vars commit 047cb00 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Tue Oct 8 09:22:05 2024 +0100 Disable schedules for deleted orgs on next tick (#1383) * disable schedules for deleted orgs * add debug logs commit 2c014f7 Author: James Ritchie <james@trigger.dev> Date: Sun Oct 6 13:02:00 2024 -0700 Override log retention (#1385) * set full log retention as admin * If run.logsDeletedAt is set, don’t bother getting the trace commit a69e04f Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 14:18:58 2024 +0100 Include push output in logs for self-hosted deploys (#1382) * include push output in logs * changeset commit c5488df Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Sat Oct 5 13:12:47 2024 +0100 Fix CLI downgrade check (#1380) * fix downgrade detection * remove unused semver package from webapp * add changeset commit 1caec27 Author: Eric Allam <eric@trigger.dev> Date: Fri Oct 4 15:33:35 2024 -0700 docs: Max duration (#1379) * maxDuration docs * Update the init command to set the maxDuration and include a commented out maxDuration in the config file commit e14c954 Author: Eric Allam <eallam@icloud.com> Date: Fri Oct 4 15:02:05 2024 -0700 Release 3.0.10 commit 8e61f5d Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri Oct 4 14:59:07 2024 -0700 chore: Update version for release (#1378) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> commit 08db565 Author: Eric Allam <eallam@icloud.com> Date: Thu Oct 3 12:38:25 2024 -0700 improve the timed out description commit 6d08842 Author: Eric Allam <eric@trigger.dev> Date: Thu Oct 3 12:43:26 2024 -0700 feat: Add maxDuration to tasks (#1377) * WIP * Get max duration working on deployed runs * Actually set the timed out runs to status = TIMED_OUT * The client status for TIMED_OUT is now MAX_DURATION_EXCEEDED * New TimedOutIcon * Added new timedout icon * Add ability to opt-out of maxDuration with timeout.None * MAX_DURATION_EXCEEDED -> TIMED_OUT * changeset * Improved styling for the status tooltip content --------- Co-authored-by: James Ritchie <james@trigger.dev> commit 665ccf8 Author: nicktrn <55853254+nicktrn@users.noreply.github.com> Date: Thu Oct 3 12:33:18 2024 +0100 Update github actions and self-hosting docs commit 1ff7b86 Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 18:26:36 2024 -0700 Add max queue depth limits (#1376) * Add runs to an env queue, as well as the actual queue * Add queue size limit guard on triggering tasks commit c531a9d Author: Eric Allam <eric@trigger.dev> Date: Wed Oct 2 15:30:39 2024 -0700 fix: cleanup ttl expire run graphile jobs (#1373) * fix: remove ttl expire run graphile jobs when a run is started or completed * Update expireEnqueuedRun.server.ts commit 0bf500f Author: Matt Aitken <matt@mattaitken.com> Date: Wed Oct 2 15:30:16 2024 -0700 Prioritize finishing waited runs (#1375) * If a tree node is missing, estimate the size as zero * Task to test prioritizing finishing existing runs after triggerAndWaits * When requeuing a run with a checkpoint, put it in the queue with the parent run time so it’s correctly prioritized * The same change but if there’s no checkpoint * Revert "Squashed commit of the following:" This reverts commit b837b5a. * Removed console logs * cleaned up conditionals * Unlock free plan state * Fixed subscribe button if you’re already github verified * Simplified the downgrade reasons logic * made periodEnd required --------- Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>
3 tasks
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes
New Features
maxDurationconfiguration option for tasks, allowing users to set a maximum execution time (default 300 seconds).audioWaveform,puppeteer,ffmpeg, andaptGet.Documentation
maxDuration, including comprehensive guides and examples for implementation.Style