Updating Apify components #19528
Updating Apify components #19528
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
WalkthroughThis PR extends the Apify integration with a new Key-Value Store record retrieval action, a task runner with webhook-based polling and rerun support, API method additions and signature changes in the Apify client, refined logic in the actor runner, and corresponding version bumps across multiple action modules and the package. Changes
Sequence DiagramsequenceDiagram
actor User
participant Client
participant Apify as Apify API
participant Webhook as Webhook Handler
participant Storage as Context Storage
User->>Client: Trigger run-task with waitForFinish=true
Client->>Apify: startTask(taskId, input, params)
Apify-->>Client: runId, status=RUNNING
Client->>Storage: Store runId, createdAt, webhookUrl
Client->>Apify: createWebhook(taskRunFinished events)
Apify-->>Client: webhookId
Client->>Storage: Store webhookId
Client->>Client: Suspend (await webhook or 1-day timeout)
Note over Webhook,Apify: Task completes on Apify side
Apify->>Webhook: POST task completion event
Webhook->>Client: Resume run with webhook event
Client->>Apify: getRun(runId)
Apify-->>Client: Run status (succeeded/failed)
alt Terminal Status
Client->>Apify: deleteWebhook(webhookId)
Client-->>User: Return final status/result
else Polling Window Active
Client->>Storage: Schedule polling rerun (30s interval)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
📜 Review details
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (11)
components/apify/actions/get-dataset-items/get-dataset-items.mjs(1 hunks)components/apify/actions/get-kvs-record/get-kvs-record.mjs(1 hunks)components/apify/actions/run-actor/run-actor.mjs(5 hunks)components/apify/actions/run-task-synchronously/run-task-synchronously.mjs(1 hunks)components/apify/actions/run-task/run-task.mjs(1 hunks)components/apify/actions/scrape-single-url/scrape-single-url.mjs(1 hunks)components/apify/actions/set-key-value-store-record/set-key-value-store-record.mjs(3 hunks)components/apify/apify.app.mjs(6 hunks)components/apify/package.json(1 hunks)components/apify/sources/new-finished-actor-run-instant/new-finished-actor-run-instant.mjs(1 hunks)components/apify/sources/new-finished-task-run-instant/new-finished-task-run-instant.mjs(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-09-12T07:58:39.628Z
Learnt from: matyascimbulka
Repo: PipedreamHQ/pipedream PR: 18308
File: components/apify/actions/run-task-synchronously/run-task-synchronously.mjs:95-0
Timestamp: 2025-09-12T07:58:39.628Z
Learning: The Apify ActorRun object always contains the defaultDatasetId property according to the official documentation, so conditional checks for its existence are not needed when calling listDatasetItems.
Applied to files:
components/apify/actions/run-task-synchronously/run-task-synchronously.mjscomponents/apify/actions/get-dataset-items/get-dataset-items.mjscomponents/apify/actions/run-actor/run-actor.mjs
📚 Learning: 2025-09-12T08:28:06.736Z
Learnt from: matyascimbulka
Repo: PipedreamHQ/pipedream PR: 18308
File: components/apify/sources/common/base.mjs:17-0
Timestamp: 2025-09-12T08:28:06.736Z
Learning: WEBHOOK_EVENT_TYPE_GROUPS.ACTOR_RUN_TERMINAL from apify/consts is an array containing all terminal Actor run event types: ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED", "ACTOR.RUN.ABORTED", "ACTOR.RUN.TIMED_OUT"]. It should be used directly in the eventTypes field when creating webhooks for future-proofing.
Applied to files:
components/apify/actions/run-task/run-task.mjscomponents/apify/actions/run-actor/run-actor.mjscomponents/apify/sources/new-finished-actor-run-instant/new-finished-actor-run-instant.mjs
🧬 Code graph analysis (3)
components/apify/actions/set-key-value-store-record/set-key-value-store-record.mjs (2)
components/apify/actions/run-actor/run-actor.mjs (1)
input(319-319)components/apify/actions/run-task/run-task.mjs (1)
input(68-68)
components/apify/actions/get-kvs-record/get-kvs-record.mjs (2)
components/apify/actions/run-task/run-task.mjs (1)
$(204-204)components/apify/actions/run-actor/run-actor.mjs (1)
record(371-374)
components/apify/sources/new-finished-actor-run-instant/new-finished-actor-run-instant.mjs (1)
components/apify/actions/run-actor/run-actor.mjs (1)
props(187-187)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Lint Code Base
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: pnpm publish
🔇 Additional comments (32)
components/apify/package.json (1)
3-3: LGTM!Version bump from 0.3.1 to 0.4.0 is appropriate for the new features and actions being added in this PR.
components/apify/actions/get-dataset-items/get-dataset-items.mjs (1)
8-8: LGTM!Version bump aligns with the package version update. The pagination logic correctly handles limits and offsets.
components/apify/actions/scrape-single-url/scrape-single-url.mjs (1)
9-9: LGTM!Version bump is consistent with the package-wide updates.
components/apify/sources/new-finished-task-run-instant/new-finished-task-run-instant.mjs (1)
9-9: LGTM!Version bump is consistent with the package-wide updates.
components/apify/actions/run-task-synchronously/run-task-synchronously.mjs (1)
8-8: LGTM!Version bump is consistent with the package-wide updates. The implementation correctly uses
defaultDatasetIddirectly without conditional checks, as per the Apify documentation.components/apify/actions/set-key-value-store-record/set-key-value-store-record.mjs (3)
6-7: LGTM!Version bump and description update appropriately reflect the action's capabilities.
20-22: LGTM!Adding
unnamed: falseto the prop definition appropriately filters out unnamed Key-Value Stores, ensuring users can only select named stores for this action.
43-55: LGTM!The type inference logic correctly handles all JavaScript primitive types and objects:
null, numbers, booleans, arrays, and objects are stored as JSON- Strings are evaluated separately for potential JSON parsing
components/apify/actions/run-actor/run-actor.mjs (6)
10-10: LGTM!Version bump reflects the significant logic changes in this action.
140-152: LGTM!Iterating over schema properties instead of input data keys ensures only valid schema-defined properties are processed, and skipping
undefinedvalues prevents sending empty fields to the API.
218-218: LGTM!Filtering out empty option values and labels prevents confusing or broken dropdown selections in the UI.
220-226: LGTM!Using
prefillas the default value aligns with Apify's input schema specification, where prefill serves as a suggestion while the actual default is handled by the platform.
228-236: LGTM!Correctly handles
string[]props with special editors by transforming default values appropriately - extracting URLs forrequestListSourcesand stringifying objects forjson/schemaBasededitors.
312-319: LGTM!The input selection logic correctly prioritizes dynamic schema props (
data) over the fallbackpropertiesobject prop, with a clean ternary chain.components/apify/sources/new-finished-actor-run-instant/new-finished-actor-run-instant.mjs (3)
3-3: LGTM: Import and prop addition for dynamic actor selection.The
apifymodule is correctly imported and added as a prop, enabling dynamic actor option generation viagetActorOptions.Also applies to: 16-16
17-33: LGTM: Well-structured dynamic prop for actor source selection.The
actorSourceprop withreloadProps: truecorrectly triggersadditionalProps()re-evaluation when the user changes the selection between "store" and "recently-used" actors.
35-51: LGTM: Dynamic actorId generation via additionalProps.The pattern correctly spreads the base
propDefinitionand provides a customoptions()function that delegates tothis.apify.getActorOptions(). This aligns with the helper method added inapify.app.mjs.components/apify/apify.app.mjs (5)
166-168: LGTM: Auth token getter for API calls.Exposing the auth token via
getAuthToken()is necessary for the direct HTTP calls in the KVS record action.
193-198: LGTM: runTask now accepts input parameter.The method signature update to pass
inputtostart()aligns with the Apify client API and enables input override functionality in the run-task action.
262-269: LGTM: KVS record access methods.
getKVSRecordandgetKVSRecordUrlcorrectly delegate to the Apify client's key-value store API, enabling the new get-kvs-record action.
270-275: LGTM: runTaskSynchronously now accepts input.Consistent with
runTaskchanges, the synchronous variant now also passesinputtocall().
183-186: Method rename successfully completed. No references to the oldgetActorRunmethod name remain in the codebase. The newgetRunmethod is properly being called incomponents/apify/actions/run-task/run-task.mjs.components/apify/actions/get-kvs-record/get-kvs-record.mjs (4)
1-14: LGTM: Well-annotated action metadata.The action metadata correctly indicates this is a read-only, non-destructive operation with proper documentation link.
37-52: LGTM: Robust HEAD request handling.Using
validateStatus: () => trueto handle all status codes manually is appropriate here, with proper 404 and error handling.
69-77: LGTM: Proper type discrimination for JSON vs text.The check
typeof data === "object" && !Array.isArray(data) && data !== nullcorrectly identifies plain JSON objects, returning them directly while wrapping primitives/arrays in a{ value }structure.
78-87: LGTM: Signed URL fallback for large/binary files.Returning a signed URL for files that can't be parsed inline is a sensible approach that avoids memory issues with large files.
components/apify/actions/run-task/run-task.mjs (6)
65-76: LGTM: Input parsing with clear error message.The JSON parsing with descriptive error handling is appropriate for user-provided input.
91-100: LGTM: Non-fatal webhook deletion.Using
console.warnfor webhook deletion failures is appropriate since cleanup failure shouldn't block the action result.
102-115: LGTM: Polling state persistence across reruns.The
schedulePollhelper correctly persistsapifyRunId,pollStartMs, andwebhookIdvia$.flow.rerun()context.
153-162: LGTM: Polling window enforcement with cleanup.Properly enforces the 1-day polling limit and cleans up the webhook before throwing the timeout error.
170-193: LGTM: Terminal status handling.The logic correctly handles all terminal statuses, cleans up the webhook, and differentiates between success and failure outcomes.
196-241: LGTM: Initial execution with webhook and fallback polling.The flow correctly:
- Starts the task and stores context
- Creates a suspend point with resume URL
- Sets up a webhook for terminal events
- Schedules fallback polling via rerun
This provides reliable completion detection with webhook as primary and polling as fallback.
michelle0927
moved this from Doing
to Ready for QA
in Component (Source and Action) Backlog
For Integration QA: |
vunguyenhung
moved this from Ready for QA
to Ready for Release
in Component (Source and Action) Backlog
|
Hi everyone, all test cases are passed! Ready for release! Test reports
|
sergio-eliot-rodriguez
left a comment
sergio-eliot-rodriguez
left a comment
There was a problem hiding this comment.
Looking great!
github-project-automation
bot
moved this from Ready for Release
to Done
in Component (Source and Action) Backlog
Applies changes from #19525
Note: Will create follow-up PR to update
apify_oauthcomponents accordingly after this is published.Summary by CodeRabbit
New Features
Bug Fixes
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.