batch action and query starts into single scheduled function#212
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
commit: |
84591e1 to
2dfc248
Compare
039fa52 to
665faef
Compare
2dfc248 to
ad4ea78
Compare
|
Warning Review limit reached
Next review available in: 13 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository: get-convex/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe component loop batches action and query starts through Sequence Diagram(s)sequenceDiagram
participant S as Scenario
participant L as Loop
participant B as runBatch
participant C as noisyOnComplete
participant M as metricsByLabel
S->>L: schedule noisy tasks
L->>B: submit action and query batch
B->>C: record terminal result
S->>M: poll run metrics
M-->>S: return status and latency metrics
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@example/convex/test/scenarios/noisyNeighbor.ts`:
- Around line 1-31: Run Prettier on the noisyNeighbor scenario file and commit
the resulting formatting changes, without altering its behavior or making
unrelated edits.
In `@src/component/worker.ts`:
- Around line 113-149: Limit concurrency in runWork so it does not launch all
items simultaneously under one parent action, using a workload-appropriate bound
that keeps the wrapper within Convex action budgets. Preserve independent
per-item completion and failure scheduling, and ensure a slow child cannot
terminate the entire batch.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: get-convex/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 39bf4510-e4a1-4d98-8752-f7a28d4222db
⛔ Files ignored due to path filters (1)
example/convex/_generated/api.d.tsis excluded by!**/_generated/**
📒 Files selected for processing (9)
example/convex/schema.tsexample/convex/test/PLAN.mdexample/convex/test/noisy.tsexample/convex/test/scenarios/noisyNeighbor.tssrc/component/complete.test.tssrc/component/complete.tssrc/component/loop.test.tssrc/component/loop.tssrc/component/worker.ts
| export const runWork = internalAction({ | ||
| args: { | ||
| items: v.array(v.object(actionOrQueryRunArgs)), | ||
| logLevel, | ||
| }, | ||
| handler: async (ctx, { items, logLevel }) => { | ||
| const console = createLogger(logLevel); | ||
| // Complete each job independently, running its onComplete inline. Only the | ||
| // job that hits an OCC gets scheduled (as a batch of one) — a conflicting | ||
| // onComplete can no longer drag the rest of the batch into a retry, since | ||
| // arbitrary user code in one onComplete shouldn't couple the others. | ||
| await Promise.all( | ||
| items.map(async (item) => { | ||
| // No single item may reject the batch: a thrown error here would fail | ||
| // the whole action and take the other in-flight items down with it. | ||
| try { | ||
| const job = await runActionOrQueryJob(ctx, console, item); | ||
| await completeJobInline(ctx, console, job); | ||
| } catch (e) { | ||
| console.error( | ||
| `[runWork] unexpected error for ${item.workId}, scheduling failed completion: ${e}`, | ||
| ); | ||
| await ctx.scheduler.runAfter(0, internal.complete.complete, { | ||
| jobs: [ | ||
| { | ||
| workId: item.workId, | ||
| runResult: { kind: "failed", error: formatError(e) }, | ||
| attempt: item.attempt, | ||
| nonRetryable: isNonRetryableError(e), | ||
| }, | ||
| ], | ||
| }); | ||
| } | ||
| }), | ||
| ); | ||
| }, | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Convex ctx.runAction from action concurrent execution time and resource limits
💡 Result:
When using ctx.runAction in Convex, it is important to understand its impact on execution and resource allocation. Execution Time and Resource Limits 1. Time Limits: Convex actions currently have execution time limits of 30 minutes for the Convex runtime and 10 minutes for the Node.js runtime [1][2]. Note that ctx.runAction is a blocking call; while it is executing, the parent action remains frozen and continues to consume its own allocated resources [3][4]. 2. Resource Overhead: The use of ctx.runAction incurs significant overhead because it initiates an entirely separate function call with its own system resource allocation [3][4]. 3. Concurrency Limits: Actions are subject to deployment-class concurrency limits, which vary depending on your plan (e.g., S16, S256, D1024, D2048) [1][2]. These limits constrain the number of functions that can run concurrently, including actions [1][2]. Additionally, individual actions can perform up to 1,000 concurrent I/O operations, such as database queries or fetch requests [3][1]. Best Practices The primary use case for ctx.runAction is to facilitate cross-runtime communication—specifically, calling a "use node" action from the default Convex runtime [3][4][5]. For logic that resides within the same runtime, using ctx.runAction is discouraged. Instead, you should extract shared code into a plain TypeScript helper function [3][4][5]. This approach is more efficient, as it avoids the overhead of a separate function invocation and the associated resource consumption of a secondary Convex execution context [3][4]. Self-Hosted Considerations If you are running a self-hosted Convex instance, execution limits and concurrency can be adjusted via environment variables. If you encounter errors related to concurrent function limits, you can configure thresholds such as APPLICATION_MAX_CONCURRENT_NODE_ACTIONS and APPLICATION_MAX_CONCURRENT_V8_ACTIONS to suit your hardware and workload requirements [6][7]. Timeout limits for self-hosted actions can also be managed via specific configuration knobs like ACTIONS_USER_TIMEOUT_SECS [8].
Citations:
- 1: https://docs.convex.dev/production/state/limits
- 2: https://docs.convex.dev/production/state/limits.md
- 3: https://docs.convex.dev/functions/actions
- 4: https://docs.convex.dev/api/interfaces/server.GenericActionCtx.md
- 5: https://docs.convex.dev/api/interfaces/server.GenericActionCtx
- 6: Self-Hosted Convex crashed with message about "16 concurrent actions need Convex Pro" convex-backend#391
- 7: https://get-convex-convex-backend.mintlify.app/self-hosting/configuration
- 8: Convex-Backend self hosted: Action execution timeout after 10 seconds not 10 minutes! convex-backend#241
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate files =="
fd -a 'worker\.ts$' . | sed 's#^\./##'
echo "== worker outline =="
worker=$(fd 'worker\.ts$' . | head -n1)
[ -n "${worker:-}" ] && ast-grep outline "$worker" --view expanded || true
echo "== relevant worker section =="
[ -n "${worker:-}" ] && nl -ba "$worker" | sed -n '1,220p'
echo "== ACTION_START_BATCH_SIZE references =="
rg -n "ACTION_START_BATCH_SIZE|ACTION_RUN_BATCH_SIZE|createAction|internalAction|ctx\.runAction|ctx\.runQuery" -S .Repository: get-convex/workpool
Length of output: 774
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant worker section =="
awk '{printf "%6d\t%s\n", NR, $0}' src/component/worker.ts | sed -n '1,260p'
echo "== batching and ctx.run* references =="
awk '{printf "%6d\t%s\n", NR, $0}' src/component/worker.ts | rg -n "ACTION_START_BATCH_SIZE|ACTION_RUN_BATCH_SIZE|int.*action|internalAction|ctx\.runAction|ctx\.runQuery|Promise\.all|map|batch" || true
echo "== all relevant action/job/config references =="
rg -n "ACTION_START_BATCH_SIZE|ACTION_RUN_BATCH_SIZE|int.*action|internalAction|ctx\.runAction|ctx\.runQuery|Promise\.all|map|completeJobInline|runActionOrQueryJob|runWork" src component . --glob '!node_modules' --glob '!dist' --glob '!build' | head -n 250Repository: get-convex/workpool
Length of output: 30929
Bound the runWork fan-out to fit Convex action budgets.
runWork starts up to ACTION_START_BATCH_SIZE (32) child ctx.runAction/ctx.runQuery calls under one parent action. Since a child ctx.runAction blocks the parent until completion, one heavy child can exhaust the parent action’s wall-clock or resource budget and terminate the entire batch. Make the batch size workload-specific or limit concurrency/long-child protection so a single slow item cannot take out the wrapper action.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/component/worker.ts` around lines 113 - 149, Limit concurrency in runWork
so it does not launch all items simultaneously under one parent action, using a
workload-appropriate bound that keeps the wrapper within Convex action budgets.
Preserve independent per-item completion and failure scheduling, and ensure a
slow child cannot terminate the entire batch.
| await ctx.runMutation(internal.complete.complete, { jobs: [job] }); | ||
| } catch (e) { | ||
| console.error( | ||
| `[runBatch] completing ${job.workId} inline failed, scheduling a batch of one instead: ${e}`, |
There was a problem hiding this comment.
minor nit, but I wonder if "batch of one" might be misleading in this error message, as a "batch" elsewhere refers to work, but here we are just scheduling the completion, not the work itself
2 participants
Actions and queries are now started together in a single batched
runWorkaction wrapper, rather than each being scheduled individually. This reduces the number of scheduled function calls when multiple actions or queries are pending at once. Batches are capped at 32 items (ACTION_START_BATCH_SIZE) to keep individual invocations bounded. Mutations continue to be scheduled one at a time viarunMutationWrapper, as before.The
beginWorkfunction has been replaced bybeginWorkBatch, which accepts a list of pending starts, separates them by function type, and schedules the appropriate wrappers. TherunActionWrappernow delegates to the sharedrunActionOrQueryJobandcompleteActionOrQueryJobshelpers, which are also used by the newrunWorkaction. Query execution is handled within the same action wrapper path as actions.