Make Agent Swarms Faster and Cheaper with Bounded Parallelism

Parallelism can shorten an agent workflow only when independent work overlaps. Past that point, more concurrency increases queueing, rate-limit failures, duplicate context, retry storms, and tail latency.

The safe pattern is a dependency graph plus a bounded worker pool. Measure it against a single-agent baseline, cancel branches that no longer matter, and optimize total successful-task cost—not the number of simultaneous model calls.

This guide includes a tested TypeScript worker pool in examples/bounded-parallelism.

Find real parallel work

Write the workflow as a directed acyclic graph (DAG):

                 ┌─ repository analysis ─┐
user request ────┼─ documentation check ─┼─ synthesis ─ review
                 └─ test-plan design ────┘

The three middle tasks can overlap if they do not need one another’s intermediate state. Synthesis waits for the required inputs. Review waits for synthesis.

A chain does not become faster when named agents are assigned to every step:

plan → fetch → interpret → write → approve → execute

Parallelizing dependent steps creates speculative work that will be discarded or, worse, acted on before its prerequisite is final.

Use the three decision tests from Agent Swarms: When Multiple Agents Help: independence, clear contracts, and evaluable synthesis.

Establish the baseline

Run the same versioned workload through the simplest correct implementation. Record:

• end-to-end success by task type;
• p50, p95, and p99 latency;
• total input and output tokens;
• model and tool calls;
• retry and rate-limit counts;
• duplicated or discarded work; and
• total cost per successful task.

Do not optimize average latency alone. A design that makes easy tasks fast but doubles p95 or lowers success may be worse for users. Google’s systems paper The Tail at Scale explains why temporary slowdowns become dominant as a request fans out across more components—the same shape created by a wide agent workflow.

Use agent evals for quality and OpenTelemetry traces for the critical path. Optimization without both is guesswork.

Use a bounded worker pool

Promise.all(items.map(run)) starts everything immediately. That is acceptable for a small constant list, but it is dangerous when a planner or user can create the list.

The repository example implements mapBounded with a fixed number of lanes:

export async function mapBounded<Input, Output>(
  items: readonly Input[],
  concurrency: number,
  worker: (
    item: Input,
    index: number,
    signal: AbortSignal | undefined,
  ) => Promise<Output>,
  signal?: AbortSignal,
): Promise<readonly TaskResult<Output>[]> {
  if (!Number.isInteger(concurrency) || concurrency < 1) {
    throw new RangeError("concurrency must be a positive integer");
  }

  const results: Array<TaskResult<Output> | undefined> =
    new Array(items.length);
  let nextIndex = 0;

  async function runLane(): Promise<void> {
    while (true) {
      signal?.throwIfAborted();

      const index = nextIndex;
      nextIndex += 1;
      if (index >= items.length) return;
      const item = items[index] as Input;

      const startedAt = performance.now();
      try {
        const value = await worker(item, index, signal);
        results[index] = {
          durationMs: performance.now() - startedAt,
          index,
          status: "fulfilled",
          value,
        };
      } catch (error) {
        results[index] = {
          durationMs: performance.now() - startedAt,
          error,
          index,
          status: "rejected",
        };
      }
    }
  }

  const laneCount = Math.min(concurrency, items.length);
  await Promise.all(Array.from({ length: laneCount }, () => runLane()));

  if (results.some((result) => result === undefined)) {
    throw new Error("A task completed without a recorded result");
  }

  return results as readonly TaskResult<Output>[];
}

The included check runs five tasks with concurrency two, verifies peak active work never exceeds two, and verifies results preserve input order:

cd examples/bounded-parallelism
bun run check

Expected output:

Bounded worker pool preserved order and capped concurrency at 2.

The function returns individual failures as results so one optional branch does not discard every successful branch. The coordinator still decides which failures make synthesis impossible.

Set more than one limit

A single concurrency number is not a complete budget. Enforce:

Use separate semaphores for different scarce resources. A model provider and a database have different capacities. One global limit can leave one resource idle while another is overloaded.

Fairness matters too. A FIFO queue can let a huge workflow occupy every slot. Reserve per-tenant capacity or use fair scheduling so small interactive requests are not trapped behind a batch job.

Choose concurrency empirically

Sweep concurrency over a fixed workload—for example 1, 2, 4, 8, and 16. Repeat each setting enough times to observe variance. Plot:

• successful workflows per minute;
• p50 and p95 end-to-end latency;
• provider 429/5xx rate;
• total tokens and retries;
• tool queue wait; and
• cost per successful workflow.

Throughput typically improves, then flattens as a provider, network, CPU, or downstream service saturates. Tail latency and errors may rise before average throughput falls. Choose a point with headroom rather than the single fastest benchmark run.

Adapt slowly. A concurrency controller that reacts to every 429 can oscillate. Reduce on sustained rate-limit or latency signals, recover gradually, and keep hard maximums.

Cancel work that no longer matters

Parallel branches should share a workflow cancellation signal. Cancel when:

• the user stops the request;
• a hard deadline expires;
• a required branch proves the task impossible;
• a sufficiently good result makes speculative alternatives unnecessary; or
• a budget or safety gate trips.

AbortSignal communicates cancellation; every worker and tool adapter must actually observe it. Node’s promise-based APIs increasingly accept signals, and custom adapters can call signal.throwIfAborted() before expensive work.

Cancellation does not make external writes disappear. For side effects, follow the durable idempotency and reconciliation pattern.

Do not launch speculative work unless you have a rule for cancelling and accounting for it. “Maybe useful later” is an expensive queueing strategy.

Deduplicate before calling a model or tool

Parallel agents often ask the same question in different words. Compute a stable key from the normalized operation and trustworthy inputs:

cache key = operation version + model/tool version + normalized arguments

Use single-flight deduplication so concurrent identical reads share one in-flight promise. Cache only operations whose semantics allow reuse, and include tenant, permissions, freshness requirements, and relevant configuration in the key.

Never cache:

• authorization decisions without the full principal and policy version;
• writes merely because arguments match;
• personal results across tenants; or
• model output when hidden context or nondeterminism changes its meaning.

Record cache hits and misses in traces so a “faster” release is not accidentally serving stale data.

Reduce context before routing models

Input tokens often dominate multi-agent cost because the same conversation is copied to every worker. Give each worker:

1. the global user outcome;
2. its narrow assignment;
3. only the trusted background it needs; and
4. references to artifacts it may fetch.

Do not send every sibling result to every worker. Synthesis can receive typed outputs after parallel work completes.

Route by task requirements, not by agent personality. Use a cheaper or faster model only after evals show it meets that task’s acceptance criteria. Keep a fallback for specific failure classes; do not blindly run two models for every task and choose the prettier response.

Stop with evidence

Every loop needs code-enforced stop conditions:

• maximum steps and attempts;
• remaining token/cost budget;
• task-specific acceptance criteria satisfied;
• no unresolved required dependency;
• no new evidence after a defined number of iterations; and
• explicit escalation when the system lacks authority or information.

A model saying “I am done” can be one signal. Deterministic validators, test results, citations, and state transitions should decide whether the workflow is actually complete.

Preserve quality during partial failure

Classify branches as required or optional. If an optional research source times out, synthesis can continue while disclosing reduced coverage. If the security review or required dataset fails, stop.

Never let synthesis turn missing branches into confident prose. Pass structured branch status:

interface BranchOutcome<T> {
  readonly coverage: readonly string[];
  readonly result?: T;
  readonly status: "complete" | "failed" | "timed_out";
  readonly uncertainty: readonly string[];
}

The final result should preserve evidence, failed coverage, and uncertainty. Parallelism is not a license to hide failure behind a fluent coordinator.

A practical optimization order

1. Make the single-agent baseline correct and observable.
2. Remove unnecessary model calls and duplicate context.
3. Split only independent, contract-shaped tasks.
4. Add bounded concurrency with resource-specific limits.
5. Propagate cancellation and total deadlines.
6. Deduplicate identical reads and cache only safe results.
7. Route models by eval-proven task requirements.
8. Sweep concurrency and select a point with tail-latency headroom.
9. Re-run quality, safety, and cost gates for every change.

The fastest agent call is the one the workflow proves it does not need.

Sources and freshness notes

The worker pool was executed on July 18, 2026 with Bun 1.3.9. Cancellation guidance follows the standard AbortController and AbortSignal API documented by Node.js, and the tail-latency rationale follows the primary The Tail at Scale paper. Performance thresholds are deliberately workload-specific: publish your workload, configuration, repetitions, and raw measurements before claiming a universal optimal concurrency.