Blog

Temporal Matching and Workers: Task Queues, Partitions, Sticky Queues, Pollers and Slots

How a Temporal task actually reaches your code: sync match vs a backlog in PostgreSQL, task queue partitions as the real unit of parallelism, the sticky queue that turns a SIGKILL into a 5–10 s stall, a one-line config that made 795 writes/s, and how to size pollers and slots with Little's law instead of guessing.
24 min readPart 4 of 10
On this page
  1. Why there's a counter in the middle
  2. What a task queue actually is
  3. How a task lands on the queue
  4. Sync match vs backlog
  5. The sync-match ratio is the health signal
  6. The handshake: History is the only judge
  7. The long poll
  8. Partitions: one shelf, several counters
  9. Forwarding, and why each partition needs pollers
  10. Placement and host failure
  11. Why more Matching hosts don't speed up one queue
  12. Sticky queues: why a worker restart feels slow
  13. The ack level and the 795 writes/s incident
  14. "Worker" means two different things
  15. Inside your SDK worker: pollers, slots and the cache
  16. Pollers: few, and at least one per partition
  17. Slots: size them with Little's law
  18. Is the wait on the shelf or at the desk?
  19. Autoscaling workers: not on CPU
  20. What I'd tell myself before go-live

TL;DR: The Temporal server never calls your workers. Workers long-poll the Matching service through Frontend, and History drops tasks onto named task queues. If a poll is already parked on the partition, the task is handed over in memory (sync match, zero database writes); if not, it's INSERTed into the tasks table and read back later (backlog). A healthy queue is almost entirely sync match. Each task queue is split into partitions (default 4), each owned by exactly one Matching host, so the partition — not the host — is the unit of parallelism: a sixth Matching host does nothing for a four-partition queue. Sticky queues keep a workflow on the worker that has it cached, which is why a SIGKILLed worker produces a 5–10 s workflow-task stall followed by full replays on everyone else. Inside your worker, pollers only fetch and slots do the work, and a poller only reaches out when a slot is free. So the first question for "Temporal is slow" is always: is the wait on the shelf, or at the desk? Slots at 0 means your workers. Never scale Matching for that.


The most expensive mistake I've watched people make with Temporal is scaling the wrong thing. Schedule-to-start latency goes up, someone says "Matching is overloaded", two more Matching hosts appear, and nothing changes — because the tasks were waiting for our workers, which had every slot full. The reverse happens too: a fleet idling at 5% CPU with one queue backing up, where the fix was a partition count, not a pod count.

Both come from the same gap: not knowing what happens between "History decided an activity should run" and "my function started". This post is that gap — Matching from the server side, then the SDK worker from the application side — with the config keys, the metrics, and the incident that taught me to do arithmetic before listing suspects.

The four services covers the overall shape, and Frontend and History covers the transfer queue that feeds Matching.

Why there's a counter in the middle

History has just recorded "next step: run activity sendEmail". That code lives in your worker processes, not in the server. History can't call the workers: it doesn't know their addresses (NAT, new IPs every deploy, scale to zero), it would have to track their health and capacity, and a slow fleet would back up History's own transfer queue. Workers can't ask History directly either — that hammers the busiest service in the cluster.

So there's a service in the middle. History puts a task on a named task queue and forgets it once Matching acknowledges. Workers make outbound gRPC calls only and wait at the queue. When a task and a waiting worker meet, the task changes hands — a job counter with named shelves. Workers come and go freely, and when they're slow, Matching keeps a durable backlog so History keeps moving.

Matching listens on gRPC 7235 and membership 6935 in the standard sample config (check yours). It talks to Frontend (polls arrive through it), to History (tasks arrive from it, and Matching calls back RecordActivityTaskStarted / RecordWorkflowTaskStarted), and to PostgreSQL.

What a task queue actually is

A task queue is never created. It exists the first time a task is added or a worker polls it, and its identity has three parts:

text
task queue identity = (namespace_id, task_queue_name, task_type)
                                                       └─ Workflow | Activity | Nexus (newer)

So a worker started with taskQueue: 'email_queue' that registers workflows and activities polls three physical queues: email_queue/Workflow (decisions: "continue this workflow's code"), email_queue/Activity ("run this function once"), and its own sticky queue named like <host>:<uuid>. The workflow and activity queues are fully independent — separate partitions, backlogs, task_queues rows, dynamic-config constraints and metrics. A million-task activity backlog doesn't slow workflow-task dispatch on the same name, though it does slow the workflows waiting on those activities.

One trap: Matching doesn't know what you registered. It hands a task to whichever poll is waiting. If that worker lacks the activity, the attempt fails ("Activity function X is not registered") and is retried on the same queue — possibly onto another worker that lacks it too. Every worker polling a queue must register everything that lands on it. Different apps, different task queues.

How a task lands on the queue

The Matching slice of "workflow code calls executeActivity(sendEmail)":

text
SDK worker ── RespondWorkflowTaskCompleted [ScheduleActivityTask] ──▶ Frontend ──▶ History (shard owner)
 
History, one DB transaction:
  append ActivityTaskScheduled + update mutable state + INSERT transfer_tasks(ActivityTask)
 
History transfer queue processor:
  matching client picks a WRITE partition (say P2), looks up its owner on the ring
  └─ AddActivityTask(email_queue/P2, scheduleToStartTimeout) ──▶ Matching
 
Matching, partition P2:
  poll parked here?  yes ─▶ RecordActivityTaskStarted on History ─▶ task in poll response ─▶ OK (sync match)
                     no  ─▶ INSERT INTO tasks ─▶ OK (backlog)
 
History acks the transfer task

The task is durable in History before Matching sees it — the transfer_tasks row commits with the event. If Matching is down, History's transfer queue keeps retrying AddActivityTask; nothing is lost, the task is just late, and that lateness is schedule-to-start latency. History acks only after Matching says OK, and OK means exactly "handed to a poller" or "persisted in tasks". Workflow tasks take the same path via AddWorkflowTask, except History may try the sticky queue first.

Sync match vs backlog

Sync match hands a task to a parked poll in memory with no PostgreSQL write; backlog match INSERTs the task into the tasks table and reads it back in batches later
Sync match hands a task to a parked poll in memory with no PostgreSQL write; backlog match INSERTs the task into the tasks table and reads it back in batches later

Sync match. The partition offers the task to a parked poll, waiting up to matching.syncMatchWaitDuration (200 ms on current versions). If a poll takes it, Matching does the History handshake, the poll returns the task, and AddTask returns OK. No row is written to tasks. Metrics: poll_success_sync, syncmatch_latency.

Backlog. If no poll takes it, Matching spools it:

  1. A task ID comes from the partition's current ID block, granted per range_id — the same "stamp that only goes up" fencing as History shards.
  2. INSERT INTO tasks (range_hash, task_queue_id, task_id, data, data_encoding), one protobuf row per task. AddTask returns OK.
  3. The partition's task reader pulls rows into memory in batches of matching.getTasksBatchSize (default 1000), and hands them to polls as they arrive.
  4. The partition tracks an ack level — "everything up to task X is done" — saved every matching.updateAckInterval (default 1 minute) in its task_queues row. Rows below it are range-deleted.

The task_queues row is also the partition's lease:

sql
-- verify with \d task_queues on your version
CREATE TABLE task_queues (
  range_hash    BIGINT NOT NULL,
  task_queue_id BYTEA  NOT NULL,   -- namespace UUID bytes + partition name + type byte
  range_id      BIGINT NOT NULL,   -- the lease / fencing token
  data          BYTEA  NOT NULL,   -- protobuf: ack level, expiry, kind ...
  data_encoding VARCHAR(16) NOT NULL,
  PRIMARY KEY (range_hash, task_queue_id)
);

A host loading a partition bumps range_id with a conditional update, and every later backlog write is conditioned on it. If another host has bumped it since, the write fails with "task queue ownership lost" and this host unloads the partition. During ring churn two hosts can believe they own a partition, but only one can write its backlog — the same pattern as shard fencing. Newer servers may also use tasks_v2 / task_queues_v2 for fairness scheduling; run \dt task* before assuming tasks is the only backlog table.

The sync-match ratio is the health signal

Sync-match ratioMeaningPostgreSQL cost per task (Matching)
~100%Pollers outnumber tasks~0
FallingWorkers can't keep up, or polls sit on different partitions than tasksINSERT + SELECT + DELETE + dead tuples
~0%, backlog growingWorkers saturated or absentfull churn; tasks bloats, autovacuum works hard
promql
# Sync-match ratio per task queue (label names version-dependent: check /metrics first)
sum by (namespace, taskqueue) (rate(poll_success_sync[5m]))
/ clamp_min(sum by (namespace, taskqueue) (rate(poll_success[5m])), 1)

The handshake: History is the only judge

A backlog can hold stale tasks: after a crash the new owner re-reads from an old ack level and may offer a task twice, and a workflow may have been cancelled while its task sat on the queue. Recording every handout durably in Matching would make every sync match a DB write, defeating the point.

So Matching records nothing. Just before giving a task to a poll, it calls Record*TaskStarted on History, which checks mutable state, writes "started" if the task is still valid, and replies. Only then does the poll return. Duplicates are harmless — History refuses, Matching drops the task (tasks_dropped) and offers the next one. Matching can lose its in-memory state freely. And the one gap — "started" in History but the poll response lost in a crash — is closed by the activity's start-to-close timeout or the 10 s workflow task timeout, which is why every activity needs a sensible StartToCloseTimeout (timeouts post).

The cost: every successful poll is one extra RPC into History, and it's a mutable-state update that takes the per-workflow lock and a shard I/O slot. A slow History therefore slows dispatch too — service_latency{operation="RecordActivityTaskStarted"} on History rises together with schedule-to-start.

The long poll

A worker doesn't ask "any work?" once a second. It asks once and keeps standing there for up to a minute.

text
SDK poller ──PollActivityTaskQueue("email_queue")──▶ load balancer ──▶ any Frontend host
Frontend ──(matching client: pick READ partition, ring lookup)──▶ owner of /_sys/email_queue/1
Matching parks the poll on partition 1; the Frontend goroutine waits on the gRPC call
  • Every poll goes through Frontend, so a poller storm shows up first as Frontend ConcurrentLimit rejections: frontend.namespaceCount defaults to 1200 concurrent long polls per namespace, per Frontend host, per API.
  • A poll is one unary gRPC call held open — a parked goroutine and an HTTP/2 stream on each of two hops. Nothing spins.
  • If nothing arrives within matching.longPollExpirationInterval (1 minute), Matching returns an empty response and poll_timeouts goes up. Not an error; the SDK re-polls.
  • The SDK's poll deadline is a bit longer than the server window (~70 s in the Go SDK; verify yours). Every LB and proxy in front of Frontend needs an idle timeout above ~70 s, or you get temporal_long_request_failure exactly at the LB timeout. I start at 2 minutes.

Idle polls cost almost no CPU and never touch PostgreSQL. What they cost is concurrency: Frontend's per-namespace count, Matching goroutines, file descriptors, LB connection tables. 200 workers × 50 pollers × 2 types is 20,000 parked polls — Frontend starts rejecting, and nothing got faster because tasks/s didn't change.

Partitions: one shelf, several counters

If a queue lived on one Matching host, all its adds, polls and backlog writes would hit one host and one lease row, and losing that host would pause the queue. Same problem as History, same fix: split it. Every queue (per type) has 4 read and 4 write partitions by default (matching.numTaskqueueReadPartitions / matching.numTaskqueueWritePartitions, dynamic and per-queue), each with its own owner, backlog rows, lease row and in-memory matcher.

text
partition 0 (root)   RPC name = "email_queue"
partition 1..N-1     RPC name = "/_sys/email_queue/<id>"
sticky               RPC name = the sticky queue name (never partitioned)

The SDK knows nothing about this. The matching client inside Frontend (polls) and History (adds) picks a partition at random (or load-aware, depending on version).

Write partitions are where History adds; read partitions are where polls spread. Keep them equal in steady state, and read ≥ write at every moment during a change. Growing 4 → 8: raise read first. Shrinking 8 → 4: lower write first, wait until partitions 4–7 drain to zero backlog, then lower read — do it the other way and tasks already spooled on 4–7 are stranded.

Forwarding, and why each partition needs pollers

With 4 partitions and one idle poll, the poll waits on P3 and the next task lands on P1. So partitions form a tree: a child with an unmatched poll or task forwards it to its parent, and the root (P0) is where stragglers meet. With 20 or fewer partitions every child's parent is P0 (matching.forwarderMaxChildrenPerNode, default 20).

Forwarding is rate-limited on purpose — 1 outstanding poll, 1 outstanding task and 10/s per partition by default — and if the parent can't match a forwarded task, the child spools it. It's a low-traffic fallback, not a throughput path. So with 16 read partitions and 8 pollers across the fleet, half the partitions have no poll at any moment and their tasks crawl through forwarding while your workers sit idle. Pollers across the fleet ≥ read partitions, with margin.

Placement and host failure

Every partition of every queue is hashed independently onto the ring of live Matching hosts:

text
routing key = "<namespaceID>:<partition RPC name>:<taskType>"
owner       = matchingRing.Lookup(routing key)

There's no ownership table in PostgreSQL; task_queues only shows which partitions exist and how often they've changed hands:

sql
-- Read-only. A range_id that climbs quickly = the partition is re-leased often (ring churn, restarts).
SELECT encode(substring(task_queue_id from 17), 'escape') AS partition_and_type, range_id
FROM task_queues
ORDER BY range_id DESC
LIMIT 20;

When a Matching host dies, its parked polls fail and SDKs re-poll; its in-memory buffer is gone but the rows are still in tasks. For the seconds until gossip notices, adds routed to it fail and History's transfer queue retries them. Then only that host's partitions rehash to survivors, each new owner bumps range_id and reloads from the saved ack level (redispatching a few tasks that History refuses), and you see a schedule-to-start spike about as long as detection plus reload. Nothing durable is lost; sync-matched-but-undelivered tasks come back via timeouts. Adding a host is the mirror image — about 1 ÷ N of all partitions move — so on a busy cluster add hosts one at a time, off-peak. Membership itself is its own post.

Why more Matching hosts don't speed up one queue

One partition, one host. A four-partition queue can use at most four hosts. Each partition has three ceilings — one host's CPU, one in-memory matcher, one lease row and backlog key range — so a queue's throughput ≈ partitions × per-partition ceiling.

Temporal doesn't publish that ceiling. The figure usually quoted is on the order of ~100 tasks/s per partition (some say a few hundred), and it depends on CPU, payload size, sync-match ratio and DB latency. I use it as a conservative start — partitions ≈ peak tasks/s ÷ 100, minimum 4 — then benchmark: push load until the queue's schedule-to-start p99 rises while worker slots are free and its Matching owners are under 70% CPU.

Symptoms of too few partitions: that queue's asyncmatch_latency and approximate_backlog_count rise while worker slots are free; the 1–2 hosts owning its partitions are hot and the rest cool; task persistence_latency is normal; adding hosts didn't help.

Three placements of one queue's partitions on Matching hosts: 4 partitions on 3 hosts leaves one host with half the load, 16 on 4 hosts lands near even, and 4 on 6 hosts leaves two hosts idle for that queue
Three placements of one queue's partitions on Matching hosts: 4 partitions on 3 hosts leaves one host with half the load, 16 on 4 hosts lands near even, and 4 on 6 hosts leaves two hosts idle for that queue

Placement is a hash, not a plan, and that changes the math. Illustrative numbers for one hot queue at 400 tasks/s, from a quick uniform-random simulation:

SetupHottest host share (typical)Per-partition load
4 partitions / 3 hosts50% at best; 75% about 30% of the time100 t/s
4 partitions / 6 hosts~3 hosts own none of this queue100 t/s
16 partitions / 4 hosts~31–44% (exact 4/4/4/4 only ~1.5% of the time)25 t/s, ~4× headroom
64 partitions / 4 hosts~30–34%~6 t/s

Smaller pieces average out. My working rule for a hot queue is partitions ≥ 2 × the Matching hosts I want serving it — an operational starting point, not a Temporal rule. Idle hosts only appear in practice when one or two queues dominate; a real cluster has sticky queues, temporal-sys-* queues and other namespaces spread everywhere.

Raise partitions on the hot queue only:

yaml
# dynamicconfig/production.yaml
matching.numTaskqueueReadPartitions:
  - value: 16
    constraints:
      namespace: "payments"
      taskQueueName: "orders-tq"
      taskType: "Activity"
  - value: 4
matching.numTaskqueueWritePartitions:
  - value: 16
    constraints:
      namespace: "payments"
      taskQueueName: "orders-tq"
      taskType: "Activity"
  - value: 4

Dynamic config reloads without a restart, but it must be the same file on every host, or History adds to 16 partitions while some Frontends poll 4. And don't raise it globally "to be safe": every partition is another lease row and ack-level writer, and on quiet queues the few polls get spread so thin that the sync-match ratio drops and more tasks spool.

Sticky queues: why a worker restart feels slow

To make each decision a worker replays the workflow function against its history (determinism and replay). For a 2,000-event workflow, doing that every step burns CPU and makes History ship the whole history every time. So the SDK keeps recent workflows in memory — the sticky cache (maxCachedWorkflows in TypeScript, worker.SetStickyWorkflowCacheSize in Go) — and History routes the next task to that worker's personal sticky queue:

  1. Completing a workflow task, the worker tells History its sticky queue name and a sticky schedule-to-start timeout (default 5 s in Go, 10 s in TypeScript).
  2. History records it in mutable state and sends the next workflow task to that sticky queue.
  3. Sticky queues have one partition and no forwarding. If nobody has polled one recently, Matching answers StickyWorkerUnavailable at once and History uses the normal queue.
  4. If a sticky task sits unpicked past the timeout — the worker is overloaded, or just died and its queue is still loaded — History moves it to the normal queue, where any worker can take it at the cost of a full replay.

Graceful stop drops the sticky queue so the next workflow task goes to the normal queue at once; after a SIGKILL the task waits 5–10 s on the dead worker's sticky queue before another worker replays the full history
Graceful stop drops the sticky queue so the next workflow task goes to the normal queue at once; after a SIGKILL the task waits 5–10 s on the dead worker's sticky queue before another worker replays the full history

Step 4 is what bites. SIGKILL a worker with a warm cache and every workflow it held gets its next task sent to a queue nobody polls, waits out the 5–10 s timeout, then replays in full somewhere else. We saw exactly this on one of our worker fleets: every restart was followed by a burst of slow workflow tasks and a CPU spike on the survivors, and the more workflows cached at the moment of the kill, the bigger the cascade — so peak-hour deploys hurt most. Replay re-runs workflow code, not activities (their results come from history): the cost is CPU and History reads, never duplicate emails.

The fix is graceful shutdown. A worker that stops properly stops polling, finishes in-flight work and tells the server to drop its sticky queue (newer SDKs and servers do this with ShutdownWorker; check your changelog), so tasks go to the normal queue immediately. Under pm2 that needs time: the TypeScript SDK handles SIGINT/SIGTERM, but pm2 sends SIGKILL after kill_timeout — 1600 ms by default (verify your version). If kill_timeout is shorter than your shutdown, every deploy is a SIGKILL. Then deploy off-peak, one worker at a time, and keep histories short with continue-as-new.

A fleet that restarts constantly — crash loops, aggressive scale-in, pm2 max_memory_restart — turns every workflow task into a replay: higher temporal_workflow_task_execution_latency, more GetWorkflowExecutionHistory on History, a falling temporal_sticky_cache_hit ratio. A worker problem that looks exactly like server load.

The ack level and the 795 writes/s incident

Each partition saves its ack level every matching.updateAckInterval, default 1 minute. Saving often means fewer redispatched tasks after a crash, but every save is a row update. Saving rarely means more redispatches — harmless, because History refuses them. Duplicates are cheap and write storms aren't, so the default leans towards rarely.

On a staging cluster someone had set it to 0.5 s. Every active partition rewrote its task_queues row twice a second, and PostgreSQL showed 795 UpdateTaskQueue writes/s of pure overhead. Back at the default: 8/s.

The write storm arithmetic: UpdateTaskQueue writes per second ≈ active partitions ÷ updateAckInterval, so about 400 partitions at 0.5 s gave the measured 795 writes/s and the 1-minute default gave 8 writes/s
The write storm arithmetic: UpdateTaskQueue writes per second ≈ active partitions ÷ updateAckInterval, so about 400 partitions at 0.5 s gave the measured 795 writes/s and the 1-minute default gave 8 writes/s

The habit I took from it: do the arithmetic before listing suspects. "800 UpdateTaskQueue/s, we have too many workflows":

text
UpdateTaskQueue/s ≈ active partitions ÷ updateAckInterval
 
800/s at the 1 min default -> 800 × 60  = 48,000 active partitions   not plausible
800/s at 0.5 s             -> 800 × 0.5 = 400 ≈ 100 queues × 4       plausible

The rate depends on active partitions and the interval, not on workflow count. matching.rps is a request limit and irrelevant here. And mind the near-name: history.<tray>ProcessorUpdateAckInterval is History's per-shard bookmark. Same idea, different service — a storm on task_queues is Matching, on shards is History. To measure without metrics:

sql
-- Run twice, 60 s apart; the n_tup_upd difference on task_queues ÷ 60 = ack-level/lease writes/s
SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_dead_tup, last_autovacuum
FROM pg_stat_user_tables
WHERE relname IN ('tasks', 'task_queues', 'tasks_v2', 'task_queues_v2');

Matching's PostgreSQL traffic is also capped by matching.persistenceMaxQPS (3000 per host on current docs; older material says 9000). Hitting it shows as PersistenceLimit throttling; raising it just moves load to the database. The real fix for backlog churn is getting back to sync match. tasks bloat and autovacuum live in the persistence post.

"Worker" means two different things

Now the other side of the counter. "The Temporal workers are down" can mean two unrelated things:

Internal worker (server role)SDK worker (your app)
What it istemporal-server start --service=workera process you build with an SDK
Whose code / queuesTemporal's system workflows on temporal-sys-*yours, plus sticky queues
In the membership ring?yesno — a gRPC client of Frontend
If all replicas diesystem jobs pause; your workflows keep running (except things like Schedules)your workflows stop progressing; tasks pile up in Matching

Scaling the internal worker does nothing for your schedule-to-start; scaling your SDK workers does nothing for a stuck namespace deletion. Always say which.

The internal worker is Temporal running its own background jobs as Temporal workflows on the Go SDK, so they get retries and durability for free. On PostgreSQL the important ones are the task queue scanner (SQL stores only — it deletes expired tasks rows and idle task_queues rows, so Matching's tables don't grow forever), the history scanner (orphan history branches), batch operations (temporal workflow terminate --query ..., throttled by worker.batcherRPS, 50 per job), namespace deletion, parent-close processing for large fan-outs, and Schedules on temporal-sys-per-ns-tq (worker.schedulerNamespaceStartWorkflowRPS, 30 per namespace). The executions scanner isn't supported on SQL. A newer Scheduler that runs inside History exists on current main but isn't GA — check where your Schedules actually run.

One correction to my own early notes: retention deletion isn't the internal worker's job. When a workflow closes, History creates a DeleteHistoryEvent timer task on its shard and runs it itself. If retention lags, look at History's timer queue, not the worker role.

Run at least two replicas so a node loss doesn't pause Schedules; Temporal's docs cite a production deployment with "5 Frontend, 15 History, 17 Matching, and 3 Worker Services". And the trap: its embedded SDK clients are Frontend clients. Enable mTLS and an authorizer on Frontend without giving the system worker its own cert and role, and it fails silently — Schedules stop firing, scanners stop running, your own workflows look fine. Grep its logs for PermissionDenied, or route it via the internal-frontend role. I page on zero internal-worker replicas for 5 minutes and warn on any temporal-sys-* backlog older than 15 minutes; Temporal publishes no thresholds, so those are my starting points.

promql
max by (namespace, taskqueue) (approximate_backlog_count{taskqueue=~"temporal-sys-.*"})
sum by (namespace) (rate(schedule_missed_catchup_window[15m]))

Inside your SDK worker: pollers, slots and the cache

An SDK worker holds no authoritative state, so it can be killed and scaled like any stateless service — provided it shuts down gracefully. Inside it are pollers (each an outstanding long poll; they only fetch), slots (permission to run one task concurrently, separately for workflow tasks, activities and local activities), and the sticky cache.

The rule that makes everything else make sense: a poller only reaches out if a slot is free. Think hands and desks — grabbing a job takes milliseconds and doing it takes much longer, so a few hands keep many desks fed.

Inside one worker: with free slots the pollers wait at the task queue and every task sync-matches; with all activity slots busy the activity pollers stop asking, tasks pile up as backlog and schedule-to-start rises
Inside one worker: with free slots the pollers wait at the task queue and every task sync-matches; with all activity slots busy the activity pollers stop asking, tasks pile up as backlog and schedule-to-start rises

When a type's slots are full the worker stops polling for it. That's backpressure, and it's why an overloaded worker shows up first as schedule-to-start latency and a Matching backlog, not as errors in the worker.

Defaults per SDK, from Temporal's worker tuning reference (version-dependent):

SDKWF slotsActivity slotsWF pollersActivity pollersCached workflows
Go100010002210,000
Java20020055600
TypeScript401001010dynamic (e.g. 2000 at 4 GiB)
Python100100551000
.NET1001005510,000

Pollers: few, and at least one per partition

Too few and slots sit free while tasks wait (and unpolled partitions spool). Too many and you park thousands of idle polls on Frontend and Matching. Count them before adding any:

text
waiting polls ≈ workers × task queues × pollers per queue
              = 200 × 3 × 10 = 6,000    (12,000 with 10 workflow AND 10 activity pollers each)

That's against a 1200 limit per Frontend host, per API, per namespace. How many do you need? Little's law, illustrative: busy pollers ≈ tasks/s × time to grab one. At 500 tasks/s and ~20 ms per grab that's about 10 pollers across the whole fleet, plus at least one per read partition per queue and type, plus margin.

Temporal's docs now recommend poller autoscaling for most cases, since hand-set counts end up too high or too low. In Go it's WorkflowTaskPollerBehavior / ActivityTaskPollerBehavior set to worker.NewPollerBehaviorAutoscaling(...) (Go SDK v1.36.0+), which overrides MaxConcurrent*TaskPollers; other SDKs have equivalents in recent releases.

Slots: size them with Little's law

Slots are either fixed-size (maxConcurrentActivityTaskExecutions in TypeScript, MaxConcurrentActivityExecutionSize in Go) or handed out by a resource-based tuner that targets CPU and memory (the docs warn the targets can still be exceeded; tuners replace the fixed options, and setting both fails at startup).

typescript
// Illustrative TypeScript worker. Values are a starting point, not a recommendation.
const worker = await Worker.create({
  connection,
  namespace: 'default',
  taskQueue: 'email_queue',
  workflowsPath: require.resolve('./workflows'),
  activities,
  maxConcurrentActivityTaskExecutions: 100, // activity slots
  maxConcurrentWorkflowTaskExecutions: 40,  // workflow slots (decisions are quick)
  maxConcurrentActivityTaskPolls: 5,
  maxConcurrentWorkflowTaskPolls: 5,
  maxCachedWorkflows: 2000,                 // memory vs replay
  stickyQueueScheduleToStartTimeout: '10s',
  shutdownGraceTime: '30s',                 // pm2 kill_timeout must exceed this
});
await worker.run(); // handles SIGINT/SIGTERM

Activity slots are also a concurrency limit on whatever the activity calls: 1000 slots on 10 workers is up to 10,000 concurrent calls into your email API or app database. Size slots to the downstream, not just the worker.

Sizing is concurrency = rate × duration. Illustrative: 300 workflows/s with 3 activities each is ~1200 workflow tasks/s (start plus three completions) and 900 activity tasks/s.

text
workflow slots busy = 1200/s × 0.015 s ≈ 18
activity slots busy =  900/s × 0.250 s ≈ 225
× 2 burst headroom  → ~40 WF slots, ~450 activity slots across the fleet
450 ÷ 100 per worker → 5 workers, + 1 for N+1 = 6

Then measure. Capacity planning applies the same law across the whole chain. On the worker, memory is dominated by the sticky cache (too big → OOM → replay storm elsewhere) and CPU by replay and your activity code; pollers cost almost nothing locally — their cost is on the server.

Is the wait on the shelf or at the desk?

Schedule-to-start (temporal_activity_schedule_to_start_latency, temporal_workflow_task_schedule_to_start_latency) runs from History scheduling a task to a worker starting it, so every hop adds to it. Alone it only says "something in between is slow". The slot gauge, temporal_worker_task_slots_available, says which side of the counter is short.

ObservationDiagnosisActionDo NOT
S2S up, 0 pollers on the queuenobody at this queueworkers down, or a typo'd queue name / wrong namespacescale the server
S2S up, slots ≈ 0, worker CPU highworker fleet saturatedadd workersadd Matching hosts or partitions
S2S up, slots ≈ 0, worker CPU lowslots too small for IO-bound workmore slots or a tuner; check downstream—
S2S up, slots free, few pollers, poll errorspoller starvationmore pollers / autoscaling; check LB timeout, namespaceCount—
S2S up, slots free, History transfer-queue lagHistory late to enqueueHistory, shards, PostgreSQLblame Matching
S2S up, slots free, one queue's backlog up, its owners hotpartitionsraise that queue's partitions, then hostsraise them globally
S2S up, slots free, task persistence latency upPostgreSQLDB firstadd partitions
Workflow-task S2S jumps to 5–10 s after a deploysticky cascadegraceful shutdown, rolling off-peak deploysscale the server
S2S low, activity still slowyour activityyour codetouch Temporal

Check in that order: worker side first, then History, then Matching, then PostgreSQL. The one rule: never add Matching hosts because schedule-to-start is high while slots available = 0.

promql
# Schedule-to-start p95 per queue (the _seconds suffix depends on your exporter)
histogram_quantile(0.95, sum by (le, task_queue) (rate(temporal_activity_schedule_to_start_latency_seconds_bucket[5m])))
 
# Free activity slots, fleet-wide (0 = the workers are the limit)
sum by (task_queue) (temporal_worker_task_slots_available{worker_type="ActivityWorker"})
 
# Server-side backlog per queue, and one hot Matching owner
max by (namespace, taskqueue, task_type) (approximate_backlog_count)
sum by (instance) (rate(process_cpu_seconds_total{job="temporal-matching"}[5m]))
 
# Too many pollers? Frontend rejections by cause
sum by (resource_exhausted_cause) (rate(service_errors_resource_exhausted[5m]))
bash
# Pollers and backlog for one queue; the workflow side is a separate queue
temporal task-queue describe --namespace default --task-queue email_queue --task-queue-type activity

Temporal publishes no universal schedule-to-start threshold; derive it from your SLO (an activity that must start within 1 s 99% of the time → alert on p99 above 1 s for 10 minutes). For batch queues a backlog is normal, so alert on approximate_backlog_age_seconds instead. Grep /metrics for the real names before building dashboards.

Autoscaling workers: not on CPU

An activity calling a slow payment API spends 99% of its time waiting: all 100 slots taken, 5% CPU, and a CPU-based autoscaler that never adds a worker while schedule-to-start grows without bound. Temporal's docs give the pairing — high schedule-to-start with high worker CPU/memory means add workers; with low CPU/memory, add pollers or slots — and the slot gauge adds the third case: slots free and pollers fine means the server side.

So scale on schedule-to-start + free slots + backlog together. Under pm2 that's pm2 scale <app> <n> by hand or script. On Kubernetes, KEDA (v2.17+) has a temporal scaler that reads task-queue backlog from Frontend (its docs warn backlog-based activation is unreliable for scale-to-zero), or an HPA on adapter-defined metrics built from temporal_worker_task_slots_used / _available.

Scale-in is where the incidents are:

  • terminationGracePeriodSeconds or pm2 kill_timeout must exceed WorkerStopTimeout / shutdownGraceTime, plus margin.
  • Long activities must heartbeat, or one killed mid-flight waits a full start-to-close before retrying.
  • Every removed worker drops its sticky cache, so aggressive scale-in shows up as CPU spikes on the survivors and read load on History. Use long scale-down stabilisation windows (10 minutes is a reasonable start) and a non-zero minimum for latency-sensitive queues.

And more pollers is not more capacity. If every slot is busy, doubling maxConcurrentActivityTaskPolls does literally nothing — the extra pollers never poll. Scale workers for slots; keep pollers small or autoscaled.

What I'd tell myself before go-live

  • History never calls a worker. Sync match costs PostgreSQL nothing; a backlog costs INSERT + SELECT + DELETE + dead tuples. Watch the sync-match ratio.
  • The partition is the unit of parallelism. One partition, one owner. Raise partitions only on the hot queue — read first when growing, write first when shrinking — and keep pollers across the fleet ≥ read partitions.
  • Is the wait on the shelf or at the desk? Slots at 0 means your workers, never Matching.
  • A SIGKILLed warm worker costs a 5–10 s stall plus replays everywhere else. pm2 kill_timeout longer than your shutdown grace; deploy off-peak, one at a time.
  • UpdateTaskQueue/s ≈ active partitions ÷ matching.updateAckInterval. Leave it at 1 minute, and do the arithmetic before listing suspects.
  • Pollers fetch, slots work. Size slots with Little's law and the downstream's capacity; never autoscale on CPU alone.
  • Two kinds of worker. The internal worker runs temporal-sys-* housekeeping, needs its own credentials when Frontend has auth, doesn't do retention deletes, and should have two replicas.

Next in the series: timeouts, retries and determinism — the start-to-close timer that quietly recovers every lost handout above, and the replay rules that make sticky caches possible.

Keep reading