Task lifecycle

A task is the smallest scheduling and execution unit in BlockX: the Client generates a taskId, submits a TaskInput to a Worker, and the Worker runs the three phases Builder → Calls → Plugin locally and converges on a TaskResult. This page covers only that main path; for component details see Worker, Coordinator, Call execution subsystem, and Plugin system; for the overall architecture see Architecture overview. The main path relies on the following constraints (see also docs/specs/architecture.md §2.1 in the blockx repo):

Task-centric: shared state, read cache, and timeouts are all anchored to the task.
Single-Worker execution: a task runs on exactly one Worker. The Worker is unaware of the Coordinator and only exposes RequestTaskSlot / SubmitTask.
All calls within a task run in parallel: there are no ordering dependencies between calls; the dispatcher ramps them up gradually in windows.
Fixed function version per task: on activation, a taskCodeEpoch (TaskContext.TaskCodeEpoch) is pinned; code updates during that time only affect subsequent tasks.
Read-only functions: user functions never write to the database directly; all writes go through a Writer Plugin.
No task-level automatic retries: the Worker only does bounded attempt retries for individual calls (CallRetryPolicy); whether to resubmit a task is decided by the Client based on TaskResult.executeResult.retryable.
No cancellation: only task timeouts (taskTimeoutMs) and optional Watch-disconnect reclamation.

Task model

The structure submitted externally is the gRPC TaskInput (api/grpc/worker/worker.proto); the Go counterpart is commontypes.TaskInput (internal/common/types/types.go):

Field	Type	Description
`task_id`	`string`	A stable unique ID generated by the Client (uuid4 recommended). The idempotency key.
`function_call_config`	`{type, config}`	`type` selects the Call Builder; `config` is opaque JSON parsed by the corresponding Builder.
`result_handler`	`{type, config}`	Optional. `type` selects the Writer Plugin. At most one per task.
`task_timeout_ms`	`int64`	Optional. `0` uses the Worker default; negative values are rejected; values above `maxTaskTimeoutMs` are also rejected.

Currently registered Builder types (internal/plugin/callbuilder/types/types.go): BlockTableCallConfig (dbScan), InputsCallConfig, BlockBundleCallConfig (bundleScan). Currently registered Writer Plugin types (internal/plugin/event/types/types.go): BlockTableWriteHandler, ReturnValueHandler, NormalTableWriteHandler. A minimal JSON example (assembled from the fields of commontypes.TaskInput and call_list.CallListCallConfigBuilderDecl; not a fixture from the repo):

{
  "task_id": "3f2b1c9e-8a4d-4c1e-9b0a-6f7e5d4c3b2a",
  "functionCallConfig": {
    "type": "InputsCallConfig",
    "config": {
      "function": "hello_world",
      "callList": [["ct2"], ["blockx"]]
    }
  },
  "resultHandler": {
    "type": "ReturnValueHandler",
    "config": {}
  },
  "taskTimeoutMs": 30000
}

TaskInput carries no block context at the top level. Block information is carried by the specific Builder’s config (for example dbscan.DBScanBuilderDecl.Block) and then written into each call by the Builder. When the Worker receives a request, submitTaskFromPB in internal/worker/adapters/wire.go converts the proto into commontypes.SubmitTaskRequest (also validating that config is valid JSON and task_timeout_ms >= 0); parseTaskPayload in internal/worker/adapters/payload.go then wraps it into commontypes.TaskCtx:

// TaskCtx carries the immutable context injected when a task is activated.
type TaskCtx struct {
	TaskInput

	Stage Stage `json:"stage"`
	DebugMode bool `json:"debugMode,omitempty"`
	Deadline time.Time          `json:"deadline"`
	Ctx      context.Context    `json:"-"`
	Cancel   context.CancelFunc `json:"-"`
}

TaskCtx is the minimal context shared by the Builder, Writer Plugin, and IO scope; Stage is rewritten by the adapter to builder / executor / plugin as phases advance. WorkerCore only hangs it on TaskContext.AdapterTaskCtx as an opaque pointer.

End-to-end sequence

The 16 steps of the main path are laid out below (see also docs/specs/architecture.md §3).

Client builds the task

The Client generates a task_id and prepares a TaskInput. BlockX is not responsible for task generation, DAG orchestration, or scheduled triggering.

Client asks the Coordinator for placement (optional)

Call CoordinatorService.ReserveWorkerSlot(task_id) (api/grpc/coordinator/coordinator.proto). The Coordinator rebuilds its Worker view from the heartbeats it watches in etcd (CoordinatorCore.ApplyWorkerUpdate) and picks candidates with SelectCandidate. Without going through the Coordinator, the Client can call RequestTaskSlot on a Worker directly, or call SubmitTask directly and let the Worker request the slot automatically.

Coordinator requests a slot from the Worker

CoordinatorServer.ReserveWorkerSlot (internal/coordinator/adapters/grpc_server.go) tries candidate Workers sequentially, calling WorkerService.RequestTaskSlot(task_id, ttl_ms) on each, with ttl_ms taken from CoordinatorCore.SlotTTLMs(). On rejection it moves to the next candidate; an RPC timeout counts as “possibly allocated” and is retried a bounded number of times against the same Worker with the same task_id.

Worker allocates a slot atomically

WorkerCore.HandleRequestTaskSlot (internal/worker/core/worker.go) checks freeCapacity() under o.mu, switches a FREE slot to ALLOCATED, writes LeaseDeadlineMs = now + ttlMs, and returns slot_id and state=ALLOCATED.

Coordinator returns worker_addr + slot_id

The Coordinator stores no task payload, stores no results, and writes no slot truth to etcd; slot_id is just an opaque string to it.

Client submits SubmitTask

The Client connects directly to worker_addr and calls WorkerService.SubmitTask(task, slot_id). Orchestrator.SubmitTask (internal/worker/adapters/orchestrator.go) first computes the effective deadline (WorkerCore.EffectiveTaskDeadlineMs) and parses the payload, then hands off to WorkerCore.HandleSubmitTask.

Worker validates and activates

HandleSubmitTask does the following in order: ready/draining check; if the same taskId is already in w.tasks, return RUNNING, and if a retained result exists, return the terminal state; if no slot_id was provided, run HandleRequestTaskSlot inline once; verify that the slot exists, is ALLOCATED, and is bound to the same taskId; once that passes, create a TaskContext (Phase = Preparing) and return the PrepareTaskActivation command. The gRPC layer returns state=RUNNING at this point.

Adapter completes activation

Orchestrator.activateAndRun (internal/worker/adapters/orchestrator_phases.go) resolves the current taskCodeEpoch in a goroutine, pins a Function Code View snapshot, then feeds back WorkerCore.HandleTaskActivationPrepared: the slot switches to RUNNING, Phase advances to Builder, and StartBuilderPhase is emitted. It then creates the TaskIOScope (WorkerIOScope.NewTaskScope) and the adapter-side taskRuntime. If any step fails, it goes through HandleTaskActivationFailed with failureCode ACTIVATION_FAILED or IO_SCOPE_FAILED.

Builder phase generates the call list

runBuilderPhase first acquires builderSem, looks up the Builder in BuilderRegistry by FunctionCallConfig.Type, and calls CallBuilder.Build(ctx, taskCtx, io) to get a cbtypes.CallList. The result is handed back to the core via HandleTaskPhaseFinished(PhaseBuilder, ...), and the core emits StartCallPhase.

Calls phase executes in parallel

runCallPhase acquires executorSem and calls DispatcherCore.PrepareCallPhaseWithDigests to set up TaskDispatchState; the dispatch loop then sends calls to the Python Executor over UDS via ExecuteCall in windows. SDK IO requests inside the Executor come back to the Worker as CallWaiting, are handled by the IO scope, and then ResumeCall; subfunction calls likewise return to the dispatcher for rebinding. See Call execution subsystem and IO access subsystem for details.

Converge outputs

Once all root calls reach a terminal state, DispatcherCore.checkConvergeTask produces ConvergeTaskCalls{Success, FailureCode, Outputs}; Outputs contains only the return values of successful root calls, and subcall return values are not included. The adapter converts it into a PhaseOutcome and calls HandleTaskPhaseFinished(PhaseCalls, ...).

Plugin phase writes results

The core emits StartWriterPhase{Outputs}; runWriterPhase acquires pluginSem, and if ResultHandler is non-empty, looks up the plugin in PluginRegistry and calls WriterPlugin.Execute(ctx, taskCtx, outputs, io), producing []PluginResult.

Converge to the terminal state

HandleTaskPhaseFinished(PhasePlugin, ...) calls convergeTask: Phase = Terminal, taskIndex switches from ActiveSlotID to Result, RetainedUntil = now + ResultRetentionMs, and within the same critical section releaseSlot returns the slot to FREE, removes the task from w.tasks, and emits ConvergeTask. The adapter’s executePhaseResult cancels unfinished calls, closes the TaskIOScope, deletes the taskRuntime, and finally triggers OnTaskTerminal.

Client fetches the result

SubmitTask does not return a TaskResult. The Client receives the terminal TaskUpdate via a WatchTasks stream, or polls GetTaskResult. OnTaskTerminal is wired to SubscriptionManager.NotifyTerminal (internal/worker/adapters/stream_subscriber.go), which pushes the terminal state once to every stream subscribed to that taskId.

Heartbeat reflects capacity

WorkerCore.DeriveHeartbeat derives reservedSlots / runningTasks from the slot table, and EtcdPublisher.RunHeartbeatLoop writes to etcd periodically; the Coordinator updates its view from this.

Slots and admission

RequestTaskSlot and SubmitTask are two steps of the same Worker protocol; whether the caller is the Coordinator or the Client makes no difference to the Worker.

RequestTaskSlot(task_id, ttl_ms) only reserves; it does not execute. ttl_ms must be positive (<= 0 returns InvalidArgument) and only controls automatic reclamation of ALLOCATED slots; it is unrelated to the task timeout. TickTimers scans ExpiredSlots every 500ms; expired slots are reclaimed by HandleSlotLeaseExpired, which also clears task records still in Preparing.
SubmitTask(task, slot_id?): when slot_id is provided it must match a record that is ALLOCATED and bound to the same taskId, otherwise InvalidSlot; when omitted, the Worker requests a slot inline once using DefaultTTLMs, returning NoSlot if there is no capacity.
Idempotency: while the same taskId is ALLOCATED / RUNNING, RequestTaskSlot returns the same slot_id; once it has entered RUNNING or its result is still within the retention window, SubmitTask returns the stable current state and never starts another execution. After the old execution finishes and its result expires, the same taskId can be allocated a slot again.
There is no release interface: if you obtain a slot and then abandon the submission, you can only wait for the TTL to reclaim it.

The boundary between admission errors and execution terminal states (see also the end of docs/specs/architecture.md §4.3.1): every failure before the slot switches from ALLOCATED to RUNNING is returned as a gRPC status and never enters TaskResult; once HandleSubmitTask has created a TaskContext, subsequent activation failures, Builder / Calls / Plugin failures, and timeouts all converge into a TaskResult terminal state.

Error code (`commontypes.ErrorCode`)	gRPC code (`workerCodeToGRPC`)	When it occurs
`InvalidArgument`	`InvalidArgument`	`taskId` is empty, `ttl_ms <= 0`, `task_timeout_ms` is invalid, `config` is not valid JSON
`NoSlot`	`ResourceExhausted`	No free slot locally
`InvalidSlot`	`FailedPrecondition`	`slot_id` does not exist, is not `ALLOCATED`, or is bound to a different `taskId`
`Unavailable`	`Unavailable`	Worker is not ready (no Executor heartbeat yet) or is draining
`NotFound`	`NotFound`	`GetTaskResult` cannot find that `taskId`

For details on the slot table, taskIndex, and heartbeat derivation see Worker; for the Coordinator’s candidate selection, circuit breaking, and uncertain-state retries see Coordinator.

The three phases

WorkerCore only decides whether to enter the next phase and when to write the terminal state; actual execution happens in the adapter. Phases advance via command / event round trips: the core returns StartBuilderPhase → the adapter executes → it feeds back HandleTaskPhaseFinished(PhaseBuilder, outcome) → the core returns StartCallPhase, and so on (internal/worker/core/types.go defines all command types).

Phase	Concurrency	Phase-level admission (`core.AdmissionConfig`, defaults)	Effect of failure on the task
Builder	Runs serially once; IO carries a Builder retry scope, and the IO core does bounded retries on retryable errors	`builderSem` (`BuilderSlots = 16`)	Immediately `FAILED`, `BUILDER_FAILED` / `BUILDER_NOT_FOUND`, `retryable` taken from the IO error
Calls	Fully parallel; per-task concurrent in-flight cap `TaskMaxInflightCalls = 1024`; bounded attempt retries per call	`executorSem` (`ExecutorTaskSlots = 32`; an empty call list does not take one)	`CallFailurePolicy` currently only has `fastFail`: as soon as any root call fails terminally, dispatching stops, outputs are discarded, and the task converges to `CALL_FAILED` (`retryable=false`); output exceeding `MaxCollectedOutputBytes` is `OUTPUT_BYTES_EXCEEDED`
Plugin	Runs serially once; IO carries a Result retry scope	`pluginSem` (`PluginSlots = 16`)	`FAILED`, `PLUGIN_FAILED` / `PLUGIN_NOT_FOUND`, `retryable` decided by `PluginError.Retryable` (unclassified errors default to retryable)

Other points:

If the task ctx is cancelled while a phase is waiting for admission (for example, it has already converged due to timeout), the phase fails with CANCELLED; before entering Calls, the deadline is also checked first, and if it has already expired the task converges directly as TIMED_OUT without taking executorSem.
After the Calls phase succeeds, the core always emits StartWriterPhase; when ResultHandler is empty, runWriterPhase still acquires pluginSem but does not execute a plugin, and pluginResults is empty.
Streaming Builders (StreamingCallBuilder, only for BlockBundleCallConfig, StreamBuildConfig.Enabled off by default) shrink the Builder phase to PrepareStream and move the scan into the Calls phase, constrained by scanSem (ScanSlots = 8); a failure midway through production means some calls have already executed.
Timeouts are handled by TickTimers scanning TimedOutTasks and calling HandleTaskTimedOut, which forces convergence to TIMED_OUT (retryable=true) without requiring immediate interruption of calls running in the Executor; in-flight calls receive CancelCall.

A task has only four externally visible states (commontypes.TaskState); the internal phases (core.TaskPhase: Preparing / Builder / Calls / Plugin / Terminal) are all folded into RUNNING:

Results and queries

The Worker core converges the task and records its terminal result. Its result index retains that result independently of slot reuse. Worker adapters serve the same result through unary queries and Watch updates. HandleWatchDetached starts a deadline only when disconnect handling is enabled and a Watch was attached. The timer can converge the task with WATCH_DISCONNECTED; the default disables this behavior. The caller contract and completion boundaries are defined in Execution semantics and results. Full result fields and errors are in the reference.

Timing semantics quick reference

Name	Default	Meaning	Defined in
slot TTL (Coordinator side)	`SlotTTLMs = 5000`	The `ttl_ms` the Coordinator passes when calling `RequestTaskSlot`; env `COORDINATOR_SLOT_TTL_MS`	`internal/coordinator/core/coordinator.go`
slot TTL (Worker inline)	`DefaultTTLMs = 5000`	TTL used for the automatic request when `SubmitTask` has no `slot_id`	`internal/worker/core/config.go`
task timeout default	`TaskDeadlineMs = 300000`	Effective timeout when `task_timeout_ms = 0`, counted from when the Worker accepts the submission; env `TASK_DEADLINE_MS`	`internal/worker/core/config.go`
task timeout cap	`MaxTaskTimeoutMs = 8h` (`DefaultMaxTaskTimeoutMs`)	Exceeding it is `InvalidArgument`; the Function Code View snapshot retention window must be no shorter than this; env `MAX_TASK_TIMEOUT_MS`	`internal/worker/core/config.go`, `internal/worker/app/config.go`
Result retention	`ResultRetentionMs = 300000`	Window during which a terminal result is queryable / can be hit idempotently; env `RESULT_RETENTION_MS`	`internal/worker/core/config.go`
Heartbeat interval	`HeartbeatIntervalMs = 200`	How often the Worker writes to etcd; 2000 recommended in production, env `WORKER_HEARTBEAT_INTERVAL`	`internal/worker/app/config.go`
Heartbeat timeout (Coordinator)	`HeartbeatTimeoutMs = 10000`	Beyond this the Worker is considered lost and no longer routed to	`internal/coordinator/core/coordinator.go`
Watch disconnect grace	`WatchDisconnectGraceMs = 0` (disabled)	Window from the last Watch disconnecting to ruling `WATCH_DISCONNECTED`; 120000 recommended in production, env `WATCH_DISCONNECT_GRACE_MS`	`internal/worker/core/config.go`
gRPC keepalive	`GRPCKeepaliveTimeMs = 30000` / `GRPCKeepaliveTimeoutMs = 10000`	Determines how long before a silent disconnect is confirmed, which affects when the Watch grace starts	`internal/worker/app/config.go`
Timer scan period	500ms	`TickTimers`: slot expiry, task timeout, Watch grace, result cleanup	`internal/worker/app/app.go`
Per-call execution budget	`CallDeadlineMs = 5000`	Counts only CPU + IO backend time, excluding queueing; independent of the task timeout	`internal/worker/core/config.go`

WorkerCore.EffectiveTaskDeadlineMs is where these values come together: a negative task_timeout_ms or one exceeding MaxTaskTimeoutMs returns InvalidArgument directly, and 0 takes TaskDeadlineMs.

Unsuitable scenarios

BlockX’s task model targets single-row state, lightweight functions, and one-shot writes; the following scenarios are not a good fit (docs/specs/architecture.md §5):

Computations that depend on a large number of rows in the same table: wide-range aggregations, window statistics, candlestick (K-line) charts.
Anything requiring arbitrary time-window state or large-scale stateful computation.
Strong dependence on a streaming engine’s exactly-once state recovery.
Arbitrary incremental streaming computation over ordinary online tables.

For streaming needs on non-onchain tables, a compromise is to have the Writer Plugin write to both the online table and Kafka, and have the business side listen to the topic and submit new tasks as needed. Specs in the blockx repo:

docs/specs/architecture.md: §2 task model, §3 main path, §4.3 results and protocols, §5 unsuitable scenarios.
docs/specs/worker.md: §2 external protocol, §3 local state model, §4 execution flow, §8 failure semantics.
docs/specs/task-resource-coordinator.md: placement, uncertain-state retries, circuit breaking.
docs/specs/call-execution-subsystem.md: dispatcher, Executor, subfunction calls.
docs/specs/plugin-system.md: declaration structures for Call Builders and Writer Plugins.

Site pages:

Architecture overview: overall system architecture.
Protocols and interfaces: gRPC / UDS / etcd protocol details.
Worker: slot table, WorkerCore events and commands, configuration.
Coordinator: the ReserveWorkerSlot processing chain.
Call execution subsystem: inside the Calls phase.
Plugin system: Builder and Writer Plugin extensions.
IO access subsystem: TaskIOScope, retry scopes, backend admission.

Get started

Core concepts

Guides

Reference

Architecture

Components

Development guide

Task model

End-to-end sequence

Slots and admission

The three phases

Results and queries

Timing semantics quick reference

Unsuitable scenarios

​Task model

​End-to-end sequence

​Slots and admission

​The three phases

​Results and queries

​Timing semantics quick reference

​Unsuitable scenarios

​Related docs

Task model

End-to-end sequence

Slots and admission

The three phases

Results and queries

Timing semantics quick reference

Unsuitable scenarios

Related docs