onchain table -> onchain table.
BlockX is only responsible for executing computations. Task generation, DAG orchestration, trigger management, and scheduled jobs are all the Client’s responsibility.
This group of pages is for developers who want to contribute code to BlockX. If you only want to know where the code is and which directory to look at for a given change, go straight to Repository layout and code layering.
The system at a glance
1
The Client requests execution resources
The Client calls
ReserveWorkerSlot(taskId) on any Coordinator. The Coordinator picks a candidate Worker based on the Worker heartbeats in etcd, calls RequestTaskSlot on it, and hands workerAddr + slotId back to the Client. The Client can also bypass the Coordinator and connect to a Worker directly.2
The Client submits the task
The Client calls
SubmitTask(task, slotId?) on the target Worker. The Worker validates and activates the slot, creates the task context, and pins the function code snapshot this task will use.3
The Worker executes in three phases
In the Builder phase, Call Builders run serially and scan trigger data to produce the
call list; in the Calls phase, the dispatcher delivers calls in windows to the long-lived Python Executor process pool for parallel execution; in the Plugin phase, the Writer Plugin writes the aggregated outputs to BlockDB in one shot.4
The Client fetches the result
SubmitTask only returns a submission acknowledgement. The Client obtains the terminal TaskResult from the same Worker through GetTaskResult or the WatchTasks stream.Component diagram
Every box in the diagram corresponds to a component page.Sync Invoker reuses the Worker’s executor adapter, Pool Manager, Function Code View, and IO subsystem code, and starts an independent Executor pool in its own process, but does not go through the slot / task flow. The bundle cluster is deployed from the same code with a different process profile and etcd registration prefix, and is dedicated to large-scale bundle backfill tasks.
Components and responsibilities
Core constraints
These constraints run through the design of every component. If you find yourself breaking one of them while changing code, go back to the corresponding spec first.- Task-centric: the task is the smallest scheduling unit and runs on exactly one Worker; all caching and state sharing is anchored to the task and released when the task ends.
- All calls within a task run in parallel: calls have no ordering dependencies between them and share the same read-only world state at a single point in time.
- Function versions are fixed within a task:
taskCodeEpochis pinned when the task is activated, and all subsequent top-level calls and sub-function calls resolve against the same code snapshot; hot code updates only affect new tasks. - Python-side IO must be hijackable: every BlockDB / RPC request a user function makes through the SDK is hooked back to the Worker for unified handling; the Executor is never allowed to connect to external storage directly.
- Functions are read-only; writes go through Plugins: user functions cannot write to BlockDB; each task has at most one Writer Plugin, which writes everything in one shot during the Plugin phase.
- No task-level automatic retry, no cancellation: the framework only provides timeouts; whether to resubmit is up to the Client, based on
TaskResult.retryable. Inside the Worker, bounded attempt retries of individual calls are allowed. - The Worker’s local slot table is the single source of truth: the Coordinator only maintains a discardable derived view, and etcd only carries registration and heartbeats. When multiple Coordinators pick the same Worker at the same time, the conflict converges at the Worker’s atomic capacity check.
TaskResultonly expresses task-level results: it does not return the return value of each call; the only terminal states areSUCCEEDED / FAILED, and failures are expressed with an error code andretryable.
Three design principles
- Sans-IO: state machines, scheduling decisions, and protocol mapping are concentrated in the pure in-memory
core/; adapters take care of RPC, UDS, storage, clocks, and observability.WorkerCore,CoordinatorCore, andDispatcherCoreall follow the one-way pattern “input event → core decision → adapter executes commands”. - Scoped Resource Context: subsystems such as IO access, which revolve around resource ownership and lifecycle reclamation, are organized into a Worker-level shared scope and task-level local scopes, binding quota, cache, singleflight, and deadline to the scope lifecycle.
- Occam’s Razor: do not multiply entities beyond necessity. Prefer reusing existing concepts, states, interfaces, error codes, and protocols.
When not to use BlockX
BlockX is not suited to the following computations:- Computations that depend on a large number of rows from the same table, such as wide-range aggregation, complex window statistics, or candlestick (K-line) computation.
- Scenarios that need arbitrary time-window state or large-scale stateful computation.
- Complex computations that depend heavily on a streaming engine’s exactly-once state recovery.
- Arbitrary incremental streaming computation over ordinary online tables.
Suggested reading order
1
Start with the main path
Task lifecycle and Protocols and interfaces. After reading them you should be able to say which RPCs, phases, and states a task goes through.
2
Then see how the repository is layered
3
Dive into the component you need to change
Use the component table above to reach the corresponding page. Every component page lists code locations, core types, extension points, and tests.
4
Read the development guides before you start
When you change behavioral semantics, update the design documents under
docs/specs/ in the BlockX repository first, then the code, and put both in the same PR.