Skip to main content
BlockX runs as EC2 + systemd + ASG: systemd manages a foreground nerdctl run container for each Worker, and the broker inside the container starts executor sandboxes through the host’s containerd. Production has two mutually isolated clusters (block and bundle, each with its own coordinator + worker fleet), plus a parallel sync-invoker fleet that doesn’t go through a coordinator. This page is written for developers, to help you build a mental model of where the code runs and how it gets started; the full operational details are in docs/deploy.md and docs/deploy/ in the blockx repository. The isolation between the two clusters comes only from the etcd registry prefix: the block coordinator watches only /blockx/workers/, the bundle coordinator watches only the bundle prefixes, and neither the request fields nor the gRPC protocol carry any “cluster” identifier. The bundle cluster being live doesn’t mean traffic has been switched over; the upstream routing scheme is designed separately. The sync-invoker doesn’t register with etcd and doesn’t go through a coordinator; one Invoke is one synchronous function call (see Sync Invoker for details).

​
Processes and binaries

The Dockerfile builds the five binaries in the table below, all in the same image, with the start command choosing the entry point. The four coordinator/worker entry points are “thin profiles”: cmd/*/main.go only declares an app.Profile; shared assembly, config loading, and shutdown ordering live in internal/coordinator/app and internal/worker/app.
The ports in the table are the defaults in internal/coordinator/app/config.go, internal/worker/app/config.go, and cmd/syncinvoker/config.go. The EC2 systemd templates override them: both the worker and the sync-invoker use gRPC 0.0.0.0:9900 and metrics 0.0.0.0:9901 (see deploy/env/*.env.example). The metrics HTTP endpoint starts only when BLOCKX_PROMETHEUS_LISTEN_ADDR is set; the path is determined by BLOCKX_PROMETHEUS_PATH and defaults to /metrics.
The only differences between the worker and the bundle worker are in app.Profile: Deployment, Service, WorkerRegistryPrefix, Builders, Plugins, and TuneDefaults. On EC2, the entry point is chosen through WORKER_BINARY in worker.env (blockx-worker or blockx-bundle-worker); deploy/scripts/blockx-worker-start accepts only these two values.

​
Dependent services

BundleWrite and TableUpserts upload their data with a direct PUT to the presigned URL returned by BlockDB; they don’t use the worker’s AWS credential chain and don’t need write permission on the target bucket. See docs/deploy.md §2 and §3.3 in the blockx repository for the IAM and address lists.

​
How configuration is released

Application configuration and cloud resources live in two separate repositories:
  • The blockx repository maintains only runtime artifacts: systemd units, host scripts, env examples, the health gate, and runbooks, all under deploy/.
  • Chaintable/blockx-ec2-manifest maintains worker.env (and later syncinvoker.env); after merging to main, it is uploaded as an immutable S3 object keyed by Git SHA, SSM stores only the three-line s3-v1 pointer (BLOCKX_WORKER_ENV_FORMAT / BLOCKX_WORKER_ENV_S3_URI / BLOCKX_WORKER_ENV_SHA256), and a State Manager association is then triggered.
  • DeBankDeFi/SRE (Terraform) maintains the ASG, launch template, User Data, IAM, SG, NLB, Prometheus scrape, and deploy bundle URI. The blockx side doesn’t change AWS resources directly.
Convergence path on an instance: User Data downloads a pinned version of the deploy bundle to /opt/blockx-deploy and calls blockx-worker-install; after that, State Manager periodically runs blockx-worker-reconcile, which skips when the env is unchanged and otherwise delegates to install to reinstall, restart, and run the health gate. Validation failures fail closed and don’t overwrite the local /etc/blockx/worker.env. The deploy/ directories:
After changing deploy/scripts/* or deploy/systemd/*, rebuild the deploy bundle and have SRE update the bundle URI; reconcile only converges the env and doesn’t sync new scripts to existing instances.

​
Image and CI

The Dockerfile is a two-stage build:
  1. go-builder (golang:1.26-bookworm): compiles blockx-coordinator, blockx-bundle-coordinator, and blockx-syncinvoker (CGO_ENABLED=0) plus blockx-worker and blockx-bundle-worker (CGO_ENABLED=1, DuckDB needs cgo), targeting linux/amd64.
  2. runtime (python:3.12-slim): pip install /app/python installs blockx_executor / blockx_sdk / blockx_audit and the private dependencies (blockdb-py, blockx-py, and so on), along with py-spy; the five Go binaries are then copied to /usr/local/bin. There is no separate venv: the Python packages go straight into the image’s system site-packages, with PYTHONPATH=/app/python.
The GitHub token for private Go modules and Python packages is injected through a BuildKit secret (--secret id=github_token) and never enters an image layer. .github/workflows/: Releasing an image means tagging and pushing (git tag v1.2.3 && git push origin v1.2.3). The EC2 side references only pinned tags or digests, never latest.

​
Graceful shutdown

All processes handle SIGTERM / SIGINT. On a signal, the coordinator calls cancel() on the root context and GracefulStop() on the gRPC server. The worker’s shutdown in internal/worker/app/app.go has two phases: first SetDraining and Deregister from etcd, then wait for in-flight tasks to finish (DRAIN_TIMEOUT_MS, code default 30s, EC2 template 90s); then cancel(), stop the executor pool, close the IO scope, close the WatchTasks subscriptions, and finally GracefulStop (falling back to Stop if it doesn’t finish within 5s). ASG scale-in has no lifecycle hook; it relies on three layers of protection: the worker’s built-in drain, systemd ExecStop, and the etcd lease TTL. See Worker for details.

​
Observability entry points

See Observability for metric definitions and how to read the dashboards. In the blockx repository:
  • docs/deploy.md — deployment overview from the SRE perspective, full environment variable table, image addresses.
  • docs/deploy/ec2-automation-plan.md — responsibility boundaries between blockx / blockx-ec2-manifest / SRE and the release path.
  • docs/deploy/worker-systemd-nerdctl.md — worker EC2 runbook (install, env, acceptance, rollback).
  • docs/deploy/syncinvoker-systemd-nerdctl.md — sync-invoker EC2 runbook.
  • docs/deploy/sync-invoker-sandbox-host.md — verification record for the sync-invoker host containerd sandbox setup.
  • docs/deploy/sync-invoker-loadtest-2026-06-29.md — sync-invoker capacity load test record.
  • docs/sync-invoker-grafana.md — metric definitions for the sync-invoker Grafana dashboard.
  • docs/specs/2026-07-21-block-bundle-clusters.md — design of the block / bundle dual-cluster split.
  • docs/specs/2026-07-28-registry-prefix-migration.md — registry prefix v2 and COORDINATOR_REGISTRY_PREFIXES.
  • docs/specs/2026-07-30-ec2-worker-profiling.md — performance sampling for EC2 workers without host dependencies.
On this site:

Local development environment

The minimal steps to build and run locally.

Observability

Code locations for logs, metrics, tracing, and usage.

Worker

Worker assembly, drain, and shutdown ordering.

Bundle clusters

bundle coordinator / worker and bundle-specific capabilities.