> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockx.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment overview

> How BlockX runs in production: which processes exist, what they depend on, how configuration is released, how the image is built, and where the shutdown and observability entry points are.

BlockX runs as EC2 + systemd + ASG: systemd manages a foreground `nerdctl run` container for each Worker, and the broker inside the container starts executor sandboxes through the host's containerd. Production has two mutually isolated clusters (`block` and `bundle`, each with its own coordinator + worker fleet), plus a parallel sync-invoker fleet that doesn't go through a coordinator. This page is written for developers, to help you build a mental model of where the code runs and how it gets started; the full operational details are in `docs/deploy.md` and `docs/deploy/` in the blockx repository.

```mermaid theme={null}
flowchart LR
    client["Caller"]
    nlb["NLB"]

    subgraph blockCluster["block cluster"]
        blockCoord["blockx-coordinator<br/>cmd/coordinator"]
        blockWorkers["worker ASG<br/>cmd/worker (systemd + nerdctl)"]
    end

    subgraph bundleCluster["bundle cluster"]
        bundleCoord["blockx-bundle-coordinator<br/>cmd/bundle_coordinator"]
        bundleWorkers["bundle worker ASG<br/>cmd/bundle_worker"]
    end

    subgraph siFleet["sync-invoker fleet"]
        si["blockx-syncinvoker<br/>cmd/syncinvoker"]
    end

    etcd[("etcd<br/>/blockx/workers/<br/>/blockx/bundle-workers/")]
    deps["BlockDB / Meta / NodeRPC<br/>Glue + S3 (Iceberg)<br/>Usage Kafka"]
    redis[("Redis<br/>function code")]

    client -->|ReserveWorkerSlot| blockCoord
    client -. "routing pending upstream confirmation" .-> bundleCoord
    client -->|"SubmitTask / WatchTasks"| blockWorkers
    client --> nlb -->|Invoke| si

    blockCoord <-->|watch| etcd
    bundleCoord <-->|watch| etcd
    blockWorkers -->|"register / heartbeat"| etcd
    bundleWorkers -->|"register / heartbeat"| etcd
    blockCoord -->|RequestTaskSlot| blockWorkers
    bundleCoord -->|RequestTaskSlot| bundleWorkers

    blockWorkers --> deps
    bundleWorkers --> deps
    blockWorkers --> redis
    si --> deps
    si --> redis
```

The isolation between the two clusters comes only from the etcd registry prefix: the block coordinator watches only `/blockx/workers/`, the bundle coordinator watches only the bundle prefixes, and neither the request fields nor the gRPC protocol carry any "cluster" identifier. The bundle cluster being live doesn't mean traffic has been switched over; the upstream routing scheme is designed separately. The sync-invoker doesn't register with etcd and doesn't go through a coordinator; one `Invoke` is one synchronous function call (see [Sync Invoker](/en/components/sync-invoker) for details).

## Processes and binaries

The `Dockerfile` builds the five binaries in the table below, all in the same image, with the start command choosing the entry point. The four coordinator/worker entry points are "thin profiles": `cmd/*/main.go` only declares an `app.Profile`; shared assembly, config loading, and shutdown ordering live in `internal/coordinator/app` and `internal/worker/app`.

| Binary                      | Entry point              | Role                                                                                                                                                                                 | Main dependencies                                                                             | Default ports (code defaults)                         |
| --------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| `blockx-coordinator`        | `cmd/coordinator`        | Scheduling entry point of the block cluster; watches `/blockx/workers/`, reserves a Slot for each Task and forwards it                                                               | etcd; optional chaintable-log Kafka, OTLP                                                     | gRPC `:8080`; metrics off by default                  |
| `blockx-bundle-coordinator` | `cmd/bundle_coordinator` | Scheduling entry point of the bundle cluster; watches both `/blockx/bundle-workers/` and `/blockx/prod/lanes/default/bundle-workers/`; exposes a separate `BundleCoordinatorService` | Same as above                                                                                 | gRPC `:8080`; metrics off by default                  |
| `blockx-worker`             | `cmd/worker`             | block worker, full / compatibility profile: registers every Builder and Writer Plugin (including the bundle capabilities and devstub)                                                | etcd, BlockDB, NodeRPC, Meta, Glue/S3, Usage Kafka, containerd (sandbox mode), optional Redis | gRPC `:8081`; metrics off by default                  |
| `blockx-bundle-worker`      | `cmd/bundle_worker`      | bundle worker profile: registers only the `BundleScan` and `CallList` Builders and the `BundleWrite` and `TableUpserts` Plugins; `StreamBuild.Enabled` defaults to `true`            | Same as above                                                                                 | Same as above                                         |
| `blockx-syncinvoker`        | `cmd/syncinvoker`        | Synchronous invocation service: reuses the worker's executor pool and UDS adapter without running the task dispatcher                                                                | BlockDB, NodeRPC, Redis (function code); no dependency on etcd                                | gRPC `localhost:18080`; metrics/health off by default |

<Note>
  The ports in the table are the defaults in `internal/coordinator/app/config.go`, `internal/worker/app/config.go`, and `cmd/syncinvoker/config.go`. The EC2 systemd templates override them: both the worker and the sync-invoker use gRPC `0.0.0.0:9900` and metrics `0.0.0.0:9901` (see `deploy/env/*.env.example`). The metrics HTTP endpoint starts only when `BLOCKX_PROMETHEUS_LISTEN_ADDR` is set; the path is determined by `BLOCKX_PROMETHEUS_PATH` and defaults to `/metrics`.
</Note>

The only differences between the worker and the bundle worker are in `app.Profile`: `Deployment`, `Service`, `WorkerRegistryPrefix`, `Builders`, `Plugins`, and `TuneDefaults`. On EC2, the entry point is chosen through `WORKER_BINARY` in `worker.env` (`blockx-worker` or `blockx-bundle-worker`); `deploy/scripts/blockx-worker-start` accepts only these two values.

## Dependent services

| Service                     | Purpose                                                                                                                                                                                                     | Used by               |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| etcd                        | Worker registration and heartbeat (with lease TTL); the coordinator watches the prefix to get its capacity view                                                                                             | coordinator, worker   |
| BlockDB                     | On-chain data reads and writes; each service type has its own gRPC address (`BLOCKDB_*_ADDR`), and workers that register BundleWrite / TableUpserts must configure `BLOCKDB_BATCH_WRITE_ADDR`               | worker, sync-invoker  |
| Meta Service                | logical-types metadata (`META_ADDR`; a local stub is used when unset)                                                                                                                                       | worker                |
| Node RPC                    | JSON-RPC calls to chain nodes (`NODE_RPC_ENDPOINT`; a dev stub is used when unset)                                                                                                                          | worker, sync-invoker  |
| AWS Glue + S3               | Read-only parsing of Iceberg table metadata / manifests and bundle parquet reads; the worker uses the EC2 instance role and needs only `glue:GetTable` and `s3:GetObject` (plus `kms:Decrypt` with SSE-KMS) | worker, bundle worker |
| Usage Kafka                 | Compute usage ledger, topic defaults to `chaintable-usage`; fail-closed: the worker refuses to start when `BLOCKX_USAGE_BROKERS` is not configured and `BLOCKX_USAGE_DISABLED=true` is not set explicitly   | worker, bundle worker |
| Redis                       | Source of function code metadata (`FUNCTION_CODE_REDIS_URL` / `FUNCTION_CODE_REDIS_KEY`), adapter in `internal/functioncode/adapters/redis`; falls back to the devstub when unset                           | worker, sync-invoker  |
| containerd                  | With `EXECUTOR_SPAWN_MODE=sandbox`, the broker creates `blockx-executor-*` sandboxes through the host's containerd                                                                                          | worker, sync-invoker  |
| chaintable-log Kafka / OTLP | Optional: `CHAINTABLE_LOG_BROKERS` forwards slog records to Kafka; `OTEL_EXPORTER_OTLP_ENDPOINT` enables tracing                                                                                            | all                   |

BundleWrite and TableUpserts upload their data with a direct `PUT` to the presigned URL returned by BlockDB; they don't use the worker's AWS credential chain and don't need write permission on the target bucket. See `docs/deploy.md` §2 and §3.3 in the blockx repository for the IAM and address lists.

## How configuration is released

Application configuration and cloud resources live in two separate repositories:

* **The blockx repository** maintains only runtime artifacts: systemd units, host scripts, env examples, the health gate, and runbooks, all under `deploy/`.
* **`Chaintable/blockx-ec2-manifest`** maintains `worker.env` (and later `syncinvoker.env`); after merging to `main`, it is uploaded as an immutable S3 object keyed by Git SHA, SSM stores only the three-line `s3-v1` pointer (`BLOCKX_WORKER_ENV_FORMAT` / `BLOCKX_WORKER_ENV_S3_URI` / `BLOCKX_WORKER_ENV_SHA256`), and a State Manager association is then triggered.
* **`DeBankDeFi/SRE` (Terraform)** maintains the ASG, launch template, User Data, IAM, SG, NLB, Prometheus scrape, and deploy bundle URI. The blockx side doesn't change AWS resources directly.

Convergence path on an instance: User Data downloads a pinned version of the deploy bundle to `/opt/blockx-deploy` and calls `blockx-worker-install`; after that, State Manager periodically runs `blockx-worker-reconcile`, which `skip`s when the env is unchanged and otherwise delegates to install to reinstall, restart, and run the health gate. Validation failures fail closed and don't overwrite the local `/etc/blockx/worker.env`.

The `deploy/` directories:

| Path              | Contents                                                                                                                                                                                                                                                                                     |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deploy/systemd/` | `blockx-worker.service`, `blockx-syncinvoker.service`: `ExecStartPre` → prestart, `ExecStart` → start (foreground `nerdctl run`), `ExecStop` → stop, `ExecStopPost` → cleanup; worker `TimeoutStopSec=140`, sync-invoker `30`                                                                |
| `deploy/scripts/` | `blockx-worker-{prestart,start,stop,cleanup,health,install,reconcile,fetch-env,drain,profile}` and the corresponding `blockx-syncinvoker-*` (no drain/profile); `build-blockx-ec2-deploy-bundle` produces a deterministic `blockx-ec2-deploy-<sha>.tar.gz` from `deploy/` at a pinned commit |
| `deploy/env/`     | `worker.env.example`, `syncinvoker.env.example`: `KEY=value` files read by the host scripts and also passed to the container with `--env-file`; only simple `KEY=value` is allowed, with no `export`, quotes, or shell expressions                                                           |
| `deploy/tests/`   | bash unit tests for the scripts above (fetch-env, install, reconcile, health, start, profile, deploy bundle), run one by one by the `Deploy Script Tests` job in CI                                                                                                                          |

<Warning>
  After changing `deploy/scripts/*` or `deploy/systemd/*`, rebuild the deploy bundle and have SRE update the bundle URI; `reconcile` only converges the env and doesn't sync new scripts to existing instances.
</Warning>

## Image and CI

The `Dockerfile` is a two-stage build:

1. `go-builder` (`golang:1.26-bookworm`): compiles `blockx-coordinator`, `blockx-bundle-coordinator`, and `blockx-syncinvoker` (`CGO_ENABLED=0`) plus `blockx-worker` and `blockx-bundle-worker` (`CGO_ENABLED=1`, DuckDB needs cgo), targeting `linux/amd64`.
2. `runtime` (`python:3.12-slim`): `pip install /app/python` installs `blockx_executor` / `blockx_sdk` / `blockx_audit` and the private dependencies (`blockdb-py`, `blockx-py`, and so on), along with `py-spy`; the five Go binaries are then copied to `/usr/local/bin`. There is no separate venv: the Python packages go straight into the image's system site-packages, with `PYTHONPATH=/app/python`.

The GitHub token for private Go modules and Python packages is injected through a BuildKit secret (`--secret id=github_token`) and never enters an image layer.

`.github/workflows/`:

| Workflow    | Name          | Trigger                                         | What it does                                                                                                                                                   |
| ----------- | ------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `build.yml` | `Build image` | push of any tag, `workflow_dispatch`            | Runs `docker buildx build --push` on a self-hosted runner and pushes `294354037686.dkr.ecr.ap-northeast-1.amazonaws.com/chaintable/blockx:<tag>` and `:latest` |
| `test.yml`  | `CI`          | PR to `dev`, push to `dev`, `workflow_dispatch` | Deploy Script Tests, Go Tests, Python Tests, the sharded Worker Process E2E, Bundle Worker / Contract / System E2E, Perf E2E                                   |

Releasing an image means tagging and pushing (`git tag v1.2.3 && git push origin v1.2.3`). The EC2 side references only pinned tags or digests, never `latest`.

## Graceful shutdown

All processes handle `SIGTERM` / `SIGINT`. On a signal, the coordinator calls `cancel()` on the root context and `GracefulStop()` on the gRPC server. The worker's shutdown in `internal/worker/app/app.go` has two phases: first `SetDraining` and `Deregister` from etcd, then wait for in-flight tasks to finish (`DRAIN_TIMEOUT_MS`, code default 30s, EC2 template 90s); then `cancel()`, stop the executor pool, close the IO scope, close the WatchTasks subscriptions, and finally `GracefulStop` (falling back to `Stop` if it doesn't finish within 5s). ASG scale-in has no lifecycle hook; it relies on three layers of protection: the worker's built-in drain, systemd `ExecStop`, and the etcd lease TTL. See [Worker](/en/components/worker) for details.

## Observability entry points

| Entry point     | Location                                                                                                                                                               |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Structured logs | JSON to stderr (`internal/obs/logger.go`); under systemd use `journalctl -u blockx-worker`; with `CHAINTABLE_LOG_BROKERS` configured they're also forwarded to Kafka   |
| Prometheus      | Enabled by `BLOCKX_PROMETHEUS_LISTEN_ADDR`, path `BLOCKX_PROMETHEUS_PATH` (default `/metrics`); `:9901/metrics` on EC2                                                 |
| pprof           | The worker mounts `/debug/pprof/` on the same HTTP server (disable with `WORKER_DEBUG_PPROF=false`); the sync-invoker mounts `/readyz` and `/healthz` on the same port |
| File profiles   | `WORKER_CPU_PROFILE` / `WORKER_TRACE` and `COORDINATOR_CPU_PROFILE` / `COORDINATOR_TRACE` write to the given paths                                                     |
| Host sampling   | `deploy/scripts/blockx-worker-profile` captures Go / Python flame graphs with a temporary profiler container; `py-spy` doesn't need to be installed on the host        |
| Tracing         | `OTEL_EXPORTER_OTLP_ENDPOINT`; noop when unset                                                                                                                         |

See [Observability](/en/components/observability) for metric definitions and how to read the dashboards.

## Related docs

In the blockx repository:

* `docs/deploy.md` — deployment overview from the SRE perspective, full environment variable table, image addresses.
* `docs/deploy/ec2-automation-plan.md` — responsibility boundaries between blockx / blockx-ec2-manifest / SRE and the release path.
* `docs/deploy/worker-systemd-nerdctl.md` — worker EC2 runbook (install, env, acceptance, rollback).
* `docs/deploy/syncinvoker-systemd-nerdctl.md` — sync-invoker EC2 runbook.
* `docs/deploy/sync-invoker-sandbox-host.md` — verification record for the sync-invoker host containerd sandbox setup.
* `docs/deploy/sync-invoker-loadtest-2026-06-29.md` — sync-invoker capacity load test record.
* `docs/sync-invoker-grafana.md` — metric definitions for the sync-invoker Grafana dashboard.
* `docs/specs/2026-07-21-block-bundle-clusters.md` — design of the block / bundle dual-cluster split.
* `docs/specs/2026-07-28-registry-prefix-migration.md` — registry prefix v2 and `COORDINATOR_REGISTRY_PREFIXES`.
* `docs/specs/2026-07-30-ec2-worker-profiling.md` — performance sampling for EC2 workers without host dependencies.

On this site:

<Columns cols={2}>
  <Card title="Local development environment" href="/en/development/getting-started">
    The minimal steps to build and run locally.
  </Card>

  <Card title="Observability" href="/en/components/observability">
    Code locations for logs, metrics, tracing, and usage.
  </Card>

  <Card title="Worker" href="/en/components/worker">
    Worker assembly, drain, and shutdown ordering.
  </Card>

  <Card title="Bundle clusters" href="/en/components/bundle">
    bundle coordinator / worker and bundle-specific capabilities.
  </Card>
</Columns>