Local development environment

This page tells you what you need to get from cloning the repository to running a worker on your machine. BlockX is a Go workspace; the python/ directory holds the Python executor and SDK. See Architecture overview for the overall architecture and Repository layout and code layering for the directory layering.

Prerequisites

Dependency	Version / notes	Source
Go	`go.mod` declares `go 1.26.4`; the `Dockerfile` builds with `golang:1.26-bookworm`	`go.mod`, `Dockerfile`
C toolchain (CGO)	`cmd/worker` and `cmd/bundle_worker` depend on `github.com/duckdb/duckdb-go-bindings`, so you need `CGO_ENABLED=1` (the default) and `gcc` or `clang`. coordinator, bundle_coordinator, and syncinvoker can build with `CGO_ENABLED=0`	`go.mod`, `Dockerfile`, `internal/duckdb/borrowed/`
Python	`python/pyproject.toml` declares `requires-python = ">=3.12"`; the runtime image uses `python:3.12-slim`	`python/pyproject.toml`, `Dockerfile`
`uv`	Manages `python/.venv/`; both the E2E tests and the local worker start the executor from this venv	`AGENTS.md`, `internal/testutil/paths.go`
`protoc` + `protoc-gen-go` + `protoc-gen-go-grpc`	Only needed to regenerate after changing `.proto` files; the generated files are committed	`Makefile`
Access to private GitHub repositories	`go.mod` depends on private `github.com/Chaintable/*` modules, and `pyproject.toml` depends on private git packages such as `blockx-py` and `blockdb-py`. Your machine must be able to reach these repositories through git	`go.mod`, `python/pyproject.toml`, `.github/workflows/test.yml`

Every external service has a stand-in for local development, so you don’t need to set up etcd, BlockDB, Meta, or a chain node first:

Production dependency	Local substitute
etcd	Set `ETCD_ENDPOINTS=none` to skip registration (`registerEtcdPublisherAfterReady` in `internal/worker/app/app.go` returns immediately on an empty value or `none`). E2E tests use `StartEmbeddedEtcd` in `internal/testutil/etcd.go` to start `go.etcd.io/etcd/server/v3/embed` in-process
BlockDB	When `STUB_BLOCKDB_DELAY_MS` is set (`0` is valid), the worker uses `DelayedBackendAdapter` from `internal/worker/devstub`; when neither any `BLOCKDB_*_ADDR` nor that variable is set, it falls back to `StubBackendAdapter` (`blockDBStubDelayFromEnv` and `blockdbBackend` in `internal/worker/app/app.go`). `cmd/blockdb_grpc_stub` is a standalone gRPC stub process for benchmarks; tests also have `StartMockFunctionBlockDB` in `internal/testutil/blockdb.go`
Meta (logical-types)	When `META_ADDR` is unset, a local stub is used; you can feed it schemas with `STUB_LOGICAL_TYPES_SCHEMAS` (`stubLogicalTypesSchemas` in `app.go`)
Node RPC	When `NODE_RPC_ENDPOINT` is unset, a dev stub is used (`docs/deploy.md` §9.2)
Function code storage	When `FUNCTION_CODE_REDIS_URL` is not configured and the BlockDB table read/subscribe addresses are incomplete, `setupFunctionCodeView` in `internal/worker/app/function_code.go` picks `devstub.MemoryFunctionCodeStore`; `FUNC_DEBUG=true` forces the devstub
Usage Kafka	`BLOCKX_USAGE_DISABLED=true`. If unset, the worker probes the broker at startup and refuses to start when the probe fails (fail-closed)
Capability backend example	`internal/sdk/localtestservice` is an in-process sample capability backend; `examples/local_test_service/` shows how to hit a local worker with blockx-py

Quick start

Clone and download Go dependencies

git clone git@github.com:Chaintable/blockx.git
cd blockx
go mod download        # or make deps (additionally runs go mod tidy)

If private module downloads fail, first confirm that git can reach github.com/Chaintable/*; CI solves this with GOPRIVATE=github.com/Chaintable/* and token injection.

Install Python dependencies

uv sync --project python

This creates the venv in python/.venv/. pythonBin() in internal/testutil/paths.go looks for /tmp/blockx-venv/bin/python and then python/.venv/bin/python, falling back to python3 only when neither exists. When the venv is missing, all process E2E tests are skipped rather than failed.

Build and run unit tests

go build ./...
go test ./...

The first build of cmd/worker links the DuckDB static library and is much slower than the other packages; later builds hit the build cache. go test ./... also runs the process / contract / system E2E tests (they don’t check -short), which need the venv from the previous step; e2e/perf is skipped by TestMain by default. To run only pure unit tests, use go test ./internal/... ./api/....

Run the Python unit tests

PYTHONPATH=python uv run --project python python -m unittest discover -s python/tests

Start a worker on your machine

The following set of variables comes from StartWorkerAt in internal/testutil/worker.go; it is the minimal set the process E2E tests use to start a real worker:

WORKER_LISTEN=127.0.0.1:9221 \
WORKER_ADDR=127.0.0.1:9221 \
EXECUTOR_COUNT=1 \
PYTHON_BIN=$PWD/python/.venv/bin/python \
PYTHONPATH=$PWD/python \
ETCD_ENDPOINTS=none \
ICEBERG_NAMESPACE=chaintable_test \
BLOCKX_USAGE_DISABLED=true \
BLOCKDB_BATCH_WRITE_ADDR=127.0.0.1:1 \
STUB_BLOCKDB_DELAY_MS=0 \
go run ./cmd/worker

You’re up when the log shows worker listening and executor started. What each item means:

ETCD_ENDPOINTS=none: don’t register with etcd, so no Coordinator is involved; clients call RequestTaskSlot / SubmitTask on the worker directly.
ICEBERG_NAMESPACE: WorkerFullConfig.Validate requires it to be non-empty (internal/worker/app/config.go).
BLOCKDB_BATCH_WRITE_ADDR: the block profile in cmd/worker registers the BundleWrite plugin, and validateProfileRuntimeConfig requires this address to be non-empty; locally, any unreachable port will do.
STUB_BLOCKDB_DELAY_MS=0: once any BLOCKDB_*_ADDR is set, the worker treats BlockDB as real; this variable switches the read path back to the devstub.

To submit a task and verify the whole path, follow examples/local_test_service/README.md to start local_proxy.py and then run submit.py. You can also run a single process E2E test to see a real call:

go test -v -timeout 300s ./cmd/worker/ -run TestWorkerProcess_RequestSlotAndSubmit

The worker must be started with the set of variables above. Without ICEBERG_NAMESPACE or BLOCKDB_BATCH_WRITE_ADDR, the process exits during config validation.

Makefile targets

Target	What it does
`make` / `make all`	Same as `make test`
`make fmt`	Checks formatting: fails if `go fmt ./...` produces output and lists `gofmt -s -l .`
`make vet`	`go vet ./...`
`make test`	Runs `fmt` and `vet` first, then `go test -coverprofile=coverage.out -short -timeout $(TEST_TIMEOUT)` (default `600s`) on every package with tests except `cmd/syncinvoker`, and finally runs `cmd/syncinvoker/*_test.go` file by file (`SYNCINVOKER_TEST_TIMEOUT`, default `3m`), merges the coverage, and prints the total and the top 10 packages
`make test-cover`	`go test ./... -v -cover -short`
`make test-bench`	`go test ./... -bench=. -benchmem -short`
`make proto`	Runs `proto-blockdb`, `proto-meta`, and `proto-localtestservice` in order, then generates the pb files for `api/grpc/{worker,coordinator,bundlecoordinator,syncinvoker}`
`make proto-blockdb` / `proto-meta` / `proto-localtestservice`	Generate `internal/sdk/{blockdb,meta,localtestservice}/gen/` respectively
`make deps`	`go mod download` + `go mod tidy`
`make clean`	Deletes `coverage.out` and `coverage.html`, and runs `go clean -cache -testcache`
`make build`	Runs `go build ./bin/...`. To build the binaries, use `go build ./cmd/...`, or `go build -o` one at a time as in the `Dockerfile`

Environment variables and config files

The worker’s load order lives in internal/worker/app/app.go: DefaultWorkerFullConfig() → the profile’s TuneDefaults → the JSON file pointed to by WORKER_CONFIG (LoadWorkerFullConfigFrom) → environment variable overrides. JSON field names match the json tags of WorkerFullConfig; see docs/deploy.md §9.3 in the blockx repository for an example. The coordinator reads only environment variables and has no config file (LoadCoordinatorRuntimeConfigFromEnv in internal/coordinator/app/config.go). The variables you’ll run into most often in local development:

Variable	Default	Description
`WORKER_LISTEN` / `WORKER_ADDR`	`:8081` / auto-detected egress IP	Listen address and the external address registered in etcd
`ETCD_ENDPOINTS`	`localhost:2379`	Shared by the worker and the coordinator; on the worker side `none` means don’t register, on the coordinator side it must be non-empty (`Validate` reports `etcdEndpoints must not be empty`)
`EXECUTOR_COUNT`	`8`	Number of Python executor child processes
`PYTHON_BIN` / `PYTHONPATH`	`python3` / —	Executor interpreter and module search path; locally point them at `python/.venv/bin/python` and `python/`
`TASK_SLOTS`	`50` (`core.DefaultConfig`)	Maximum concurrent tasks per worker
`TASK_DEADLINE_MS`	`300000`	Task timeout when the user doesn’t specify one
`EXECUTOR_SPAWN_MODE`	`process`	`sandbox` uses the containerd sandbox; keep `process` for local development
`BLOCKX_USAGE_DISABLED`	`false`	Must be `true` locally, otherwise the worker probes Kafka at startup
`STUB_BLOCKDB_DELAY_MS` / `META_ADDR` / `NODE_RPC_ENDPOINT`	—	Control whether BlockDB / Meta / NodeRPC use real endpoints or stubs
`COORDINATOR_LISTEN`	`:8080`	coordinator gRPC listen address
`COORDINATOR_REGISTRY_PREFIXES`	profile default	etcd prefixes the coordinator watches, comma-separated
`OTEL_EXPORTER_OTLP_ENDPOINT`	—	Trace export is disabled when unset

For the full table (dozens of entries covering AIMD, gRPC keepalive, the Iceberg cache, and more), see docs/deploy.md §9 in the blockx repository; deploy/env/worker.env.example is a sample env file for EC2 deployment that contains many keys read only by the host scripts, so it isn’t suitable for running locally as-is.

Code generation

Regenerate the Go bindings after changing .proto files:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
make proto          # everything
make proto-blockdb  # only generate internal/sdk/blockdb/gen/

The Makefile adds $(go env GOBIN) (or $(go env GOPATH)/bin) to PATH, so installing the plugins in the default location is enough. Proto source locations: api/grpc/*/ and internal/sdk/{blockdb,meta,localtestservice}/proto/. Commit the generated output together with the change.

Common issues

All E2E tests show SKIP with Python dependencies not available

RequirePythonModules in internal/testutil/python.go tries an import with the detected interpreter and calls t.Skip on failure. The process E2E tests in cmd/worker require all seven modules greenlet, blockx_sdk, blockx_executor, blockdb, blockx, leafage, chaintable to be importable (cmd/worker/worker_process_helpers_test.go); the last four come from the private git dependencies in pyproject.toml. Run uv sync --project python first and confirm that python/.venv/bin/python exists.

go build ./cmd/worker fails with build constraints exclude all Go files in duckdb-go-bindings/lib

This is a symptom of CGO_ENABLED=0. worker and bundle_worker must have CGO enabled, and the machine needs a C compiler. A slow first link is normal.

The worker exits immediately: invalid worker config

Read the error string: for icebergNamespace must not be empty, add ICEBERG_NAMESPACE; for plugin BlockBundleWriteResultHandler requires blockdb.batchWriteAddr, add BLOCKDB_BATCH_WRITE_ADDR; for usage-related errors, add BLOCKX_USAGE_DISABLED=true.

The worker is running but the Coordinator can't see it

Check that ETCD_ENDPOINTS points to the same etcd and that WORKER_ADDR is an address the Coordinator can reach; with container network isolation it must be set explicitly (docs/deploy.md §7). For single-machine local debugging, you can bypass the Coordinator with ETCD_ENDPOINTS=none.

The Python executor child processes don't start

Confirm that the interpreter PYTHON_BIN points to can import blockx_executor (PYTHONPATH must include python/). The default python3 is usually the system interpreter and lacks dependencies such as greenlet.

go mod download can't fetch github.com/Chaintable/*

These are private modules. Configure git credentials (for example git config --global url."git@github.com:".insteadOf "https://github.com/") and set GOPRIVATE=github.com/Chaintable/*. The Python-side blockx-py and friends are also private git dependencies and need credentials too.

Testing: commands, durations, and prerequisites for each test layer.
Contributing: commit and PR conventions.
Repository layout and code layering: what goes in cmd/, internal/, api/, and python/.
Deployment overview: image build and the EC2 runtime shape.
docs/deploy.md in the blockx repository: full environment variable table, Docker startup examples, and common issues.
AGENTS.md in the blockx repository: the authoritative checklist for build, test, and collaboration conventions.

Get started

Architecture

Components

Development guide

Local development environment

Prerequisites

Quick start

Makefile targets

Environment variables and config files

Code generation

Common issues

​Prerequisites

​Quick start

​Makefile targets

​Environment variables and config files

​Code generation

​Common issues

​Related docs

Prerequisites

Quick start

Makefile targets

Environment variables and config files

Code generation

Common issues

Related docs