Testing

BlockX splits its tests by which layer of truth they protect: core unit tests protect the state machines, adapter tests protect protocol mapping, process E2E tests protect the external contract of a single process, contract / system E2E tests protect the boundaries between components, and perf tests separately protect throughput and latency. This page only covers how to run them, where they live, and what they need; for environment setup see Local development environment.

Test layers at a glance

Except for e2e/perf, every layer is covered by go test ./...; layers that need the Python executor call t.Skip instead of failing when the venv is missing (RequirePythonModules in internal/testutil/python.go).

Layer	Location	Command	Duration	Prerequisites	`-short`
Go unit tests (core / adapter / app)	`internal/*/_test.go` (about 190 files), `api/*/_test.go`	`go test ./internal/... ./api/...`	Seconds to minutes	No external dependencies	Only one check each in `internal/sdk/noderpc` and `internal/obs`
Worker process E2E	`cmd/worker/worker_process_*_test.go`	`go test -v -timeout 300s ./cmd/worker/...`	CI splits it into 8 shards, 90s per test	`uv sync --project python`; must be able to `import greenlet, blockx_sdk, blockx_executor, blockdb, blockx, leafage, chaintable`	Only skips `TestWorkerProcess_ConcurrentWorkers_And_Executors` and the task_finished scenario archiving
Bundle worker startup smoke	`cmd/bundle_worker/bundle_worker_process_test.go`	`go test -v -timeout 120s ./cmd/bundle_worker/...`	Under a minute	Same as above	Not skipped
Coordinator process	`cmd/coordinator/coordinator_process_lifecycle_test.go`, `cmd/bundle_coordinator/bundle_coordinator_process_test.go`	`go test ./cmd/coordinator/... ./cmd/bundle_coordinator/...`	Seconds	Embedded etcd; no Python needed	Not skipped
Sync-invoker E2E	`cmd/syncinvoker/syncinvoker_process_test.go` (23 `TestSyncInvoker_E2E_*` tests), `syncinvoker_perf_test.go`	`go test -v -timeout 300s ./cmd/syncinvoker/...`; `make test` runs it file by file, `3m` per file	Minutes	venv (`greenlet, blockx_sdk, blockx_executor`)	Not skipped; the perf cases scale with `SYNC_PERF_N` (default 200)
Contract E2E	`e2e/contract/coordinator-etcd/`, `e2e/contract/coordinator-worker/`	`go test -v -timeout 300s ./e2e/contract/...`	Minutes	venv + embedded etcd; CI runs `go build ./cmd/worker ./cmd/coordinator` first to warm the cache	Not skipped
System E2E	`e2e/system/{smoke,routing,recovery,logical_type}/`	`go test -v -timeout 600s ./e2e/system/...`; quick entry `go test ./e2e/system/smoke/...`	Minutes	venv + embedded etcd; `logical_type` can switch to a real BlockDB with `TEST_BLOCKDB_*_ADDR`	Not skipped
Perf	`e2e/perf/`, `cmd/perf` (a CLI that targets a live environment)	`./test.sh` or `BLOCKX_RUN_PERF=1 go test -v -timeout 600s ./e2e/perf/...`	Around ten minutes	venv, `sudo` (cgroup); see below	`TestMain` skips the whole package under `-short`; `BLOCKX_RUN_PERF_SHORT=1` overrides that
Python unit tests	`python/tests/test_*.py` (43 files, no subdirectories)	`PYTHONPATH=python uv run --project python python -m unittest discover -s python/tests`	Seconds to minutes	`uv sync --project python`	N/A

A few additional notes:

make test adds -short to every package, but no layer other than perf is skipped because of it, so make test is the full E2E run.
cmd/syncinvoker is removed from FAST_TEST_PACKAGES by the Makefile and run file by file: every test starts its own process + Python executor, and the perf cases aren’t guarded by -short, so running the whole package at once would exceed the budget; running per file makes SYNCINVOKER_TEST_TIMEOUT the budget for each file and produces coverage per file that is merged afterwards.
Example of -run filtering for a single module: go test ./internal/worker/core/ -run TestWorkerCore_RequestTaskSlot.
A single Python module: PYTHONPATH=python uv run --project python python -m unittest python.tests.test_executor_protocol_contract.

Day-to-day iteration
Before committing
Run only one E2E layer

go fmt ./... && go vet ./...
go test ./internal/... ./api/...
PYTHONPATH=python uv run --project python python -m unittest discover -s python/tests

uv sync --project python
make test                     # fmt + vet + everything (including E2E, syncinvoker file by file)

go test -v -timeout 300s ./cmd/worker/... -run TestWorkerProcess_RequestSlotAndSubmit
go test -v -timeout 300s ./e2e/contract/coordinator-worker/...
go test -v -timeout 600s ./e2e/system/smoke/...

Test organization conventions

Three files define where tests go and how they’re named:

docs/specs/worker-test-organization.md in the blockx repository: applies to internal/worker/**, cmd/worker/, and cmd/bundle_worker/. Three tiers: core unit (tests only the Sans-IO core; HTTP, processes, and Python are forbidden), adapter integration (handler / subscriber / orchestrator / executor pool; fakes and stubs allowed), and process e2e (a real cmd/worker process, only under cmd/<entrypoint>/). File names express the implementation boundary: worker_test.go, dispatcher_retry_test.go, handler_protocol_test.go, worker_process_<slice>_test.go; weakly named files like misc_test.go and e2e_test.go are forbidden. Top-level functions are Test<Boundary>_<Theme>, and t.Run names must state “scenario + expected outcome”. Split thresholds: a file with more than 3 logical blocks, a top-level test with more than 8 t.Runs, or a process E2E file with more than 6 independent contracts.
docs/specs/python-test-organization.md in the blockx repository: one implementation boundary per file in python/tests/, named test_<boundary>.py; TestCase names are <Boundary><Theme>Test; method names are “scenario + expected outcome”, for example test_deadline_after_resume_prevents_followup_waiting.
docs/specs/system-e2e-testing.md in the blockx repository: contract files are named contract_<behavior>_test.go with functions TestContractE2E_<Boundary>_<Behavior>; system files are named system_<behavior>_test.go with functions TestSystemE2E_<Behavior>; a system file holds only one scenario family. When unsure which layer a test belongs in, default to the contract layer.

The testing guidelines in AGENTS.md add four rules: Go tests live in *_test.go in the same directory as the implementation; prefer table-driven tests for state machines, protocol mapping, and core/adapter boundaries; new behavior must cover both the happy path and the failure / timeout paths from the spec; and when you change protocol or lifecycle semantics, update the corresponding spec in the same change.

Different layers assert different truths: core unit tests assert state transitions, idempotency, command output, TTLs, and result convergence; adapter tests assert protocol mapping and error translation; process E2E tests assert only externally observable behavior (gRPC responses, terminal states, log fields). Don’t enumerate the DispatcherCore retry matrix in E2E tests, and don’t use process tests as a substitute for handler error-mapping assertions.

Test utilities and doubles

Location	Contents
`internal/testutil/etcd.go`	`StartEmbeddedEtcd` / `StartEmbeddedEtcdAt`: start etcd inside the test process with `go.etcd.io/etcd/server/v3/embed`, reclaimed by `t.Cleanup`; `WaitForWorkerHeartbeat` and `WaitForWorkerHeartbeatGone` observe registration
`internal/testutil/worker.go`, `coordinator.go`	`StartWorker` / `StartWorkerAt`, `StartCoordinator` / `StartBundleCoordinator`: build and start real processes, inject the minimal environment variables, and wait for readiness. `BlockWorkerBlockDBEnv` adds `BLOCKDB_BATCH_WRITE_ADDR` and `STUB_BLOCKDB_DELAY_MS=0`
`internal/testutil/process.go`	`Process`: stop, logs, `RSSKB` / `VmPeakKB`, `KillOneChildPython` (fault injection)
`internal/testutil/paths.go`	`BuildBinary` (builds once per test process and caches), `FreeAddr`, `RunTestMain` (cleans up the binaries when done), `pythonBin()` venv detection
`internal/testutil/python.go`	`RequirePythonModules`: `t.Skip` when detection fails, results cached per interpreter
`internal/testutil/rpc.go`, `rpc_types.go`	`RequestTaskSlot`, `SubmitTask`, `PollUntilTerminal`, `GRPCCoordClient` / `GRPCWorkerClient`, plus reverse lookup from gRPC codes to business error codes
`internal/testutil/blockdb.go`	`StartMockFunctionBlockDB`: in-process fake BlockDB over gRPC that implements only `GetRow` / `BatchGetRows` / `FilterRows` / `Scan`, used for function code reads
`internal/testutil/sandbox_e2e.go`	`RequireExecutorSandboxE2E`: prerequisite checks for the containerd sandbox E2E tests (`ctr`, image, snapshotter); skips when they aren’t met. The related tests carry `//go:build sandbox_e2e` and aren’t compiled by default
`internal/worker/devstub/`	Pluggable stubs for dev / e2e: `StubBackendAdapter`, `DelayedBackendAdapter` (IO backend), `StaticCallBuilder` / `PayloadCallBuilder` (Builder), `LogWriterPlugin` / `TestWriterPlugin` (Plugin), `MemoryFunctionCodeStore` (function code), `NoopEpochResolver` / `FailingEpochResolver`. Registered only by the `cmd/worker` and `cmd/syncinvoker` profiles; the bundle entry points don’t register them
`cmd/blockdb_grpc_stub/`	Standalone BlockDB gRPC stub process (flags such as `-listen`, `-read-delay`, `-probe-address`) for benchmarks; no Scan / Subscribe
`internal/sdk/localtestservice/`	Sample capability backend: `Service` + `Adapter`, registered by the worker by default; disable with `LOCAL_TEST_SERVICE_DISABLED=true`. Covered by `cmd/worker/worker_process_localtestservice_test.go`
`examples/local_test_service/`	Example of hitting a local worker with blockx-py (`local_proxy.py` + `submit.py`); not an automated test

testutil has no fake clock. Core methods take now explicitly (for example WorkerCore.HandleRequestTaskSlot(taskID string, ttlMs int64, now int64)), so core unit tests simply pass a time value.

CI

.github/workflows/test.yml runs on PRs to dev, pushes to dev, and manual triggers; every job except Deploy Script Tests configures access to the private github.com/Chaintable/* modules through a GitHub App token:

Job	Contents
`Deploy Script Tests`	`bash deploy/tests/*.sh`: script tests for the worker / syncinvoker entrypoint, profile, env resolver, reconcile, health, and deploy bundle
`Go Tests`	`go test ./internal/... ./api/... ./cmd/...`. This job doesn’t install Python dependencies; the process E2E tests under `cmd/*` that depend on the executor are skipped when `RequirePythonModules` detection fails, while the coordinator process tests and the various `profile_test.go` files run normally
`Python Tests`	`uv sync` followed by `uv run python -m unittest discover -s tests` (`working-directory: python`)
`Worker Process E2E (1/8 … 8/8)`	`bash .github/scripts/run-go-test-shard.sh ./cmd/worker <index> 8 90s`: compiles the test binary, splits into 8 shards by `cksum` of the test name, and runs each test individually with a 90s limit
`Bundle Worker Process E2E`	`go test -v -timeout 120s ./cmd/bundle_worker/...`
`Contract E2E`	`go build ./cmd/worker ./cmd/coordinator && go test -v -timeout 300s ./e2e/contract/...`
`System E2E`	`go test -v -timeout 600s ./e2e/system/...`
`Perf E2E`	`go test -v -short -timeout 900s ./e2e/perf/...` with `PERF_N=8 PERF_C=2 PERF_EXECUTOR_COUNT=2 PERF_SLO_MULTIPLIER=5`

.github/workflows/build.yml only builds and pushes the image when a tag is pushed; it doesn’t run tests.

Performance testing

Each e2e/perf case starts its own etcd + coordinator + worker and uses PERF_* environment variables to control scale; checkSLO asserts latency against thresholds scaled up by PERF_SLO_MULTIPLIER, and the thresholds come from docs/specs/perf-methodology.md. The whole package is gated by TestMain in e2e/perf/short_test.go: when BLOCKX_RUN_PERF is not 1, it returns immediately and runs no cases at all.

# Standard entry: runs inside a cgroup (default CPUQuota=400%, MemoryMax=8G, swap disabled), requires sudo
./test.sh
./test.sh -run TestPerf_SchedulerConcurrency
PERF_CPU=200% PERF_MEM=4G ./test.sh

# Run directly without resource limits (results skew optimistic); the gate must be opened explicitly
BLOCKX_RUN_PERF=1 go test -v -timeout 600s ./e2e/perf/...
BLOCKX_RUN_PERF=1 PERF_N=10 PERF_IO_READS=3 go test -v -timeout 300s -run TestPerf_CacheLocality ./e2e/perf/...

Main variables: PERF_N (tasks per scenario, default 20), PERF_C (concurrency, default 4), PERF_EXECUTOR_COUNT (default 4), PERF_IO_READS (default 5), PERF_SLO_MULTIPLIER (default 1.0, CI uses 5). test.sh itself reads PERF_CPU, PERF_MEM, and PERF_TIMEOUT, and passes through WORKER_CPU_PROFILE, WORKER_TRACE, COORDINATOR_CPU_PROFILE, and COORDINATOR_TRACE.
Files are named after the Stages in docs/specs/perf-methodology.md §3: s0_framework_*, s1_builder_*, s2_call_count_*, s3_cpu_*, s4_io_*, s5_plugin_*; phase_*, m3_stream_build_test.go, and calibration_test.go are legacy files that predate the spec. helpers_test.go holds scaffolding such as runScenario / startPerfInfra* / checkSLO.
Artifacts land in e2e/perf/_artifacts/ (gitignored): <TestName>_<subtest>/spans.jsonl (OTel spans from the with_trace subtests, enabled by BLOCKX_OTEL_SPAN_FILE + BLOCKX_OTEL_UDS_EVENTS=1), <TestName>/*.pb.gz (pprof), pyspy/.
Offline span analysis: python3 analyze_spans.py e2e/perf/_artifacts/<TestName>_with_trace/spans.jsonl. The script lives at the repository root, accepts a single positional argument, and defaults to /tmp/blockx_spans.jsonl.
Reports: docs/specs/perf-test-report.md (v1 main report), perf-test-report-appendix.md (v1 appendix), perf-test-report-v2.md (the only current baseline, cgroup 12 vCPU / 32G), perf-capability-backend-report.md (capability backend framework overhead). Reproduction steps are in docs/specs/perf-test-how-to-v2.md and the methodology in docs/specs/perf-methodology.md.
cmd/perf is a load CLI that targets a deployed environment (-coordinator, -n, -c, -mode) and isn’t part of go test. cmd/bundle-loadgen is the load generator for Bundle clusters and comes with its own pure unit tests.

Local development environment: venv, CGO, minimal worker startup.
Contributing: PRs must include the verification commands.
Worker, Coordinator, Sync Invoker, Bundle clusters: the “Testing” section on each component page points to the corresponding files.
docs/specs/worker-test-organization.md, docs/specs/python-test-organization.md, docs/specs/system-e2e-testing.md in the blockx repository: organization conventions.
docs/specs/perf-methodology.md, docs/specs/perf-test-how-to-v2.md, e2e/perf/README.md in the blockx repository: performance testing methodology, reproduction, and case index.
AGENTS.md in the blockx repository: the authoritative source for commands and testing guidelines.

Get started

Architecture

Components

Development guide

Test layers at a glance

Test organization conventions

Test utilities and doubles

CI

Performance testing

​Test layers at a glance

​Test organization conventions

​Test utilities and doubles

​CI

​Performance testing

​Related docs

Test layers at a glance

Test organization conventions

Test utilities and doubles

CI

Performance testing

Related docs