> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockx.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Testing

> BlockX's test layers, the commands and prerequisites for each layer, test doubles, CI, and the entry points for performance testing

BlockX splits its tests by which layer of truth they protect: core unit tests protect the state machines, adapter tests protect protocol mapping, process E2E tests protect the external contract of a single process, contract / system E2E tests protect the boundaries between components, and perf tests separately protect throughput and latency. This page only covers how to run them, where they live, and what they need; for environment setup see [Local development environment](/en/development/getting-started).

## Test layers at a glance

Except for `e2e/perf`, every layer is covered by `go test ./...`; layers that need the Python executor call `t.Skip` instead of failing when the venv is missing (`RequirePythonModules` in `internal/testutil/python.go`).

| Layer                                | Location                                                                                                             | Command                                                                                           | Duration                                 | Prerequisites                                                                                                                    | `-short`                                                                                                 |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| Go unit tests (core / adapter / app) | `internal/**/*_test.go` (about 190 files), `api/**/*_test.go`                                                        | `go test ./internal/... ./api/...`                                                                | Seconds to minutes                       | No external dependencies                                                                                                         | Only one check each in `internal/sdk/noderpc` and `internal/obs`                                         |
| Worker process E2E                   | `cmd/worker/worker_process_*_test.go`                                                                                | `go test -v -timeout 300s ./cmd/worker/...`                                                       | CI splits it into 8 shards, 90s per test | `uv sync --project python`; must be able to `import greenlet, blockx_sdk, blockx_executor, blockdb, blockx, leafage, chaintable` | Only skips `TestWorkerProcess_ConcurrentWorkers_And_Executors` and the task\_finished scenario archiving |
| Bundle worker startup smoke          | `cmd/bundle_worker/bundle_worker_process_test.go`                                                                    | `go test -v -timeout 120s ./cmd/bundle_worker/...`                                                | Under a minute                           | Same as above                                                                                                                    | Not skipped                                                                                              |
| Coordinator process                  | `cmd/coordinator/coordinator_process_lifecycle_test.go`, `cmd/bundle_coordinator/bundle_coordinator_process_test.go` | `go test ./cmd/coordinator/... ./cmd/bundle_coordinator/...`                                      | Seconds                                  | Embedded etcd; no Python needed                                                                                                  | Not skipped                                                                                              |
| Sync-invoker E2E                     | `cmd/syncinvoker/syncinvoker_process_test.go` (23 `TestSyncInvoker_E2E_*` tests), `syncinvoker_perf_test.go`         | `go test -v -timeout 300s ./cmd/syncinvoker/...`; `make test` runs it file by file, `3m` per file | Minutes                                  | venv (`greenlet, blockx_sdk, blockx_executor`)                                                                                   | Not skipped; the perf cases scale with `SYNC_PERF_N` (default 200)                                       |
| Contract E2E                         | `e2e/contract/coordinator-etcd/`, `e2e/contract/coordinator-worker/`                                                 | `go test -v -timeout 300s ./e2e/contract/...`                                                     | Minutes                                  | venv + embedded etcd; CI runs `go build ./cmd/worker ./cmd/coordinator` first to warm the cache                                  | Not skipped                                                                                              |
| System E2E                           | `e2e/system/{smoke,routing,recovery,logical_type}/`                                                                  | `go test -v -timeout 600s ./e2e/system/...`; quick entry `go test ./e2e/system/smoke/...`         | Minutes                                  | venv + embedded etcd; `logical_type` can switch to a real BlockDB with `TEST_BLOCKDB_*_ADDR`                                     | Not skipped                                                                                              |
| Perf                                 | `e2e/perf/`, `cmd/perf` (a CLI that targets a live environment)                                                      | `./test.sh` or `BLOCKX_RUN_PERF=1 go test -v -timeout 600s ./e2e/perf/...`                        | Around ten minutes                       | venv, `sudo` (cgroup); see below                                                                                                 | `TestMain` skips the whole package under `-short`; `BLOCKX_RUN_PERF_SHORT=1` overrides that              |
| Python unit tests                    | `python/tests/test_*.py` (43 files, no subdirectories)                                                               | `PYTHONPATH=python uv run --project python python -m unittest discover -s python/tests`           | Seconds to minutes                       | `uv sync --project python`                                                                                                       | N/A                                                                                                      |

A few additional notes:

* `make test` adds `-short` to every package, but no layer other than perf is skipped because of it, so `make test` is the full E2E run.
* `cmd/syncinvoker` is removed from `FAST_TEST_PACKAGES` by the `Makefile` and run file by file: every test starts its own process + Python executor, and the perf cases aren't guarded by `-short`, so running the whole package at once would exceed the budget; running per file makes `SYNCINVOKER_TEST_TIMEOUT` the budget for each file and produces coverage per file that is merged afterwards.
* Example of `-run` filtering for a single module: `go test ./internal/worker/core/ -run TestWorkerCore_RequestTaskSlot`.
* A single Python module: `PYTHONPATH=python uv run --project python python -m unittest python.tests.test_executor_protocol_contract`.

<Tabs>
  <Tab title="Day-to-day iteration">
    ```bash theme={null}
    go fmt ./... && go vet ./...
    go test ./internal/... ./api/...
    PYTHONPATH=python uv run --project python python -m unittest discover -s python/tests
    ```
  </Tab>

  <Tab title="Before committing">
    ```bash theme={null}
    uv sync --project python
    make test                     # fmt + vet + everything (including E2E, syncinvoker file by file)
    ```
  </Tab>

  <Tab title="Run only one E2E layer">
    ```bash theme={null}
    go test -v -timeout 300s ./cmd/worker/... -run TestWorkerProcess_RequestSlotAndSubmit
    go test -v -timeout 300s ./e2e/contract/coordinator-worker/...
    go test -v -timeout 600s ./e2e/system/smoke/...
    ```
  </Tab>
</Tabs>

## Test organization conventions

Three files define where tests go and how they're named:

* `docs/specs/worker-test-organization.md` in the blockx repository: applies to `internal/worker/**`, `cmd/worker/`, and `cmd/bundle_worker/`. Three tiers: `core unit` (tests only the Sans-IO core; HTTP, processes, and Python are forbidden), `adapter integration` (handler / subscriber / orchestrator / executor pool; fakes and stubs allowed), and `process e2e` (a real `cmd/worker` process, only under `cmd/<entrypoint>/`). File names express the implementation boundary: `worker_test.go`, `dispatcher_retry_test.go`, `handler_protocol_test.go`, `worker_process_<slice>_test.go`; weakly named files like `misc_test.go` and `e2e_test.go` are forbidden. Top-level functions are `Test<Boundary>_<Theme>`, and `t.Run` names must state "scenario + expected outcome". Split thresholds: a file with more than 3 logical blocks, a top-level test with more than 8 `t.Run`s, or a process E2E file with more than 6 independent contracts.
* `docs/specs/python-test-organization.md` in the blockx repository: one implementation boundary per file in `python/tests/`, named `test_<boundary>.py`; `TestCase` names are `<Boundary><Theme>Test`; method names are "scenario + expected outcome", for example `test_deadline_after_resume_prevents_followup_waiting`.
* `docs/specs/system-e2e-testing.md` in the blockx repository: contract files are named `contract_<behavior>_test.go` with functions `TestContractE2E_<Boundary>_<Behavior>`; system files are named `system_<behavior>_test.go` with functions `TestSystemE2E_<Behavior>`; a system file holds only one scenario family. When unsure which layer a test belongs in, default to the contract layer.

The testing guidelines in `AGENTS.md` add four rules: Go tests live in `*_test.go` in the same directory as the implementation; prefer table-driven tests for state machines, protocol mapping, and core/adapter boundaries; new behavior must cover both the happy path and the failure / timeout paths from the spec; and when you change protocol or lifecycle semantics, update the corresponding spec in the same change.

<Note>
  Different layers assert different truths: core unit tests assert state transitions, idempotency, command output, TTLs, and result convergence; adapter tests assert protocol mapping and error translation; process E2E tests assert only externally observable behavior (gRPC responses, terminal states, log fields). Don't enumerate the `DispatcherCore` retry matrix in E2E tests, and don't use process tests as a substitute for handler error-mapping assertions.
</Note>

## Test utilities and doubles

| Location                                        | Contents                                                                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/testutil/etcd.go`                     | `StartEmbeddedEtcd` / `StartEmbeddedEtcdAt`: start etcd inside the test process with `go.etcd.io/etcd/server/v3/embed`, reclaimed by `t.Cleanup`; `WaitForWorkerHeartbeat` and `WaitForWorkerHeartbeatGone` observe registration                                                                                                                                                                           |
| `internal/testutil/worker.go`, `coordinator.go` | `StartWorker` / `StartWorkerAt`, `StartCoordinator` / `StartBundleCoordinator`: build and start real processes, inject the minimal environment variables, and wait for readiness. `BlockWorkerBlockDBEnv` adds `BLOCKDB_BATCH_WRITE_ADDR` and `STUB_BLOCKDB_DELAY_MS=0`                                                                                                                                    |
| `internal/testutil/process.go`                  | `Process`: stop, logs, `RSSKB` / `VmPeakKB`, `KillOneChildPython` (fault injection)                                                                                                                                                                                                                                                                                                                        |
| `internal/testutil/paths.go`                    | `BuildBinary` (builds once per test process and caches), `FreeAddr`, `RunTestMain` (cleans up the binaries when done), `pythonBin()` venv detection                                                                                                                                                                                                                                                        |
| `internal/testutil/python.go`                   | `RequirePythonModules`: `t.Skip` when detection fails, results cached per interpreter                                                                                                                                                                                                                                                                                                                      |
| `internal/testutil/rpc.go`, `rpc_types.go`      | `RequestTaskSlot`, `SubmitTask`, `PollUntilTerminal`, `GRPCCoordClient` / `GRPCWorkerClient`, plus reverse lookup from gRPC codes to business error codes                                                                                                                                                                                                                                                  |
| `internal/testutil/blockdb.go`                  | `StartMockFunctionBlockDB`: in-process fake BlockDB over gRPC that implements only `GetRow` / `BatchGetRows` / `FilterRows` / `Scan`, used for function code reads                                                                                                                                                                                                                                         |
| `internal/testutil/sandbox_e2e.go`              | `RequireExecutorSandboxE2E`: prerequisite checks for the containerd sandbox E2E tests (`ctr`, image, snapshotter); skips when they aren't met. The related tests carry `//go:build sandbox_e2e` and aren't compiled by default                                                                                                                                                                             |
| `internal/worker/devstub/`                      | Pluggable stubs for dev / e2e: `StubBackendAdapter`, `DelayedBackendAdapter` (IO backend), `StaticCallBuilder` / `PayloadCallBuilder` (Builder), `LogWriterPlugin` / `TestWriterPlugin` (Plugin), `MemoryFunctionCodeStore` (function code), `NoopEpochResolver` / `FailingEpochResolver`. Registered only by the `cmd/worker` and `cmd/syncinvoker` profiles; the bundle entry points don't register them |
| `cmd/blockdb_grpc_stub/`                        | Standalone BlockDB gRPC stub process (flags such as `-listen`, `-read-delay`, `-probe-address`) for benchmarks; no Scan / Subscribe                                                                                                                                                                                                                                                                        |
| `internal/sdk/localtestservice/`                | Sample capability backend: `Service` + `Adapter`, registered by the worker by default; disable with `LOCAL_TEST_SERVICE_DISABLED=true`. Covered by `cmd/worker/worker_process_localtestservice_test.go`                                                                                                                                                                                                    |
| `examples/local_test_service/`                  | Example of hitting a local worker with blockx-py (`local_proxy.py` + `submit.py`); not an automated test                                                                                                                                                                                                                                                                                                   |

testutil has no fake clock. Core methods take `now` explicitly (for example `WorkerCore.HandleRequestTaskSlot(taskID string, ttlMs int64, now int64)`), so core unit tests simply pass a time value.

## CI

`.github/workflows/test.yml` runs on PRs to `dev`, pushes to `dev`, and manual triggers; every job except `Deploy Script Tests` configures access to the private `github.com/Chaintable/*` modules through a GitHub App token:

| Job                              | Contents                                                                                                                                                                                                                                                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Deploy Script Tests`            | `bash deploy/tests/*.sh`: script tests for the worker / syncinvoker entrypoint, profile, env resolver, reconcile, health, and deploy bundle                                                                                                                                                                   |
| `Go Tests`                       | `go test ./internal/... ./api/... ./cmd/...`. This job doesn't install Python dependencies; the process E2E tests under `cmd/*` that depend on the executor are skipped when `RequirePythonModules` detection fails, while the coordinator process tests and the various `profile_test.go` files run normally |
| `Python Tests`                   | `uv sync` followed by `uv run python -m unittest discover -s tests` (`working-directory: python`)                                                                                                                                                                                                             |
| `Worker Process E2E (1/8 … 8/8)` | `bash .github/scripts/run-go-test-shard.sh ./cmd/worker <index> 8 90s`: compiles the test binary, splits into 8 shards by `cksum` of the test name, and runs each test individually with a 90s limit                                                                                                          |
| `Bundle Worker Process E2E`      | `go test -v -timeout 120s ./cmd/bundle_worker/...`                                                                                                                                                                                                                                                            |
| `Contract E2E`                   | `go build ./cmd/worker ./cmd/coordinator && go test -v -timeout 300s ./e2e/contract/...`                                                                                                                                                                                                                      |
| `System E2E`                     | `go test -v -timeout 600s ./e2e/system/...`                                                                                                                                                                                                                                                                   |
| `Perf E2E`                       | `go test -v -short -timeout 900s ./e2e/perf/...` with `PERF_N=8 PERF_C=2 PERF_EXECUTOR_COUNT=2 PERF_SLO_MULTIPLIER=5`                                                                                                                                                                                         |

`.github/workflows/build.yml` only builds and pushes the image when a tag is pushed; it doesn't run tests.

## Performance testing

Each `e2e/perf` case starts its own etcd + coordinator + worker and uses `PERF_*` environment variables to control scale; `checkSLO` asserts latency against thresholds scaled up by `PERF_SLO_MULTIPLIER`, and the thresholds come from `docs/specs/perf-methodology.md`.

The whole package is gated by `TestMain` in `e2e/perf/short_test.go`: when `BLOCKX_RUN_PERF` is not `1`, it returns immediately and runs no cases at all.

```bash theme={null}
# Standard entry: runs inside a cgroup (default CPUQuota=400%, MemoryMax=8G, swap disabled), requires sudo
./test.sh
./test.sh -run TestPerf_SchedulerConcurrency
PERF_CPU=200% PERF_MEM=4G ./test.sh

# Run directly without resource limits (results skew optimistic); the gate must be opened explicitly
BLOCKX_RUN_PERF=1 go test -v -timeout 600s ./e2e/perf/...
BLOCKX_RUN_PERF=1 PERF_N=10 PERF_IO_READS=3 go test -v -timeout 300s -run TestPerf_CacheLocality ./e2e/perf/...
```

* Main variables: `PERF_N` (tasks per scenario, default 20), `PERF_C` (concurrency, default 4), `PERF_EXECUTOR_COUNT` (default 4), `PERF_IO_READS` (default 5), `PERF_SLO_MULTIPLIER` (default 1.0, CI uses 5). `test.sh` itself reads `PERF_CPU`, `PERF_MEM`, and `PERF_TIMEOUT`, and passes through `WORKER_CPU_PROFILE`, `WORKER_TRACE`, `COORDINATOR_CPU_PROFILE`, and `COORDINATOR_TRACE`.
* Files are named after the Stages in `docs/specs/perf-methodology.md` §3: `s0_framework_*`, `s1_builder_*`, `s2_call_count_*`, `s3_cpu_*`, `s4_io_*`, `s5_plugin_*`; `phase_*`, `m3_stream_build_test.go`, and `calibration_test.go` are legacy files that predate the spec. `helpers_test.go` holds scaffolding such as `runScenario` / `startPerfInfra*` / `checkSLO`.
* Artifacts land in `e2e/perf/_artifacts/` (gitignored): `<TestName>_<subtest>/spans.jsonl` (OTel spans from the `with_trace` subtests, enabled by `BLOCKX_OTEL_SPAN_FILE` + `BLOCKX_OTEL_UDS_EVENTS=1`), `<TestName>/*.pb.gz` (pprof), `pyspy/`.
* Offline span analysis: `python3 analyze_spans.py e2e/perf/_artifacts/<TestName>_with_trace/spans.jsonl`. The script lives at the repository root, accepts a single positional argument, and defaults to `/tmp/blockx_spans.jsonl`.
* Reports: `docs/specs/perf-test-report.md` (v1 main report), `perf-test-report-appendix.md` (v1 appendix), `perf-test-report-v2.md` (the only current baseline, cgroup 12 vCPU / 32G), `perf-capability-backend-report.md` (capability backend framework overhead). Reproduction steps are in `docs/specs/perf-test-how-to-v2.md` and the methodology in `docs/specs/perf-methodology.md`.
* `cmd/perf` is a load CLI that targets a **deployed** environment (`-coordinator`, `-n`, `-c`, `-mode`) and isn't part of `go test`. `cmd/bundle-loadgen` is the load generator for Bundle clusters and comes with its own pure unit tests.

## Related docs

* [Local development environment](/en/development/getting-started): venv, CGO, minimal worker startup.
* [Contributing](/en/development/contributing): PRs must include the verification commands.
* [Worker](/en/components/worker), [Coordinator](/en/components/coordinator), [Sync Invoker](/en/components/sync-invoker), [Bundle clusters](/en/components/bundle): the "Testing" section on each component page points to the corresponding files.
* `docs/specs/worker-test-organization.md`, `docs/specs/python-test-organization.md`, `docs/specs/system-e2e-testing.md` in the blockx repository: organization conventions.
* `docs/specs/perf-methodology.md`, `docs/specs/perf-test-how-to-v2.md`, `e2e/perf/README.md` in the blockx repository: performance testing methodology, reproduction, and case index.
* `AGENTS.md` in the blockx repository: the authoritative source for commands and testing guidelines.