> ## Documentation Index > Fetch the complete documentation index at: https://docs.blockx.chaintable.com/llms.txt > Use this file to discover all available pages before exploring further. # Coordinator > Task Resource Coordinator: selects a Worker for the Client from the etcd heartbeat view and reserves a short-lived slot on it The Task Resource Coordinator is BlockX's control-plane process. It does exactly one thing: when a Client calls `ReserveWorkerSlot(taskId)`, it picks a candidate from the Worker view rebuilt from etcd heartbeats, calls `RequestTaskSlot` on that Worker, and hands `workerAddr + slotId` back to the Client. From then on, the Client submits the task directly to the Worker and the Coordinator is no longer involved. See [Architecture overview](/en/architecture/overview) for where it sits in the overall architecture, and [Task lifecycle](/en/architecture/task-lifecycle) for the main path. ## Responsibilities and boundaries Responsible for: * Watching Worker registrations and heartbeats in etcd and maintaining a local, disposable `WorkerView`. * For each `ReserveWorkerSlot`, performing validation, the readiness check, candidate filtering and ranking, and then trying a small number of candidates serially. * Passing the `slotId` returned by the Worker through to the Client as-is, without interpreting its internal meaning. * Applying local penalties and short circuit-breaking to Workers based on recent failures. Not responsible for: * Parsing the task payload, generating `taskId`, or receiving `SubmitTask` or `TaskResult`. * Maintaining a centralized slot ledger, a pending queue, or global `taskId` deduplication. * Writing any data to etcd; etcd only carries Worker discovery and heartbeats. Core invariants: * The Worker's local slot state machine is the single source of truth; a Coordinator restart only loses the view cache and routing bias, and does not affect correctness. * `CoordinatorCore` is a Sans-IO pure decision engine: it never reads the clock or touches the network, and every method receives `now` (UTC milliseconds) explicitly. * Before the first full Worker view is established, and after a watch outage exceeds the grace period, `ReserveWorkerSlot` always returns `Unavailable`. * A single request tries candidates serially with no hedged requests; when the outcome is uncertain, it retries only against the same Worker with the same `taskId`, and never switches Workers. * The same code runs as two sets of processes with different `app.Profile` values: `cmd/coordinator` (block cluster) and `cmd/bundle_coordinator` (bundle cluster). See [Bundle clusters](/en/components/bundle) for bundle cluster details. ## Code location | Path | Purpose | | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `cmd/coordinator/main.go` | block Coordinator entry point; only declares `app.Profile{Deployment: "block", ...}` and calls `app.Run` | | `cmd/bundle_coordinator/main.go` | bundle Coordinator entry point, `Deployment: "bundle"`; watches both the legacy and the prod v2 bundle registries by default | | `internal/coordinator/core/` | Pure decision logic: `Config`, `CoordinatorCore`, `WorkerView`, `ValidateReserveRequest` | | `internal/coordinator/adapters/grpc_server.go` | Public gRPC handler `CoordinatorServer.ReserveWorkerSlot`; holds the orchestration loop and the core's read-write lock | | `internal/coordinator/adapters/bundle_grpc_server.go` | `BundleCoordinatorServer`, forwards bundle proto requests to the same `CoordinatorServer` | | `internal/coordinator/adapters/worker_rpc.go` | `WorkerRPCClient`, the gRPC client that sends `RequestTaskSlot` to Workers and classifies gRPC errors into `SlotError` | | `internal/coordinator/adapters/etcd_watcher.go` | `EtcdWorkerWatcher`, one list+watch loop per registry prefix, merged and fed to the core | | `internal/coordinator/adapters/etcd_watcher_metrics.go` | The `blockx_coordinator_registry_workers{format}` metric | | `internal/coordinator/app/app.go` | `Profile`, `Run`, `registerCoordinatorService`; process wiring and lifecycle | | `internal/coordinator/app/config.go` | `CoordinatorRuntimeConfig`, `LoadCoordinatorRuntimeConfigFromEnv`, environment variable parsing | | `api/grpc/coordinator/coordinator.proto` | Public protocol for `CoordinatorService.ReserveWorkerSlot`; generated code lives in `coordinatorpb/` | | `api/grpc/bundlecoordinator/` | `BundleCoordinatorService`; same fields as the block version, only the service name differs | | `api/etcd/types.go`, `api/etcd/registry.go` | The `WorkerHeartbeat` struct, registry prefix constants, and `ParseRegistryPrefix` | | `internal/common/types/types.go` | Business error codes `ErrorCode` (`ErrInvalidArgument`, `ErrUnavailable`, `ErrNoExecutableSlot`, `ErrSlotUncertain`, `ErrNoSlot`, etc.) | | `internal/coordinator/**/*_test.go`, `cmd/coordinator/*_test.go`, `cmd/bundle_coordinator/*_test.go` | Unit tests and process-level startup tests | | `e2e/contract/coordinator-etcd/`, `e2e/contract/coordinator-worker/` | Contract-level E2E: etcd disconnect/recovery, timeout uncertainty, failover, slot TTL, etc. | | `docs/specs/task-resource-coordinator.md`, `docs/specs/architecture.md` §4.1 | Design specs | | `docs/specs/2026-07-28-registry-prefix-migration.md` | legacy / v2 registry prefix migration and multi-prefix merge semantics | ## Core types and interfaces ### core * `core.Config` (`internal/coordinator/core/coordinator.go`): selection and protection parameters; `DefaultConfig()` provides the defaults, and `Validate()` requires `WatchGracePeriodMs <= HeartbeatTimeoutMs`. * `core.CoordinatorCore`: holds `map[string]*WorkerView`, `initialized`, and `watchDisconnectedAtMs`. Event entry points: `ApplyFullWorkerList`, `ApplyWorkerUpdate`, `ApplyWorkerRemoval`, `MarkWatchDisconnected`, `MarkWatchReconnected`. Decision outputs: `IsReady`, `SelectCandidate`. Feedback entry points: `RecordSlotSuccess`, `RecordSlotRejection`, `RecordSlotBackpressure`, `RecordSlotTimeout`. * `core.WorkerView` (`internal/coordinator/core/worker_view.go`): embeds `etcd.WorkerHeartbeat` and adds three fields that exist only locally and are never written back to etcd. ```go theme={null} type WorkerView struct { WorkerHeartbeat // Local-only fields — not from etcd, not written back. CircuitBreakerDeadlineMs int64 RecentFailureCount int Penalty float64 } ``` * `WorkerView.UsedTotal()` = `ReservedSlots + RunningTasks`; `WorkerView.FreeCapacity()` = `TaskSlots - UsedTotal()`. * `core.CandidateResult{WorkerAddr, Err}` and `core.CoordinatorError{Code, Message}`: the output of `SelectCandidate`; `Code` is a `commontypes.ErrorCode`. * `core.ValidateReserveRequest(taskID)` (`internal/coordinator/core/validate.go`): only checks that `taskId` is non-empty. ### adapters * `adapters.CoordinatorServer` (`grpc_server.go`): implements `coordinatorpb.CoordinatorServiceServer`; holds `core`, a `SlotReserver`, and a `sync.RWMutex`. `Lock/Unlock` are exposed so the watcher can write to the core. * `adapters.HandlerConfig{MaxTimeoutRetries}`: number of retries against the same Worker when the outcome is uncertain; defaults to `1`. * `adapters.SlotReserver` interface: `RequestTaskSlot(ctx, workerAddr, req) (*commontypes.TaskSnapshot, *SlotError)`; replaced with a mock in tests. * `adapters.WorkerRPCClient` (`worker_rpc.go`): the gRPC implementation of `SlotReserver`; caches one `grpc.ClientConn` per `workerAddr`. `grpcErrToSlotError` decides `SlotError.MayHaveAllocated`. * `adapters.SlotError{Code, Message, MayHaveAllocated}`: `MayHaveAllocated=true` means the request may have reached the Worker, so retries must stay on the same Worker. * `adapters.EtcdWorkerWatcher` / `EtcdWatcherConfig{Endpoints, KeyPrefixes, DialTimeout, RetryDelay}` (`etcd_watcher.go`): `Run(ctx)` blocks while running list+watch. * `adapters.BundleCoordinatorServer` (`bundle_grpc_server.go`): a thin translation layer from the bundle proto to `CoordinatorServer`. ### app and api * `app.Profile{Deployment, Service, WorkerRegistryPrefixes}` (`internal/coordinator/app/app.go`): the entry point's identity. `Deployment` must be `"block"` or `"bundle"`, and determines which gRPC service is registered and which registry kind is allowed. * `app.Run(p Profile)`: wires obs, core, `WorkerRPCClient`, `CoordinatorServer`, the etcd client, the watcher, and the gRPC server, then blocks until SIGINT/SIGTERM. * `app.CoordinatorRuntimeConfig` (`config.go`): the process-level configuration aggregate; `LoadCoordinatorRuntimeConfigFromEnv(p)` applies environment variable overrides and calls `Validate()`. * `etcd.WorkerHeartbeat` (`api/etcd/types.go`): the JSON value the Worker writes to etcd, with fields `workerAddr`, `taskSlots`, `reservedSlots`, `runningTasks`, `cpuPercent`, `memoryPercent`, `memoryAvailableBytes`, `memoryTotalBytes`, `processRssBytes`, `errorRate`, and `lastHeartbeatMs`. * `etcd.ParseRegistryPrefix` and the constants `LegacyBlockWorkerRegistryPrefix` (`/blockx/workers/`), `LegacyBundleWorkerRegistryPrefix` (`/blockx/bundle-workers/`), `ProdDefaultBlockWorkerRegistryPrefix`, `ProdDefaultBundleWorkerRegistryPrefix`, `TestDefaultBundleWorkerRegistryPrefix` (`api/etcd/registry.go`). ## Data flow / execution flow The etcd key is ` + workerAddr`, and the value is the `WorkerHeartbeat` JSON. The Worker side refreshes it periodically with a leased `Put`; the write logic lives in `internal/worker/adapters/etcd_publisher.go` (lease TTL defaults to 10s; see the Worker configuration `HeartbeatIntervalMs` for the heartbeat interval). The Coordinator is read-only. ```mermaid theme={null} sequenceDiagram participant C as Client participant S as CoordinatorServer participant K as CoordinatorCore participant E as etcd participant W as Worker Note over E,K: Background: EtcdWorkerWatcher runs list+watch for each prefix E-->>K: ApplyFullWorkerList / ApplyWorkerUpdate / ApplyWorkerRemoval C->>S: ReserveWorkerSlot(taskId) S->>K: ValidateReserveRequest, IsReady(now) loop Up to MaxCandidates candidates S->>K: SelectCandidate(taskId, excludeAddrs, now) K-->>S: WorkerAddr or CoordinatorError S->>W: RequestTaskSlot(taskId, ttlMs) alt Success with a non-empty slotId W-->>S: TaskSnapshot{SlotID} S->>K: RecordSlotSuccess S-->>C: taskId + workerAddr + slotId else Explicit rejection S->>K: RecordSlotRejection or RecordSlotBackpressure Note over S: Add to excludeAddrs and try the next candidate else Uncertain outcome Note over S: Retry the same Worker with the same taskId up to MaxTimeoutRetries times S->>K: RecordSlotTimeout if still uncertain S-->>C: gRPC Aborted end end S-->>C: Candidates exhausted: gRPC ResourceExhausted ``` The orchestration logic of `CoordinatorServer.ReserveWorkerSlot` runs in the following order (`internal/coordinator/adapters/grpc_server.go`): `core.ValidateReserveRequest` rejects an empty `taskId`; when `core.IsReady(now)` is false, it returns `Unavailable` immediately, without sending a request to any Worker. Calls `core.SelectCandidate` under the read lock. The hard filters, in order: already in `excludeAddrs`, heartbeat timed out (`now - LastHeartbeatMs > HeartbeatTimeoutMs`), `CPUPercent`/`MemoryPercent`/`ErrorRate` above the red line, circuit breaker not yet expired, `FreeCapacity() <= 0`. The remaining Workers are ranked by `candidateRanksBefore`: larger `FreeCapacity` first, then more recent `LastHeartbeatMs`, then smaller `Penalty`, and finally lexicographic order of `WorkerAddr`. `WorkerRPCClient.RequestTaskSlot` calls the Worker with `WorkerRPCTimeout`. `grpcErrToSlotError` splits gRPC status codes into two classes: `InvalidArgument`/`ResourceExhausted`/`FailedPrecondition`/`NotFound` are explicit rejections; `DeadlineExceeded`/`Canceled`/`Aborted`/`Internal`/`Unknown` plus timeout-style `Unavailable` are marked `MayHaveAllocated=true`. A successful response with an empty `SlotID` is also treated as uncertain. On success, `RecordSlotSuccess` resets the failure count. On an explicit rejection: a Worker `ResourceExhausted` (mapped to `ErrNoSlot`) goes through `RecordSlotBackpressure` and does not count toward the circuit breaker; other rejections go through `RecordSlotRejection`. An uncertain outcome is first retried on the same Worker; if it is still uncertain, `RecordSlotTimeout` is called and `Aborted` is returned. An explicit rejection received during the retries moves on to the next candidate. `CoordinatorCore` has no locking of its own; `CoordinatorServer.mu` is the only synchronization point. The watcher writes to the core via `srv.Lock()/Unlock()`, and the handler selects candidates under the read lock and records feedback under the write lock. ### Public error codes The public protocol is gRPC. Business error codes are mapped to gRPC status codes in `coordCodeToGRPC`: | Scenario | Business code (`commontypes`) | gRPC status | Request sent to a Worker? | | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------- | | `taskId` is empty | `ErrInvalidArgument` | `InvalidArgument` | No | | core not initialized, watch grace period exceeded, or all Worker heartbeats timed out | `ErrUnavailable` | `Unavailable` | No | | No unfiltered Worker with `FreeCapacity > 0` in the trusted view | `ErrNoExecutableSlot` | `ResourceExhausted` | No | | All `MaxCandidates` candidates explicitly rejected | Directly `status.Error(codes.ResourceExhausted, "all candidates rejected")` | `ResourceExhausted` | Yes | | Still unconfirmed after retrying the same Worker | Directly `status.Error(codes.Aborted, "unable to confirm slot reservation")` | `Aborted` | Yes; the target Worker may hold a dangling slot until its TTL reclaims it | When the reservation is still unconfirmed after retries, the Coordinator returns gRPC `Aborted`; `coordCodeToGRPC` maps `ErrSlotUncertain` to `Aborted`, and the test utility `internal/testutil/rpc_types.go` interprets it back as `ErrSlotUncertain`. Callers should treat `Aborted` as "possibly reserved; wait at least one slot TTL before retrying" and `Unavailable` as "safe to retry immediately". ## State and lifecycle `CoordinatorCore` has only two global state flags and no explicit state machine: * `initialized`: becomes true after the first `ApplyFullWorkerList` call and never reverts. The watcher calls it only after every configured prefix has completed its initial list. * `watchDisconnectedAtMs`: when the watch on any prefix disconnects, `MarkWatchDisconnected(now)` records the time of the first disconnect; `IsReady` and `SelectCandidate` return `Unavailable` once `now - watchDisconnectedAtMs > WatchGracePeriodMs`. After every prefix has re-listed successfully and all are connected again, `MarkWatchReconnected` clears it. Locally derived state per Worker: * `RecentFailureCount` and `Penalty`: `RecordSlotRejection`/`RecordSlotTimeout` each increment by one, and `Penalty = float64(RecentFailureCount)`. * `CircuitBreakerDeadlineMs`: set to `now + CircuitBreakerCooldownMs` (default 10s) when `RecentFailureCount >= CircuitBreakerThreshold` (default 3); the Worker is hard-filtered for that period. * `RecordSlotSuccess` clears all three together; `RecordSlotBackpressure` leaves all three untouched. * `ApplyFullWorkerList` preserves these three fields for surviving Workers; a Worker that reappears after being deleted by `ApplyWorkerRemoval` starts from zero. Relationships between the timing parameters: * The slot TTL (`SlotTTLMs`, default 5s) is the reservation window the Coordinator passes to the Worker in `RequestTaskSlot`; it is not a task execution timeout, and the Coordinator does not interpret it further. * A single `ReserveWorkerSlot` has no separate overall time budget; the upper bound is determined jointly by `MaxCandidates × (1 + MaxTimeoutRetries) × WorkerRPCTimeout` and the caller's ctx, roughly `3 × 2 × 3s` in the default worst case. * Heartbeat freshness is computed by subtracting the Worker-written `lastHeartbeatMs` from the Coordinator's local clock, so clock skew between machines directly affects the filtering result. ## Configuration All configuration comes from `internal/coordinator/app/config.go`; defaults come from `DefaultCoordinatorRuntimeConfig`, `core.DefaultConfig`, `adapters.DefaultHandlerConfig`, and `adapters.DefaultEtcdWatcherConfig`. | Environment variable | Struct field | Default | Description | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `COORDINATOR_LISTEN` | `ListenAddr` | `:8080` | gRPC listen address | | `ETCD_ENDPOINTS` | `EtcdEndpoints` / `Watcher.Endpoints` | `localhost:2379` | Comma-separated | | `COORDINATOR_REGISTRY_PREFIXES` | `Watcher.KeyPrefixes` | From `Profile.WorkerRegistryPrefixes` | Comma-separated list of exact prefixes that replaces the profile defaults as a whole; each must pass `ParseRegistryPrefix`, and its kind must match `Deployment` | | `COORDINATOR_WORKER_RPC_TIMEOUT_MS` | `WorkerRPCTimeout` | `3000` | Timeout for a single `RequestTaskSlot` | | `COORDINATOR_ETCD_DIAL_TIMEOUT_MS` | `EtcdDialTimeout` / `Watcher.DialTimeout` | `5000` | | | `COORDINATOR_ETCD_WATCH_RETRY_DELAY_MS` | `Watcher.RetryDelay` | `1000` | Retry interval after a list+watch failure | | `COORDINATOR_HEARTBEAT_TIMEOUT_MS` | `Core.HeartbeatTimeoutMs` | `10000` | A timed-out heartbeat means the Worker is hard-filtered | | `COORDINATOR_WATCH_GRACE_PERIOD_MS` | `Core.WatchGracePeriodMs` | `10000` | Must be `<= HeartbeatTimeoutMs` | | `COORDINATOR_SLOT_TTL_MS` | `Core.SlotTTLMs` | `5000` | The `ttlMs` passed to the Worker | | `COORDINATOR_MAX_CANDIDATES` | `Core.MaxCandidates` | `3` | Maximum number of candidates tried per request | | `COORDINATOR_MAX_TIMEOUT_RETRIES` | `Handler.MaxTimeoutRetries` | `1` | Retries on the same Worker for uncertain outcomes; 0 is allowed | | `COORDINATOR_CIRCUIT_BREAKER_THRESHOLD` | `Core.CircuitBreakerThreshold` | `3` | Consecutive failure count | | `COORDINATOR_CIRCUIT_BREAKER_COOLDOWN_MS` | `Core.CircuitBreakerCooldownMs` | `10000` | Circuit breaker duration | | `COORDINATOR_CPU_RED_LINE` / `COORDINATOR_MEMORY_RED_LINE` / `COORDINATOR_ERROR_RATE_RED_LINE` | `Core.CPURedLine` / `MemoryRedLine` / `ErrorRateRedLine` | `0.90` / `0.90` / `0.50` | Valid range `(0, 1]` | The observability-related `BLOCKX_PROMETHEUS_LISTEN_ADDR`, `BLOCKX_PROMETHEUS_PATH`, `CHAINTABLE_LOG_BROKERS`, and `CHAINTABLE_LOG_TOPIC` are handled centrally by `internal/obs`; see [Observability](/en/components/observability). `COORDINATOR_CPU_PROFILE` / `COORDINATOR_TRACE` make `app.Run` write pprof / runtime trace files for the entire process lifetime, and are only for performance troubleshooting. Profile default registries: * `cmd/coordinator`: `/blockx/workers/`. * `cmd/bundle_coordinator`: `/blockx/bundle-workers/` and `/blockx/prod/lanes/default/bundle-workers/`. * Test clusters replace them entirely with `COORDINATOR_REGISTRY_PREFIXES=/blockx/test/lanes/default/bundle-workers/`. With multiple prefixes, `EtcdWorkerWatcher` merges by `WorkerAddr`: the entry with the newer `LastHeartbeatMs` wins; on a tie, the v2 format beats legacy; on a further tie, the prefix strings are compared. A Worker is removed from the view only after it has disappeared from every prefix. See `docs/specs/2026-07-28-registry-prefix-migration.md` in the blockx repo for details. ## Extension points * **Changing the selection policy**: touch only `SelectCandidate` and `candidateRanksBefore` in `internal/coordinator/core/coordinator.go`, and add `TestSelectCandidate_*` cases in `coordinator_test.go`. The core has no IO, so it can be tested as pure functions. Ranking is currently deterministic; the absolute memory field in the heartbeat (`memoryAvailableBytes`) is only parsed and takes no part in filtering or ranking. * **Adding a Worker feedback type**: add a `RecordSlotXxx` to the core and dispatch on `SlotError.Code` in `recordSlotOutcome` in `adapters/grpc_server.go`. That is how `RecordSlotBackpressure` was added. * **Changing Worker error classification**: `grpcErrToSlotError` in `adapters/worker_rpc.go` and `workerCodeToGRPC` on the Worker side (`internal/worker/adapters/wire.go`) must remain inverses of each other; if you change one, check the other (see [Worker](/en/components/worker)). * **Adding a deployment form**: add a `cmd//main.go` that declares an `app.Profile`, add branches in `app.registerCoordinatorService` and `validateRegistryPrefixes`, and allow the new worker kind in `ParseRegistryPrefix` in `api/etcd/registry.go`. * **Replacing the Worker RPC implementation**: implement the `adapters.SlotReserver` interface; `CoordinatorServer` does not depend on a concrete client. The current `WorkerRPCClient` opens exactly one `grpc.ClientConn` per `workerAddr` and does not use the multi-connection balancer from `internal/sdk/grpcclient`: a point-to-point client targeting a single Worker does not enable a multi-connection strategy. See `docs/specs/2026-07-23-grpc-client-balancing.md` in the blockx repo for background. * **Adding a configuration item**: add the field and its `Validate` rule in `core.Config` / `adapters.HandlerConfig` / `adapters.EtcdWatcherConfig`, then add environment variable parsing in `LoadCoordinatorRuntimeConfigFromEnv` in `app/config.go`, and extend `config_test.go`. ## Testing ```bash theme={null} # core / adapters / app unit tests, plus registry prefix parsing go test ./internal/coordinator/... ./api/etcd/... # Process-level tests: build the real binary, start embedded etcd, verify gRPC readiness and error mapping go test ./cmd/coordinator/... ./cmd/bundle_coordinator/... # Contract-level E2E (includes coordinator-etcd and coordinator-worker) go test -v -timeout 300s ./e2e/contract/... ``` Test organization: * `internal/coordinator/core/coordinator_test.go`: every `SelectCandidate` filter condition and ranking rule, circuit breaker trip/recovery, the grace period semantics of `IsReady`, `ApplyFullWorkerList` preserving local fields. * `internal/coordinator/adapters/handler_test.go`: covers the orchestration loop with a mock `SlotReserver` (switching candidates after a timeout, returning `Aborted` when retries still time out, returning `ResourceExhausted` when all reject, retrying on an empty `SlotID`), plus the error codes and compression behavior of the real gRPC server. * `internal/coordinator/adapters/worker_rpc_test.go`: `MayHaveAllocated` classification in `grpcErrToSlotError` (connection refused vs timeout/cancel/`Aborted`). * `internal/coordinator/adapters/etcd_watcher_test.go`: list+watch on embedded etcd, multi-prefix merge and deduplication, readiness state across disconnect/reconnect, skipping unparseable values. * `internal/coordinator/app/config_test.go`, `profile_test.go`: environment variable parsing and `Profile` validation. * `e2e/contract/coordinator-etcd/`: when etcd starts after the Coordinator, first `Unavailable` and then recovery; routing still works when the watch recovers within the grace period; becomes `Unavailable` beyond the grace period. * `e2e/contract/coordinator-worker/`: an explicit transport failure switches to the next Worker, a `RequestTaskSlot` timeout returns `SlotUncertain` (`Aborted`), `SubmitTask` is rejected after the slot expires, an old `slotId` is invalid after a Worker restart, and rejection when the unified slot pool is full. E2E tests start real processes via `StartCoordinator` / `StartBundleCoordinator` in `internal/testutil`, using `COORDINATOR_*` environment variables to shorten the timing parameters. See [Testing](/en/development/testing) for the overall test organization. ## Related docs Specs in the blockx repo: * `docs/specs/task-resource-coordinator.md`: detailed Coordinator design; protocol, error semantics, selection rules, consistency boundaries. * `docs/specs/architecture.md` §4.1: the control plane's position in the overall architecture and its three boundaries. * `docs/specs/2026-07-28-registry-prefix-migration.md`: legacy / v2 registry prefixes, multi-prefix merging, and rollout order. * `docs/specs/2026-07-23-bundle-coordinator-api.md`: why the bundle Coordinator has a separate service identity. * `docs/specs/2026-07-23-grpc-client-balancing.md`: the scope of the multi-connection balancer. * `docs/specs/worker.md`: the authoritative definition of `RequestTaskSlot` and the slot state machine. Site pages: * [Worker](/en/components/worker): the Worker-side implementation of `RequestTaskSlot` and etcd heartbeat publishing. * [Bundle clusters](/en/components/bundle): bundle clusters and `cmd/bundle_coordinator`. * [Protocols and interfaces](/en/architecture/protocols): gRPC / etcd protocol summary. * [Task lifecycle](/en/architecture/task-lifecycle): the full path from `ReserveWorkerSlot` to `SubmitTask`. * [Observability](/en/components/observability): how metrics, logs, and tracing are wired in.