> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockx.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Sync Invoker
> Synchronous function-call entry point: one gRPC unary request executes one Python function and returns the result, call_id, and attributed execution duration in place
Sync Invoker (binary `syncinvoker`) is BlockX's synchronous function-call service. The caller sends one gRPC unary `Invoke`, the server executes the function once in its local Python executor pool, and the same RPC returns the result, `call_id`, and attributed execution duration. It is deployed independently, bypasses the Coordinator, and does not go through the Worker's slot / task flow; it only reuses the Worker's executor pool, UDS protocol, Function Code View, and read-only IO subsystem. See [Architecture overview](/en/architecture/overview) for where it sits overall.
## Responsibilities and boundaries
Responsible for:
* Accepting `Invoke` / `DebugInvoke`, executing a registered function (`function_id`) or ad-hoc source (`inline_source`), and returning `result_json` / `Failure`, `execution_duration_ms`, `queue_wait_ms`.
* admission: among `N` executors × `K` context slots, pick the executor with the lowest load and dispatch to it; return `RESOURCE_EXHAUSTED` when full.
* deadline: at the deadline only send a soft `CancelCall`, then wait for the executor's real terminal state (cooperative cancel / SIGALRM) or the fallback sweep (wedge / heartbeat timeout) to converge.
* subcall: subcalls issued by a parent function are dispatched on the same executor; parent and children share one deadline, one code epoch, and one IO scope.
* `Precheck`: only statically audits the source without executing it; returns `pass + findings`.
* Execution attribution: `execution_duration_ms` = effective CPU of the whole subcall tree + this call's IO backend duration; `EXECUTOR_LOST` always counts as 0 and is retryable.
* Failure-origin labeling: every failure carries a `FailureOrigin` (`USER` / `SYSTEM` / `CAPACITY` / `UNKNOWN`); admission-layer failures carry it via `google.rpc.ErrorInfo.metadata`.
Not responsible for:
* Authentication, API keys, rate limiting, billing, business-level concurrency control (gateway side).
* Any write path: `HandleIO` rejects `mode == "write"`; no BlockDB writer / Event Writer / ResultHandler is wired in.
* task orchestration: no readyQueue, fairness, failure retry, call result cache, callbuilder / writer plugin.
* Server-side retry: a call enters `EXECUTING` at most once; retries are new calls issued by the SDK.
Core invariants:
* Terminal states are irreversible: `core.Registry.Finish` is the single terminal gate and lets each call through only once; late executor / IO / subcall results are discarded.
* A call enters `EXECUTING` at most once (`Registry.Bind` only accepts `ACCEPTED`).
* Every sync-call (including inline) pins the current FCV epoch at `ACCEPTED`, sends it down via `ExecuteCallPayload.TaskCodeEpoch`, and subcalls inherit it.
* Metering numbers come only from the executor's real terminal payload, or are explicitly 0 on `EXECUTOR_LOST`; they are never inferred from heartbeat snapshots.
* Configuration constraint: `MaxCallDeadlineMs + heartbeat_interval < SchedulerStallTimeoutMs`, otherwise the process refuses to start (`cmd/syncinvoker/main.go`).
* `[reg.Bind → SendExecuteCall]`, subtree cleanup on deadline / client-gone, and executor-lost fan-out all share `Service.dispatchMu` and are pairwise mutually exclusive.
## Code location
| Path | Purpose |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cmd/syncinvoker/main.go` | Process assembly: executor UDS adapter, IO scope, FCV, audit pool, `Service`, pool manager, reclaim sweeper, gRPC server, drain shutdown |
| `cmd/syncinvoker/config.go` | `Config` / `DefaultConfig` / `LoadConfigFromEnvChecked`, all environment variables and defaults |
| `cmd/syncinvoker/function_code.go` | `setupFunctionCodeView`: one of Redis hash source, BlockDB syncer, or in-memory devstub |
| `cmd/syncinvoker/io.go` | `ioBackendModules` / `setupIOScope`: read-only IO backend assembly (noderpc / blockdb / blockdb bridge / localtestservice / router) |
| `cmd/syncinvoker/health.go` | `/healthz` liveness, `/readyz` readiness (FCV not devstub, ≥1 healthy executor, not draining) |
| `internal/syncinvoker/core/call.go` | `CallState`, `FailureCode`, `FailureOrigin`, `Failure`, `Terminal`, `DebugOutput` |
| `internal/syncinvoker/core/registry.go` | `Registry`: call table + executor index + attach generation, terminal gate |
| `internal/syncinvoker/core/admission.go` | `PickExecutor` / `RankExecutors`: pick an executor by in-flight count |
| `internal/syncinvoker/adapters/service.go` | `Service`: admission → dispatch → wait main flow, subcall, deadline soft cancel / hard kill, terminal-state classification, duration aggregation, `Precheck` |
| `internal/syncinvoker/adapters/grpc_server.go` | `GRPCServer`: proto ↔ `InvokeInput` / `InvokeOutput` mapping, backfills `call_id` into the trailer |
| `internal/syncinvoker/adapters/exec_adapter.go` | `ExecutorAdapter` interface seam; the production implementation is the worker's `*executor.Adapter` |
| `api/grpc/syncinvoker/sync_invoker.proto` | `SyncInvokerService` protocol; generated code in `syncinvokerpb/` |
| `internal/obs/metrics_syncinvoke.go` | All `blockx_syncinvoke_*` metric definitions |
| `internal/syncinvoker/**/*_test.go` | core / adapters unit tests (stub executor adapter) |
| `cmd/syncinvoker/*_test.go` | Process-level E2E (real Python executor), perf, sandbox e2e, config tests |
| `docs/specs/sync-invoker.md` | Main spec |
| `docs/specs/sync-invoker-failure-origin.md` | `FailureOrigin` classification scheme |
| `docs/sync-invoker-grafana.md` | Metric semantics and troubleshooting paths |
`cmd/syncinvoker/` is about 1100 lines of non-test code, heavier than a typical `cmd/` entry point. The reason is that it has no `internal/syncinvoker/app/` package: the executor pool, the three FCV sources, IO backend assembly, and health probes all mirror `cmd/worker`'s assembly logic directly inside `cmd/`. Before changing the assembly, check whether the corresponding `cmd/worker` file already has an equivalent implementation.
## Core types and interfaces
### core (Sans-IO)
* `core.CallState` (`internal/syncinvoker/core/call.go`): `ACCEPTED` / `EXECUTING` + six terminal states; `IsTerminal()`.
* `core.FailureCode` / `core.FailureOrigin` / `core.Failure`: same values as the proto enums but without depending on proto; `FailureTerminal(f)` picks the terminal state by code and fills in the origin with `defaultFailureOrigin`.
* `core.Terminal`: the final result handed to the waiter, carrying `EffectiveDurationUs`, `QueueWaitUs`, `IOBackendDurationMs`, `Debug`.
* `core.Registry`: `Accept` / `AcceptSubcall` / `Bind` / `Finish` / `Drop` / `Subtree` / `CallsByRoot` / `InflightCount` / `MarkTerminating` / `ObserveSnapshot`. Executor health is only `HealthReady` / `HealthTerminating`; the attach generation distinguishes a replacement from the ghost of a killed process.
* `core.CallMeta`: the `Epoch` pinned at accept, `CacheKey` (cycle detection), `DeadlineMs`, `Root`.
* `core.PickExecutor(loads, maxInflight)` / `core.RankExecutors(loads, maxInflight, shuffle)` (`admission.go`): the default strategy and the admit-spread strategy.
### adapters
* `adapters.ServiceConfig` (`service.go`): `DefaultDeadlineMs`, `ExecutorMaxInflight`, `MaxCallDeadlineMs`, `MaxAcceptedDeadlineMs`, `SchedulerStallTimeoutMs`, `AdmitSpread`.
* `adapters.Service`: `NewService(cfg, exec, codeView)` hooks the callbacks onto the executor adapter at construction time (`wireExecutorCallbacks`). Main methods:
```go theme={null}
func (s *Service) Invoke(ctx context.Context, in InvokeInput) (InvokeOutput, error)
func (s *Service) Precheck(ctx context.Context, in PrecheckInput) (PrecheckOutput, error)
func (s *Service) HandleIO(ctx context.Context, taskID, callID string, req *executor.IORequest) (any, error)
func (s *Service) HandleSubcall(ctx context.Context, executorID, parentCallID, requestID, functionID string, argsJSON json.RawMessage, ancestry []string, _ int64) error
func (s *Service) CheckWedgedExecutors()
```
* `adapters.ExecutorAdapter` (`exec_adapter.go`): `Bind` / `BindSubcall` / `Unbind` / `MarkDraining` / `SendExecuteCall` / `SendResumeCall` / `SendCancelCall` / `Snapshots` + `SetCallbacks` / `SetExecutorLostCallback` / `SetIOHandler` / `SetSubcallHandler`. Tests inject a stub; production passes `*executor.Adapter`.
* `adapters.FunctionCodeView`: `CurrentEpoch()` + `Fetch(functionID, epoch)`; `fccore.SnapshotStore` satisfies it directly. May be nil (inline only).
* `adapters.GRPCServer` (`grpc_server.go`): the three handlers `Invoke` / `DebugInvoke` / `Precheck`; on admission failure `mirrorCallID` writes `call_id` into the trailer `x-blockx-sync-call-id`.
* `callScope` (`service.go`, unexported): a per-sync-call `iocore.TaskIOScope` + cancel + in-flight IO counter; the whole subcall tree shares the root's scope.
### Protocol (`api/grpc/syncinvoker/sync_invoker.proto`)
| RPC | Request | Response | Notes |
| ------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| `Invoke` | `InvokeRequest{oneof function_id / inline_source, args_json, deadline_unix_ms}` | `InvokeResponse{call_id, success, result_json, failure, execution_duration_ms, queue_wait_ms}` | Open API path |
| `DebugInvoke` | `DebugInvokeRequest` (same fields as above) | `DebugInvokeResponse` (adds `debug_output`: last 256KB of stdout/stderr, `error_stack`, `local_vars_json`) | Page-debugging path; the only place a traceback is returned |
| `Precheck` | `PrecheckRequest{source_code}` | `PrecheckResponse{pass, findings[]}` | Audit only, no execution |
Two layers of error semantics: admission-layer failures return a non-OK gRPC status (`INVALID_ARGUMENT` / `NOT_FOUND` / `RESOURCE_EXHAUSTED` / `UNAVAILABLE`) with `call_id` and `failure_origin` in `ErrorInfo.metadata`; execution-layer failures return `OK + success=false + Failure{code, message, retryable, origin}`.
## Data flow / execution flow
`Service.Invoke` is the main path. core only makes decisions (`Registry` state transitions, `PickExecutor`); the adapter does Bind / Send / timing / IO around it.
```mermaid theme={null}
sequenceDiagram
participant C as Client
participant G as GRPCServer
participant S as Service
participant R as core.Registry
participant X as executor.Adapter
participant P as Python executor
participant IO as callScope
C->>G: Invoke(function_id or inline_source, args_json, deadline)
G->>S: Invoke(InvokeInput)
S->>S: pinEpoch, resolveSource(FCV Fetch), auditSource
S->>R: Accept(callID, CallMeta)
S->>X: Snapshots, readyLoads, PickExecutor
S->>X: Bind(callID, executorID)
S->>R: Bind(callID, executorID) enters EXECUTING
S->>IO: openCallScope(deadline)
S->>X: SendExecuteCall(ExecuteCallPayload)
X->>P: ExecuteCall over UDS
P-->>X: IO request or subcall request
X->>S: HandleIO or HandleSubcall
S->>IO: read-only Read
S->>X: BindSubcall + SendExecuteCall(child) on the same executor
P-->>X: CallCompleted or CallFailed
X->>S: onCallTerminal, Registry.Finish, signal(w.ch)
alt terminal state arrives first
S-->>G: InvokeOutput(result, durations)
G-->>C: InvokeResponse
else deadline arrives first
S->>X: SendCancelCall(subtree and root)
S->>S: bounded wait on w.ch, deadlineEscalationWindow
S-->>G: InvokeOutput(Failure TIMED_OUT)
G-->>C: InvokeResponse success=false
end
```
Key steps (`service.go`):
`pinEpoch` reads `FunctionCodeView.CurrentEpoch()`; `resolveSource` checks that exactly one oneof field is set, `Fetch`es `function_id` at that epoch, and for inline returns the source directly while still pinning the epoch.
`auditSource` decides whether to gate according to `FUNCTION_CODE_AUDIT_MODE`: off skips auditing, enforce fails closed, dark only logs; functions whose space is in the allowlist are skipped. `normalizeArgsJSON` validates the JSON array and compacts whitespace in a single scan; the same bytes serve as the cycle key and are sent to the executor.
`admit`: `readyLoads` filters `Snapshots` for executors that are `Healthy` with `AvailableContexts > 0`, reconciles attach generations via `Registry.ObserveSnapshot`, then uses `Registry.InflightCount` (Bind +1 / Finish -1) as the load key; `PickExecutor` picks the lowest load, a failed `exec.Bind` is retried once with a new pick, and if it still fails the result is `UNAVAILABLE`. With `ADMIT_SPREAD=1` it switches to `RankExecutors` and tries `Bind` on each candidate in turn; exhausting the candidates yields `RESOURCE_EXHAUSTED`.
`openCallScope` creates the IO scope; `SendExecuteCall` sends `uds.ExecuteCallPayload{TaskID: callID, TaskCodeEpoch, CallDeadlineMs, Debug, Audit}`. Then a three-way `select`: terminal state on `w.ch`, the deadline timer, and `ctx.Done()`.
`onCallTerminal` passes through the single-shot `Registry.Finish` gate, `Unbind`s, folds `treeEffective` (accumulated effective µs of subcalls) into the root, closes the scope and `FinalizeCallIO`, and `signal`s the waiter. `buildOutput` computes `execution_duration_ms = effective/1000 + IOBackendDurationMs`; `EXECUTOR_LOST` is always 0. `queue_wait_ms` is taken directly from the terminal payload's `QueueWaitUs`, reported by the executor, and is not folded into `execution_duration_ms`.
## State and lifecycle
Transitions of `core.CallState` are owned exclusively by `Registry`. `TERMINAL_ADMISSION_DENIED` lands in two ways: a failure before Bind goes through `Registry.Drop` (no terminal state is recorded; the call is simply removed from the table); only when `openCallScope` / `SendExecuteCall` fails after Bind does it `Finish(StateTerminalAdmissionDenied)`.
```mermaid theme={null}
stateDiagram-v2
[*] --> ACCEPTED: Registry.Accept
ACCEPTED --> EXECUTING: Registry.Bind
ACCEPTED --> TERMINAL_ADMISSION_DENIED: Registry.Drop, validation, audit, no capacity
EXECUTING --> TERMINAL_ADMISSION_DENIED: openCallScope or SendExecuteCall failed
EXECUTING --> TERMINAL_SUCCESS: CallCompleted
EXECUTING --> TERMINAL_FAILURE: CallFailed, CODE_LOAD, CALL, IO_PROXY
EXECUTING --> TERMINAL_TIMEOUT: terminal state received after deadline soft cancel
EXECUTING --> TERMINAL_EXECUTOR_LOST: fanOutExecutorLost
EXECUTING --> TERMINAL_CLIENT_DISCONNECTED: onClientGone
TERMINAL_SUCCESS --> [*]
TERMINAL_FAILURE --> [*]
TERMINAL_TIMEOUT --> [*]
TERMINAL_EXECUTOR_LOST --> [*]
TERMINAL_CLIENT_DISCONNECTED --> [*]
TERMINAL_ADMISSION_DENIED --> [*]
```
The deadline path (`onDeadline`) converges in layers, and every layer yields a real terminal state:
1. Only send a soft `SendCancelCall` to the `Subtree` and the root, without `Finish`ing either; `quiesceCallScope` stops new IO.
2. Bounded wait on `w.ch`, with window `deadlineEscalationWindow = max(5s, 2×SchedulerStallTimeoutMs + 2s)`. All four paths deliver the signal: the executor's cooperative cancel or SIGALRM self-heal sends a real terminal state; `CheckWedgedExecutors` detects a scheduler frozen for longer than `SchedulerStallTimeoutMs` whose `RunningCallId` is past its deadline → `reapExecutorCause("wedge_sweep")`; heartbeat timeout → the adapter's executor-lost callback → `reapExecutor`; window exceeded → `reapExecutorCause("deadline_escalation")` forces convergence.
3. After the terminal state arrives, `CallsByRoot` clears subcalls added during the grace period, and the final `Failure` is uniformly overridden to `TIMED_OUT` (origin `UNKNOWN`).
Client disconnect (`onClientGone`) is different: it immediately `Finish`es the root, cancels the subtree, `MarkDraining`s, and returns the gRPC context error; only subcall CPU and IO durations already observed on the Go side are recorded.
Executor side: `Registry` tracks only `{ready, terminating}`. `reapExecutor` marks the executor terminating, fans out `EXECUTOR_LOST` to all in-flight calls, and calls `PoolManager.KillExecutor`; the pool's `waitLoop` spawns a replacement process, which rejoins admission only once `ObserveSnapshot` sees a higher attach generation.
## Differences from Worker
| Dimension | Worker | Sync Invoker |
| ---------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
| Entry point | Coordinator placement → `RequestTaskSlot` → `SubmitTask`, async callback | gRPC unary straight to the instance, synchronous return; horizontal scaling via gateway LB |
| Abstraction | task → callList → call, orchestrated by the task dispatcher | Only sync-call; at the protocol layer `taskId` is an alias for `sync_call_id` |
| Retry / cache / plugin | Yes | No |
| IO | `TaskIOScope`, read and write | One `callScope` (`TaskIOScope`) per call, read-only, `DisableSharedReadCache: true` |
| Write path | BlockDB / Event Writer / ResultHandler | All disabled |
Reuse boundary (`docs/specs/sync-invoker.md` §6):
* `internal/worker/adapters/executor`: `Adapter` (UDS server, `Bind` / `BindSubcall` / `Unbind` / `MarkDraining`, heartbeat, `Snapshots`), `PoolManager` (forks N Python executors, restarts on crash, `KillExecutor`), `IOHandler` / `SubcallHandler` / `IORequest` interfaces.
* `internal/worker/core`: only `ExecutorSnapshot` is used.
* `internal/worker/devstub`: `StubBackendAdapter`, the IO stub when no external backend is configured.
* `internal/functioncode/*`: `SnapshotStore`, Redis / BlockDB syncer, `audit.Pool`.
* `internal/io/core`, `internal/io/assembly`, `internal/io/adaptive`: `WorkerIOScope` / `TaskIOScope`, backend assembly, AIMD admission.
* `internal/sdk/*`: blockdb / noderpc / localtestservice / router clients.
* `api/uds`: `ExecuteCallPayload` / `CallCompletedPayload` / `CallFailedPayload` / `ResumeCallPayload`.
For Worker internals see [Worker](/en/components/worker), [Call execution subsystem](/en/components/call-execution), [IO access subsystem](/en/components/io-subsystem), and [Function Code View](/en/components/function-code).
## Precheck and audit gate
`Precheck` and Invoke's audit gate share the same `audit.Auditor` (`audit.NewPool`, started at process startup for all three modes off / enforce / dark, `Size: 2`), but the two are decoupled:
* `FUNCTION_CODE_AUDIT_MODE` only controls the Invoke gate: `SetAuditGateOff(true)` (off) / `SetAuditDark(true)` (dark) / neither (enforce).
* `Precheck` ignores the mode and the space allowlist, always audits under the `_` entry, and returns the real verdict. A failed audit is `OK + pass=false + findings`; only when no verdict can be produced is the status non-OK: `FAILED_PRECONDITION` (no auditor), `UNAVAILABLE` (auditor down), `INVALID_ARGUMENT` (empty source), `CANCELED` / `DEADLINE_EXCEEDED` (caller ctx canceled, counted separately).
* In enforce mode, an auditor startup failure prevents the process from starting; off / dark degrade to no auditor (Invoke is not gated, Precheck returns `FAILED_PRECONDITION`).
## Failure classification and attribution
Execution-layer failures are mapped in `failedTerminal` (`service.go`) from `uds.CallFailedPayload.ErrorKind`, without parsing the message:
* `classifyFailure`: `execute_call_rejected` → `CODE_LOAD_FAILED`; `runtime_error` / `cancelled` → `CALL_FAILED`; `deadline_exceeded` → `TIMED_OUT`; `transport_lost` / `resume_send_failed` / `terminal_decode_failed` → `EXECUTOR_LOST`; `executor_capacity_exhausted` → `EXECUTOR_CAPACITY_EXHAUSTED`; the IO classifier's `system_error` / `retryable_io` / `param_error` / `non_retryable` / `timeout` / `scope_closed` → `IO_PROXY_FAILED`.
* `failureRetryable`: `EXECUTOR_LOST` / `EXECUTOR_CAPACITY_EXHAUSTED` are always true; `IO_PROXY_FAILED` honors the payload's retryable flag; everything else is false.
* `classifyFailureOrigin`: by `ErrorKind` + internal `DetailCode` (e.g. `grpc:NotFound` → `USER`, `syncinvoker:io_not_configured` → `SYSTEM`) + `ChildOrigin` (`subcall_failed` inherits the subcall's origin); `deadline_exceeded` is `UNKNOWN`. The full table is in `docs/specs/sync-invoker-failure-origin.md`.
* Admission layer: `denyAdmission` → `withCallID` writes `call_id` and `failure_origin` into `ErrorInfo.metadata`; `classifyAdmissionOrigin` determines the origin by reason / gRPC code.
## Configuration
Everything lives in `cmd/syncinvoker/config.go`; environment variables override `DefaultConfig()`.
| Environment variable | Field | Default | Description |
| ------------------------------------------------------------ | --------------------------------- | ------------------------- | ---------------------------------------------------------------------------- |
| `SYNC_INVOKER_LISTEN` | `ListenAddr` | `localhost:18080` | gRPC listen address |
| `EXECUTOR_COUNT` | `ExecutorCount` | `4` | N: number of Python executor processes; ≤0 falls back to the default |
| `EXECUTOR_MAX_INFLIGHT` | `ExecutorMaxInflight` | `4` | K: max concurrent top-level calls per executor; `0` = unlimited |
| `EXECUTOR_MAX_RUNNABLE` | `ExecutorMaxRunnable` | `128` | Cap on the RUNNABLE queue inside the executor |
| `DEFAULT_DEADLINE_MS` | `DefaultDeadlineMs` | `100` | Default deadline when the request has `deadline_unix_ms=0` |
| `MAX_CALL_DEADLINE_MS` | `MaxCallDeadlineMs` | `100` | Deadline cap; `0` = bounded only by FCV retention |
| `HEARTBEAT_INTERVAL_S` | `HeartbeatIntervalS` | `0.2` | Executor heartbeat interval |
| `HEARTBEAT_TIMEOUT_MS` | `HeartbeatTimeoutMs` | `2000` | Heartbeat timeout after which the executor is considered lost |
| `SCHEDULER_STALL_TIMEOUT_MS` | `SchedulerStallTimeoutMs` | `2000` | wedge sweep threshold; `0` disables wedge reaping |
| `FUNCTION_SNAPSHOT_RETENTION_MS` | `FunctionSnapshotRetentionMs` | `DefaultDeadlineMs + 10s` | FCV snapshot retention; also serves as `MaxAcceptedDeadlineMs` |
| `ADMIT_SPREAD` | `AdmitSpread` | `false` | Enables the admit-spread strategy |
| `FUNCTION_CODE_AUDIT_MODE` | `FunctionCodeAuditMode` | `off` | `off` / `enforce` / `dark` |
| `FUNCTION_CODE_AUDIT_SPACE_ALLOWLIST` | `FunctionCodeAuditSpaceAllowlist` | empty | Spaces that skip the audit gate, comma-separated |
| `FUNCTION_CODE_REDIS_URL` | `FunctionCodeRedis.URL` | empty | When set, FCV uses Redis, taking precedence over BlockDB |
| `BLOCKDB_TABLE_READ_ADDR` / `_SCAN_ADDR` / `_SUBSCRIBE_ADDR` | `BlockDB.*` | empty | All three set: BlockDB FCV; all empty: devstub; partially set: startup error |
| `NODE_RPC_ENDPOINT` | `NodeRPCEndpoint` | empty | When set, the `rpc` backend uses the real node RPC, otherwise a stub |
| `EXECUTOR_SPAWN_MODE` | `ExecutorSpawnMode` | `process` | `sandbox` requires `EXECUTOR_SANDBOX_*` |
| `BLOCKX_PROMETHEUS_LISTEN_ADDR` | `Prometheus.Addr` | empty | Enables `/metrics`, `/readyz`, `/healthz` |
`ShutdownTimeoutMs` is fixed at `5000` with no environment variable override. `FunctionSnapshotRetention()` feeds both the FCV syncer and `ServiceConfig.MaxAcceptedDeadlineMs`.
## Observability
Metrics are defined in `internal/obs/metrics_syncinvoke.go`; semantics and dashboards are in `docs/sync-invoker-grafana.md` in the blockx repo. The most important ones:
* `blockx_syncinvoke_calls_total{status, failure_code}`: execution-layer terminal-state count (excludes admission rejections).
* `blockx_syncinvoke_admission_denied_total{reason}`: rejections before dispatch.
* `blockx_syncinvoke_call_duration_milliseconds` / `blockx_syncinvoke_billed_duration_milliseconds` / `blockx_syncinvoke_queue_wait_milliseconds`: end-to-end wall clock, attributed duration, queueing.
* `blockx_syncinvoke_deadline_overruns_total{resolved_by}`: which layer converged the calls that overran their deadline (`completed_late` / `cooperative_cancel` / `in_process_timeout` / `executor_reaped` / `escalation_reap`).
* `blockx_syncinvoke_executor_reaps_total{cause}` / `blockx_syncinvoke_executor_lost_victims_total{cause}`: hard-kill frequency and collateral count; should be 0 when healthy.
* `blockx_syncinvoke_precheck_total{result}`.
tracing: `obs.Tracer("blockx-syncinvoker")`; under the `syncinvoker.invoke` span are `resolve_source` / `admission` / `dispatch` / `wait_terminal` / `build_output`.
## Deployment shape
sync-invoker is deployed as an independent fleet parallel to the worker fleet (EC2 systemd + `nerdctl` + host containerd sandbox), gRPC on `9900`, metrics/health on `9901`; see [Deployment overview](/en/development/deployment) and `docs/deploy/syncinvoker-systemd-nerdctl.md` in the blockx repo for details.
## Extension points
* Adding a failure classification: change `classifyFailure` / `failureRetryable` / `classifyFailureOrigin` (`service.go`); a new code also requires updating `core.FailureCode`, the proto `FailureCode`, `failureCodeToProto`, and `failureCodeLabel`.
* Adding an admission strategy: add a pure function in `core/admission.go` and switch on it in `Service.admit` via a `ServiceConfig` flag; depend only on `ExecutorLoad`.
* Adding a read-only IO backend: append an `assembly.Module` to `ioBackendModules` in `cmd/syncinvoker/io.go`, keeping the registration identical to `cmd/worker`.
* Adding an FCV source: add a branch to `setupFunctionCodeView` in `cmd/syncinvoker/function_code.go` that returns `functionCodeSetup{view, start, close, desc, devstub}`.
* Adding an RPC: change `api/grpc/syncinvoker/sync_invoker.proto` → `make proto` → add a handler in `GRPCServer` → add transport-agnostic `XxxInput` / `XxxOutput` to `Service`.
* Adding a config option: a `Config` field + `DefaultConfig` + `envXxx` in `LoadConfigFromEnvChecked`, passed through to `ServiceConfig` when needed.
## Testing
```bash theme={null}
# First use, or after python/pyproject.toml changes
uv sync --project python
# core / adapters unit tests (stub executor, no Python started)
go test ./internal/syncinvoker/...
# Process-level E2E (real Python executor; pythonBin() prefers python/.venv)
go test -v -timeout 300s ./cmd/syncinvoker/...
```
* `internal/syncinvoker/core/*_test.go`: `Registry` terminal gate, executor health / attach generation, `PickExecutor` / `RankExecutors`.
* `internal/syncinvoker/adapters/service_test.go`: admission, subcall races, `onDeadline` / `onClientGone`, duration aggregation, failure classification; `audit_gate_test.go`: enforce / dark / allowlist; `precheck_test.go`: the various Precheck return codes.
* `cmd/syncinvoker/syncinvoker_process_test.go`: `startSyncInvoker` launches a real process and covers inline / function\_id / debug / read-only IO / write rejection / subcall / `RESOURCE_EXHAUSTED` / CPU hog self-heal / wedge hard kill / graceful shutdown / health probes.
* `cmd/syncinvoker/syncinvoker_perf_test.go`: latency-breakdown perf test; `syncinvoker_sandbox_e2e_test.go` carries the build tag `sandbox_e2e`.
* The `Makefile` lists `cmd/syncinvoker` under `SLOW_TEST_PACKAGES`, and `make test` runs it file by file (each file with its own `SYNCINVOKER_TEST_TIMEOUT`, default 3m) to avoid a whole-package timeout.
See [Testing](/en/development/testing) for more.
## Related docs
* blockx repo `docs/specs/sync-invoker.md`: shape, protocol, admission, deadline layering, subcall, lifecycle, reuse boundary, attribution, capacity.
* blockx repo `docs/specs/sync-invoker-failure-origin.md`: `FailureOrigin` semantics and mapping table.
* blockx repo `docs/sync-invoker-grafana.md`: metric list, PromQL, troubleshooting paths.
* blockx repo `docs/deploy/syncinvoker-systemd-nerdctl.md`: EC2 runbook.
* On this site: [Worker](/en/components/worker), [Call execution subsystem](/en/components/call-execution), [Python Executor](/en/components/python-executor), [Function Code View](/en/components/function-code), [IO access subsystem](/en/components/io-subsystem), [Observability](/en/components/observability), [Protocols and interfaces](/en/architecture/protocols).