> ## Documentation Index > Fetch the complete documentation index at: https://docs.blockx.chaintable.com/llms.txt > Use this file to discover all available pages before exploring further. # Backend Adapter > The BlockDB, NodeRPC, logical-types, Iceberg, and other backend adapters under internal/sdk: how to implement the BackendAdapter interface, wire it into the IO subsystem, and add a new one Backend adapters are the client layer through which BlockX accesses external services. All of the code lives in `internal/sdk/` (the directory name is historical; it contains clients for each service, not a published SDK). Each subpackage wraps one remote service (BlockDB gRPC, chain node JSON-RPC, Meta gRPC, Iceberg/Glue) as an `iocore.BackendAdapter` that the IO subsystem understands, and the Worker or sync-invoker assembles it into `WorkerIOScope` at startup. Builders, Writer Plugins, and Function Code View use them through `TaskIO.Read/Write` or by holding an adapter directly. See [Architecture overview](/en/architecture/overview) for the overall picture and [IO access subsystem](/en/components/io-subsystem) for the upper half of the call chain. ## Responsibilities and boundaries * Each SDK package does exactly one thing: translate `iocore.ReadReq` / `iocore.WriteReq` into one remote call and put the response into `ReadResult.Data` / `WriteResult.Data`. Request types are defined by the SDK package, not by `internal/io/core`. * The SDK does no caching, no singleflight, no task windows, and no admission control. All of that lives in the IO subsystem; the SDK only provides `Registration(...)` or an `assembly.Module` to help callers wrap the adapter in `adaptive.WrapBackend`. * The SDK is responsible for error classification: it implements `iocore.ErrorClassifier.ClassifyError` (deciding retryable / non-retryable / param) and `adaptive.OutcomeClassifier.ClassifyAdaptiveOutcome` (deciding whether AIMD should shrink). Local context cancellation / timeout never counts as downstream overload. * The BlockDB SDK exposes only atomic RPCs. The BatchWrite Init → PUT → Commit orchestration lives in `internal/plugin/event/batchwrite`, not in the SDK (`docs/specs/2026-07-30-bundle-write-batch-api.md` §2). * The bridge paths for the Python executor (BlockDB bridge, NodeRPC bridge, capability backends) only pass protobuf/JSON-RPC bytes through; they never rebuild request bodies. * Connection policy belongs to the adapter instance: gRPC clients pointing at an NLB use `grpcclient.NewMultiConn`, the same address within one adapter shares one `ClientConn`, and there is no process-level connection registry. * `internal/duckdb/borrowed` only provides a connection pool and result decoding on top of the native DuckDB bindings; SQL assembly, S3 credentials, and filter pushdown stay in `internal/plugin/callbuilder/bundlescan`. ## Code location | Path | Purpose | Main consumers (confirm by grepping imports) | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `internal/sdk/blockdb/` | BlockDB gRPC client: typed `Adapter`, executor-facing `BridgeAdapter`, streaming `ScanClient`, error classification, development stub | `internal/worker/app`, `cmd/syncinvoker`, `internal/plugin/event/{batchwrite,blockdbwrite,tableupserts,bundlewrite}`, `internal/plugin/callbuilder/dbscan`, `internal/functioncode/adapters/blockdb`, `internal/sdk/localtestservice` | | `internal/sdk/blockdb/proto/`, `gen/` | The five `blockdb.v1` proto files and the generated `blockdbpb` | All of the above + `internal/testutil/blockdb.go` | | `internal/sdk/noderpc/` | Chain node JSON-RPC HTTP client, transport rotation, semantic batching facade (`BatchingBackend`) | `internal/worker/app/app.go`, `cmd/syncinvoker/io.go` | | `internal/sdk/noderpc/batchobserved/` | Binds the batching facade's metric hooks to `internal/obs` | Same as above | | `internal/sdk/meta/proto/`, `gen/` | Meta service gRPC definitions (`TableMetaService`, `BlockMetaService`, `TimeMetaService`) | `internal/sdk/logicaltypes`, `e2e/system/logical_type`, `cmd/worker/*_test.go` | | `internal/sdk/meta/` | Legacy Meta HTTP adapter (`meta.Adapter`, `MetaReadReq`) | No production importer; see the note below | | `internal/sdk/logicaltypes/` | Backend that fetches table schemas through `TableMetaService.GetTable`; includes a local stub | `internal/worker/app`, `internal/plugin/schema/columns.go`, `internal/plugin/callbuilder/bundlescan` | | `internal/sdk/iceberg/` | Worker-side read-only Iceberg/Glue planner and bundle file index backend | `internal/worker/app/app.go`, `internal/plugin/callbuilder/bundlescan/source_resolution.go` | | `internal/sdk/localtestservice/` (with `proto/`, `gen/`) | Business-free sample capability backend (`HelloWorld`, `BlockDBGetRow`) | `internal/worker/app`, `cmd/syncinvoker/io.go`; the example client is in `examples/local_test_service/` | | `internal/sdk/router/` | Function discovery capability backend wrapping `github.com/chaintable/router/go` | `internal/worker/app`, `cmd/syncinvoker/io.go` | | `internal/sdk/grpcclient/` | Generic gRPC connection construction (`New`, `NewMultiConn`) and `DelimitedWriteRowEncoder` | `blockdb`, `logicaltypes`, `internal/plugin/event/batchwrite/delimited_rows.go`, `internal/worker/app` | | `internal/duckdb/borrowed/` | Connection pool, context interruption, and zero-copy result reading for the native DuckDB bindings | `internal/plugin/callbuilder/bundlescan/duckdb.go` | | `cmd/blockdb_grpc_stub/` | Standalone-process BlockDB benchmark stub server | Performance tests | | `docs/specs/2026-07-30-bundle-write-batch-api.md` | BatchWrite lifecycle and SDK contract | — | | `docs/specs/noderpc-semantic-batching.md` | NodeRPC semantic batching | — | | `docs/specs/2026-08-06-noderpc-binary-sidecar.md` | NodeRPC bridge binary pass-through | — | | `docs/specs/2026-07-23-grpc-client-balancing.md` | Multi-connection gRPC client | — | ## Core types and interfaces All adapters implement the same interface (`internal/io/core/model/adapter.go`): ```go theme={null} type BackendAdapter interface { Read(ctx context.Context, req ReadReq) (*ReadResult, error) Write(ctx context.Context, req WriteReq) (*WriteResult, error) Close() error } ``` ### blockdb * `AdapterConfig` (`adapter.go`): one address field per gRPC service (`TableReadAddr`, `TableWriteAddr`, `TableScanAddr`, `BlockReadAddr`, `BlockWriteAddr`, `BatchWriteAddr`, `BlockSubscribeAddr`, `L2BlockAddr`, `TimeReadAddr`, `TimeWriteAddr`) plus `MaxRecvMsgSize` (0 means 32 MiB). Identical addresses share one `ClientConn`; an unconfigured service returns `blockdb:service_not_configured` (a param-class error) when called. The `TableSubscribeAddr` field exists and is read from `BLOCKDB_TABLE_SUBSCRIBE_ADDR`, but `NewAdapter` never dials it; it is only used in `internal/worker/app/function_code.go` as one of the conditions for enabling the BlockDB Function Code View. * `Adapter` (`adapter.go`): the typed backend, `BackendBlockDB = "blockdb"`. `Read` uses a type switch to dispatch `ReadOp[*blockdbpb.GetRowRequest]`, `BatchGetRowsRequest`, `FilterRowsRequest`, `GetBlockRowsRequest`, `GetEventRequest`, `GetStateRequest`, and `GetValueRequest`, converting the `ResultSet` into a JSON row array; JSON/LIST/DICT logical columns are restored to objects according to `ColumnMeta.logical_type`. `Write` dispatches `WriteOp[*UpsertRowsRequest]`, `WriteOp[*DeleteRowsRequest]`, `WriteOp[*UpsertTimeRowsRequest]`, `BlockWriteReq`, `InitWriteJobReq`, `CommitWriteJobReq`, and `CancelWriteJobReq`. * `ReadOp[T]` / `WriteOp[T]` (`types.go`): generic wrappers that turn any proto request into an `iocore.ReadReq/WriteReq`, carrying `Key` (cache key), `Op` (stats name), and `RTO/WTO` (timeouts). `Filter`, `EqFilter`, `AndFilter`, and `FilterJSON` generate parameterized WHERE clauses; `Value(any)` converts to `*blockdbpb.Value`. * Three table semantics (`proto/table.proto`, `block.proto`, `time.proto`): Table (L1 row tables: `GetRow/BatchGetRows/FilterRows/UpsertRows/DeleteRows/Scan`), Block (L2 block-scoped: `GetEvent/GetState` take `current_block`, `UpsertBlock` takes `Block{block_id,height,timestamp}`, `L2BlockReadService.GetBlockRows`), and Time (`GetValue(row_id,time_at)`, `UpsertTimeRows(time_at)`). `BlockWriteReq` is a multi-table L2 write: `Data map[table][]*WriteRow`, and the adapter issues one `UpsertBlock` per table. * BatchWrite (`batch_write.go`, `proto/batch.proto`): `InitWriteJobReq/CommitWriteJobReq/CancelWriteJobReq` each wrap one `BatchWriteService` request, and the response is placed into `WriteResult.Data` as proto bytes. Uploading to the presigned URL, checking `job_id` consistency, and similar orchestration live in `internal/plugin/event/batchwrite/writer.go`. * `BridgeAdapter` (`bridge_adapter.go`): `BackendBlockDBBridge = "blockdb_executor"`. It receives the raw request protobuf sent by the Python `blockdb_bridge.BridgeChannel` through the UDS binary sidecar, extracts the gRPC method path with `capabilitywire.DecodeBinaryRequest`, and forwards it according to the `bridgeReadMethods` / `bridgeWriteMethods` whitelists; `bridgeBlockedWriteMethods` rejects BatchWrite lifecycle calls initiated by function code; the streaming methods `Scan`/`Subscribe` are unsupported. Response bytes are placed into `ReadResult.Data` unchanged. * `ScanClient` / `Adapter.ScanAll` (`online.go`): streaming reads over `TableScanService.Scan`; Function Code View reuses the typed `Adapter.ScanAll`. `SubscribeClient` currently only has a constructor and holds the connection; it has no `Subscribe` method. * Error classification (`adapter.go`, `adaptive.go`): `classifyBlockDBError` parses `Error NNNN (SQLSTATE)` from the gRPC status into a `mysql:NNNN` / `proxysql:NNNN` code, then maps by gRPC code; `classifyAdaptiveOutcome` treats `ResourceExhausted/DeadlineExceeded/Unavailable`, transport errors, and a set of MySQL/TiDB/ProxySQL timeout phrases as overload. * `Registration(cfg, adaptive.Config, opts...)` / `BridgeRegistration(...)`: return an `iocore.BackendRegistration`; `WithMultiConnConfig` injects a `grpcclient.MultiConnConfig`. * Test doubles: `DelayedGRPCStub` (`grpc_stub.go`, mounted on a real gRPC server by `cmd/blockdb_grpc_stub`) and `DelayedBridgeStubAdapter` (`bridge_stub.go`, an in-process bridge stub enabled by `STUB_BLOCKDB_DELAY_MS`). ### noderpc * `NodeRPCReq` (`types.go`): a JSON-RPC 2.0 request invoked directly from Go, `BackendNodeRPC = "rpc"`; `Operation()` returns the method and `Timeout()` is fixed at 10s. * `Adapter` (`adapter.go`): `Read` POSTs the request body to `endpoint`; for bridge requests (a `rawBridgeRequester` sent by the Python executor) it takes `params.chain` from the JSON metadata, POSTs to `endpoint/`, and uses the sidecar bytes directly as the body. `Write` is unsupported. Every request carries the `x-load-deadline/x-load-priority/x-load-retries` headers; an `{"error":...}` inside an HTTP 2xx response is parsed into `jsonRPCError`, and non-2xx becomes `HTTPStatusError`. * `HTTPClientConfig{TransportCount, RefreshIntervalMs}` + `rotatingHTTPTransport` (`transport.go`): 8 independent `http.Transport` shards used round-robin, with one replaced smoothly every 30s; `HTTPPoolSizesForAdmission` derives the idle connection budget from the admission initial / maximum concurrency. * `Registration(endpoint, adaptive.Config, opts...)`. * Semantic batching (`batching_backend.go`, `batch_core.go`, `batch_protocol.go`, `batch_config.go`, `batch_window*.go`): `BatchingBackend` implements `BackendAdapter` and wraps outside admission (`assembly.Module.OuterWrap`). `BatchConfig{Mode, MaxWait, MaxItems, FallbackCooldown}`, `BatchMode` = `off|shadow|on`, `BatchConfigFromEnv()`, `NewBatchingBackend(inner, cfg, BatchObserver)`. The only batchable methods are `getAddressCode`, `getStorageAt`, `getAddressBalance`, and `getAddressNonce`, and the block context must be a fixed hash/height; the physical request is the internal method `blockx_stateReadBatch` with a BSRB/1 binary payload. `batchobserved.Observer()` provides the obs metric hooks. * Binary sidecar: the pass-through path for bridge requests is `Adapter.readFromBridge`; a missing `RawBinaryRequest()` returns `noderpc:invalid_bridge_request` directly, with no JSON-rebuild fallback. The Python side is in `python/blockx_executor/noderpc_bridge.py`. ### meta and logicaltypes * The `go_package` in `internal/sdk/meta/proto/*.proto` points at the blockdb repo; `make proto-meta` uses `--go_opt=M...` to remap the three files into `internal/sdk/meta/gen` (package name `gen`, usually aliased as `metapb` by callers). * `logicaltypes.Adapter` (`logicaltypes/adapter.go`): `BackendLogicalTypes = "logicalTypes"`, `ReadReq{TableID, RTimeout}` → `metapb.TableMetaServiceClient.GetTable` → JSON-encoded `TableSchema{Type string, Columns []logical_types.Column}`. `ClassifyError` treats schema / configuration gRPC codes as non-retryable. `LocalStubAdapter` returns a local schema by table ID (`"*"` as the fallback). `Registration(host, adaptive.Config, opts...)`. * `meta.Adapter` (`meta/meta.go`): the legacy client for HTTP GET `/api/v1/meta/get_table_metadata?id=`, `BackendMeta = "meta"`, `MetaReadReq`. No production code in the repo imports it; only the in-package unit tests and the `//go:build live` `meta_live_test.go` do. The Worker's `META_ADDR` goes through `logicaltypes`. ### iceberg * `Config{Namespace, Region}`, `NewPlanner(ctx, cfg)` (`planner.go`): builds a Glue catalog using the AWS default credential chain (the instance role on EC2) and does not touch Glue/S3 at construction time. The `DataFilePlanner` interface has only `PlanDataFiles(ctx, tableName) ([]string, error)`; `Planner.PlanDataFiles` loads the table and plans all data files of the current snapshot, and errors out immediately if it encounters a delete file. * `ResolveDataFilesReq{TableID, PhysicalTableID, RequiredBundle, RTimeout}` (`io_adapter.go`): `BackendIcebergResolve = "iceberg"`; `CacheKey()` uses only the logical `TableID`. It implements `CachedReadValidator.AcceptCachedRead`: the cached index is reused only if its high water is ≥ `RequiredBundle`, otherwise a full-table refresh is triggered (corresponding to the commit "iceberg: refresh indexes above bundle high water"). * `IcebergIOAdapter`: read-only; encodes the path list into a `BFI1` index (`bundle_file_index.go`: shared prefix + one record per bundle; two live files in the same bundle is an error). `LookupBundleFileIndex(encoded, bundle)` and `BundleFileIndexHighWater(encoded)` are used by `bundlescan` for decoding. * The Worker registers it as a module with `SystemCache: true` and `ReadOnly: true`; admission is the fixed concurrency `IO.SystemIOCacheRefreshConcurrency` (default 2). See `docs/deploy.md` §3.3 in the blockx repo for the required IAM (`glue:GetTable`, `s3:GetObject`, plus `kms:Decrypt` for SSE-KMS). ### localtestservice * `Service` (`service.go`) directly implements the generated `LocalTestServiceServer`: `HelloWorld` returns `"hello, "`; `BlockDBGetRow` issues one `ReadOp[*blockdbpb.GetRowRequest]` through the injected `BlockDBReader` (the host's raw blockdb adapter or a stub). * `Adapter` (`adapter.go`): `BackendLocalTestService = "localtestservice"`; unwraps the capabilitywire binary envelope, runs `proto.Unmarshal` by method path, and calls `Service` in-process. `Module(blockDB BlockDBReader) assembly.Module`: read-only, fixed admission of 4. * `examples/local_test_service/` is the Python-side end-to-end example (blockx-py's `LocalTestService` class → BridgeChannel → worker); see `docs/capability-backend-guide.md` in the blockx repo for a file-by-file template for adding a capability backend. ### router * `Adapter` (`router/adapter.go`): `BackendRouter = "router"`; supports only `routerpkg.RouterFindFunctionsFullMethodName`, and the business logic is in `Service.FindFunctions` of the `github.com/chaintable/router/go` dependency. `Module()` is read-only with a fixed admission of 100. ### grpcclient * `New(target, opts...)`: an insecure ClientConn with DNS `round_robin`. * `NewMultiConn(target, MultiConnConfig, opts...)`: registers a custom balancer named `blockx_multi_conn` that maintains `Connections` SubConns inside one `ClientConn` and swaps one every `RefreshIntervalMs`; `DefaultMultiConnConfig()` = 8 connections / 10000 ms. `grpc.WithDisableServiceConfig()` prevents the resolver-delivered service config from overriding the policy. * `DelimitedWriteRowEncoder` (`delimited_write_row.go`): decides the `Value` oneof kind from `logical_types.Column` and writes length-delimited `blockdb.v1.WriteRow` bytes directly, without constructing the generated proto message; the BatchWrite upload file uses it. ### duckdb/borrowed * `Pool` (`pool.go`): `NewPool(db, maxOpen, maxIdle)`, `Acquire(ctx)`, `Release(conn)`, `Close()`; `WatchContext(ctx, conn)` (`context.go`) propagates context cancellation to DuckDB through `duckdb_interrupt`; `ScanResult(ctx, res, yield)` (`rows.go`) iterates chunk by chunk and hands VARCHAR/BLOB values to the callback as zero-copy views; the values become invalid once the callback returns, so copy them if you need to keep them. * The conversion logic is adapted from `duckdb-go` and calls `duckdb-go-bindings` directly; `LICENSE.duckdb-go`, `LICENSE.duckdb-go-bindings`, and `README.md` record the origin and ownership. ### cmd/blockdb\_grpc\_stub * `main.go` starts a real gRPC server with `blockdb.RegisterDelayedGRPCStub(registrar, readDelay)`: read RPCs wait `-read-delay` and then return a fixed token shape (`GetRow`, `GetState`), write RPCs return immediately; `-probe-address` mode only probes an already-running stub and verifies its delay. See `cmd/blockdb_grpc_stub/README.md` for its purpose and the Worker-side environment requirements. ## Data flow / execution flow ```mermaid theme={null} flowchart LR subgraph callers["Callers"] PY["Python function code
(BridgeChannel / noderpc_bridge)"] GO["Go Builder / Writer Plugin
(blockdb.ReadOp / InitWriteJobReq / ResolveDataFilesReq)"] end TIO["TaskIOScope
cache / singleflight / windows"] OUTER["OuterWrap
(noderpc.BatchingBackend)"] ADM["adaptive admission wrapper
AIMD / fixed concurrency"] SDK["SDK adapter
Read/Write + ClassifyError"] REMOTE["Remote services
BlockDB gRPC / node RPC HTTP / Meta gRPC / Glue+S3"] PY --> TIO GO --> TIO TIO --> OUTER --> ADM --> SDK --> REMOTE ``` `internal/worker/app/app.go` (`cmd/syncinvoker/io.go` for the sync-invoker) declares each backend as an `assembly.Module{Kind, Enabled, Build, Stub, Admission, OuterWrap}`. When `Enabled` is false, `Stub` is used (e.g. devstub when `NODE_RPC_ENDPOINT` is empty, `logicaltypes.LocalStubAdapter` when `META_ADDR` is empty). `assembly.Build` uniformly applies the metrics-instrumented admission wrapper through `adaptiveobserved.WrapBackendRegistration` (`internal/io/assembly/assembly.go`). Go callers construct an SDK request type (e.g. `blockdb.ReadOp[*blockdbpb.GetBlockRowsRequest]`) and hand it to `TaskIO.Read`; Python callers' gRPC/JSON-RPC bytes reach the Worker through the UDS sidecar and are wrapped into an IORequest that carries `RawBinaryRequest()`, whose `Backend()` is determined by the backend name on the wire. An adapter only looks at request types it recognizes and returns an error for anything else. Before a gRPC call the client id is propagated with `clientid.IntoOutgoingGRPC(ctx)`; HTTP uses `clientid.IntoHTTP`. The error returned by the adapter is first fed to AIMD through `ClassifyAdaptiveOutcome`, then turned into an `IOError` through `ClassifyError`, which decides retries at the Call/IO layer. Both treat local context termination as Ignore / timeout rather than downstream overload. ## State and lifecycle * gRPC connections: the `NewMultiConn` balancer first creates a candidate SubConn for each slot, and only promotes it and drains the old connection once it is READY; the first refresh is randomly delayed within `(0, refreshInterval]`. `Adapter.Close()` closes all `ClientConn`s. * NodeRPC transport: `rotatingHTTPTransport` retires one shard every refresh interval; a retired shard waits for in-flight requests to finish before closing; `Close()` rejects new requests and waits for everything to drain. * Batching groups: `groupCollecting → groupFlushing → groupDone`; requests with the same chain, same fixed block, same timeout, and same client id join the same group, and a flush is triggered by `MaxWait` (default 500µs, timed with timerfd on Linux) or `MaxItems` (default 16). Receiving `-32601` enters a Worker-level cooldown (default 60s), after which only one half-open probe batch is let through. `Close()` fails the groups still collecting and cancels in-flight physical requests. * Idempotency: every BlockDB RPC is idempotent, so the IO layer can do limited retries of Init/Commit/Cancel in the Result phase; NodeRPC has only the Read path. ## Configuration The Worker fields are in `internal/worker/app/config.go` (`WorkerFullConfig`), the environment variable overrides are in `internal/worker/app/app.go`, and defaults come from `DefaultWorkerFullConfig()`. | Environment variable | JSON field | Default | Description | | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NODE_RPC_ENDPOINT` | `nodeRpcEndpoint` | Empty (uses devstub) | Node RPC HTTP(S) address | | `NODE_RPC_INITIAL_LIMIT` | `nodeRpcInitialLimit` | `1024` | Initial NodeRPC admission concurrency; `NODE_RPC_AIMD_*` tunes AIMD | | `NODE_RPC_HTTP_TRANSPORT_COUNT` / `NODE_RPC_HTTP_REFRESH_INTERVAL_MS` | `nodeRpcHttp.*` | `8` / `30000` | Number of transport shards and rotation interval | | `NODE_RPC_SEMANTIC_BATCH_MODE` / `_MAX_WAIT_US` / `_MAX_ITEMS` / `_FALLBACK_COOLDOWN_MS` | None (env only, `noderpc.BatchConfigFromEnv`) | `off` / `500` / `16` / `60000` | Invalid values fail startup; `MaxWait ≤ 2ms`, `MaxItems ∈ [2,64]` | | The 11 `BLOCKDB_*_ADDR` variables such as `BLOCKDB_TABLE_READ_ADDR` | `blockdb.*Addr` | Empty | An independent address per service; when all are empty the blockdb backend uses devstub. `BLOCKDB_BATCH_WRITE_ADDR` is required when registering BundleWrite / bundle TableUpserts | | `BLOCKDB_MAX_RECV_MSG_SIZE` | `blockdb.maxRecvMsgSize` | `0` (32 MiB) | Maximum size of a single response | | `BLOCKDB_INITIAL_LIMIT` | `blockdbInitialLimit` | `16` | The typed and bridge backends each have their own limiter sharing the same policy; `BLOCKDB_AIMD_*` tunes AIMD | | `BLOCKDB_GRPC_CONNECTIONS` / `_REFRESH_INTERVAL_MS`, `BLOCKDB_BRIDGE_GRPC_*`, `LOGICAL_TYPES_GRPC_*` | `blockdbGrpc` / `blockdbBridgeGrpc` / `logicalTypesGrpc` | `8` / `10000` | Independent multi-connection policy per adapter | | `META_ADDR` | `metaAddr` | Empty (uses `LocalStubAdapter`) | logical-types Meta gRPC `host:port`; `LOGICAL_TYPES_INITIAL_LIMIT` defaults to `16` | | `ICEBERG_NAMESPACE` / `ICEBERG_REGION` | `icebergNamespace` / `icebergRegion` | Empty (required; `Validate` rejects it) / `ap-northeast-1` | Glue namespace and region | | `SYSTEM_IO_CACHE_REFRESH_CONCURRENCY` | `io.systemIoCacheRefreshConcurrency` | `2` | Fixed admission concurrency for the Iceberg resolve backend | | `STUB_BLOCKDB_DELAY_MS` | — | Unset | When present, the in-process BlockDB stub is used (both typed and bridge are replaced); invalid values fail startup | | `STUB_LOGICAL_TYPES_SCHEMAS` | — | Empty | JSON schema map for `LocalStubAdapter`; read only when `META_ADDR` is empty | | `LOCAL_TEST_SERVICE_DISABLED` / `ROUTER_DISABLED` | `localTestServiceDisabled` / `routerDisabled` | `false` | Disables the corresponding capability backend | | `IO_FAIR_QUEUE_BACKENDS` | `ioFairQueueBackends` | Empty | The listed backend kinds (e.g. `rpc`) split their admission queue into lanes by client id | ## Extension points * Add a BlockDB RPC: edit `internal/sdk/blockdb/proto/*.proto` → `make proto-blockdb` → add a `ReadOp[*NewRequest]` branch in `Adapter.dispatchRead/dispatchWrite` → if it should be exposed to Python, add the `FullMethodName` to `bridgeReadMethods` / `bridgeWriteMethods` in `bridge_adapter.go` → add cases in `blockdb_test.go` (bufconn + gomock server). * Add a backend: implement `BackendAdapter` (optionally `ErrorClassifier` and `OutcomeClassifier`), provide a `Registration` or `Module`, then add a declaration to `backendModules` in `internal/worker/app/app.go`. For a gRPC-style capability backend, copy `internal/sdk/localtestservice` directly; the steps are in `docs/capability-backend-guide.md`. * Add a batchable NodeRPC method: edit `batchMethods` in `batch_core.go` and the BSRB kind in `batch_protocol.go`, change the Leafage-side handler at the same time, and follow the version evolution rules in `docs/specs/noderpc-semantic-batching.md` §7.3. * Adjust connection policy: only change the `grpcclient.MultiConnConfig` of the corresponding adapter; do not add a global connection pool. * Regenerate proto: requires `protoc`, `protoc-gen-go`, and `protoc-gen-go-grpc` in `$(go env GOPATH)/bin` (the install commands are at the top of the `Makefile`). `make proto` first runs `proto-blockdb`, `proto-meta`, and `proto-localtestservice`, then generates `api/grpc/*`. All three SDK targets use `require_unimplemented_servers=false`. ```bash theme={null} go install google.golang.org/protobuf/cmd/protoc-gen-go@latest go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest make proto-blockdb # or make proto-meta / make proto-localtestservice / make proto ``` ## Testing ```bash theme={null} go test ./internal/sdk/... ./internal/duckdb/... ./cmd/blockdb_grpc_stub/... go test -run 'TestBatching' ./internal/sdk/noderpc/ ``` * `blockdb`: `blockdb_test.go` starts a gRPC server with `bufconn`, and `mock_server_test.go` is the mockgen-generated service mock; `partial_config_test.go` covers clean failures for unconfigured services; `batch_write_test.go` verifies that the proto matches the BlockDB repo contract; `bridge_*_test.go` covers the bridge whitelists and the raw-bytes path. * `noderpc`: `noderpc_test.go` uses `httptest.Server`; `batching_backend_test.go` covers cross-task batching, error isolation, the `-32601` fallback, and the cooldown probe; `batch_window_linux_test.go` calls `t.Skip` when timerfd is unavailable. * Tests that need external services are skipped: `TestAdapter_ReadReal` needs `NODERPC_REAL_URL` (and not `-short`); `meta_live_test.go` needs `-tags live` and `META_API_URL`. CI's `make test` uses `-short`, so it never touches real services. * `iceberg`: `planner_test.go` only tests delete-file rejection and index encoding; it does not access Glue. * `duckdb/borrowed`: `pool_test.go`, `cells_test.go`, and `rows_test.go` need cgo and the DuckDB bindings, and compile as part of `go test`. * E2E: `e2e/system/logical_type/` starts a local Meta gRPC service with `metapb` to test logical types and dbscan; process-level tests each start their own mock gRPC server (`cmd/worker/worker_process_blockdb_bridge_test.go` implements `TableReadServiceServer`, `cmd/worker/worker_process_function_code_test.go` uses `internal/testutil.StartMockFunctionBlockDB`); performance benchmarks use `cmd/blockdb_grpc_stub`. ## Related docs * blockx repo `docs/specs/2026-07-30-bundle-write-batch-api.md`: the BatchWrite Job lifecycle; the SDK only provides the three atomic RPCs. * blockx repo `docs/specs/noderpc-semantic-batching.md`: admission rules, group state machine, BSRB/1 protocol, configuration, and fallback for the batching facade. * blockx repo `docs/specs/2026-08-06-noderpc-binary-sidecar.md`: NodeRPC bridge binary pass-through. * blockx repo `docs/specs/2026-07-23-grpc-client-balancing.md`: the design of `NewMultiConn` and per-component configuration. * blockx repo `docs/specs/io-subsystem.md`, `docs/io-backend-module-design.md`, `docs/capability-backend-guide.md`: the backend adapter contract and the guide for adding a backend. * blockx repo `docs/deploy.md` §3.3 (Iceberg IAM), §9.2 (Worker environment variables). * On this site: [IO access subsystem](/en/components/io-subsystem), [Plugin system](/en/components/plugins), [Function Code View](/en/components/function-code), [Python Executor](/en/components/python-executor), [Bundle clusters](/en/components/bundle), [Protocols and interfaces](/en/architecture/protocols), [Testing](/en/development/testing).