Backend Adapter

​

Responsibilities and boundaries

• Each SDK package does exactly one thing: translate iocore.ReadReq / iocore.WriteReq into one remote call and put the response into ReadResult.Data / WriteResult.Data. Request types are defined by the SDK package, not by internal/io/core.
• The SDK does no caching, no singleflight, no task windows, and no admission control. All of that lives in the IO subsystem; the SDK only provides Registration(...) or an assembly.Module to help callers wrap the adapter in adaptive.WrapBackend.
• The SDK is responsible for error classification: it implements iocore.ErrorClassifier.ClassifyError (deciding retryable / non-retryable / param) and adaptive.OutcomeClassifier.ClassifyAdaptiveOutcome (deciding whether AIMD should shrink). Local context cancellation / timeout never counts as downstream overload.
• The BlockDB SDK exposes only atomic RPCs. The BatchWrite Init → PUT → Commit orchestration lives in internal/plugin/event/batchwrite, not in the SDK (docs/specs/2026-07-30-bundle-write-batch-api.md §2).
• The bridge paths for the Python executor (BlockDB bridge, NodeRPC bridge, capability backends) only pass protobuf/JSON-RPC bytes through; they never rebuild request bodies.
• Connection policy belongs to the adapter instance: gRPC clients pointing at an NLB use grpcclient.NewMultiConn, the same address within one adapter shares one ClientConn, and there is no process-level connection registry.
• internal/duckdb/borrowed only provides a connection pool and result decoding on top of the native DuckDB bindings; SQL assembly, S3 credentials, and filter pushdown stay in internal/plugin/callbuilder/bundlescan.

​

Code location

​

Core types and interfaces

type BackendAdapter interface {
	Read(ctx context.Context, req ReadReq) (*ReadResult, error)
	Write(ctx context.Context, req WriteReq) (*WriteResult, error)
	Close() error
}

​

blockdb

• AdapterConfig (adapter.go): one address field per gRPC service (TableReadAddr, TableWriteAddr, TableScanAddr, BlockReadAddr, BlockWriteAddr, BatchWriteAddr, BlockSubscribeAddr, L2BlockAddr, TimeReadAddr, TimeWriteAddr) plus MaxRecvMsgSize (0 means 32 MiB). Identical addresses share one ClientConn; an unconfigured service returns blockdb:service_not_configured (a param-class error) when called. The TableSubscribeAddr field exists and is read from BLOCKDB_TABLE_SUBSCRIBE_ADDR, but NewAdapter never dials it; it is only used in internal/worker/app/function_code.go as one of the conditions for enabling the BlockDB Function Code View.
• Adapter (adapter.go): the typed backend, BackendBlockDB = "blockdb". Read uses a type switch to dispatch ReadOp[*blockdbpb.GetRowRequest], BatchGetRowsRequest, FilterRowsRequest, GetBlockRowsRequest, GetEventRequest, GetStateRequest, and GetValueRequest, converting the ResultSet into a JSON row array; JSON/LIST/DICT logical columns are restored to objects according to ColumnMeta.logical_type. Write dispatches WriteOp[*UpsertRowsRequest], WriteOp[*DeleteRowsRequest], WriteOp[*UpsertTimeRowsRequest], BlockWriteReq, InitWriteJobReq, CommitWriteJobReq, and CancelWriteJobReq.
• ReadOp[T] / WriteOp[T] (types.go): generic wrappers that turn any proto request into an iocore.ReadReq/WriteReq, carrying Key (cache key), Op (stats name), and RTO/WTO (timeouts). Filter, EqFilter, AndFilter, and FilterJSON generate parameterized WHERE clauses; Value(any) converts to *blockdbpb.Value.
• Three table semantics (proto/table.proto, block.proto, time.proto): Table (L1 row tables: GetRow/BatchGetRows/FilterRows/UpsertRows/DeleteRows/Scan), Block (L2 block-scoped: GetEvent/GetState take current_block, UpsertBlock takes Block{block_id,height,timestamp}, L2BlockReadService.GetBlockRows), and Time (GetValue(row_id,time_at), UpsertTimeRows(time_at)). BlockWriteReq is a multi-table L2 write: Data map[table][]*WriteRow, and the adapter issues one UpsertBlock per table.
• BatchWrite (batch_write.go, proto/batch.proto): InitWriteJobReq/CommitWriteJobReq/CancelWriteJobReq each wrap one BatchWriteService request, and the response is placed into WriteResult.Data as proto bytes. Uploading to the presigned URL, checking job_id consistency, and similar orchestration live in internal/plugin/event/batchwrite/writer.go.
• BridgeAdapter (bridge_adapter.go): BackendBlockDBBridge = "blockdb_executor". It receives the raw request protobuf sent by the Python blockdb_bridge.BridgeChannel through the UDS binary sidecar, extracts the gRPC method path with capabilitywire.DecodeBinaryRequest, and forwards it according to the bridgeReadMethods / bridgeWriteMethods whitelists; bridgeBlockedWriteMethods rejects BatchWrite lifecycle calls initiated by function code; the streaming methods Scan/Subscribe are unsupported. Response bytes are placed into ReadResult.Data unchanged.
• ScanClient / Adapter.ScanAll (online.go): streaming reads over TableScanService.Scan; Function Code View reuses the typed Adapter.ScanAll. SubscribeClient currently only has a constructor and holds the connection; it has no Subscribe method.
• Error classification (adapter.go, adaptive.go): classifyBlockDBError parses Error NNNN (SQLSTATE) from the gRPC status into a mysql:NNNN / proxysql:NNNN code, then maps by gRPC code; classifyAdaptiveOutcome treats ResourceExhausted/DeadlineExceeded/Unavailable, transport errors, and a set of MySQL/TiDB/ProxySQL timeout phrases as overload.
• Registration(cfg, adaptive.Config, opts...) / BridgeRegistration(...): return an iocore.BackendRegistration; WithMultiConnConfig injects a grpcclient.MultiConnConfig.
• Test doubles: DelayedGRPCStub (grpc_stub.go, mounted on a real gRPC server by cmd/blockdb_grpc_stub) and DelayedBridgeStubAdapter (bridge_stub.go, an in-process bridge stub enabled by STUB_BLOCKDB_DELAY_MS).

​

noderpc

• NodeRPCReq (types.go): a JSON-RPC 2.0 request invoked directly from Go, BackendNodeRPC = "rpc"; Operation() returns the method and Timeout() is fixed at 10s.
• Adapter (adapter.go): Read POSTs the request body to endpoint; for bridge requests (a rawBridgeRequester sent by the Python executor) it takes params.chain from the JSON metadata, POSTs to endpoint/<chain>, and uses the sidecar bytes directly as the body. Write is unsupported. Every request carries the x-load-deadline/x-load-priority/x-load-retries headers; an {"error":...} inside an HTTP 2xx response is parsed into jsonRPCError, and non-2xx becomes HTTPStatusError.
• HTTPClientConfig{TransportCount, RefreshIntervalMs} + rotatingHTTPTransport (transport.go): 8 independent http.Transport shards used round-robin, with one replaced smoothly every 30s; HTTPPoolSizesForAdmission derives the idle connection budget from the admission initial / maximum concurrency.
• Registration(endpoint, adaptive.Config, opts...).
• Semantic batching (batching_backend.go, batch_core.go, batch_protocol.go, batch_config.go, batch_window*.go): BatchingBackend implements BackendAdapter and wraps outside admission (assembly.Module.OuterWrap). BatchConfig{Mode, MaxWait, MaxItems, FallbackCooldown}, BatchMode = off|shadow|on, BatchConfigFromEnv(), NewBatchingBackend(inner, cfg, BatchObserver). The only batchable methods are getAddressCode, getStorageAt, getAddressBalance, and getAddressNonce, and the block context must be a fixed hash/height; the physical request is the internal method blockx_stateReadBatch with a BSRB/1 binary payload. batchobserved.Observer() provides the obs metric hooks.
• Binary sidecar: the pass-through path for bridge requests is Adapter.readFromBridge; a missing RawBinaryRequest() returns noderpc:invalid_bridge_request directly, with no JSON-rebuild fallback. The Python side is in python/blockx_executor/noderpc_bridge.py.

​

meta and logicaltypes

• The go_package in internal/sdk/meta/proto/*.proto points at the blockdb repo; make proto-meta uses --go_opt=M... to remap the three files into internal/sdk/meta/gen (package name gen, usually aliased as metapb by callers).
• logicaltypes.Adapter (logicaltypes/adapter.go): BackendLogicalTypes = "logicalTypes", ReadReq{TableID, RTimeout} → metapb.TableMetaServiceClient.GetTable → JSON-encoded TableSchema{Type string, Columns []logical_types.Column}. ClassifyError treats schema / configuration gRPC codes as non-retryable. LocalStubAdapter returns a local schema by table ID ("*" as the fallback). Registration(host, adaptive.Config, opts...).
• meta.Adapter (meta/meta.go): the legacy client for HTTP GET /api/v1/meta/get_table_metadata?id=, BackendMeta = "meta", MetaReadReq. No production code in the repo imports it; only the in-package unit tests and the //go:build live meta_live_test.go do. The Worker’s META_ADDR goes through logicaltypes.

​

iceberg

• Config{Namespace, Region}, NewPlanner(ctx, cfg) (planner.go): builds a Glue catalog using the AWS default credential chain (the instance role on EC2) and does not touch Glue/S3 at construction time. The DataFilePlanner interface has only PlanDataFiles(ctx, tableName) ([]string, error); Planner.PlanDataFiles loads the table and plans all data files of the current snapshot, and errors out immediately if it encounters a delete file.
• ResolveDataFilesReq{TableID, PhysicalTableID, RequiredBundle, RTimeout} (io_adapter.go): BackendIcebergResolve = "iceberg"; CacheKey() uses only the logical TableID. It implements CachedReadValidator.AcceptCachedRead: the cached index is reused only if its high water is ≥ RequiredBundle, otherwise a full-table refresh is triggered (corresponding to the commit “iceberg: refresh indexes above bundle high water”).
• IcebergIOAdapter: read-only; encodes the path list into a BFI1 index (bundle_file_index.go: shared prefix + one record per bundle; two live files in the same bundle is an error). LookupBundleFileIndex(encoded, bundle) and BundleFileIndexHighWater(encoded) are used by bundlescan for decoding.
• The Worker registers it as a module with SystemCache: true and ReadOnly: true; admission is the fixed concurrency IO.SystemIOCacheRefreshConcurrency (default 2). See docs/deploy.md §3.3 in the blockx repo for the required IAM (glue:GetTable, s3:GetObject, plus kms:Decrypt for SSE-KMS).

​

localtestservice

• Service (service.go) directly implements the generated LocalTestServiceServer: HelloWorld returns "hello, <name>"; BlockDBGetRow issues one ReadOp[*blockdbpb.GetRowRequest] through the injected BlockDBReader (the host’s raw blockdb adapter or a stub).
• Adapter (adapter.go): BackendLocalTestService = "localtestservice"; unwraps the capabilitywire binary envelope, runs proto.Unmarshal by method path, and calls Service in-process. Module(blockDB BlockDBReader) assembly.Module: read-only, fixed admission of 4.
• examples/local_test_service/ is the Python-side end-to-end example (blockx-py’s LocalTestService class → BridgeChannel → worker); see docs/capability-backend-guide.md in the blockx repo for a file-by-file template for adding a capability backend.

​

router

• Adapter (router/adapter.go): BackendRouter = "router"; supports only routerpkg.RouterFindFunctionsFullMethodName, and the business logic is in Service.FindFunctions of the github.com/chaintable/router/go dependency. Module() is read-only with a fixed admission of 100.

​

grpcclient

• New(target, opts...): an insecure ClientConn with DNS round_robin.
• NewMultiConn(target, MultiConnConfig, opts...): registers a custom balancer named blockx_multi_conn that maintains Connections SubConns inside one ClientConn and swaps one every RefreshIntervalMs; DefaultMultiConnConfig() = 8 connections / 10000 ms. grpc.WithDisableServiceConfig() prevents the resolver-delivered service config from overriding the policy.
• DelimitedWriteRowEncoder (delimited_write_row.go): decides the Value oneof kind from logical_types.Column and writes length-delimited blockdb.v1.WriteRow bytes directly, without constructing the generated proto message; the BatchWrite upload file uses it.

​

duckdb/borrowed

• Pool (pool.go): NewPool(db, maxOpen, maxIdle), Acquire(ctx), Release(conn), Close(); WatchContext(ctx, conn) (context.go) propagates context cancellation to DuckDB through duckdb_interrupt; ScanResult(ctx, res, yield) (rows.go) iterates chunk by chunk and hands VARCHAR/BLOB values to the callback as zero-copy views; the values become invalid once the callback returns, so copy them if you need to keep them.
• The conversion logic is adapted from duckdb-go and calls duckdb-go-bindings directly; LICENSE.duckdb-go, LICENSE.duckdb-go-bindings, and README.md record the origin and ownership.

​

cmd/blockdb_grpc_stub

• main.go starts a real gRPC server with blockdb.RegisterDelayedGRPCStub(registrar, readDelay): read RPCs wait -read-delay and then return a fixed token shape (GetRow, GetState), write RPCs return immediately; -probe-address mode only probes an already-running stub and verifies its delay. See cmd/blockdb_grpc_stub/README.md for its purpose and the Worker-side environment requirements.

​

Data flow / execution flow

​

State and lifecycle

• gRPC connections: the NewMultiConn balancer first creates a candidate SubConn for each slot, and only promotes it and drains the old connection once it is READY; the first refresh is randomly delayed within (0, refreshInterval]. Adapter.Close() closes all ClientConns.
• NodeRPC transport: rotatingHTTPTransport retires one shard every refresh interval; a retired shard waits for in-flight requests to finish before closing; Close() rejects new requests and waits for everything to drain.
• Batching groups: groupCollecting → groupFlushing → groupDone; requests with the same chain, same fixed block, same timeout, and same client id join the same group, and a flush is triggered by MaxWait (default 500µs, timed with timerfd on Linux) or MaxItems (default 16). Receiving -32601 enters a Worker-level cooldown (default 60s), after which only one half-open probe batch is let through. Close() fails the groups still collecting and cancels in-flight physical requests.
• Idempotency: every BlockDB RPC is idempotent, so the IO layer can do limited retries of Init/Commit/Cancel in the Result phase; NodeRPC has only the Read path.

​

Configuration

​

Extension points

• Add a BlockDB RPC: edit internal/sdk/blockdb/proto/*.proto → make proto-blockdb → add a ReadOp[*NewRequest] branch in Adapter.dispatchRead/dispatchWrite → if it should be exposed to Python, add the FullMethodName to bridgeReadMethods / bridgeWriteMethods in bridge_adapter.go → add cases in blockdb_test.go (bufconn + gomock server).
• Add a backend: implement BackendAdapter (optionally ErrorClassifier and OutcomeClassifier), provide a Registration or Module, then add a declaration to backendModules in internal/worker/app/app.go. For a gRPC-style capability backend, copy internal/sdk/localtestservice directly; the steps are in docs/capability-backend-guide.md.
• Add a batchable NodeRPC method: edit batchMethods in batch_core.go and the BSRB kind in batch_protocol.go, change the Leafage-side handler at the same time, and follow the version evolution rules in docs/specs/noderpc-semantic-batching.md §7.3.
• Adjust connection policy: only change the grpcclient.MultiConnConfig of the corresponding adapter; do not add a global connection pool.
• Regenerate proto: requires protoc, protoc-gen-go, and protoc-gen-go-grpc in $(go env GOPATH)/bin (the install commands are at the top of the Makefile). make proto first runs proto-blockdb, proto-meta, and proto-localtestservice, then generates api/grpc/*. All three SDK targets use require_unimplemented_servers=false.

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
make proto-blockdb   # or make proto-meta / make proto-localtestservice / make proto

​

Testing

go test ./internal/sdk/... ./internal/duckdb/... ./cmd/blockdb_grpc_stub/...
go test -run 'TestBatching' ./internal/sdk/noderpc/

• blockdb: blockdb_test.go starts a gRPC server with bufconn, and mock_server_test.go is the mockgen-generated service mock; partial_config_test.go covers clean failures for unconfigured services; batch_write_test.go verifies that the proto matches the BlockDB repo contract; bridge_*_test.go covers the bridge whitelists and the raw-bytes path.
• noderpc: noderpc_test.go uses httptest.Server; batching_backend_test.go covers cross-task batching, error isolation, the -32601 fallback, and the cooldown probe; batch_window_linux_test.go calls t.Skip when timerfd is unavailable.
• Tests that need external services are skipped: TestAdapter_ReadReal needs NODERPC_REAL_URL (and not -short); meta_live_test.go needs -tags live and META_API_URL. CI’s make test uses -short, so it never touches real services.
• iceberg: planner_test.go only tests delete-file rejection and index encoding; it does not access Glue.
• duckdb/borrowed: pool_test.go, cells_test.go, and rows_test.go need cgo and the DuckDB bindings, and compile as part of go test.
• E2E: e2e/system/logical_type/ starts a local Meta gRPC service with metapb to test logical types and dbscan; process-level tests each start their own mock gRPC server (cmd/worker/worker_process_blockdb_bridge_test.go implements TableReadServiceServer, cmd/worker/worker_process_function_code_test.go uses internal/testutil.StartMockFunctionBlockDB); performance benchmarks use cmd/blockdb_grpc_stub.

​

Related docs

• blockx repo docs/specs/2026-07-30-bundle-write-batch-api.md: the BatchWrite Job lifecycle; the SDK only provides the three atomic RPCs.
• blockx repo docs/specs/noderpc-semantic-batching.md: admission rules, group state machine, BSRB/1 protocol, configuration, and fallback for the batching facade.
• blockx repo docs/specs/2026-08-06-noderpc-binary-sidecar.md: NodeRPC bridge binary pass-through.
• blockx repo docs/specs/2026-07-23-grpc-client-balancing.md: the design of NewMultiConn and per-component configuration.
• blockx repo docs/specs/io-subsystem.md, docs/io-backend-module-design.md, docs/capability-backend-guide.md: the backend adapter contract and the guide for adding a backend.
• blockx repo docs/deploy.md §3.3 (Iceberg IAM), §9.2 (Worker environment variables).
• On this site: IO access subsystem, Plugin system, Function Code View, Python Executor, Bundle clusters, Protocols and interfaces, Testing.