Remote APIs, Embedded Runtimes, and Inference Services

There is no universally best place to run a model.

Embedding a runtime in the Rust process removes an inference-network boundary but couples model memory, startup and failure to every application replica. A private inference service centralizes accelerators and batching but adds a protocol, queue and operational service. A remote API offers managed capacity and access to models you may not be able to host, while widening the data/trust boundary and adding an external dependency.

“Local is private,” “remote is scalable,” and “a service is production-ready” are slogans. The real decision depends on stated requirements and measured implementations.

This chapter adds a constraint-based planner to mosaic-runtime. It calculates model-memory copies and rejects topologies that violate data-boundary, external-network, scaling, runtime-isolation, memory or hop constraints. It deliberately does not produce invented latency, price or quality scores. Those require benchmarks for a concrete model, hardware, provider and traffic distribution.

Learning objectives

By the end you will be able to:

distinguish embedded inference, a private inference service and a remote model API;
map each topology’s process, network, trust and failure boundaries;
calculate lower-bound local model-memory duplication from process ownership;
state whether a memory budget is per process, host or placement group;
express allowed data egress as an ordered boundary;
separate external-network independence from private-network availability;
explain why embedded runtimes do not scale independently from application replicas;
explain why a service can batch across callers but adds queue/transport failure;
treat remote-provider capacity as external, not infinite;
design deadlines, cancellation, backpressure and retry rules end to end;
preserve release identity across process/provider boundaries;
secure private services with workload identity, authorization and encryption;
benchmark quality, TTFT, throughput, memory and failure behavior under the real workload;
choose hybrid task placement without collapsing capabilities;
recognize when constraints leave no feasible topology;
state what the runnable planner simplifies.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-runtime --example inference_topology

The scenario declares:

application replicas                 4
private inference-service replicas   2
resident model bytes per copy        6 GiB
local model-memory budget            16 GiB across the placement group
maximum data boundary                private network
independence from external network   required
independent model scaling            required
runtime failure isolation            required
maximum logical inference hops       1

The planner returns:

Topology	Local copies	Local model bytes	Feasible?	Why
Embedded	4	24 GiB	No	memory, independent scaling, failure isolation
Private inference service	2	12 GiB	Yes	satisfies declared constraints
Remote API	0	0	No	external data boundary and network dependency

This does not declare the private service fastest or cheapest. It proves it is the only topology among the three that satisfies this exact constraint set. When the tests relax data, scaling, isolation and memory requirements, all three become feasible.

Define the three topologies

Embedded

application process
  ├─ request/harness/business logic
  ├─ tokenizer/media processor
  ├─ inference runtime
  └─ model state + caches

Inference is an in-process function or library call. The runtime may use CPU, GPU, Metal, CUDA, ONNX Runtime, Candle, llama.cpp bindings or another native backend.

Private inference service

application process(es)
  → private protocol/network
  → separately deployed inference process(es)
      ├─ admission queue
      ├─ batch/scheduler
      ├─ runtime
      └─ model state + caches

The service may be a sidecar, daemon on one host, cluster service or dedicated accelerator fleet. This chapter’s planner models a private-network service. A same-host Unix socket sidecar has different hop and failure details and should be represented as another profile rather than quietly relabelled.

Remote API

application
  → egress/proxy/public or private provider link
  → external provider control/data plane
  → provider-selected serving runtime

You manage the client and product harness. The provider manages opaque portions of model delivery, scheduling and hardware. Private connectivity can reduce exposure to the public Internet, but data still crosses an organizational/provider trust boundary.

First-principles comparison

Property	Embedded	Private service	Remote API
Minimum logical inference hops	0	1	1
Model bytes owned locally	yes	yes	no
Shares model across app replicas	no, unless runtime architecture does so	yes	provider-managed
Independent model scaling	no	yes	provider-managed
Runtime crash isolation	no	yes	yes from app process
External network dependency	no	no for self-contained private deployment	yes
Data boundary	same process	private network	external provider
Direct access to internals	strongest	strong if self-hosted	limited by provider
Operational burden	coupled to app	separate serving platform	vendor/client integration

Each cell needs qualification. “No network hop” does not mean no queueing or device contention. “Runtime crash isolated” does not mean product impact is isolated; the dependency can still be unavailable. “No local model bytes” does not mean no memory, tokenization or media-processing cost.

Step 1: state requirements as data

The runnable type is:

pub struct TopologyRequirements {
    pub application_replicas: u32,
    pub inference_service_replicas: u32,
    pub model_resident_bytes: u64,
    pub local_model_memory_budget_bytes: u64,
    pub maximum_data_boundary: DataBoundary,
    pub external_network_independence_required: bool,
    pub independent_model_scaling_required: bool,
    pub runtime_failure_isolation_required: bool,
    pub maximum_network_hops: u8,
}

Every count/byte budget must be non-zero. Arithmetic uses checked_mul.

These fields are constraints, not preferences. If remote inference is legally forbidden, a price discount should not override it. If no topology passes, return no feasible result and change the requirements, model or architecture explicitly.

Ordered data boundaries

pub enum DataBoundary {
    SameProcess,
    PrivateNetwork,
    ExternalProvider,
}

The enum is ordered by widening trust scope for this planner:

same_process < private_network < external_provider

This is still coarse. Production policy may distinguish:

same memory space;
same host/Unix socket;
same namespace;
same VPC/region;
same legal entity;
approved subprocessor;
geographic residency;
content versus derived embeddings;
encrypted versus confidential-compute boundary.

Extend the type and policy. Do not encode these as comments beside a Boolean named private.

External-network independence

The planner asks whether operation must be independent of an external provider/network. A private service satisfies this narrow condition, but it still depends on internal networking, DNS/service discovery, certificates, accelerator hosts and artifact availability.

“Offline” is ambiguous. Define the failure you must survive:

public Internet outage?
provider outage?
entire private network outage?
single host disconnected?
control plane unavailable but data plane warm?

Then test that event.

Step 2: calculate model-memory ownership

The example treats model_resident_bytes as six GiB per runtime copy:

embedded copies = application replicas = 4
embedded bytes  = 4 × 6 GiB = 24 GiB

service copies  = inference-service replicas = 2
service bytes   = 2 × 6 GiB = 12 GiB

remote local copies = 0
remote local model bytes = 0

The 16 GiB budget is explicitly total local model memory across the compared placement group. If your budget is per host, the arithmetic must model placement and copies per host. Summing memory across four different hosts does not prove any individual host is over capacity.

model_resident_bytes must include the measured steady-state components relevant to capacity:

weights after loading/conversion;
runtime graph/allocator state;
tokenizer/processor if material;
KV cache reservation;
scratch/workspace buffers;
batch-dependent activations;
device-driver/context overhead;
fragmentation and safety headroom.

The planner uses a fixed number as a transparent lower-level input. It does not estimate these components. Feed it measurements from the target runtime.

Shared memory is not automatic

Operating-system page cache or memory mapping may share physical read-only pages between processes. GPU contexts, converted weights, runtime arenas and caches may still duplicate. Measure proportional set size and device memory; do not simply multiply file size or assume perfect sharing.

Remote zero is local-only

0 means no local model residency. The provider still allocates model/cache memory and charges or rate-limits accordingly. Your client still uses memory for requests, streaming buffers, media and retries.

Step 3: model topology properties honestly

The code derives:

pub struct TopologyProperties {
    pub local_model_copies: u32,
    pub local_model_memory_bytes: u64,
    pub minimum_network_hops: u8,
    pub data_boundary: DataBoundary,
    pub depends_on_external_network: bool,
    pub independently_scalable: bool,
    pub runtime_failure_isolated: bool,
}

The hop is a minimum logical application-to-inference hop. Real paths can include:

client middleware
  → service mesh sidecar
  → load balancer
  → gateway
  → inference router
  → worker

Retries multiply attempts. TLS, serialization, queueing and streaming flush behavior contribute latency. Measure them separately rather than assigning “1 ms per hop.”

Independent scaling

Embedded inference scales with the app:

more web replicas → more model copies
fewer web replicas → less model capacity

That can be ideal for a small CPU classifier with aligned traffic. It is wasteful when stateless web requests scale differently from a six-GiB generator.

A service separates:

application replicas based on HTTP/business load
inference replicas based on token/pair/media workload and accelerator capacity

Independent scaling is not automatically effective. Model cold start may take seconds or minutes; accelerators may be unavailable; scale-to-zero can destroy latency. Maintain warm capacity based on measured startup and demand.

Failure isolation

An embedded native-runtime panic, abort, allocator failure or device fault can terminate or poison the application process. Rust memory safety does not make foreign kernels, drivers or model parsers infallible.

A separate process isolates address space and restart policy. It also creates dependency failures: connection refusal, timeout, overload, version skew and partial streaming. Isolation changes the failure shape; it does not eliminate failure.

Embedded inference in depth

Embedded is compelling when:

the model is small enough for each required process/host;
same-process/same-device data locality is important;
zero inference-network hops materially helps;
offline or edge execution is required;
traffic is modest or aligned with application replicas;
one release can ship with the app;
the runtime has a safe supported Rust/native binding.

Design requirements:

Verify the model bundle before activation.
Load once into shared immutable state such as Arc<Model>.
Bound concurrent inference with semaphores.
move blocking/accelerator work off async executor workers.
serialize access if the runtime/session is not thread safe.
enforce context/media/output bounds before allocation.
make cancellation semantics explicit; dropping a future may not stop a native kernel.
distinguish startup/readiness from liveness.
expose release identity and device/runtime metadata.
drain work before process shutdown where possible.

Thread pools and oversubscription

An embedded CPU runtime may create its own intra-op/inter-op pools. If every web process and runtime session creates many threads, the host can oversubscribe dramatically. Configure and measure:

application workers
× model sessions per process
× inference threads per session

ONNX Runtime documents session/thread-pool and allocator controls because defaults interact with deployment shape. There is no universal best thread count.

Startup and rollout coupling

Every application restart reloads the model. Rolling out unrelated web code can churn accelerator memory. Readiness must stay false until verification and warm-up pass. Large embedded artifacts can make horizontal autoscaling too slow for bursts.

Private inference service in depth

A service is compelling when:

many callers share one model;
accelerators should be pooled;
continuous/dynamic batching improves utilization;
model/app scaling and releases differ;
runtime faults need process isolation;
one platform serves multiple language stacks;
centralized admission and observability are valuable.

The API is a compatibility boundary. Version:

request/response schema;
capability-specific endpoint;
release identity;
token/media/byte limits;
streaming event format;
usage and stop reasons;
error taxonomy;
idempotency/retry semantics.

Admission before expensive work

The service needs bounded:

request bytes
decoded media
input tokens
reserved output
queue length
active sequences
KV/cache memory
batch tokens/pixels/frames
tenant concurrency and usage

Return overload honestly. An unbounded queue changes overload into latency and OOM.

Batching is task-specific

embeddings/classification can batch complete requests with similar shapes;
rerankers batch query/candidate pairs;
generators need prefill/decode scheduling and per-sequence KV state;
image/video generation has long iterative steps and large output buffers.

A universal Vec<Request> batch hides the resource unit. Admission should account in tokens, pixels, frames, samples, steps and expected resident bytes.

Protocol and cancellation

Carry one absolute deadline through every hop. If each layer starts a fresh 30-second timeout, a request can exceed the user’s budget.

Client disconnect is evidence of lost interest, not proof that compute stopped. Propagate cancellation to the scheduler, remove queued work, and define whether active accelerator kernels are interruptible. Record wasted work.

Retries are safe only when:

the operation has no side effect, or has an idempotency key;
attempt budgets are bounded;
the original deadline remains;
overload responses use backoff/jitter;
partial streamed output is not duplicated.

Never retry every timeout immediately; that amplifies overload.

Service security

“Private network” is not authorization. Use:

workload identity and mutually authenticated transport;
per-tenant capability authorization;
quotas and cost accounting;
request/body/media bounds before decoding;
model-route allowlists;
no arbitrary filesystem/URL inputs;
redacted structured logs;
least-privilege artifact access;
network policy;
signed/verified images and model artifacts.

Remote APIs in depth

Remote APIs are compelling when:

a managed/frontier capability is required;
self-hosting hardware/runtime expertise is not justified;
demand is bursty and provider limits fit;
time to first implementation matters;
the provider’s data, residency and retention terms satisfy policy;
multiple providers can be evaluated/routed.

The application still owns:

input validation and minimization;
secrets and credential rotation;
authorization to invoke a model/tool;
deadlines, rate limits, retries and circuit breaking;
schema validation and safe output handling;
release metadata/change detection;
evaluation and incident response;
cost controls;
fallback truthfulness.

Capacity is not infinite

Providers expose quotas, per-minute units, concurrency limits, overload and regional incidents. Design a local admission queue that is bounded and fair. Separate retryable transport failures from invalid requests, safety refusals and exhausted budgets.

Data minimization

Before sending:

remove fields the task does not need;
avoid secrets and unrelated conversation history;
use tenant-approved provider/region;
understand retention/training controls;
hash or pseudonymize only when the task still works and policy allows;
treat embeddings and outputs as potentially sensitive derived data.

Encryption in transit does not change who receives plaintext at the endpoint.

Provider change

Record requested and resolved model plus runtime fingerprint when available. A stable API schema does not prove stable model behavior. Apply the release audit from the previous chapter and maintain canaries.

Hybrid placement

One product can use all three:

embedded small classifier
  → route/abstain locally

private embedding + reranking service
  → keep corpus inside private boundary

remote generator
  → only approved minimized evidence

Or:

remote primary for highest quality
local verified fallback for outage

Fallback is a different release and often a different capability. Do not report success if the fallback cannot meet the output/safety contract. Re-evaluate routing and fallback paths, including the transition itself.

Avoid per-request arbitrary topology choice. Policy should consider:

tenant/data classification;
required capability;
latency/quality target;
current capacity;
external-provider permission;
release approval;
cost budget.

Record the selected route and reason without sensitive content.

Benchmark the real candidates

Once constraints identify feasible options, benchmark each on the same cases and load shape.

Quality

task success and instruction following;
calibration/abstention;
retrieval/reranking metrics;
multimodal fidelity;
shifted/adversarial slices.

Latency and throughput

client-observed end-to-end;
queue time;
serialization/upload;
preprocessing;
prefill/first token;
inter-token latency;
terminal validation;
items/tokens/pairs/frames per second;
p50/p95/p99 under steady and burst load.

Resources and operations

host/device memory and fragmentation;
model cold start/warm-up;
power/cost per successful task;
failure/retry/wasted-compute rate;
autoscaling lag;
deployment/rollback time;
operator burden.

Warm-cache microbenchmarks do not answer cold-start or overload behavior. Test realistic input lengths, concurrency, cancellation and provider limits.

Failure investigation

Symptom	Likely topology checks
App replicas OOM during rollout	embedded model copies, concurrent old/new pods, runtime arena/cache
Private service p99 spikes	bounded queue, prefill/decode contention, batch policy, network retries
Remote costs double	retry amplification, output length, fallback duplication, provider route
Cancellation does not free GPU	scheduler/kernel interruptibility and queued-vs-active state
Local mode corrupts async latency	blocking inference running on executor workers
Remote works, private fails quality	tokenizer/template/release mismatch, not topology alone
Service is “healthy” but rejects all work	readiness/admission capacity versus liveness
Sensitive tenant reaches remote	route authorization/data classification failed before adapter

Diagnose by stage and release identity. “Network issue” and “model slow” are not root causes.

Exercises

Exercise 1: recompute the example

Change application replicas to two and local budget to 16 GiB. Disable independent scaling and fault-isolation requirements but retain private data and external-network independence. Which topologies pass?

Solution

Embedded uses 2 × 6 = 12 GiB, stays same-process, and no longer violates scaling/isolation, so it passes. The private service uses 12 GiB and passes. Remote still violates data boundary and external network independence.

Exercise 2: model per-host placement

Extend the planner with host count, copies per host and per-host memory budget. Reject a plan where fleet total fits but one host is overcommitted. Include old+new replicas during rolling update.

Exercise 3: sidecar topology

Add SameHostSidecar with one Unix-socket hop, same-host data boundary, independent process failure but scaling coupled to the app. Explain which existing Boolean properties are insufficient.

Exercise 4: deadline propagation

Implement a Deadline based on monotonic time. Simulate client → gateway → service → worker and prove each hop uses remaining time rather than resetting the budget.

Exercise 5: retry storm

Model 100 clients timing out together against an overloaded service. Compare immediate retry, exponential backoff with jitter, and no retry. Count attempts and useful completions.

Exercise 6: hybrid privacy route

Design a typed policy that allows public text to a remote generator, private documents only to a private reranker, and secrets nowhere. Test an indirect prompt instruction attempting to override the route.

Check your understanding

Why can embedded inference multiply model memory with app replicas?
What does the planner’s memory budget aggregate?
Why is a private network not an authorization boundary?
What new failure modes does process isolation introduce?
Why can a larger inference queue reduce reliability?
Which operations can safely retry after partial streaming?
Why does remote zero local model memory not mean zero resource cost?
What measurements must follow constraint filtering?
When should a hybrid route fail instead of falling back?

Completion checklist

Sources and further study

Hugging Face, Candle: a native Rust ML framework focused on performance and lightweight deployments, with CPU/GPU support and model examples.
ONNX Runtime, Performance tuning: latency, throughput, memory and application/model size as deployment measurement dimensions.
ONNX Runtime, Thread management: intra/inter-op thread pools and avoiding contention across sessions.
ONNX Runtime, Memory consumption: allocators and sharing arenas across sessions.
vLLM, official project documentation: a model-serving system with scheduling, continuous batching and an API boundary.
KServe, official documentation: inference services, autoscaling, model serving and staged deployment features.
The earlier blocking/media/accelerator scheduling chapter: isolating blocking work and bounding permits in an async Rust service.
The earlier release-identity chapter: preserving actual model/provider identity through topology changes.

What this chapter does not claim

The planner models one embedded process, one private-network service shape and one external-provider shape. It does not model per-host placement, shared pages, multi-GPU sharding, service meshes, sidecars, confidential computing, price, latency, quality or availability probability. Its “network hops” are minimum logical boundaries. The six-GiB resident input must come from measurement. The planner’s value is that it rejects architecture by declared invariants before subjective optimization begins.

The next chapter derives memory effects from numeric formats, quantization, batching and device placement, giving the topology planner better measured inputs.