Quantization, Memory, Batching, and Device Placement

“It is a 7B four-bit model, so it needs 3.5 GB” is not a deployment plan.

It is one multiplication about one memory region. It omits quantization scales and zero points, runtime state, converted or duplicated weights, graph workspaces, activations, padding, KV caches, allocator behavior, driver reservations, concurrent requests, and a safety margin. It also confuses compact storage with the precision and kernels used for computation.

This chapter builds a checked lower-bound memory planner in mosaic-ml. It makes each declared byte region visible, compares the result with a user-supplied device profile, and fails closed on invalid capabilities or arithmetic overflow. The planner does not predict speed, quality, peak graph memory, or universal hardware support. Those facts must come from evals and measurements on the exact model, runtime, kernel, driver, and device.

Learning objectives

By the end you will be able to:

distinguish checkpoint format, weight storage, compute precision, activation precision, and KV-cache precision;
calculate packed dense and block-quantized weight payloads with ceiling division;
include per-group scale and zero-point metadata rather than quoting payload alone;
explain affine quantization and the accuracy/overhead trade-off of tensor, channel, and group granularity;
distinguish post-training quantization, quantization-aware training, and quantized fine-tuning;
inventory resident weights, fixed runtime state, per-sequence scratch, padded activations, KV reservations, and headroom;
calculate padding waste for a heterogeneous batch;
explain why static, dynamic, continuous, and chunked batching have different latency/memory behavior;
represent device capabilities as measured input rather than infer them from the label “GPU”;
reject a compute format unsupported by the declared runtime/device profile;
compare CPU, CUDA, Metal, and other accelerators without assuming one universal winner;
build quality, latency, throughput, memory, and thermal gates for a quantized release;
diagnose out-of-memory failures that a checkpoint-file size estimate misses;
state exactly what the runnable planner does not model.

Prerequisites and dependency order

Read these chapters first:

What a model artifact actually contains separates weights, architecture, tokenizer/processor, template, runtime, and license.
Tokens, embeddings, tensors, and transformer mechanics establishes tensor shapes and operations.
Prefill, decode, KV caches, and context cost derives dense decoder KV bytes and phase-specific latency.
Remote, embedded, and service execution decides how many processes own model copies and where the device boundary lives.

This chapter plans one model instance on one declared device. The topology planner answers how many such instances a deployment owns. Multiply deliberately; do not silently treat an eight-replica application as one model copy.

Completion artifact

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-ml --example quantization_batch_device

The companion compares the same seven-billion-parameter model, batch, runtime, cache, and 8 GiB device profile under two weight layouts:

device capacity                         8,589,934,592 bytes
10% headroom leaves                     7,730,941,132 usable bytes

batch sequence lengths                  [128, 512, 300, 64]
actual tokens                           1,004
rectangular padded tokens               2,048
padding efficiency                      49.02%

four-bit payload                        3,500,000,000 bytes
54,687,500 groups × 3 metadata bytes      164,062,500 bytes
resident quantized weights              3,664,062,500 bytes
fixed runtime                             536,870,912 bytes
per-sequence scratch                      134,217,728 bytes
padded-token activations                  134,217,728 bytes
reserved KV cache                         536,870,912 bytes
required                               5,006,239,780 bytes
result                                  admitted

FP16 resident weights                  14,000,000,000 bytes
FP16-layout total                      15,342,177,280 bytes
shortfall against usable device        7,611,236,148 bytes
result                                  rejected

Both variants declare BF16 computation. That is intentional: four-bit weight storage does not mean every multiply and accumulation is four-bit.

First principles: quantization is a representation contract

An ideal real-valued tensor contains values (x). A common affine quantizer represents them with integers (q), a positive scale (s), and a zero point (z):

q        = clamp(round(x / s) + z)
x_approx = s × (q - z)

The integer range is finite. Mapping and rounding introduce error. A symmetric scheme often fixes or eliminates the zero point and centers the representable range around zero. An asymmetric scheme can use the integer range more effectively when the observed real range is not symmetric.

ONNX Runtime’s quantization documentation defines the scale/zero-point mapping, distinguishes operator-oriented and quantize/dequantize graph representations, and separates dynamic from static activation quantization. Its warning matters: quantization is not lossless, so compare original and quantized weights, activations, and downstream behavior.

Granularity creates an accuracy/metadata trade-off

One scale for a whole tensor is cheap but forces values with different ranges through one mapping. Per-channel or per-group quantization can fit local distributions more closely, but every channel or group needs metadata and a kernel that understands the layout. The Transformers quantization guide describes this trade-off and notes that two four-bit values are commonly packed into one byte for storage because ordinary hardware does not generally expose arbitrary four-bit memory objects as a drop-in scalar type.

For (P) parameters, (b) bits per stored weight, group size (g), and (m) metadata bytes per group:

payload_bytes  = ceil(P × b / 8)
groups         = ceil(P / g)
metadata_bytes = groups × m
resident_weight_lower_bound = payload_bytes + metadata_bytes

Ceiling division is required. Three four-bit values occupy twelve bits and therefore two whole bytes. Three weights grouped in pairs require two groups, not one and a half.

The companion uses checked integer arithmetic:

let payload_bits = parameter_count
    .checked_mul(bits_per_weight)
    .ok_or(DeploymentPlanError::ArithmeticOverflow)?;
let payload_bytes = checked_ceil_div(payload_bits, 8)?;
let groups = checked_ceil_div(parameter_count, weights_per_group)?;
let metadata_bytes = groups
    .checked_mul(metadata_per_group)
    .ok_or(DeploymentPlanError::ArithmeticOverflow)?;

Never perform capacity arithmetic with wrapping release-mode integers. An overflowed byte count can look small and incorrectly admit work.

What can be quantized?

Treat at least these as separate choices:

Region	Examples	Main trade-off
Weights	FP32, FP16, BF16, INT8, packed INT4	resident size/bandwidth versus approximation error and kernel availability
Activations	FP32, FP16/BF16, INT8	temporary memory/compute versus range calibration and accuracy
Accumulators	often wider than operands	numerical stability versus throughput
KV cache	FP16/BF16, FP8/INT8/lower in supported systems	long-context concurrency versus attention quality and conversion overhead
Embeddings/index	FP32/FP16, scalar/product/binary quantization	index size/search speed versus recall/ranking fidelity
Optimizer states	training-specific wide or quantized states	training memory versus update stability

“W4A16” usually means four-bit weights and sixteen-bit activations, not an entirely four-bit program. Record the actual scheme, granularity, calibration method, excluded layers, compute dtype, KV dtype, backend, and kernel version in release metadata.

Storage format is not compute format

The runnable API prevents one enum from pretending to describe both:

pub enum WeightLayout {
    Dense(DenseStorageFormat),
    BlockQuantized(BlockQuantization),
}

pub enum ComputeFormat {
    Float32,
    Float16,
    BFloat16,
    Int8,
}

A runtime may read packed INT4 blocks, load scales, dequantize blocks into FP16/BF16 registers, and accumulate in a wider format. Another runtime may have fused weight-only kernels. A file converter may generate a backend-specific layout. The same nominal “four-bit” label can therefore have different memory overhead, supported operations, quality, and speed.

The Transformers example explicitly configures four-bit loading and BF16 compute separately. Preserve that distinction in your Rust config and trace.

Never infer a speedup from fewer bits

Lower storage can reduce memory traffic and permit a model or larger batch to fit. It is not a proof of lower latency:

the device may lack a compatible instruction or fused kernel;
unpack/dequantize operations add work;
the graph may fall back to a slower execution provider;
conversions can cross the host/device boundary;
unsupported operators may run at another precision;
small batches may not saturate the accelerator;
the bottleneck may be tokenization, media decoding, network, queueing, or sampling;
thermal or power limits can change sustained performance.

ONNX Runtime documents that hardware support determines whether quantization accelerates execution and that overhead can make it slower on unsuitable hardware. The only valid conclusion before measurement is “the declared representation requires fewer payload bytes.”

PTQ, QAT, and quantized fine-tuning are different procedures

Post-training quantization

PTQ transforms an already-trained model. Weight-only schemes may need no task training, while activation quantization often uses calibration data or dynamic ranges. It is operationally attractive because it avoids full retraining, but still needs representative quality evaluation.

Important families include:

GPTQ, a one-shot weight quantization method using approximate second-order information;
AWQ, which uses activation observations to protect salient weights while targeting hardware-friendly weight-only quantization;
SmoothQuant, which moves quantization difficulty between activations and weights to enable W8A8 execution.

Names do not guarantee identical artifacts across tools. Pin algorithm/configuration, calibration dataset identity, converter version, source revision, output digest, and runtime compatibility.

Quantization-aware training

QAT simulates quantization effects during training so parameters can adapt to them. It costs a training workflow and can recover quality when PTQ does not meet the product gate. It does not remove the need to test the exported graph and actual deployment kernels.

Quantized fine-tuning

QLoRA backpropagates through a frozen four-bit quantized base model into trainable low-rank adapters. That is a training-memory technique and adaptation method. It is not synonymous with serving a generic four-bit checkpoint, and it does not imply that optimizer states, gradients, activations, or compute all use four bits.

The fine-tuning part of this book will implement the dataset and release gates. Here the lesson is to name the procedure precisely.

Build a complete memory inventory

For one runtime instance, begin with:

required =
    resident weights
  + quantization metadata
  + fixed runtime state
  + per-sequence scratch
  + padded-token activations
  + reserved KV cache
  + other measured graph/device regions

Then compare against:

usable = floor(device_capacity × (10,000 - headroom_bps) / 10,000)

The companion models the first six terms and exposes headroom. It deliberately leaves backend-specific “other” terms for measurement.

Resident weights

Do not substitute download size:

compression such as ZIP/Zstandard is removed in memory;
memory mapping can share physical pages, but residency changes with access and eviction;
conversion may create a second layout before releasing the first;
tensor/device sharding changes placement;
some runtimes prepack weights;
adapter merge can allocate a new tensor;
multiple processes or devices may hold copies.

Inspect runtime allocation after warmup and after the largest supported request.

Fixed runtime state

This can include graph metadata, tokenizer/media processor state, execution contexts, command buffers, allocator pools, compiled kernels, CUDA/Metal contexts, staging buffers, and backend workspaces. “Fixed” means independent of the batch in this model, not constant across runtime versions or devices.

Scratch and activations

Temporary tensors depend on graph, sequence shape, batch, algorithm, fusion, precision, and memory reuse. Peak live memory is not the sum of every tensor ever created, but it is also not zero simply because outputs are small.

The companion takes two measured coefficients:

pub scratch_bytes_per_sequence: u64,
pub activation_bytes_per_padded_token: u64,

These are intentionally inputs. Obtain them by controlled sweeps and conservative deltas, then re-profile when graph/runtime/device changes.

KV reservation

The previous chapter derives dense decoder K+V bytes:

bytes/token/sequence =
    2 × layers × key_value_heads × head_dimension × element_bytes

The example gets the coefficient from KvCacheGeometry and reserves 1,024 tokens for each of four sequences. Real systems may use paged blocks, prefix sharing, sliding attention, offload, architecture-specific caches, or quantized KV. Make those separate plan types rather than editing a dense formula until it appears to match.

Transformers’ cache documentation explains the key/value update mechanics, while its cache strategy guide compares dynamic, fixed-size, offloaded, sliding-window, and quantized approaches. Fixed caches can unlock compilation but reserve/mask unused positions; offload saves device memory by moving state across a slower boundary.

Headroom

Headroom is not wasted memory. It absorbs unmodeled peaks, allocator fragmentation, background device use, runtime changes, and estimation error. Choose it from soak tests and failure budgets. A percentage is not magic: record maximum allocated and reserved bytes, OOM rate, batch/context distribution, and the exact workload that justified it.

The planner uses basis points and rejects 100% or greater:

if headroom_basis_points >= 10_000 {
    return Err(DeploymentPlanError::InvalidHeadroom);
}

Batching is a shape and scheduling decision

Batching amortizes fixed work and can improve device utilization, but it changes queueing, padding, memory, and failure coupling.

Static rectangular batches

A conventional tensor batch pads sequences to one length. For lengths 128, 512, 300, and 64:

actual = 128 + 512 + 300 + 64 = 1,004
padded = 4 × max(...)         = 2,048
waste  = 1,044 tokens
efficiency = floor(1,004 × 10,000 / 2,048) = 4,902 basis points

The companion keeps this observable:

pub struct BatchShapePlan {
    pub sequence_count: u64,
    pub longest_sequence_tokens: u64,
    pub actual_tokens: u64,
    pub padded_tokens: u64,
    pub padding_tokens: u64,
    pub padding_efficiency_basis_points: u64,
}

Length bucketing can improve padding efficiency. It can also delay rare long requests or create priority inversion. Measure queue time per bucket and tenant, not just kernel throughput.

Dynamic batching

A server waits for compatible requests up to a batch-size or time limit, forms a batch, then runs it. A longer wait may improve throughput while worsening time to first token. The batcher must honor deadlines: do not spend a request’s remaining budget waiting for an ideal batch.

Continuous batching

Autoregressive sequences finish at different times. Continuous batching admits new work as slots or token blocks become available instead of waiting for the whole original batch to finish. This improves utilization but requires a scheduler, per-sequence state, cancellation cleanup, fair admission, and correct cache block ownership.

The PagedAttention/vLLM paper motivates block-based KV memory management. The current vLLM configuration exposes explicit KV bytes/tokens and max-context concurrency rather than treating “GPU utilization” as a complete capacity statement.

Chunked prefill

One long prefill can delay decode work. Chunking splits it so decode and prefill tokens can share a scheduling budget. Current vLLM optimization guidance explains the TTFT/ITL trade-off: the batched-token budget and scheduling policy affect the two phases differently.

Do not report one latency percentile for a mixed scheduler. At minimum report queue, preprocessing, prefill, first decode, remaining decode, TTFT, ITL, and end-to-end distributions by input/output shape.

Device placement is a capability declaration

The companion requires an explicit profile:

let device = DeviceProfile::try_new(
    DeviceKind::Cuda,
    8 * GIB,
    vec![
        ComputeFormat::BFloat16,
        ComputeFormat::Float16,
        ComputeFormat::Int8,
    ],
)?;

It never defines “all CUDA supports BF16” or “all Metal supports INT8.” DeviceKind is only a placement label. Exact support depends on accelerator generation, operating system, driver, runtime/execution provider, compiled features, model operators, tensor shapes, and available kernels.

If the requested compute format is absent:

if !device.supports(compute_format) {
    return Err(DeploymentPlanError::UnsupportedComputeFormat {
        device: device.kind(),
        format: compute_format,
    });
}

That rejection occurs before capacity is considered. Memory that fits is not useful if the execution contract cannot run.

CPU

CPU deployment can be operationally simple, widely available, and efficient for small models, embeddings, rerankers, OCR stages, or low concurrency. Performance depends on vector/dot-product instructions, core/memory topology, thread settings, NUMA placement, quantized kernels, and contention with the host application. Benchmark physical-core configurations and sustained memory bandwidth. More threads can make tail latency worse.

CUDA

CUDA devices offer high-throughput tensor kernels but have discrete memory, driver/runtime compatibility, transfer costs, stream synchronization, and generation-specific precision support. Observe device allocated/reserved memory and host pinned memory. A silent CPU fallback or frequent host-device copy can erase expected gains.

Metal and unified-memory systems

Unified memory removes the simple “copy across PCIe into VRAM” mental model, not capacity limits or data movement. The CPU, GPU, other accelerators, and applications compete for a shared physical budget; pressure can cause compression or paging. Query the exact feature set and keep working-set headroom. Apple publishes per-family tensor formats and alignment requirements in its Metal feature-set tables; do not generalize the newest table to older hardware or operating systems.

Multi-device placement

Tensor, pipeline, data, and expert parallelism have different communication and memory behavior. Dividing parameter bytes by device count is insufficient because:

some state is replicated;
partitions can be imbalanced;
collective communication adds buffers and latency;
KV placement follows the serving architecture;
the slowest stage can set throughput;
a device loss can invalidate the whole group.

Model each rank/device separately, then run a real distributed soak.

Step-by-step Rust implementation

Step 1: make invalid weight arithmetic unrepresentable at the boundary

WeightLayout::plan requires a nonzero parameter count. Block quantization accepts 1–16 bits and a nonzero group size. It calculates payload and metadata separately and returns both:

pub struct WeightMemoryPlan {
    pub parameter_count: u64,
    pub payload_bytes: u64,
    pub quantization_groups: u64,
    pub quantization_metadata_bytes: u64,
    pub total_resident_bytes: u64,
}

The range is a planner bound, not a claim that the runtime implements every 1–16-bit scheme.

Step 2: validate bounded batch input

An empty batch, a zero-token sequence, or more than 4,096 declared sequences fails. The maximum is a defensive planning bound. Production request admission should bound decoded array size before constructing this value.

The planner sums actual tokens with checked_add, multiplies sequence count by longest length with checked_mul, and reports efficiency in integer basis points. This keeps tests deterministic and avoids floating-point boundary disagreements.

Step 3: make KV optional, not ambiguous

Encoder-only embeddings/classification may have no autoregressive KV cache:

pub kv_reservation: Option<KvReservation>

None means the planner includes zero KV bytes. Some requires nonzero bytes per token and reserved tokens per sequence. A pair of zeros is not overloaded to mean “unknown” or “not applicable.”

Step 4: sum named memory regions

let required_bytes = [
    weight.total_resident_bytes,
    fixed_runtime_bytes,
    scratch_bytes,
    activation_bytes,
    kv_cache_bytes,
]
.into_iter()
.try_fold(0_u64, |total, bytes| total.checked_add(bytes))
.ok_or(DeploymentPlanError::ArithmeticOverflow)?;

The result exposes every term plus usable bytes, remaining bytes, shortfall, and admission. A caller can render a useful rejection rather than “CUDA out of memory.”

Step 5: separate planning from enforcement

plan() returns an over-capacity report so diagnostics can show the shortfall. ensure_admitted() turns that result into a typed error:

let plan = inputs.plan()?;
let admitted = plan.ensure_admitted()?;

Use the first in capacity tools and the second in admission paths.

Turn an estimate into a release gate

The Rust calculation is only the preflight. A production candidate should pass four coupled gates.

1. Functional and quality gate

Compare the quantized candidate with the pinned baseline on:

frozen ordinary, boundary, adversarial, and long-context cases;
task success and instruction compliance;
abstention/calibration, not only average accuracy;
structured-output validity and tool-selection correctness;
embedding recall/ranking or media-specific perceptual metrics;
slices known to be numerically sensitive;
exact harness, tokenizer/processor, and generation settings.

Set allowed regression per critical slice before inspecting results. One mean score can hide a severe minority or rare-workflow regression.

2. Performance gate

Warm both variants, then sweep:

input and output length;
batch/concurrency;
cold and warm start;
prefill and decode separately;
latency percentiles and achieved tokens/items per second;
CPU threads or accelerator streams;
power/thermal steady state;
host/device transfers and fallback operators.

Hold offered load constant when comparing latency. Hold latency SLO constant when comparing throughput. Otherwise queue saturation can make the “faster” variant look arbitrarily slow.

3. Memory and stability gate

Record process RSS and, where applicable:

allocated versus reserved device bytes;
peak after load, warmup, longest input, longest output, and highest admitted concurrency;
allocator retries and OOMs;
cache block utilization and fragmentation;
host pinned/staging memory;
unified-memory pressure/page activity;
recovery after cancellation and failed requests;
a sustained soak, not just one inference.

Tune headroom from this evidence. Verify that the admission limit rejects the next unsafe request before the runtime allocator fails.

4. Compatibility and rollback gate

Pin:

source model and immutable revision;
quantizer algorithm and exact configuration;
calibration dataset digest;
converter/runtime/library versions;
output artifact digests;
device/driver/OS/backend identity;
processor/tokenizer/template;
harness and eval-suite releases.

Test rollback by loading the previous complete bundle and restoring its routing/admission policy. Retaining only yesterday’s weight file is not a rollback.

Failure investigations

Failure: the file is 4 GB, but an 8 GB device OOMs

Investigate resident/prepacked weights, conversion copies, runtime workspace, graph temporaries, padded activations, KV reservation, allocator reserve/fragmentation, other processes, and device context. Reproduce after warmup at the maximum supported shape. Compare measured regions with the planner; add a named coefficient instead of lowering headroom until it “works.”

Failure: INT8 is slower than FP16

Confirm the exact graph uses INT8 kernels. Inspect operator placement and fallbacks. Check conversion and Q/DQ overhead, tensor shapes, batch size, host/device copies, supported instructions, and thread/stream configuration. Benchmark phases separately. The representation can be smaller while the execution path is worse.

Failure: average quality is stable, production complaints rise

Slice by task, language, length, media condition, confidence, tool route, and rare labels. Compare calibration and overconfident errors. Inspect layers/operators excluded or quantized differently. Rebuild calibration data from representative inputs without leaking the final test set.

Failure: short requests become slow after batching

Inspect queue time, length buckets, max batch wait, prefill scheduling, decode priority, and tenant fairness. A high-throughput batch can violate an interactive TTFT SLO. Add deadline-aware admission and separate service classes where justified.

Failure: memory grows after cancellations

Trace sequence/cache-block ownership from admission through cancel, timeout, provider error, stream disconnect, and normal completion. Await cleanup. Test repeated cancellation under load and assert that allocated cache blocks return to the baseline.

Failure: the model fits once but fails in a second process

Revisit topology. Device contexts, runtime arenas, converted weights, and caches may be per process. Memory-mapped host pages do not guarantee shared accelerator residency. Account for every process and device placement.

Security implications

Treat model/converter artifacts as untrusted until digest, size, path, and format validation pass.
Bound parameter counts, tensor dimensions, group counts, batch arrays, sequence lengths, and multiplication before allocation.
Do not let request-supplied precision/device flags bypass an evaluated release profile.
Authorize expensive context, batch, image resolution, video duration, and generation limits per tenant.
Prevent one tenant from monopolizing a batch or KV pool; preserve identity through the scheduler.
Avoid logging prompts, media, raw tensors, embeddings, or model-provider secrets in memory traces.
Isolate native converters and model inspection from trusted services.
Treat OOM as availability and possibly multi-tenant interference risk; admission must fail before allocator collapse.
Record artifact and runtime identity without exposing sensitive local paths or credentials.

What the companion intentionally does not model

The plan is a lower bound over declared regions, not an allocator/runtime simulator. It excludes:

tensor alignment and backend-specific block packing beyond supplied group metadata;
sharding, replication, offload, prefix sharing, and paged-cache fragmentation;
temporary conversion copies and graph-specific peak liveness;
allocator bins, reserved pools, driver/OS memory, and other processes;
fused-kernel availability, transfers, bandwidth, compute utilization, power, and latency;
quality loss or calibration adequacy;
different activation/KV shapes for encoder, cross-attention, MoE, diffusion, audio, or video architectures;
dynamic arrival distributions and scheduler fairness;
physical sharing behavior of memory-mapped or unified memory.

This honesty is a feature. Extend it with measured, named terms for a concrete runtime. Do not turn it into a false universal estimator.

Exercises

Recall

Why can three four-bit values require two bytes?
What memory does a quantization scale consume?
Why is four-bit storage compatible with BF16 computation?
What does W4A16 describe?
Why can INT8 be slower than FP16?
What is the difference between actual and padded batch tokens?
Why is KV memory a concurrency term?
What fact does a Cuda label fail to establish?
Why should headroom survive even after a successful load?
How do PTQ, QAT, and QLoRA differ?

Implementation

Add a PerChannelQuantization layout and tests for metadata bytes.
Add separate activation and KV storage-format fields without conflating them with compute.
Extend batch planning with length buckets and compare total padded tokens with one rectangular batch.
Build a sweep that finds the largest admitted batch for fixed lengths.
Feed KvCacheGeometry for full multi-head and grouped-query attention into the deployment plan; explain the ratio.
Add JSON serialization through a versioned output type and a golden fixture.
Add a property test that admitted plans always satisfy required <= usable.
Add a fuzz target for public planner JSON and prove malformed values never panic.
Measure your actual runtime’s peak bytes across batch/context and fit conservative coefficients.
Add a cancellation soak that asserts cache/allocator state returns below a declared threshold.

Design

Design deadline-aware length bucketing for interactive and batch tenants. State starvation rules.
Decide whether a 1.5% quality regression is acceptable for a twofold concurrency gain. Define the product slices and severity weights needed to answer.
Design release identity for a quantized local model including quantizer, calibration data, runtime, and hardware.
Compare CPU, CUDA, and Metal placement for an embedding service. Define workload and metrics before choosing.
Design admission for one model sharing a device with another service. Explain why independent percentage knobs may overcommit.
Design a multi-device plan that accounts for replicated and sharded regions separately.

Solution guidance

A packed payload still allocates whole bytes; use ceiling division.
Scales and optional zero points are metadata, multiplied by group/channel count.
Storage is unpacked/dequantized for a declared kernel/accumulator path.
Usually four-bit weights and sixteen-bit activations; verify the scheme’s exact definition.
Missing/slow kernels, conversion, fallbacks, transfers, or inadequate shape/utilization can dominate.
Actual tokens carry input; padded tokens occupy rectangular compute/storage positions.
Each active sequence owns cache state across its reserved/realized context.
It establishes neither generation-specific dtype support nor backend/operator compatibility.
Load is not peak execution, and allocators/devices have unmodeled consumers and variation.
PTQ converts trained parameters, QAT trains through simulated quantization, and QLoRA adapts a frozen quantized base via low-rank trainable parameters.

For the bucketing exercise, sort or assign by declared length bands, compute each bucket’s maximum times its count, and include queue/deadline behavior in the result. Minimizing padding alone can starve a rare long sequence.

For empirical coefficients, vary one factor at a time after warmup, sample multiple repetitions, record exact runtime/device identity, and choose a conservative upper envelope. Do not fit on the same single configuration used to validate the estimate.

Check your understanding

Can a candidate use less weight memory and more peak total memory?
Does a quantized artifact prove the runtime executed quantized kernels?
Can two four-bit formats have different resident bytes?
Is padding only a performance cost, or can it also consume activation memory?
Why does a static KV cache trade memory for compilation-friendly shapes?
Which terms grow with batch, context, processes, and devices?
How would you detect a silent CPU fallback?
What must be evaluated again after changing only KV precision?
What exact evidence would justify reducing headroom?
When should an over-capacity planner return a report rather than an immediate error?

Completion checklist

Authoritative references

The next chapter turns these concrete capability and capacity contracts into provider-neutral Rust traits. Scripted, embedded, inference-service, and remote adapters will remain interchangeable only where they honestly implement the same capability.