Benchmarking, Allocations, Flamegraphs, and Capacity

Performance engineering is a chain of claims:

representative workload
  → controlled measurement
  → distribution + resource evidence
  → causal profile
  → one change
  → repeated correctness and performance gates
  → capacity hypothesis
  → production validation

Breaking any link produces persuasive-looking fiction. A fast debug-only stub says nothing about a release build. An average hides a disastrous tail. A microbenchmark cannot include queueing. A flamegraph identifies where sampled CPU time accumulated; it does not prove why end-to-end latency changed. Little’s Law relates arrival, concurrency, and time; it does not promise a latency SLO under bursts.

The companion mosaic-performance crate makes the complete chain inspectable. It contains an allocation-free scalar cosine scorer, typed invalid-vector failures, deterministic percentile aggregation, a stable custom Cargo benchmark, and a capacity calculator. The measured numbers are machine-local evidence. The implementation and experiment identity are the reusable result.

Learning objectives

write a benchmark question and correctness contract before timing;
distinguish latency, throughput, utilization, queue delay, cost, and quality;
choose micro, component, end-to-end, load, soak, and stress experiments deliberately;
keep setup, allocation, serialization, logging, and aggregation outside a timed hot path;
use optimized builds, warmup, repeated samples, percentiles, and black_box responsibly;
identify allocator traffic, retained memory, RSS, mapped model pages, and accelerator memory;
read CPU flamegraphs without treating width, color, or horizontal order incorrectly;
benchmark text, embeddings, images, audio, and video with modality-specific work units;
derive a first capacity estimate from arrival rate and measured service time;
understand when batching, burstiness, queueing, retries, and shared accelerators invalidate a simple plan;
gate regressions with stable experiment identity and practical/statistical significance;
investigate “faster” results that actually removed work or broke quality.

Dependencies and mental model

This chapter depends on release-quality builds, deterministic tests, bounded concurrency, durable jobs, and versioned model/harness identities. It precedes observability because benchmarks define the metrics and workload labels production must emit.

An AI request usually visits several queues:

admission
  → decode/normalize
  → retrieval/embedding
  → model batch queue
  → accelerator execution
  → validation/repair
  → encode/persist/stream

For one stage:

response time = queue wait + service time
throughput    = completed units / elapsed time
utilization   = busy capacity / available capacity

Reducing service time may not reduce response time if the real wait is admission or GPU batching. Increasing a batch may improve items/second while making the first item slower. Lowering resolution may appear faster while destroying OCR or small-object recall. Performance is therefore a vector, not one number:

(quality, latency distribution, throughput, memory, compute, energy, money, reliability)

Hold the required dimensions fixed or report the frontier. Never call a lower-quality configuration an optimization without naming the changed product contract.

Step 1: state the performance question

Bad:

How fast is Mosaic?

Testable:

On the pinned Apple Silicon host, does release-build scalar cosine similarity over 768 finite f32 components remain below the reviewed p95 threshold, with identical validation semantics and zero allocation inside the scorer?

An experiment specification includes:

operation and work unit;
input distribution, sizes, modality, and confidentiality class;
expected result or quality gate;
cold/warm cache and model-loading state;
concurrency, queue discipline, batch policy, and duration;
compiler, target, profile, features, source and lockfile;
CPU/GPU/accelerator, memory, OS, drivers/runtime and power state;
local/provider model, tokenizer/preprocessor, quantization, harness and dataset versions;
network/storage topology;
measured metrics and stopping rule;
number of independent runs;
output artifact schema and comparison policy.

Without this identity, two p95 values may describe different programs.

Step 2: choose the smallest experiment that answers it

Microbenchmark

Times a pure or narrow operation: token scan, cosine score, tensor conversion, JSON decoding, resampler, image resize. It is fast and causal but excludes queueing, I/O and orchestration.

Component benchmark

Exercises a meaningful boundary: retrieval against a fixed index, one local inference engine, one storage adapter, or one media preprocessing pipeline. It reveals copies and boundary costs while remaining controllable.

End-to-end benchmark

Runs the user-visible operation through HTTP/CLI, durable state, model, validation and artifacts. It measures actual response behavior but is harder to attribute and reproduce.

Load, stress and soak

load: expected and peak traffic; measures latency/throughput/error/queue behavior;
stress: increases work until overload; finds saturation and tests graceful rejection;
soak: holds representative work for hours/days; finds leaks, fragmentation, thermal effects, stale connections and slow queue growth.

Trace replay

Replays sanitized production shapes and arrival timing. It improves representativeness, but a trace can contain sensitive prompts/media and can reproduce old traffic rather than future traffic. Keep raw content out when size/type/timing summaries are enough.

Use multiple layers. A microbenchmark finds a regression in a scoring loop; a load test proves whether that change matters once queueing and model execution dominate.

Step 3: preserve correctness before timing

The companion scorer validates the same boundary it benchmarks:

pub fn cosine_similarity(left: &[f32], right: &[f32]) -> Result<f64, ScoreError> {
    if left.is_empty() || right.is_empty() {
        return Err(ScoreError::EmptyVector);
    }
    if left.len() != right.len() {
        return Err(ScoreError::DimensionMismatch {
            left: left.len(),
            right: right.len(),
        });
    }

    let mut dot = 0.0_f64;
    let mut left_squared = 0.0_f64;
    let mut right_squared = 0.0_f64;

    for (index, (&left_value, &right_value)) in left.iter().zip(right).enumerate() {
        if !left_value.is_finite() {
            return Err(ScoreError::NonFinite {
                vector: "left",
                index,
            });
        }
        if !right_value.is_finite() {
            return Err(ScoreError::NonFinite {
                vector: "right",
                index,
            });
        }

        let left_value = f64::from(left_value);
        let right_value = f64::from(right_value);
        dot = left_value.mul_add(right_value, dot);
        left_squared = left_value.mul_add(left_value, left_squared);
        right_squared = right_value.mul_add(right_value, right_squared);
    }
    // Zero-norm and finite-result checks follow before returning.
    # Ok(0.0)
}

It borrows slices, accumulates in f64, and performs no heap allocation. Tests prove:

identical finite vectors score approximately one;
dimensions must match;
NaN/infinity is rejected at its vector/index;
zero magnitude is rejected;
output remains finite and clamped to the mathematical range.

The scalar implementation is a reference contract. SIMD, BLAS, GPU or quantized kernels may use different accumulation order and rounding, so compare with documented absolute/relative tolerance over a representative corpus. Exact float equality is usually the wrong oracle. A faster kernel that silently accepts NaN or changes ranking beyond tolerance has failed.

Run correctness gates before and after measurement:

cd rust/rust-ai-engineering
cargo test -p mosaic-performance --all-targets
cargo clippy -p mosaic-performance --all-targets -- -D warnings

Step 4: construct a benchmark that measures the intended work

Stable Rust’s built-in #[bench] attribute remains nightly-only. Cargo supports a custom benchmark target with its harness disabled:

[[bench]]
name = "evidence_scoring"
harness = false

The companion target provides main, builds deterministic vectors once, warms the operation, then records 40 batch samples:

for dimension in [128, 768, 1_536] {
    let (left, right) = deterministic_vectors(dimension);

    for _ in 0..WARMUP_ITERATIONS {
        black_box(cosine_similarity(black_box(&left), black_box(&right))?);
    }

    let mut per_operation_nanos = Vec::with_capacity(SAMPLES);
    for _ in 0..SAMPLES {
        let started = Instant::now();
        for _ in 0..ITERATIONS_PER_SAMPLE {
            black_box(cosine_similarity(
                black_box(&left),
                black_box(&right),
            )?);
        }
        let elapsed = started.elapsed().as_nanos();
        per_operation_nanos.push(
            u64::try_from(elapsed / ITERATIONS_PER_SAMPLE as u128)?
        );
    }
    // Aggregation and JSON serialization happen after the timed region.
}

Release-like optimized profile

cargo bench uses the optimized bench profile. Do not compare it to cargo test timing: tests normally use an unoptimized profile, and the custom benchmark is also executed as a target by some all-target test commands.

Compile without execution:

cargo bench -p mosaic-performance --bench evidence_scoring --no-run

Execute:

cargo bench -p mosaic-performance --bench evidence_scoring

On one July 28, 2026 development run, the optimized 768-component case reported p50 around 0.90 µs and p95 around 0.92 µs. That is not a portable promise or a production SLO. The durable evidence is the source, environment, sample report and rerunnable command.

Warmup

Warmup reduces one-time contamination from code/data page faults, caches, lazy initialization and dynamic runtime setup. It must match the question:

include cold model load when measuring startup/readiness;
exclude it when measuring steady-state per-request service;
report both for CLI tools and scale-to-zero services;
treat JIT/graph compilation and accelerator autotuning as named lifecycle phases.

Warmup is not “discard samples until they look good.” Define it before observing results.

Batch timing

A timer call can cost as much as a tiny operation. The benchmark times 5,000 operations and divides by the count. This amortizes timer overhead, but it means each recorded value is a batch average, not an individual-request tail. For end-to-end latency, instrument each operation or use a load generator that records request histograms.

Prevent dead-code elimination

std::hint::black_box discourages the optimizer from assuming inputs/results and deleting the work. It is best-effort and platform/backend dependent. It guarantees identity semantics, not a cryptographic constant-time barrier. Never use it to claim timing-attack resistance.

Wrap prepared inputs and consumed output. Do not construct the input inside black_box and assume the preceding expression cannot be folded. Inspect generated assembly when compiler behavior is central to the claim.

Deterministic inputs versus representative inputs

Deterministic vectors make regression reproduction easy. They are not a population. Add corpora covering:

embedding dimensions and numeric distributions actually served;
aligned/misaligned lengths for SIMD tail behavior;
normalized and non-normalized vectors;
early-invalid positions if validation cost matters;
candidate-set sizes and cache footprints;
query/document reuse patterns.

Seeded pseudorandom data is reproducible, but record the generator and seed. Do not include data generation inside the timed region unless generation is the operation under test.

Step 5: summarize distributions without hiding them

The companion uses an explicit nearest-rank definition:

fn nearest_rank(sorted: &[u64], percentile: usize) -> u64 {
    let rank = sorted.len().saturating_mul(percentile).div_ceil(100);
    sorted[rank.saturating_sub(1)]
}

LatencySummary::from_nanos copies and sorts outside the timed operation, then reports sample count, minimum, integer mean, p50, p95, p99 and maximum.

Percentiles need context:

p50: typical observation;
p95/p99: tail at the observed load;
maximum: sensitive to sample count and outliers;
mean: useful for resource/capacity arithmetic, poor as a tail promise.

Forty batch samples make a compact local smoke report, not a statistically strong p99 estimate. There may be only one observation in the top 2.5%. For service tail claims, collect enough independent requests and report confidence/variation across repeated runs.

Do not average percentiles from shards. Merge histogram/sample distributions with compatible buckets, then compute the percentile. Do not coordinate omission: a closed-loop load generator that waits for each response may send less work exactly when the service stalls and understate the experienced tail. Reproduce the intended arrival process.

Clock resolution, monotonicity, scheduler preemption, frequency scaling, thermal throttling, background activity and virtualized noisy neighbors all matter. Pinning cores and performance power state can reduce noise but may make the test unlike deployment. Record rather than conceal the choice.

Step 6: make the report a versioned artifact

Each benchmark result should be machine-readable:

{
  "schema_version": 1,
  "benchmark_id": "cosine_similarity/f32/scalar",
  "source_revision": "<git-sha>",
  "dirty": false,
  "rustc": "1.96.0",
  "target": "<target-triple>",
  "profile": "bench",
  "lockfile_sha256": "<digest>",
  "host": {
    "cpu": "<model>",
    "logical_cores": 0,
    "memory_bytes": 0,
    "os": "<version>"
  },
  "workload": {
    "dimension": 768,
    "warmup_iterations": 2000,
    "samples": 40,
    "iterations_per_sample": 5000,
    "seed": "<generator-version>"
  },
  "quality_contract": "<test-or-eval-report-id>",
  "latency_nanos": {
    "p50": 0,
    "p95": 0,
    "p99": 0
  }
}

The current companion emits the workload and latency core. A release pipeline enriches it with source, toolchain and host identity. Sign or attest the artifact when benchmark gates protect a high-value release boundary.

Store raw samples/histograms when size permits. Summary-only artifacts cannot be reanalyzed when the percentile method or suspected multimodal distribution changes.

Step 7: allocation and memory analysis

Time answers “how long?” Allocation evidence answers “what memory traffic and ownership caused it?”

Count the right things

allocations and deallocations per operation;
allocated bytes per operation;
live/retained heap after steady state;
peak heap;
resident set size (RSS);
virtual mappings and page cache;
model weights and key/value cache;
pinned/transfer buffers;
GPU/accelerator allocated and reserved memory;
temporary image/audio/video frames;
fragmentation and allocator contention.

“No Rust heap allocation in this function” does not mean no memory cost. A provider SDK, C library, GPU runtime, kernel, memory mapping or page fault may allocate elsewhere.

Read the hot path first

cosine_similarity accepts slices and uses scalar locals. It neither constructs Vec, String, Box, nor a trait object. That makes an allocation-free source-level contract plausible. The benchmark allocates vectors, sample storage and JSON outside the timed loop.

Then verify dynamically with a suitable allocator/profiler for the target:

Instruments Allocations on macOS;
heaptrack or allocator-specific profiling on Linux;
DHAT/Valgrind where supported;
jemalloc/mimalloc profiling if that allocator is actually deployed;
accelerator vendor/runtime memory tools for device allocations.

Profile a long-enough repeated workload. Separate initialization from steady state. Check that the profiler itself does not change the claim being measured; use profiles to find causes, then compare performance again without instrumentation.

Common accidental allocations

format! or .to_string() in logging before level checks;
cloning prompt/media bytes across task boundaries;
collecting an iterator that could remain streaming;
growing a Vec repeatedly without known capacity;
converting &str to owned strings at every adapter;
serializing intermediate JSON between internal typed stages;
rebuilding tokenizer/regex/client/model state per request;
boxing small futures/closures in a very high-frequency path;
image colorspace/layout copies;
decoding an entire video when sampled frames suffice.

Do not remove every allocation mechanically. Ownership clarity, bounded lifetime and isolation can be worth a copy. Pooling can retain sensitive data, inflate RSS, create contention and make peak memory worse. Measure the full lifecycle.

Peak memory versus throughput

A 16-request batch may maximize accelerator throughput while exceeding device memory once model weights, KV cache, input tensors, output buffers and runtime workspace coexist. Sweep batch size and sequence/resolution/duration jointly. Record out-of-memory rejection behavior, not only successful points.

Step 8: profile CPU with flamegraphs

A CPU flamegraph aggregates sampled call stacks:

vertical position is stack depth;
box width is the proportion of captured stacks containing that frame;
horizontal order is not chronological;
default colors do not indicate “hot” or “bad.”

Use it to select a target, not to measure the optimization. If another function becomes much slower, the original box may look narrower even when its absolute time did not improve.

For the companion benchmark:

cargo install --locked flamegraph
cargo flamegraph \
  --package mosaic-performance \
  --bench evidence_scoring \
  --output evidence-scoring.svg

The tool uses platform profilers (for example perf on Linux and xctrace on macOS). Current platform/toolchain requirements can differ; follow the tool’s current documentation. To preserve symbols while profiling optimized code:

[profile.bench]
debug = true

Do not silently ship that profile change if binary size/build policy matters. Record:

exact executable and arguments;
release/bench profile and symbols;
sampling backend/frequency;
workload/concurrency/duration;
source/host identity;
profiler overhead and permissions.

CPU sampling will not explain time spent waiting in an async queue, blocked on network/storage, or executing on a GPU. Pair it with async/runtime traces, queue metrics, system I/O counters and accelerator timelines. Off-CPU profiling answers a different question from on-CPU flamegraphs.

A disciplined profile loop

reproduce the end-to-end symptom;
capture baseline metrics and profile;
locate a wide relevant stack under the target workload;
form a causal hypothesis;
change one boundary;
rerun correctness/evals;
rerun the same benchmark repeatedly;
inspect memory/cost/tail trade-offs;
keep or revert using predefined criteria.

Guess-driven rewrites often make code harder while LLVM had already removed the supposed cost.

Step 9: benchmark each AI modality honestly

Text generation

Record time to first token, inter-token latency, output tokens/second, input prefill tokens/second, total latency, prompt/output token distributions, context length, batch/concurrency, KV-cache state, tool/repair calls, quality and cost. Provider “tokens” are tokenizer-specific; record tokenizer and model revision.

Embeddings and retrieval

Record vectors/second, query and candidate dimensions, batch size, index size, top-k, warm/cold cache, recall@k/nDCG against labels, memory and update cost. A brute-force microbenchmark does not represent an ANN index, and ANN speed without recall is meaningless.

Image

Record width/height/channels, decode/colorspace/layout, preprocessing, steps/sampler/guidance, batch, seed, time to preview/final, images/second, peak RAM/VRAM and a reviewed quality/diversity evaluation. A smaller image is a different workload.

Audio

Record sample rate/channels/bit depth/duration, streaming chunk size, real-time factor (processing_time / media_duration), first partial/final latency, word/segment quality, peak memory and boundary behavior. A real-time factor below one is necessary but not sufficient for live UX if chunk latency is poor.

Video

Record codec/container, resolution, frame rate/duration, sampling policy, decode versus model versus encode time, frames/second, real-time factor, peak RAM/VRAM/temp disk and temporal quality. Report dropped/skipped frames explicitly.

Multimodal alignment

Record each encoder/preprocessor revision, candidate count, modality mix, retrieval quality by slice, per-stage time and transfer/copy overhead. Total latency alone cannot identify whether decode, embedding, fusion or ranking dominates.

Provider benchmarks must also record region, network path, rate-limit state, retries, server-reported usage and billed cost. Do not load-test a third-party service without authorization and limits.

Step 10: turn service time into a capacity hypothesis

Little’s Law for a stable long-running system relates average work in the system L, average arrival/completion rate λ, and average time in system W:

L = λ × W

For one service stage, 12 arrivals/second and 120 ms average service time imply:

offered concurrency = 12 × 0.120 = 1.44 worker-equivalents

Running every worker near 100% leaves no room for variation, bursts, maintenance or retries. With a target utilization of 65%:

required workers = ceil(1.44 / 0.65) = 3

Four provisioned workers have projected average utilization:

1.44 / 4 = 0.36 = 36%

Run the exact companion:

cargo run -q -p mosaic-performance --example capacity_plan

Expected semantic result:

{
  "stage": "video-frames",
  "offered_concurrency": 1.44,
  "required_workers": 3,
  "provisioned_workers": 4,
  "projected_utilization": 0.36,
  "target_utilization": 0.65,
  "within_target": true
}

The implementation rejects blank stages, negative/non-finite rates, non-positive service time, zero workers, invalid utilization and worker-count overflow. Capacity configuration is untrusted operational input too.

Why this is only a first estimate

The calculation uses averages. It does not derive a p99 wait. Queue latency rises nonlinearly as utilization approaches saturation, and real AI service times are often heavy-tailed and correlated with input size.

Model at least:

burst/seasonal arrival distribution;
per-class service-time distribution;
priority/fairness and head-of-line blocking;
bounded queue and explicit rejection;
batch formation timeout and maximum batch;
retries/repairs/escalations and retry amplification;
per-tenant quotas;
shared CPU/memory/accelerator constraints;
replica failures, rollout and maintenance headroom;
cold start/model load;
provider limits;
downstream storage/network limits;
autoscaler observation/startup delay.

For a GPU, “worker” may mean a model replica with a memory-feasible batch scheduler, not one OS task. For a provider, it may mean an approved requests/tokens-per-minute envelope. Load-test the complete admission and queue policy at expected, peak and overload traffic.

Service time versus response time

If 120 ms includes queue wait and you multiply it by arrival rate as if it were pure service, the planning model double-counts queueing. Instrument:

accepted_at
queued_at
service_started_at
first_output_at
service_finished_at
persisted_at

Then derive queue, service, first-output and total separately with one clock domain or explicit cross-host time assumptions.

Step 11: performance regression gates

A useful gate answers:

Did a reviewed code/config/dependency change cause a practically important regression under the same identified workload while correctness and quality stayed acceptable?

Do not fail on one noisy comparison. Use:

dedicated or sufficiently controlled runners;
repeated interleaved baseline/candidate runs;
same toolchain/profile/host/workload/power policy;
raw distributions and confidence/variation;
an absolute budget plus a relative regression threshold;
minimum practical effect size;
quality/correctness/eval gates;
manual investigation path for borderline/noisy results.

Example policy:

fail if:
  quality gate fails
  OR p95 exceeds product budget
  OR repeated candidate median regresses > reviewed threshold
     and confidence/noise policy is satisfied

A benchmark library such as Criterion.rs can provide statistical analysis, baselines, throughput units and plots. A custom harness gives complete schema/control with fewer dependencies but must implement sampling/comparison discipline itself. Use the tool whose evidence you can operate; do not equate a sophisticated report with a representative workload.

Pin benchmark dependencies and never execute untrusted pull-request benchmarks with production credentials or privileged profiler access.

Failure investigation: benchmark suddenly becomes much faster

verify the result is consumed and work was not optimized away;
verify release/bench profile, features, dimensions, iterations and dataset identity;
rerun correctness and quality/evaluation gates;
inspect errors/early returns, cache hits and skipped stages;
compare generated assembly/profile only after confirming workload equivalence;
repeat baseline/candidate in alternating order;
preserve raw reports and classify the cause.

A removed validation or empty candidate set is not a win.

Failure investigation: local benchmark is stable, CI is noisy

record runner model, virtualization, co-tenancy, power/frequency and background work;
compare timer resolution and run duration;
increase work per sample before merely increasing sample count;
separate compile/setup from timed work;
use repeated paired runs on a controlled performance runner;
set thresholds above measured noise and meaningful to users;
keep correctness gates mandatory if the performance result is inconclusive.

Do not keep rerunning CI until a random pass.

Failure investigation: p99 collapses under modest load

split admission, queue, service, first output, persistence and total time;
graph arrival, completion, active workers, queue depth/age, utilization and errors together;
slice by input tokens, media size/duration, model, tenant and repair count;
inspect batch wait, head-of-line blocking and long tasks;
inspect retries, rate limits, connection pools, allocator/GC in dependencies and thermal/device behavior;
confirm load generator does not coordinate omission;
reproduce with a bounded trace, repair queue/fairness/admission, and retest overload behavior.

Scaling workers without identifying a shared GPU, memory or provider bottleneck can make it worse.

Failure investigation: RSS grows while heap looks flat

separate live heap, allocator retained pages, memory maps, page cache, thread stacks and device allocations;
capture time series through warmup, steady state and idle;
inspect decoded media/tensor pools and model/KV caches;
verify bounded cache eviction and task/resource cleanup;
compare allocator and OS resident-memory semantics;
soak under a representative mix and prove the plateau/cleanup condition.

Calling every RSS increase a Rust leak skips most of the memory system.

Alternatives and trade-offs

Custom stable benchmark

Small dependency surface and exact artifact schema. Requires you to build robust sampling, comparison and environment capture.

Criterion-style statistical harness

Strong analysis, plots and baseline workflow. Adds dependency/build time and still cannot choose a representative workload for you.

Scalar Rust reference

Portable, auditable validation and oracle. May leave SIMD/BLAS/accelerator throughput unused.

SIMD/native/accelerator kernel

Can improve throughput substantially. Adds target dispatch, native/unsafe boundaries, transfer cost, numeric drift and platform testing.

Buffer reuse/pooling

Reduces allocation churn. Retains memory/content, complicates ownership/cancellation, can contend, and must scrub sensitive buffers when required.

Larger batches

Improve throughput and accelerator occupancy. Increase waiting, memory, fairness complexity and failure blast radius.

Lower utilization target

More tail/burst/failure headroom. Costs more idle capacity. Derive it from SLO/load tests and failure policy rather than copying 70% blindly.

Security and performance implications

Security:

never publish raw customer prompts, embeddings, images, audio, video or secrets in benchmark artifacts;
treat embeddings and traces as potentially sensitive/reidentifiable;
sanitize host paths/environment and logs;
bound malicious dimensions, decompression, duration, sample count and arithmetic before allocation;
reject NaN/infinity so ranking and capacity policy cannot be poisoned;
do not grant untrusted CI privileged performance-counter access;
protect benchmark baselines/provenance against tampering;
do not claim black_box or ordinary compiler output is constant-time;
zero or avoid pooling high-sensitivity buffers according to threat model.

Performance:

benchmark the optimized supported target and actual features;
include validation rather than timing an unsafe fantasy API;
keep aggregation/logging outside the measured hot path;
observe queue wait separately from service time;
pair latency with throughput, utilization, memory, errors, quality and cost;
profile before optimizing and remeasure after one change;
load-test bounded overload/rejection, not only steady successful traffic;
preserve cold-start and steady-state results separately.

Exercises

Add a dot-product kernel sharing validation with cosine similarity; benchmark both at dimensions 128, 768 and 1,536.
Extend BenchmarkReport with source revision, target triple, host identity, lockfile digest and a schema version without leaking environment secrets.
Replace scalar scoring with an optional platform-optimized provider behind a trait. Prove tolerance and ranking equivalence on a corpus before comparing speed.
Add an allocation counter around 10,000 scorer calls and prove the hot path allocates zero on the supported allocator/toolchain.
Profile the benchmark and explain the three widest relevant stacks. Then create one optimization hypothesis and verify it with repeated baseline/candidate runs.
Sweep batch sizes for an embedding or image model. Plot quality, p50/p95, items/second and peak memory as a Pareto frontier.
Extend the capacity example with two traffic classes and bounded priority/fair scheduling.
Build an open-loop load test for the API queue. Demonstrate expected load, saturation and rejection without coordinated omission.
Design a video benchmark separating decode, frame sampling, inference and encode. Include real-time factor and peak temporary storage.
Define a regression policy that distinguishes practical importance from measurement noise and includes a quality gate.

Solution guidance

Keep dot-product accumulation allocation-free and put shared finite/dimension checks in a private helper only if it does not obscure the loop being measured. Capture Git with a build/release manifest, not arbitrary runtime shell interpolation; hash the committed lockfile and allow only reviewed host fields. A provider trait should expose the same score contract; compare absolute error, rank ordering and invalid-input behavior over fixed fixtures.

Allocation instrumentation must wrap only scorer calls after input creation and warmup. A profile explanation names work represented by stacks and corroborating metrics; box color/horizontal order is not evidence. Batch sweeps hold dataset/model/preprocessing/quality policy fixed. Two-class capacity planning needs per-class arrival/service estimates plus an explicit fairness/reservation policy; independent averages alone do not prevent starvation. Open-loop load should schedule arrivals independently of completion and report offered versus accepted/completed load. Video timers surround named stages and report the same input manifest. A regression gate combines an absolute product budget, repeated paired comparison, a noise/effect rule and correctness/eval identity.

Check your understanding

Why can a microbenchmark not establish an API p99?
What work belongs outside the companion’s timed scorer loop?
Why does cargo test timing differ from cargo bench timing?
What does black_box guarantee, and what does it explicitly not guarantee?
Why are 40 batch samples weak evidence for an individual-request p99?
Why can RSS grow while the measured Rust heap stays flat?
What does box width mean in a CPU flamegraph?
Why must ANN speed be paired with recall?
What is the difference between service time and response time?
How does Little’s Law produce 1.44 offered concurrency in the example?
Why is three workers a hypothesis rather than an SLO guarantee?
How can a closed-loop load generator hide stalls?
Why can larger batches improve throughput and worsen user latency?
What identity must accompany a benchmark comparison?