Image Generation, Diffusion, Latents, Guidance, and Provenance
Image generation is not a single model call. It is an artifact-production pipeline whose visible output depends on a text processor, one or more text encoders, a denoiser, a scheduler, a latent decoder, random-number generation, precision, device kernels, output encoding, policy, and often adapters. If any one of those changes, “the same prompt and seed” may no longer describe the same experiment.
This chapter derives diffusion sampling, then implements a small latent image generator in Rust. The companion creates seeded Gaussian noise, runs twelve classifier-free-guided denoising steps, decodes a four-channel latent into RGB, writes a real PNG, validates the artifact, and emits a content-addressed receipt. The implementation is deliberately pedagogical: its analytic denoiser is not trained and its 32×32 output is not evidence of model quality. Its value is that every otherwise-hidden mechanism is short enough to inspect and strict enough to test.
Production systems can replace that denoiser with a U-Net or diffusion transformer and replace the decoder with a trained VAE while retaining the request policy, release identity, scheduler contract, output validation, evaluation, and provenance boundary.
Learning objectives
Section titled “Learning objectives”By the end you will be able to:
- derive the closed-form diffusion forward process from a beta schedule;
- distinguish DDPM training from DDPM, DDIM, and other inference schedulers;
- explain epsilon, clean-sample, and velocity prediction without mixing their equations;
- describe why latent diffusion reduces computation and what the autoencoder discards;
- trace text conditioning through tokenization, encoding, cross-attention, and denoising;
- implement classifier-free guidance and explain why a larger scale is not always better;
- preserve meaningful seed determinism without promising cross-device bit identity;
- bind every model component, adapter, scheduler, precision, and runtime to a release;
- preflight dimensions, pixel count, steps, guidance, batch size, and output bytes;
- validate an encoded image rather than trusting a successful model call;
- design image-to-image, inpainting, ControlNet, and adapter contracts;
- profile memory, latency, throughput, and quality as separate dimensions;
- evaluate composition, spatial relations, counting, text, identity, diversity, and safety;
- treat generated images as derived artifacts rather than observed evidence;
- use Content Credentials without claiming that cryptographic provenance proves truth.
Prerequisites and dependency order
Section titled “Prerequisites and dependency order”Read bounded image processing first so decoded dimensions, orientation, color, OCR, region geometry, and allocation bounds are already familiar. Read Candle fundamentals for tensor shapes and device transfer, model artifacts for immutable bundles, and sampling and reproducibility for the difference between a replay contract and a probability distribution.
This chapter generates pixels. It does not make those pixels factual. The evidence-envelope and provenance chapters remain authoritative when a generated image enters Mosaic.
Completion artifact
Section titled “Completion artifact”Run:
cd rust/rust-ai-engineeringcargo run -q -p mosaic-ml --example latent_image_generationThe example writes a content-addressed file under:
target/generated-images/<png-sha256>.pngFor the pinned example release, seed, and request, the receipt reports:
integration mosaic-image-generation-mechanics-v1+latent4+ddim-eta0+cfg+rgb8-pngprompt SHA-256 only; raw text is not copied into the receiptcanvas 32 × 32 RGB8latent [1, 4, 8, 8]steps 12guidance 3.500seed 42media type image/pngPNG digest 4bc57123e0fe8efccdb6afd743896c2423a4c974a6e4856c64a97cf90a1fe513PNG bytes 554Run the focused checks:
cargo test -p mosaic-ml --all-features image_generationcargo test -p mosaic-ml --all-features --example latent_image_generationThe implementation is in crates/mosaic-ml/src/image_generation.rs; the executable is
crates/mosaic-ml/examples/latent_image_generation.rs.
The pipeline, not the product name
Section titled “The pipeline, not the product name”A text-to-image release commonly contains these stages:
- A tokenizer converts prompt text to bounded token IDs and attention masks.
- One or more text encoders produce contextual conditioning vectors.
- A seeded random-number generator creates the initial latent noise.
- A scheduler chooses timesteps and maps one latent state to the next.
- A denoiser predicts noise, velocity, a clean sample, or another trained target.
- Conditioning enters the denoiser, often through cross-attention.
- A decoder maps the final latent into pixels.
- Postprocessing converts the model range and color layout into an image.
- A policy layer checks the request and result.
- An encoder writes PNG, JPEG, WebP, AVIF, or another declared format.
- A provenance layer binds the artifact to the generation receipt.
Some models work directly in pixels. Some use a diffusion transformer rather than a U-Net. Some condition on pooled embeddings plus token embeddings, or use multiple text encoders. Some predict rectified-flow velocity rather than DDPM noise. “Diffusion pipeline” is therefore a family name, not an interchangeable function signature.
Do not create one untyped bag of options that can be sent to every image model. Expose capabilities such as text-to-image, image-to-image, masked inpainting, structural control, and variation with typed, release-specific parameters.
Diffusion from first principles
Section titled “Diffusion from first principles”The forward process
Section titled “The forward process”Let (x_0) be a clean training image or latent. A forward diffusion process adds a small amount of Gaussian noise at each timestep:
[ q(x_t \mid x_{t-1}) = \mathcal{N}\left(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I\right) ]
The schedule chooses (\beta_t), with (0 < \beta_t < 1). Define:
[ \alpha_t = 1-\beta_t \qquad \bar{\alpha}t = \prod{s=1}^{t}\alpha_s ]
Because Gaussian transitions compose, training does not need to simulate every preceding step. It can sample a timestep and noise (\epsilon \sim \mathcal{N}(0,I)), then construct:
[ x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon ]
At small (t), much of the clean signal remains. At large (t), the state approaches noise. The original DDPM paper connects the learned reverse process to denoising score matching. The implementation lesson is simpler: the beta schedule and its exact indexing determine the signal-to-noise ratio seen by the model. They are part of the model contract, not a cosmetic runtime preference.
What the denoiser predicts
Section titled “What the denoiser predicts”A denoiser may be trained to predict:
- epsilon: the noise (\epsilon) added to the sample;
- x-zero: the clean sample (x_0);
- velocity: a parameterization combining noise and clean signal;
- a flow or model-specific target.
For epsilon prediction, recover a clean estimate with:
[ \hat{x}_0 = \frac{x_t-\sqrt{1-\bar{\alpha}t}\epsilon\theta(x_t,t,c)} {\sqrt{\bar{\alpha}_t}} ]
Here (c) is conditioning, such as encoded text. A scheduler written for epsilon prediction cannot
silently consume velocity prediction. The tensors may have identical shapes while their meanings
differ. Persist prediction_type in the release manifest and assert it before sampling.
Reverse sampling
Section titled “Reverse sampling”A sampler begins from noise and repeatedly evaluates the denoiser at decreasing timesteps. A DDPM sampler normally includes stochastic variance. DDIM constructs a non-Markovian process with the same training objective and can take fewer sampling steps. With DDIM’s stochasticity parameter (\eta=0), a simplified deterministic transition is:
[ x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\hat{x}0 + \sqrt{1-\bar{\alpha}{t-1}}\hat{\epsilon} ]
“Deterministic” here means the transition adds no fresh random sample. It does not guarantee that different GPU kernels, precisions, compiler versions, or attention implementations produce bit-identical PNGs.
Modern scheduler families make different speed, stability, and detail trade-offs. Never identify a run only as “30 steps.” Record:
- scheduler algorithm and implementation version;
- training schedule and inference timestep spacing;
- prediction type;
- sigma or beta parameters;
- step count;
- stochasticity parameters;
- rescaling and clipping behavior.
The Diffusers scheduler documentation shows why a scheduler is an interchangeable software component only when its configuration remains compatible with the model.
Why generate in a latent space?
Section titled “Why generate in a latent space?”Pixel diffusion applies the denoiser to a tensor such as [batch, 3, 1024, 1024]. Repeated spatial
computation at that resolution is expensive. Latent Diffusion
Models first compress images through a trained autoencoder, run
diffusion in its smaller latent representation, then decode the result.
For an eight-times spatial reduction, a 1024×1024 image may become a four-channel 128×128 latent:
pixel values 3 × 1024 × 1024 = 3,145,728latent values 4 × 128 × 128 = 65,536That is 48 times fewer scalar positions before considering layer widths. The savings enable more powerful denoisers and higher resolution, but compression has a cost. Fine text, faces, thin lines, small hands, and exact geometry may be distorted by the autoencoder even when denoising is good.
The latent contract includes:
- channel count and layout;
- spatial scale factor;
- latent scaling and shift constants;
- decoder revision and digest;
- expected value distribution;
- output range and color order;
- tiling or slicing behavior.
Using the wrong VAE can create washed colors, contrast shifts, or severe artifacts without a shape error. The decoder is part of the release, not a postprocessing detail.
Conditioning and classifier-free guidance
Section titled “Conditioning and classifier-free guidance”From prompt to conditioning
Section titled “From prompt to conditioning”A prompt reaches the image only after tokenization and text encoding. Important boundaries include:
- Unicode normalization, maximum tokens, and truncation direction;
- special-token insertion;
- padding and attention-mask behavior;
- one versus multiple text encoders;
- token-level versus pooled embeddings;
- prompt weighting syntax, if any;
- clip-skip or hidden-layer selection;
- learned textual-inversion embeddings.
Prompt display text is not sufficient for replay. Preserve the tokenizer and encoder identities, the normalized/tokenized prompt digest, truncation outcome, and any prompt-template release.
Cross-attention commonly lets spatial image features query encoded text tokens. It does not guarantee compositional correctness. The model can associate “red cube beside blue sphere” with the right objects while swapping colors or spatial relations. That is an evaluation problem, not something a longer adjective list reliably fixes.
Classifier-free guidance
Section titled “Classifier-free guidance”Classifier-free guidance evaluates a conditional prediction and an unconditional or negative prediction:
[ \epsilon_\text{guided} = \epsilon_\text{uncond} + w(\epsilon_\text{cond}-\epsilon_\text{uncond}) ]
The scale (w) amplifies the direction associated with the prompt. A value of zero ignores the positive condition. Larger values can improve prompt adherence up to a point, but may reduce diversity, oversaturate colors, crush contrast, or amplify anatomy and texture artifacts. “CFG 7.5” is not universally optimal, and newer distilled or guidance-free models may use a different range or no CFG at all. The foundational classifier-free guidance paper frames this as a fidelity/diversity trade-off.
A negative prompt is conditioning, not a denylist. Writing “no watermark” does not guarantee the absence of a watermark. Product policy must inspect outputs and block prohibited requests before generation; it cannot delegate enforcement to negative text.
Walk through the Rust companion
Section titled “Walk through the Rust companion”1. Validate before allocating
Section titled “1. Validate before allocating”ImageGenerationRequest::validate runs before latent or pixel allocation. It rejects:
- empty, NUL-containing, or oversized prompts;
- dimensions below the latent scale;
- dimensions not divisible by four;
- width, height, or total pixels above policy;
- fewer than two or too many denoising steps;
- guidance above the configured limit;
- an output whose conservative raw-size estimate exceeds the PNG budget.
Use checked multiplication for width × height × channels. Checking width and height separately
does not prevent their product from overflowing or exceeding memory. Production policies should
also bound batch size, control images, masks, adapter count, and aggregate decoded input bytes.
2. Bind the release
Section titled “2. Bind the release”ImageModelRelease binds:
pub struct ImageModelRelease { pub model_id: String, pub upstream_revision: String, pub text_encoder_sha256: String, pub denoiser_sha256: String, pub decoder_sha256: String, pub scheduler_release: String,}The fixture uses synthetic digests because there are no trained weights. A production manifest
also needs tokenizer files, configuration files, every text encoder, VAE, safety model, adapters,
precision, quantization scheme, runtime, kernel set, and prompt-template release. Store exact
digests, not mutable repository names such as main or latest.
3. Construct seeded Gaussian noise
Section titled “3. Construct seeded Gaussian noise”The companion uses SplitMix64 for uniform bits and Box-Muller for normal samples. It produces a
latent of shape [1, 4, height/4, width/4]. This is an educational generator, not a
cryptographically secure RNG and not a promise to match PyTorch, NumPy, CUDA, or another Rust RNG.
A production replay receipt should name:
- RNG algorithm and implementation release;
- seed and per-image seed derivation;
- generator device;
- batch ordering;
- whether adapters or schedulers draw additional randomness.
Do not say “seeded, therefore reproducible” without defining the environment. The Diffusers reproducibility guide explicitly distinguishes reproducibility within constrained environments from bit-perfect portability across platforms.
4. Schedule denoising
Section titled “4. Schedule denoising”LinearBetaSchedule creates twelve beta values from 0.001 to 0.020 and accumulates alpha products.
The sampler iterates them in reverse. At every step it records:
pub struct DenoisingStepReceipt { pub step: u16, pub training_timestep: u16, pub alpha_cumulative: f32, pub latent_l2: f32,}This trace catches non-finite states and provides a compact trajectory diagnostic. Do not log full latents by default: they are large, may leak input information in image-conditioned workflows, and turn observability into an artifact store.
The toy predicted_noise function is a bounded analytic stand-in. It deliberately does not recover
a known clean target perfectly; if it did, it would erase the initial noise and different seeds
would converge to the same pixels. A real denoiser is a trained network whose imperfect conditional
estimate naturally leaves the initial latent influential.
5. Decode and validate a real artifact
Section titled “5. Decode and validate a real artifact”The toy decoder maps four latent channels into RGB8. The image crate encodes those bytes as PNG.
The boundary then:
- confirms the RGB buffer has exactly
width × height × 3bytes; - computes per-channel minimum, maximum, and mean;
- limits encoded bytes;
- hashes the exact PNG bytes;
- declares
image/png; - includes every denoising step in the receipt.
In production, decode the encoded output again in an independent validation path. Check media signature, dimensions, frame count, color model, alpha, ICC profile policy, metadata size, and decoder limits. A successful encoder call does not prove that a downstream decoder accepts the file or that the declared metadata matches it.
Reproducibility has levels
Section titled “Reproducibility has levels”Use precise claims:
- Request reproducibility: the normalized request and release identity are preserved.
- Statistical reproducibility: reruns have comparable quality distributions.
- Perceptual reproducibility: reruns are visually very similar.
- Numerical reproducibility: tensors stay within a declared tolerance.
- Bit reproducibility: encoded bytes and digest are identical.
The companion achieves bit reproducibility in its supported Rust environment and tests the exact PNG digest. A production GPU pipeline may only promise request identity plus a tolerance or perceptual envelope. Sources of drift include fused attention, reduced precision, nondeterministic kernels, tensor-core modes, library upgrades, device architecture, parallel reduction order, VAE tiling, and encoder metadata.
If exact artifact replay is a requirement, retain the generated artifact in content-addressed storage. Regeneration is not storage.
Extending the generation contract
Section titled “Extending the generation contract”Image-to-image
Section titled “Image-to-image”Image-to-image encodes a source image into a latent, adds noise at a chosen strength, then denoises under new conditioning. Its manifest must preserve the source artifact digest, orientation normalization, resize/crop transform, encoder identity, strength semantics, starting timestep, and output geometry.
“Strength 0.6” is meaningless without a scheduler-specific mapping to timesteps. Hash the normalized input actually sent to the encoder as well as the original asset.
Inpainting
Section titled “Inpainting”Inpainting adds a mask. Declare:
- whether white or black means replace;
- mask channel and threshold semantics;
- feathering and blur;
- source-to-model resize/crop transform;
- whether unmasked pixels are copied, re-noised, or regenerated;
- padding around the selected region.
Validate source and mask geometry together before allocation. Preserve the mask as a derived artifact and map output regions back to the original source.
Structural controls
Section titled “Structural controls”ControlNet-like modules and adapters condition on edges, depth, pose, segmentation, or reference images. Each control has its own preprocessing model, digest, coordinate transform, scale, and active timestep interval. “Used pose control” is not enough provenance.
LoRA and other adapters
Section titled “LoRA and other adapters”Adapters modify denoiser or text-encoder behavior. Record adapter digest, target modules, scale,
merge state, load order, and base-model compatibility. Floating-point addition is not associative;
changing merge order can change output. A user-supplied adapter is also executable model data from
a supply-chain perspective: restrict formats, parse under bounds, and prefer safetensors over
pickle-capable formats.
Memory, batching, and throughput
Section titled “Memory, batching, and throughput”Image generation stresses memory differently from autoregressive text generation. There is no growing token KV cache in the basic U-Net pipeline, but large spatial activations recur for every denoising step. Account for:
resident weights+ text-encoder activations+ latent batch+ denoiser activations and attention workspace+ conditional/unconditional batch expansion+ VAE decode activations+ control/adaptor activations+ runtime allocator cache+ encoded and decoded host buffers+ safety/OCR model residency+ safety marginCFG may run unconditional and conditional branches sequentially to reduce peak memory or concatenate them to improve throughput. The latter can roughly double relevant batch activations. Measure it.
Optimization techniques include:
- reduced-precision weights and activations;
- attention implementations with lower memory growth;
- attention slicing;
- VAE slicing across batch;
- VAE tiling for large images;
- model or sequential CPU offload;
- component-level device placement;
- compiled or fused graphs;
- latent preview decoding at reduced cadence;
- shape-bucketed batching;
- warm sessions and cached text embeddings.
Each can alter latency or numerics. The Diffusers memory guide documents practical offloading and sharding choices, while Candle’s Stable Diffusion examples show that Rust can host SD 1.5, 2.1, SDXL, and Turbo-family mechanics. Support in an example is not a production quality or compatibility guarantee; pin the exact code and bundle you validate.
Measure these separately:
| Metric | Question |
|---|---|
| queue latency | Did capacity delay admission? |
| cold load | How long did weights and kernels take to initialize? |
| text encoding | Is repeated prompt work cacheable? |
| denoiser step p50/p95 | Which shapes or controls slow the core loop? |
| steps per second | Is compute throughput stable? |
| VAE decode | Does high-resolution output dominate the tail? |
| policy checks | Are safety classifiers or OCR the bottleneck? |
| encode/store | Is PNG compression or object storage delaying completion? |
| peak accelerator bytes | Which configuration actually fits? |
| end-to-end success | How many admitted requests publish valid artifacts? |
Do not optimize only images per second. A fast release with worse instruction following, unsafe outputs, or frequent malformed artifacts is a regression.
Safety is a lifecycle
Section titled “Safety is a lifecycle”Image generation can create sexual content, child sexual abuse material, targeted harassment, non-consensual intimate imagery, deceptive impersonation, graphic violence, self-harm material, extremist propaganda, privacy violations, and fraud assets. Exact requirements depend on product, jurisdiction, and deployment context; treat legal review as a release input, not an afterthought.
A defensible boundary can include:
- authenticate and authorize the capability;
- classify request intent before spending generation capacity;
- deny prohibited identity, age, or abuse requests;
- minimize and protect prompts and reference images;
- generate in an isolated, bounded worker;
- decode through hardened limits;
- run output safety classifiers appropriate to the product;
- run OCR/logo/face checks when relevant;
- route uncertain cases to review;
- attach provenance and policy outcome;
- publish only after all required gates pass;
- support abuse reporting, takedown, and release rollback.
Input and output classifiers have false positives and false negatives. Evaluate them by category and demographic slice, calibrate thresholds, and preserve appeal/review paths. Do not store raw sensitive prompts in ordinary logs. A prompt digest supports correlation without turning telemetry into a privacy breach, although low-entropy prompts may still be guessable.
Licensing is also part of safety and release engineering. Verify base-model, VAE, text-encoder, adapter, and dataset terms for the intended commercial, hosted, or derivative use. A technically loadable checkpoint is not necessarily legally deployable.
Provenance: what a signature can and cannot say
Section titled “Provenance: what a signature can and cannot say”The receipt hashes the exact PNG and generation identity. That provides content addressing and a local lineage edge. It is not yet a signed Content Credential.
C2PA 2.2 defines a cryptographically bound manifest containing assertions about an asset’s origin and modifications. A production publication flow can:
- hash the final encoded asset;
- create assertions describing AI generation and ingredients;
- identify the generation application or organizational signer;
- include model/release references consistent with privacy policy;
- sign the manifest with protected credentials;
- embed it or store it through a supported durable binding;
- verify the binding before delivery.
The critical limit is semantic: a valid signature establishes that assertions and asset binding have not been altered under the trust model. It does not prove that the assertions are complete, that the signer is honest, or that the depicted event happened. Generated imagery must never be promoted to observed evidence merely because its provenance is intact.
For Mosaic, represent the PNG as a derived artifact:
source prompt/reference artifact(s) ↓ declared generation transformmodel bundle + scheduler + policy release ↓generated PNG digest ↓ optional validation/OCR/safety derivativespublication artifact + signed provenanceIf a generated diagram illustrates a research report, cite the report for factual claims and label the image as an illustration. Never cite the generated pixels as independent corroboration.
Evaluation beyond “looks good”
Section titled “Evaluation beyond “looks good””Create a versioned prompt suite before tuning. Include:
- single-object fidelity;
- multiple entities with distinct attributes;
- left/right/above/below and containment relations;
- counting;
- rare combinations;
- long-prompt truncation;
- legible text and typography;
- faces, hands, and fine structure;
- culturally diverse people and settings;
- reference-image identity preservation;
- inpainting boundary consistency;
- negative-prompt stress cases;
- safety-policy positives, negatives, and ambiguous cases;
- repeated seeds for diversity and collapse checks.
Store expected rubrics, not one “golden image.” Valid generation is one-to-many. Automated similarity scores can help detect large regressions but are not complete quality measures:
- text-image embedding similarity can reward keyword presence while missing composition;
- distribution metrics such as FID depend on feature extractor and sample set and do not score one prompt reliably;
- OCR can measure text accuracy but not design quality;
- face or identity similarity can be useful and simultaneously create privacy risk;
- aesthetic predictors encode their training preferences.
Use blinded paired human comparison for subjective quality. Randomize left/right ordering, measure inter-rater agreement, retain category-level results, and separate prompt adherence, visual quality, factual/structural correctness, safety, and preference. Report confidence intervals and sample counts; do not turn a tiny win rate into a universal claim.
A release gate should combine:
- artifact validity rate;
- policy false-negative and false-positive ceilings;
- prompt-suite slice floors;
- paired human non-inferiority or improvement;
- latency and peak-memory budgets;
- crash/OOM/timeout rates;
- deterministic replay checks where promised;
- model and dependency supply-chain approval.
Failure investigation
Section titled “Failure investigation”Every seed produces the same picture
Section titled “Every seed produces the same picture”Check whether the denoiser accidentally receives the clean target, whether initial noise is overwritten, whether seed derivation collapses, or whether cache keys omit the seed. The companion has a regression test asserting different seeds produce different PNG digests.
The same seed drifts after deployment
Section titled “The same seed drifts after deployment”Compare the complete release: scheduler, timestep spacing, model components, precision, runtime, kernel selection, device, batching, VAE tiling, and encoder. If exact bytes were never promised, evaluate drift against the declared numerical/perceptual contract rather than an imaginary one.
Images are oversaturated or brittle
Section titled “Images are oversaturated or brittle”Sweep guidance by category. Confirm that the model expects CFG and that guidance rescaling or distillation-specific settings are correct. Inspect negative-prompt templating. More guidance is not a monotonic quality improvement.
Output has the right shape but wrong colors
Section titled “Output has the right shape but wrong colors”Check decoder revision, latent scale/shift, RGB versus BGR order, model-range conversion, color profile, alpha handling, and quantization. Shape validation cannot catch semantic tensor mismatch.
OOM appears only for some prompts
Section titled “OOM appears only for some prompts”Group by dimensions, batch, CFG mode, text length, controls, adapters, VAE tiling, and concurrent residency. Measure peak allocated and reserved memory at each stage. Admission must reserve the worst supported path, not the mean path.
Inpainting changes protected pixels
Section titled “Inpainting changes protected pixels”Inspect mask polarity, thresholding, blur, resizing, padding, and compositing semantics. Compare in the normalized coordinate space and preserve the exact transform chain.
The safety gate passes an obvious violation
Section titled “The safety gate passes an obvious violation”Quarantine the artifact, preserve access-controlled evidence, classify the failure category, check the policy/model release, add a minimal redacted regression case, assess similar published artifacts, and follow incident/takedown procedures. Do not simply add the exact prompt to a string denylist.
Alternatives and trade-offs
Section titled “Alternatives and trade-offs”| Approach | Strength | Cost or risk |
|---|---|---|
| hosted image API | fastest product integration, managed accelerators | data boundary, provider drift, less runtime control |
| local full diffusion pipeline | privacy and release control | large weights, accelerator operations, safety ownership |
| distilled/few-step model | lower latency | different guidance/scheduler contract, possible quality loss |
| pixel diffusion | avoids lossy VAE boundary | substantially higher spatial compute |
| latent diffusion | efficient high-resolution synthesis | decoder artifacts and latent contract complexity |
| autoregressive image tokens | familiar sequence abstraction | long sequences and different sampling behavior |
| retrieval + composition | exact approved assets and brand control | limited novelty, asset licensing/indexing |
| procedural/vector rendering | exact geometry and text | narrower visual domain |
Use generation only when synthesis is the actual requirement. A chart should usually be rendered from data. A logo should come from an approved asset library. A factual photograph should come from a source with acquisition provenance. Model novelty is not a substitute for the correct medium.
Exercises
Section titled “Exercises”Recall
Section titled “Recall”- Why can (x_t) be sampled directly from (x_0) during training?
- What changes when a model predicts velocity instead of epsilon?
- Why is the scheduler part of the release identity?
- What does CFG scale zero do in the companion equation?
- Why does a valid C2PA signature not prove that an image is truthful?
Implementation
Section titled “Implementation”- Add
batch_sizeto the request and preflight every multiplication before allocation. Derive per-image seeds from a named algorithm and test that reordering does not silently change them. - Add a PPM encoder beside PNG. Bind encoder kind and release into generation identity, then prove that equal pixels but different encodings have different artifact digests.
- Add a typed inpainting request with a one-channel mask. Reject mismatched dimensions and test both mask polarities.
- Add an independent post-encode decode check and verify signature, dimensions, RGB color type, and decoded-byte ceiling.
- Add a
prediction_typeenum and implement separate epsilon and clean-sample branches. Write a test proving a release/request mismatch fails before denoising.
Design
Section titled “Design”- Design a release manifest for an SDXL-class pipeline with two text encoders, tokenizer files, U-Net, VAE, scheduler, LoRA adapters, precision, runtime, safety classifiers, and C2PA signer.
- Design an evaluation that can detect color/attribute swapping while not treating one reference image as the only correct output.
- Capacity-plan a service supporting 1024×1024 text-to-image and inpainting. Separate admission memory, warm residency, queue policy, and publication latency.
- Write an incident runbook for an output-policy false negative discovered after publication.
Solution guidance
Section titled “Solution guidance”- Gaussian transitions compose; (\bar{\alpha}_t) summarizes the preceding schedule, so one noise sample constructs (x_t).
- Its tensor shape can stay the same, but conversion to (\hat{x}_0) and scheduler updates must use the velocity parameterization.
- It selects timesteps and update equations; changing it changes the trajectory and often output.
- The conditional-minus-unconditional term disappears, so positive prompt changes cannot steer the result.
- The signature protects binding and assertion integrity under a trust model, not completeness, signer honesty, or correspondence with the physical world.
- Use checked
u64/usizearithmetic and derive seeds from(root_seed, stable_image_id), not mutable batch position. - Hash encoded bytes for artifact identity and hash normalized RGB separately if pixel-equivalence queries are needed.
- Put polarity in an enum, normalize once, preserve the normalized mask digest, and test edge pixels.
- Use a decoder path that does not trust in-memory encoder metadata. Enforce limits before and during decode.
- Make prediction type a release field; exhaustive
matchselects equations and rejects an unsupported type. - Every file and behavioral setting needs an immutable digest or version. Include load order and adapter scale; do not collapse the bundle to one marketing model name.
- Parse or annotate requested entities, attributes, and relations, combine automated checks with blinded human rubrics, and evaluate multiple seeds.
- Reserve for the maximum admitted combination including CFG/control/VAE workspace; report queue, sampling, validation, and publication stages independently.
- Quarantine, assess blast radius, preserve restricted evidence, remediate policy/model, add a regression slice, republish or remove, notify as required, and document rollback.
Check your understanding
Section titled “Check your understanding”- Can you derive clean-sample recovery from the forward-process equation?
- Can you name every component required to replay your deployed image pipeline?
- Can your request be rejected before allocating a latent or decoding a reference image?
- Do you know which exact RNG owns the seed?
- Can you explain why two runs with equal prompts and seeds may differ?
- Can you detect CPU/GPU OOM risk before admission?
- Does the artifact survive an independent decoder?
- Are prompt adherence, aesthetics, safety, and diversity measured separately?
- Is generated media labeled as derived rather than acquired evidence?
- Can a verifier distinguish intact provenance from factual truth?
Practical completion checklist
Section titled “Practical completion checklist”- Model, tokenizer, text encoder, denoiser, decoder, scheduler, adapters, and runtime are pinned.
- Prediction type, latent geometry, scale/shift, precision, and device are explicit.
- Prompt normalization, token limits, truncation, and template release are recorded.
- Dimensions, pixels, steps, guidance, batch, controls, masks, and output bytes are bounded.
- Allocation arithmetic is checked.
- Seed algorithm and derivation are named.
- Scheduler timesteps and configuration are part of the receipt.
- Non-finite latents terminate the run.
- Encoded output is independently decoded and validated.
- Exact encoded bytes are content-addressed.
- Raw sensitive prompts and references do not leak into ordinary logs.
- Input and output policy gates have category-level evaluation.
- Model and adapter licenses are approved for the deployment.
- Prompt-suite, human, safety, latency, memory, and failure gates block releases.
- Generated images are derived artifacts with complete lineage.
- Content Credentials are presented as provenance, never proof of truth.
- Rollback, quarantine, takedown, and abuse-reporting procedures are exercised.
Authoritative references
Section titled “Authoritative references”- Ho, Jain, and Abbeel, Denoising Diffusion Probabilistic Models.
- Song, Meng, and Ermon, Denoising Diffusion Implicit Models.
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models.
- Ho and Salimans, Classifier-Free Diffusion Guidance.
- Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.
- Hugging Face Diffusers, Schedulers, Reproducibility, and memory optimization.
- Hugging Face Candle, Stable Diffusion Rust examples.
- Coalition for Content Provenance and Authenticity, C2PA Specification 2.2.