Serde, Derive Macros, and Generated Contracts

Rust macros generate Rust code. Serde’s derive macros generate serialization implementations from types and attributes. This is powerful at AI boundaries, where tasks, traces, tools, eval cases, and provider messages need repeatable structured formats.

Generated code does not generate trustworthy semantics. Deserialization proves the input can populate a Rust shape; validation, version policy, authorization, and evidence checks remain explicit.

Learning objectives

After this chapter, you will be able to:

• distinguish declarative, derive/procedural, and attribute macros;
• derive Serialize/Deserialize with intentional wire representation;
• use tagged enums, defaults, renames, flattening, and unknown-field policy carefully;
• validate deserialized newtypes whose constructors were bypassed;
• separate wire DTOs from domain types when evolution/trust requires it;
• design versioned task, trace, and tool contracts;
• use macro_rules! for small syntax repetition without hiding control flow;
• inspect macro expansion and test exact fixtures;
• run the strict companion ProgramSpec.

Dependencies and downstream use

This chapter depends on enums, traits, errors, modules, and semantically valid states. It prepares for:

• task and eval fixture files;
• append-only trace events;
• provider request/response adapters;
• JSON tool schemas and structured outputs;
• database/HTTP DTOs;
• generated API/schema artifacts.

untrusted JSON
    │ Serde: syntax + Rust shape
    ▼
decoded boundary type
    │ explicit domain validation/version checks
    ▼
validated domain value
    │ policy/evidence/authorization
    ▼
allowed program action

Serde occupies one stage, not the whole trust boundary.

What a macro is

A macro receives Rust syntax/tokens and expands into Rust code before or during compilation.

Three important categories:

Declarative macro_rules!

Pattern-based token transformation:

macro_rules! required_fields {
    ($($field:literal),+ $(,)?) => {
        vec![$($field.to_owned()),+]
    };
}

This removes local repetition. It should not conceal significant policy or I/O.

Derive procedural macros

Generate trait implementations:

#[derive(Debug, Clone, Serialize, Deserialize)]
struct ProgramSpec { /* ... */ }

Serde derive inspects fields and attributes and emits implementations.

Attribute/function-like procedural macros

Attributes such as #[tokio::main] transform an item. Function-like proc macros look like some_macro!(...). They can perform arbitrary compile-time token processing but live in dedicated proc-macro crates.

Macros are hygienic in defined ways, but errors and generated behavior can still be complex. Prefer functions/generics when runtime values and ordinary types suffice.

Serde is a pair of traits

Serialize maps a Rust value into a serializer’s data model. Deserialize<'de> constructs a Rust value from a deserializer:

#[derive(serde::Serialize, serde::Deserialize)]
struct Usage {
    input_units: u64,
    output_units: u64,
}

The same derived type can work with JSON and other Serde formats when their data models support the representation. Format differences still matter: map keys, integer ranges, binary data, and non-finite floats may behave differently.

Do not claim “Serde-compatible” means one stable wire format. Name the format, version, canonical rules, and compatibility policy.

Tagged enums preserve alternatives

The companion defines:

#[derive(Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
enum Input {
    Text { text: String },
    Image { artifact_id: String },
}

Example JSON:

{"type":"image","artifact_id":"sha256:abc123"}

An internally tagged enum gives one explicit discriminator and variant fields. Other representations include externally tagged, adjacently tagged, and untagged.

Avoid #[serde(untagged)] for security-sensitive ambiguous inputs unless variants are structurally disjoint and tests prove precedence. Serde tries variants in order; adding a permissive variant can reinterpret old inputs.

For multimodal evidence, a tag makes unsupported modality a precise error rather than a guessed shape.

Unknown fields are a compatibility decision

The companion uses:

#[serde(deny_unknown_fields)]
struct ProgramSpec { /* ... */ }

This rejects misspellings and unexpected contract fields. It is valuable for:

• executable task/program specifications;
• security/policy configuration;
• tool arguments;
• eval fixtures where silent drift is dangerous.

Forward-compatible event consumers may instead ignore unknown fields while requiring known meaning. Choose per boundary.

Risks of ignoring:

• typo max_model_call silently falls back to default;
• an authorization field intended by a caller is discarded;
• mixed-version producers believe a feature was honored.

Risks of denying:

• old consumers fail on additive producer fields;
• rolling upgrades require careful sequencing.

Version the contract and test producer/consumer combinations. Do not use one global rule.

Defaults can hide missing intent

#[serde(default = "default_risk")]
risk: RiskClass

A default is safe only when omission has one stable meaning. Good candidates:

• optional display metadata;
• an explicitly versioned conservative policy;
• fields added with backward-compatible semantics.

Dangerous defaults:

• permissions;
• external-write enablement;
• tenant identity;
• provider/model selection;
• budgets where zero/unbounded ambiguity matters.

Record whether a value was supplied if audit semantics require it. A separate raw DTO can use Option<T> and validation can apply a documented default with provenance.

Deserialization can bypass constructors

The domain macro generates TaskId::new, but a derived transparent Deserialize can populate its private inner field through generated code without calling that constructor.

Therefore the companion ProgramSpec::from_json revalidates:

let spec = serde_json::from_str::<ProgramSpec>(raw)?;
TaskId::new(spec.task_id.as_str())?;

Stronger alternatives:

1. implement custom Deserialize for the newtype and call new;
2. deserialize a raw DTO with String, then TryFrom<Raw> into domain types;
3. use a reviewed helper that integrates validated construction.

For public domain types, custom deserialization or raw-to-domain conversion is usually safer than requiring every caller to remember validate.

The current companion deliberately exposes this lesson; later hardening will move validation into the domain decoding boundary.

Raw DTO to domain conversion

#[derive(Deserialize)]
struct RawTask {
    id: String,
    objective: String,
    budget: RawBudget,
}

impl TryFrom<RawTask> for Task {
    type Error = DomainError;

    fn try_from(raw: RawTask) -> Result<Self, Self::Error> {
        let task = Task {
            id: TaskId::new(raw.id)?,
            objective: raw.objective,
            budget: raw.budget.try_into()?,
            // ...
        };
        task.validate()?;
        Ok(task)
    }
}

Advantages:

• wire optionality/defaults remain separate;
• invalid domain values never escape conversion;
• API versions can map into one domain model;
• provider/database fields do not become public domain fields;
• error paths can attach JSON field locations.

The extra type is valuable at an untrusted or evolving boundary.

Version contracts explicitly

The companion requires version == 1. A production program format might use:

{
  "schema_version": 1,
  "program_id": "incident-summary",
  "program_revision": "sha256:...",
  "task": {}
}

Version different identities separately:

• schema version: shape/meaning of serialized data;
• program revision: prompts/tools/validators/policy;
• model/adapter revision;
• eval suite version;
• artifact transformation version.

Do not use one application version as a substitute for all component identity.

Evolution options:

• additive compatible fields with safe defaults;
• explicit tagged enum of versions;
• migration from old DTO to current domain;
• reject unsupported future/ancient versions;
• preserve raw bytes for audit.

Append-only traces especially need readers that handle known old versions and fail clearly on unknown semantics.

flatten can blur boundaries

Serde flatten merges fields from nested structs/maps:

#[serde(flatten)]
extra: BTreeMap<String, Value>

It can support extension fields, but:

• typoed known fields may land in extra;
• canonical ordering/signing becomes more complex;
• deny_unknown_fields interactions are constrained;
• ownership of field names becomes unclear.

Use an explicit "extensions" object with namespaced keys for policy/tool contracts. Flattening is more appropriate when the wire format is already fixed and tested.

Serialization is not canonicalization

serde_json::to_string emits valid JSON. It does not automatically provide a canonical signed/hash representation across map order, float spelling, library versions, or future field changes.

For content addressing or signatures:

1. define canonical data model and field ordering;
2. forbid/normalize unsupported numeric values;
3. specify Unicode/escaping policy;
4. include schema version;
5. test exact bytes with golden fixtures;
6. use a canonical format/library with a reviewed specification.

Alternatively hash the original immutable artifact bytes and store parsed representations as derived data.

Secrets and derives

Deriving Debug or Serialize on a broad configuration struct can expose tokens:

#[derive(Debug, Serialize)] // dangerous if api_key is included
struct ProviderConfig {
    api_key: SecretString,
}

Use a secret type with redacted Debug and avoid serializing it. #[serde(skip)] can help, but a separate runtime secret object is often safer.

Review generated trait implementations as part of the API. Clone on an approval token, Default on a security policy, or Deserialize on an invariant-bearing type may be unsafe even though the derive is convenient.

Declarative macros in the companion

The real mosaic-domain crate uses one macro to generate validated ID newtypes:

macro_rules! identifier {
    ($name:ident, $label:literal) => {
        // struct, constructor, access, Display
    };
}

This is a good use when behavior must be identical for several nominal types. Tests must cover the expanded contract.

The example’s required_fields! macro is intentionally small. A function taking an iterator might also be suitable; the macro accepts literal syntax and builds owned strings.

Avoid a macro that generates the entire harness state machine. Hidden control flow, error mapping, and effects become harder to review and instrument.

Inspect generated code

Tools:

• cargo expand (external Cargo subcommand) shows macro-expanded Rust;
• rustdoc shows the public API after expansion;
• compiler diagnostics identify expansion sites;
• tests exercise generated implementations.

Pin proc-macro dependencies. Procedural macros execute during compilation and are part of the supply chain.

Do not paste enormous expansion output into the textbook. Inspect the specific derived impl or macro when debugging.

Runnable companion implementation

Run:

cd rust/rust-ai-engineering
cargo run -q -p mosaic-domain --example serde_and_macros
cargo test --workspace --examples

The program prints a strict JSON specification similar to:

{"version":1,"task_id":"caption-image","input":{"type":"image","artifact_id":"sha256:abc123"},"required_output_fields":["caption","evidence"],"risk":"medium"}

The code is crates/mosaic-domain/examples/serde_and_macros.rs.

It:

• uses a tagged multimodal enum;
• denies unknown top-level fields;
• supplies an explicit conservative risk default;
• checks schema version;
• revalidates a deserialized TaskId;
• rejects empty output contracts;
• round-trips the accepted specification;
• tests unknown fields and constructor-bypassing identifier input.

Failure investigation: misspelled budget silently accepted

Symptom: a task sets max_model_call, but the harness uses default max_model_calls.

Root cause: unknown fields were ignored while defaults filled the real field.

Investigation:

1. preserve exact input fixture;
2. enable/inspect unknown-field behavior;
3. compare decoded value with supplied fields;
4. trace which defaults were applied;
5. add typo fixture;
6. decide version-compatible strictness.

For executable specifications, reject the unknown field with a location. The caller should not believe an ignored safety budget was honored.

Failure investigation: empty ID enters domain

Symptom: code assumes TaskId is non-empty, but deserialized fixture contains whitespace.

Cause: derived deserialization bypassed validating constructor and the caller forgot validate.

Fix options: custom Deserialize calling TaskId::new, or deserialize raw DTO then TryFrom. Add property tests and all ingestion-path tests (JSON, DB, message queue), not only constructor unit tests.

Common failure modes

Derive everything

Every derived trait expands the public/semantic surface. Review Clone, Default, Debug, Serialize, and Deserialize.

Serde validation conflated with domain validation

Shape is not business truth, evidence, or authority.

untagged ambiguous enums

Prefer explicit discriminators for executable contracts.

Silent defaults in safety configuration

Use raw Option plus explicit policy when omission matters.

JSON hashing without canonical rules

Define canonicalization or hash original bytes.

Macros hiding policy

Keep state transitions and effects in ordinary typed functions.

Security and performance implications

Security:

• cap input bytes and nesting before/while decoding;
• deny unknown fields where ignored policy is dangerous;
• validate after deserialization;
• never serialize/debug secrets by convenience;
• pin/audit procedural macro crates and build scripts;
• reject duplicate/ambiguous fields according to parser behavior;
• avoid unbounded arrays/maps in task/tool input.

Performance:

• derive implementations are usually efficient but still allocate owned strings/collections;
• borrowed deserialization can reduce copies within a request scope;
• recursion/nesting can exhaust stack or CPU;
• serde_json::Value is flexible but loses typed validation and can allocate heavily;
• streaming deserializers help large sequences, with explicit item/byte limits;
• schema validation plus domain validation adds work but prevents expensive downstream inference on invalid input.

Measure representative payloads; do not replace typed structs with ad hoc parsing for speculative speed.

Exercises

Recall

1. What does deriving Deserialize prove?
2. Why can derived deserialization bypass a constructor?
3. What is the risk of untagged?
4. Is JSON serialization automatically canonical?

Implementation

1. Implement custom Deserialize for a validated ID newtype.
2. Replace ProgramSpec::from_json with RawProgramSpec -> TryFrom.
3. Add a schema_version enum supporting migration from v1 to v2.
4. Add size/cardinality validation for every string/vector.
5. Write exact golden JSON and compatibility tests.

Design

Define versioned formats for:

• task/program specification;
• trace event;
• tool descriptor and invocation;
• eval case/result;
• model response;
• artifact provenance.

For each, choose unknown-field/default/tagging policy and migration behavior. Identify secrets and unbounded fields.

Failure lab

Test:

• unknown field typo;
• missing required field;
• ambiguous enum shape;
• empty deserialized newtype;
• unsupported future version;
• excessive array/nesting;
• secret redaction;
• canonical artifact byte stability.

Solution guidance

Custom newtype deserialization should deserialize a string, call the constructor, and map its error through serde::de::Error::custom. Ensure every format path uses it.

Raw-to-domain conversion is clearer when multiple fields interact. Keep the raw type private to the boundary module.

For migrations, decode the explicit version first, parse the corresponding DTO, then convert into the current domain. Preserve original bytes and migration identity for audit.

Golden tests should be supplemented by semantic round trips and backward-compatibility fixtures; exact bytes alone can make harmless display changes look catastrophic unless byte stability is the contract.

Check your understanding

• Can private newtype fields be populated by a derived impl?
• Why might deny_unknown_fields improve safety but complicate rolling upgrades?
• What should a defaulted authorization field do?
• What identity must accompany a migrated trace?
• When is flatten dangerous?
• Why should a provider DTO not derive directly into a domain response?
• What generated derives could expose a secret?
• When is a function clearer than macro_rules!?

Completion checklist

• I treat decoding and validation as separate stages.
• I use explicit tags for executable sum types.
• I choose unknown/default policy per versioned boundary.
• I cannot bypass newtype invariants through deserialization.
• I separate raw DTOs from domain values when trust/evolution requires it.
• I define canonical bytes before hashing/signing serialized data.
• I review macro dependencies and generated trait surfaces.
• I ran the strict spec example and both negative tests.

Authoritative references

• Serde derive
• Serde attributes
• Serde enum representations
• Serde custom deserialization
• The Rust Programming Language: Macros
• The Rust Reference: Macros by example
• The Rust Reference: Procedural macros
• serde_json documentation