Configuration Layering and Secret Handling

Configuration is input to the program and must be validated like an API request. AI systems add cost, privacy, residency, model, tool, and media limits whose unsafe defaults can materially change behavior.

The companion mosaic-config crate resolves precedence, records source, revalidates cross-field budgets, and keeps secret debug output redacted.

Learning objectives

• define deterministic configuration precedence;
• retain provenance of resolved settings;
• separate parse DTOs, resolved config, and runtime capabilities;
• validate cross-field and environment-dependent invariants;
• model secrets with redacted diagnostics and narrow exposure;
• decide startup-only versus reloadable settings;
• test precedence, redaction, and invalid merged states;
• run the companion layered configuration.

Dependency and model

Depends on domain validation, Serde concepts, CLI, and ownership.

safe defaults
   < file
   < environment
   < CLI explicit values
        │ merge with source
        ▼
resolved raw settings
        │ cross-field validate + initialize capabilities
        ▼
runtime config / adapters

Parsing each layer independently is insufficient: individually valid values can become invalid after merge.

Precedence must be one function

Do not let each subsystem call std::env::var or read files. Central resolution makes behavior testable and auditable.

The companion:

pub fn resolve(
    file: ConfigLayer,
    environment: ConfigLayer,
    cli: ConfigLayer,
) -> Result<ResolvedConfig, ConfigError>

applies layers in increasing precedence. None means “not supplied,” not a default value.

Clap/file parser defaults should not overwrite higher-level provenance. Apply safe defaults only in the resolver.

Track provenance

pub struct Setting<T> {
    value: T,
    source: ConfigSource,
}

Provenance helps:

• explain why routing/budget is active;
• debug environment overrides;
• audit policy rollout;
• warn on deprecated file fields;
• show safe effective config without secrets.

Do not log raw environment/config. Render an allowlisted redacted view.

For policy, provenance may also need file path/version, secret reference ID, configuration revision, and rollout assignment—not only File/Environment.

Validate after merge

Example:

• file sets max_model_calls = 1;
• CLI sets max_repairs = 1;
• each integer is valid alone;
• combined budget violates repairs < calls.

The companion constructs RunBudget and calls its domain validation after merging.

Validation categories:

• scalar: nonzero limits, valid URL;
• cross-field: timeout ordering, repair/call relationship;
• capability: selected provider supports modality;
• environment: directories writable, model artifact hash exists;
• policy: remote provider allowed for data class;
• secret: reference exists and authorized.

Perform deterministic validation first. Expensive/network initialization should return typed startup errors and obey deadlines.

Separate configuration from runtime objects

Do not pass a huge Config everywhere. Convert it into:

• provider client with secret held internally;
• RunBudget;
• model registry metadata;
• capacity/semaphore objects;
• storage paths/pools;
• immutable public policy snapshot.

Components receive narrow values/capabilities. This prevents a tool from accessing provider secrets merely because both live in application state.

Secret values

The companion SecretString:

• rejects empty;
• does not implement Display/serialization;
• implements redacted Debug;
• exposes plaintext only through explicit expose.

assert_eq!(
    format!("{secret:?}"),
    "SecretString([REDACTED])"
);

This reduces accidental leakage, not all leakage. Plaintext exists in process memory and any caller of expose can copy/log it.

Production source preference:

• workload identity/short-lived credentials;
• dedicated secret manager;
• mounted secret file/file descriptor;
• environment with platform controls;
• never committed config;
• avoid CLI argument/shell history.

Store a secret reference/version in config, resolve at startup/rotation boundary, and restrict who can read it.

Redaction boundaries

Redact:

• debug/error/log output;
• panic/crash reports;
• HTTP client request/response headers;
• traces and provider request capture;
• config/status endpoints;
• metrics labels;
• subprocess environment dumps.

Do not use simple substring replacement as the only defense. Structured logging should never attach secret fields. URLs can contain credentials/query tokens. Provider error bodies can echo requests.

Test canary secrets: insert a recognizable fake value and assert it does not appear in every public artifact.

Files and environment

Configuration files:

• explicit path and format version;
• strict unknown-field policy for safety settings;
• owner/permission checks where appropriate;
• size limit before parse;
• atomic update;
• no following untrusted symlinks under threat model;
• relative paths resolved against documented base.

Environment:

• Unicode/OS-string behavior is platform-specific;
• empty versus absent needs policy;
• variable names are public contract;
• child subprocesses inherit environment unless cleared;
• tests must serialize or inject an environment map to avoid races.

mosaic-config accepts ConfigLayer values instead of reading global environment, keeping library tests deterministic.

Reloading configuration

Classify settings:

Startup-only

• database URL/migrations;
• bind address;
• native runtime/device;
• feature-compiled backend;
• artifact root;
• thread pool sizes.

Reloadable snapshot

• routing weights;
• safe rate limits;
• approved model list;
• prompt/program release assignment.

Reload process:

1. parse/validate complete candidate;
2. build dependent resources if needed;
3. atomically swap immutable snapshot;
4. retain revision/provenance;
5. keep old snapshot for in-flight runs;
6. roll back on error.

Never mutate many globals field-by-field and expose half-new policy. Runs should record the snapshot revision they used.

Secret rotation may require rebuilding clients and draining old connections with a deadline.

Defaults are policy

Changing default model, repair count, timeout, remote-provider allowance, or max output affects quality/cost/privacy. Treat defaults as versioned behavior:

• explicit in trace/effective config;
• eval/load/security gated;
• announced in release notes;
• override/migration path;
• conservative for authority/cost.

Do not default missing tenant or permission.

Runnable companion implementation

cd rust/rust-ai-engineering
cargo run -q -p mosaic-config --example layered_config
cargo test -p mosaic-config

Expected:

calls=3 (File); repairs=2 (Cli); log=mosaic=debug (File); key=Some(SecretString([REDACTED]))

The API key comes from environment layer but is never printed. Tests prove precedence, redaction, and invalid merged budget rejection.

Failure investigation: production uses unexpected model budget

1. output safe effective config with setting sources;
2. check CLI parser supplied defaults versus explicit flags;
3. inspect file/env names and deployment injection;
4. compare config revision across replicas;
5. validate after full merge;
6. add exact deployment fixture/test.

Do not print full environment to debug.

Failure investigation: token appears in trace

1. revoke/rotate credential immediately under incident policy;
2. identify all copies/artifacts/retention;
3. inspect structured field source, error bodies, URLs;
4. patch allowlisted capture/redaction;
5. run canary-secret regression across logs/traces/errors;
6. delete/restrict contaminated artifacts under audit policy.

Common failure modes

• configuration read ad hoc inside components;
• parser defaults erasing provenance;
• secret String deriving Debug/Serialize;
• environment dumped in logs/subprocess;
• reload mutating fields non-atomically;
• unsafe policy defaults;
• file/env/CLI values not cross-validated.

Security and performance implications

Security:

• principle of least secret access;
• strict file permissions/paths;
• no secrets in metrics labels or traces;
• record policy revision and source;
• fail closed on invalid/missing security config;
• rotate with bounded drain.

Performance:

• parse/validate once, pass typed config;
• immutable snapshots avoid hot-path locks;
• reload should not rebuild expensive models unnecessarily;
• secret-manager/network lookup has startup/cache/rotation latency;
• high-cardinality provenance must not become metric labels.

Exercises

1. Add parsing DTOs for file/env/CLI without global reads.
2. Add provider URL/model plus cross-field capability validation.
3. Generate a redacted effective-config view.
4. Add immutable hot reload with revision.
5. Add canary-secret scans for CLI errors/traces.

Solution guidance

Keep raw layer fields optional. Resolve once, validate domain relationships, then construct runtime objects. Redacted view is a dedicated type/serializer, not Debug on the full config. Atomic reload uses Arc snapshot swap and records revision per run.

Check your understanding

• When are defaults applied?
• Why validate after merge?
• Does redacted Debug erase memory?
• Why inject environment maps in tests?
• Which config changes require run revision identity?

Completion checklist

• One resolver owns precedence.
• Every important setting records source/revision.
• Merged config is cross-validated.
• Secrets have narrow, redacted types.
• Components receive narrow capabilities.
• Reload is atomic and versioned.
• Companion config tests pass.

Authoritative references

• std::env
• Clap environment/value sources
• Serde attributes
• OWASP Secrets Management Cheat Sheet