Day 15 — Extract Specialist Signals

Today Cutroom extracts signals that a later reasoning model can inspect.

video
├── speech recognizer ─▶ transcript spans
├── OCR engine ────────▶ text regions
├── cut detector ──────▶ shot boundaries
└── audio classifier ──▶ timestamped events

Give each specialist a narrow trait

pub trait FrameOcr: Send + Sync {
    fn observe(&self, frame: &FrameEvidence) -> Result<Vec<TextRegion>, OcrError>;
}

pub trait SpeechRecognizer: Send + Sync {
    fn transcribe(&self, audio: &AudioEvidence) -> Result<Vec<SpeechSpan>, SpeechError>;
}

Return typed observations with locators, confidence, and model release. Do not return a paragraph that mixes recognition with interpretation.

Add concurrency only after budgets

OCR frames and audio windows can run concurrently, but bound work with a semaphore or worker queue. Cancellation should stop new dispatch and let in-flight native calls reconcile safely. Never spawn one task per frame for an untrusted upload.

Use fixtures before accelerators

Store a tiny specialist output fixture and test the join logic without loading a model. Then add local Candle, ONNX, or a remote service behind the same trait. Record preprocessing and model release because changing either can change the observation set.

Break it deliberately

• OCR returns a region outside frame bounds;
• transcript spans overlap unexpectedly;
• a specialist returns NaN confidence;
• cancellation arrives during a native call;
• one worker repeatedly fails while others succeed.

Classify per-item failure versus whole-run failure. Preserve enough evidence to retry or degrade without duplicating successful expensive work.

Completion proof

Run the image and audio pipelines with scripted observations, then explain capacity in frames/second, audio-seconds/second, queue depth, and memory—not “it seems fast.”

Deep references: Candle fundamentals, ONNX specialist models, and Blocking media work.

Next: Day 16 — Evidence Timeline →.