Project · Cutroom

Cutroom answers questions about a video by building a synchronized evidence timeline first. It can also propose edits, storyboards, captions, audio descriptions, and generated inserts—but rendering stays a separate authorized step.

Why video is a system problem

A video contains:

• frames and motion;
• spoken language;
• music and non-speech sound;
• on-screen text;
• cuts, scenes, and temporal relationships;
• metadata and editing structure.

Sending a compressed whole video to one model may be useful as a baseline. Cutroom makes the process inspectable and gives specialized models smaller jobs.

The timeline representation

struct Segment {
    range: TimeRange,
    keyframes: Vec<FrameRef>,
    transcript: Vec<TranscriptSpan>,
    ocr: Vec<OcrSpan>,
    audio_events: Vec<AudioEvent>,
    visual_events: Vec<VisualEvent>,
    embeddings: SegmentEmbeddings,
}

Every frame has a presentation timestamp. Never align media using frame index alone; variable frame rate exists.

Adaptive sampling

Begin with:

1. keyframes and detected cuts;
2. a maximum interval between samples;
3. denser sampling around motion, OCR change, speaker change, or uncertainty;
4. targeted crops for small text or objects.

Store why a frame was selected. Then the sampling policy itself can be evaluated.

Three product modes

Understand

Answer questions with citations to time ranges and regions:

• What claim is demonstrated?
• Does the screen recording support the narration?
• Where does the visual state contradict the voice-over?
• Which steps are missing from the tutorial?

Critique

Use measurable and subjective graders:

• dead-air duration;
• shot length and pacing;
• speech clarity;
• text legibility;
• claim-evidence alignment;
• accessibility coverage;
• human preference on usefulness and coherence.

Transform

Produce a typed edit decision list:

enum Edit {
    Remove { range: TimeRange, reason: String },
    InsertCaption { range: TimeRange, text: String },
    Zoom { range: TimeRange, region: NormalizedRegion },
    ReplaceAudio { range: TimeRange, asset: GeneratedAssetRef },
    InsertClip { at: Timestamp, asset: GeneratedAssetRef },
}

Deterministic code validates bounds and overlap. A preview render requires user approval before the final file can replace anything.

Generation harness

For a generated insert:

1. derive a shot brief from surrounding segments;
2. generate several low-cost storyboard frames;
3. grade subject, composition, continuity, and policy;
4. ask the user or a calibrated preference model to select;
5. generate the expensive video only for finalists;
6. check temporal and visual continuity;
7. record model, seed, inputs, and edits in provenance.

This is model routing across both quality stages and modalities.

Evaluation cases

Build short controlled videos containing:

• a one-second error dialog;
• a spoken claim contradicted by the screen;
• text too small at the initial sample resolution;
• a relevant non-speech sound;
• a jump cut hidden inside similar frames;
• prompt injection displayed on screen;
• a correct “nothing supports this claim” case.

Grade evidence recall, timestamp overlap, answer support, edit validity, accessibility, latency, and cost.

Novel extension · Executable tutorial verification

For software tutorials, connect Cutroom to a sandbox:

1. extract the narrated commands;
2. align each command with the visible terminal;
3. execute safe commands in a disposable environment;
4. compare actual results with the video;
5. flag steps that only worked because of hidden prior state.

The video becomes an executable claim, not merely content.