Adversarial Detection
Benchmarks
How invisible-watermarking and perceptual-hash systems handle the 9 attacks that show up in real AI-laundering chains. Hallmark.AI vs PhotoDNA/PDQ, Hive, Reality Defender, Vermillio.
Swipe to see all vendors →
| Transform | Hallmark.AI | PhotoDNA / PDQ | Hive | Reality Defender | Vermillio |
|---|---|---|---|---|---|
| // AI generation | |||||
| AI inpainting | ✓ 0.95 | — | unpublished | unpublished | unpublished |
| AI upscaling | ✓ 0.75 | — | unpublished | unpublished | unpublished |
| AI video regeneration | ✓ 1.00 | — | unpublished | unpublished | unpublished |
| // Capture (analog hole) | |||||
| Screen recapture | ✓ 1.00 | ✗ 0.00 | unpublished | unpublished | unpublished |
| // Compression / re-encoding | |||||
| Heavy JPEG re-encode (Q30) | ✓ 1.00 | unpublished | unpublished | unpublished | unpublished |
| Multi-platform chain | ✓ 0.78 | unpublished | unpublished | unpublished | unpublished |
| // Geometric | |||||
| Mirror + crop | ✓ 0.96 | ✗ 0.00 | unpublished | unpublished | unpublished |
| // Composition | |||||
| Picture-in-picture | ✓ 1.00 | unpublished | unpublished | unpublished | unpublished |
| Split-screen | ✓ 1.00 | unpublished | unpublished | unpublished | unpublished |
What we measure, and how.
Each row reports one of four metrics, depending on what the transform is testing: bit-accuracy (how many of the 256 embedded bits survive), R@1 (recall-at-one in retrieval against a distractor pool), F1 (threshold-pass detection score), or detection rate (binary pass/fail at production threshold τ=0.40). Hallmark.AI cells cite the specific JSON result file the number came from.
The test set is the DISC21 image distractor pool (1M+ images) plus the UCF101 video distractor pool. The augmentation suite includes the 9 transforms shown above plus an additional set covered in our extended bench (perspective skew, letterbox-and-crop, color shift, frame interpolation) for which v1 of this leaderboard does not yet show competitor comparison rows.
View raw data ↗ /leaderboard.json · License: CC-BY-4.0
- [hive]
- Hive has not published adversarial-robustness numbers for any of the listed transforms as of 2026-05-28. Their integration in Bluesky's pipeline is publicly confirmed in Bluesky's 2025 transparency report without performance figures. Source ↗
- [reality]
- Reality Defender's published materials describe deepfake detection capabilities without specific adversarial-transform robustness numbers. Source ↗
- [vermillio]
- Vermillio's TraceID product page lists capabilities without per-transform robustness benchmarks. Sony Music investor disclosure confirms scale ("2 trillion items scanned yearly") but not per-transform performance. Source ↗
- [pdq]
- PhotoDNA + PDQ together form StopNCII (the federal compliance default per the May 2026 TAKE IT DOWN Act enforcement). Hallmark measured PDQ as a prefilter on the DISC21 + UCF101 test set: confirmed failure on screen_capture_realistic and mirror_crop (0% survival), partial failure on letterbox (48%). PhotoDNA itself is Microsoft's SaaS endpoint; we did not benchmark that service directly — only the open-source PDQ implementation. Transforms not in pdq_aug_survival.json (e.g. pip_corner, splitscreen) show as "unpublished" rather than "fail". Source ↗
Run detection? Submit your system for independent measurement.
Quarterly ADB (Adversarial Detection Benchmark) reports cite vendors that participate; vendors that decline are marked unpublished indefinitely. Submission inquiries are reviewed within 5 business days.