What is watermark robustness? An AI company provides two services: (1) generate watermarked images, i.e., embed invisible messages, either through a post-processing method or an in-processing method, and (2) detect these messages when shown any of their watermarked images. There is an attack stage between the watermarking and detection stages. The watermarked images may experience natural distortions (e.g., compression, re-scaling) or manipulated by malicious users attempting to remove the watermarks. A robust watermarking method should still be able to detect the original message after an attack. WAVES can evaluate the robustness of any watermarks.

Evaluation of a single attack on a watermarking method. We first attack watermarked images over a variety of strengths (also labeled 'stg'). Then, we evaluate the detection performance (TPR@0.1%FPR) and a collection of image quality metrics such as PSNR, and plot a set of performance vs. quality plots. By normalizing and aggregating these quality metrics, we derive a consolidated 2D plot that represents the overall performance vs. quality for the evaluation.

Benchmarking watermarks and attacks. For each watermarking method, we plot all attacks on a unified performance vs. quality 2D plot to facilitate a detailed comparison. Based on this, we provide two additional analytical perspectives. We compare watermarks' robustness through the averaged performance under different attacks. We evaluate attacks' potency by ranking the quality at a specific performance threshold.

Watermark detection performance (i.e., TPR@0.1%FPR) of Stable Signature, StegaStamp, and Tree-Ring watermarks after attacks via WAVES. We compute the Average TPR@0.1%FPR across all strength levels and further averaged this metric across different attacks and datasets. Lower Average TPR@0.1%FPR indicates higher vulnerability of the watermark to a certain type of attack. Right figure shows the distribution of quality degradation for each type of attack. Lower quality degradation is preferred.

User identification performance (i.e., accuracy) of Stable Signature, StegaStamp, and Tree-Ring watermarks after attacks via WAVES. We simulate identification tasks with total user counts of 100, 1,000, and 1000000.

Distortions.

Regeneration Attacks.

Regeneration Attacks.

Adversarial Attacks.

Adversarial Attacks.