| CARVIEW |
Osmosis: RGBD Diffusion Prior for
Underwater Image Restoration
School of Marine Sciences
University of Haifa, Israel 2Dept. of Computer Science,
University of Haifa, Israel
ECCV 2024
Abstract
Underwater image restoration is a challenging task because of water effects that increase dramatically with distance.
This is worsened by lack of ground truth data of clean scenes without water.
Diffusion priors have emerged as strong image restoration priors.
However, they are often trained with a dataset of the desired restored output, which is not available in our case.
We also observe that using only color data is insufficient, and therefore augment the prior with a depth channel.
We train an unconditional diffusion model prior on the joint space of color and depth, using standard RGBD datasets of natural outdoor scenes in air.
Using this prior together with a novel guidance method based on the underwater
image formation model, we generate posterior samples of clean images, removing the water effects.
Even though our prior did not see any underwater images during training, our method outperforms state-of-the-art baselines for image restoration on very challenging scenes.
Our code, models and data are available on the project’s website.
Underwater Image Restoration
Results Comparison for Underwater Image Restoration
Comparison with several underwater methods.
Select a scene from the upper row and a method from the right column.
The input underwater image is labeled '1. Underwater,' and our result is '8. Osmosis'.
The Osmosis result is shown on the left side, with the compared method displayed on the right.
Underwater Monocular Depth Estimation
Results Comparison for Depth Estimation
Comparison with several underwater monocular depth methods.
Select a scene from the upper row and a method from the right column.
The input underwater image is labeled '1. Underwater,' and our result is '7. Osmosis'.
The Osmosis result is shown on the left side, with the compared method displayed on the right.
Osmosis Algorithm Overview
The iterative sampling process starts in \(t = T\) with random noise in 4 channels. The denoising step outputs denoised samples \(\hat{x}_0=\hat{J}_0, \hat{D}_0\). We use the physical image formation model together with \(\hat{x}_0\) to optimize the water parameters \(\phi\), and to guide the sampling towards the observed image. This process repeats itself, gradually updating both the estimated image and depth, until \(t = 0\), in which \(\hat{x}_0\) hold the method’s estimate for the reconstructed scene \(\hat{J}_0\) and its depth \(\hat{D}_0\)
Background
In water, we observe two wavelength- and distance- dependent effects. First, the \(\textit{direct}\) signal reflected from the object is attenuated. Second, light is scattered onto the object's line-of-sight (LOS), creating an additive signal termed \(\textit{backscatter}\) that increases with distance. The occluding backscatter layer is independent of the scene content. Thus, the visibility and contrast of further objects is significantly reduced and their colors are distorted.
Citation
@article{nathan2024osmosis,
title={Osmosis: RGBD Diffusion Prior for Underwater Image Restoration},
author={Bar Nathan, Opher and Levy, Deborah and Treibitz, Tali and Rosenbaum, Dan},
journal={arXiv preprint arXiv:2403.14837},
year={2024}
}References
- Peng, Yan-Tsung, et al. "Generalization of the dark channel prior for single image restoration." IEEE TIP, 2018.
- Li, Chongyi, et al. "Underwater image enhancement via medium transmission-guided multi-color space embedding." IEEE TIP, 2021.
- Han, Junlin, et al. "Underwater Image Restoration via Contrastive Learning and a Real-world Dataset." Remote Sensing, 2022.
- Huang, Shirui, et al. "Contrastive semi-supervised learning for underwater image restoration via reliable bank." In Proc. IEEE CVPR, 2023.
- Tang, Yi, et al. "Underwater image enhancement by transformer-based diffusion model with non-uniform sampling for skip strategy." In Proc. ACM International Conference on Multimedia, 2023.
- Peng, Yan-Tsung, et al. "Underwater image restoration based on image blurriness and light absorption." IEEE TIP, 2017.
- Bekerman, Yael, et al. "Unveiling optical properties in underwater images." ICCP, 2020.
- Gupta, Honey, et al. "Unsupervised single image underwater depth estimation." ICIP, 2019.
- Amitai, Shlomi, et al. "Self supervised monocular depth underwater." ICRA, 2023.