Discovering and Mitigating Biases in CLIP-based Image Editing
Md Mehrab Tanjim¹, Krishna Kumar Singh¹, Kushal Kafle¹, Ritwik Sinha¹, and Garrison W. Cottrell²

¹Adobe Research, ²UCSD

At this moment, we are unable to release the official code.

We will welcome any adaptation/reproduction of our work created by third parties.

Technical Abstract

In recent years, the use of CLIP (Contrastive Language-Image Pre-Training) has become increasingly popular in a wide range of downstream applications, including zero-shot image classification and text-to-image synthesis. Despite being trained on a vast dataset, the CLIP model has been found to exhibit biases against certain protected attributes, such as gender and race. While previous research has focused on the impact of such biases on image classification, there has been little investigation into their effects on CLIP-based generative tasks. In this paper, we aim to address this gap in the literature by uncovering the queries for which the CLIP model introduces biases in the text-based image editing task. Through a series of experiments, we demonstrate that these biases can have a significant impact on the quality and content of the generated images. To mitigate these biases, we propose a debiasing technique that does not require retraining either the CLIP model or the underlying generative model. Our results show that our proposed framework can effectively reduce the impact of biases in CLIP-based image editing models. Overall, this paper highlights the importance of addressing biases in CLIP-based generative tasks and provides practical solutions that can be readily adopted by researchers and practitioners working in this area.

Paper and Bibtex

Discovering and Mitigating Biases in CLIP-based Image Editing
Md Mehrab Tanjim, Krishna Kumar Singh, Kushal Kafle, Ritwik Sinha, Garrison W. Cottrell
In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024.

PDF: Full Paper, Supplementary

Bibtex:


					@InProceedings{Tanjim_2024_WACV,

  					title={Discovering and Mitigating Biases in CLIP-based Image Editing},

  					author={Tanjim, Md Mehrab and Singh, Krishna Kumar and Kafle, Kushal and Sinha, Ritwik and Cottrell, Garrison W.},

  					booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},

  					month={January},

					year={2024},

					pages={2984-2993}

The template was originally made by Phillip Isola and Richard Zhang; the code can be found here.