Conference Tools of Tomorrow Highlights of SIGGRAPH
ONLINE Raum Tübingen Friday, April 28, 11:30

CLIPasso: Semantically Aware Object Sketching

Video on demand

Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize the remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill semantic concepts from sketches and images alike. We define a sketch as a set of Bézier curves and use a differentiable rasterizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss. The abstraction degree is controlled by varying the number of strokes. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual components of the subject drawn.

Please note that this presentation will only be available on April 28.

PhD student Yael Vinker, PhD student, Tel Aviv University

Yael Vinker is a PhD student at Tel Aviv University, advised by Prof. Daniel Cohen-Or and Prof. Ariel Shamir. She is currently a student researcher at Google Research in the Creative Camera team.
She received her MSc BSc in Computer Science from the Hebrew University of Jerusalem. During her BSc she also studied Visual Communication at Bezalel Academy of Arts and Design.
She was an intern at EPFL (VILAB), supervised by Prof. Amir Zamir. Prior to that she was a research intern at Disney Research, Zurich.