MCS Thesis Examination: Nasir Khalid

Abstract

    The following thesis introduces a novel technique for generating textured mesh models without any 3D supervision based solely on a text prompt. This is done by deforming the control shape of a limit subdivided surface along with its texture and normal map to match an input text prompt. The generated mesh asset can be easily integrated into games or modeling applications that rely on widespread rasterization based rendering techniques. The approach relies on a pre-trained Contrastive Language-Image Pre-Training (CLIP) model to compare the input text prompt with differentiably rendered images of our initialized 3D model. Unlike previous works that focused on stylization or required training of generative models, it performs optimization on mesh parameters directly to generate shape, texture, or both. To ensure that the optimization produces plausible meshes and textures, this work introduces several techniques including image augmentations, camera tuning and use of a pre-trained prior that generates CLIP image embeddings given a text embedding. Overall, this method offers a promising solution for zero-shot generation of 3D models, demonstrating the potential of CLIP-based techniques for the field of computer graphics.

Examining Committee

• Dr. Yiming Xiao (Chair) 
• Dr. Eugene Belilovsky & Tiberiu Popa (Supervisor)
  
• Dr. Yang Wang (Examiner)
• Dr. Yiming Xiao (Examiner)