ByteDance AI Research has proposed a groundbreaking self-supervised learning framework that enables the creation of high-quality stylized 3D avatars. These avatars, characterized by a mix of continuous and discrete parameters, can be customized to match the user’s appearance with just a single selfie taken from the front.
Creating visually appealing and animate 3D avatars has become increasingly important as more people engage in digital activities such as socializing, shopping, and gaming. While popular avatar systems like Zepeto and ReadyPlayer employ cartoonized and stylized looks for their fun and user-friendly appeal, the process of manually selecting and modifying avatars can be time-consuming and challenging, especially for novice users.
To address this issue, ByteDance AI Research’s algorithm predicts an avatar vector based on a selfie image, which serves as the complete configuration for a graphics engine to generate a 3D avatar. The avatar vector consists of parameters specific to the 3D assets, which can be either continuous (e.g., head length) or discrete (e.g., hair types). Traditionally, annotating a large set of images and training a model through supervised learning was the solution. However, this approach requires extensive annotations to accommodate the wide variety of assets.
The research team proposes a self-supervised approach to minimize annotation costs. They train a differentiable imitator that replicates the graphics engine’s renderings, matching the produced avatar image with the selfie image using identification and semantic segmentation losses. By doing so, they automate the generation of styled 3D avatars, reducing the need for extensive annotations.
The research involves three main steps: Avatar Vector Conversion, Self-supervised Avatar Parameterization, and Portrait Stylization. These steps gradually close the domain gap and maintain identification information throughout the pipeline. The Portrait Stylization stage focuses on transforming the 2D real-to-stylized visual appearance, ensuring expression and other crucial elements are retained without complicating the subsequent phases.
To achieve convergence and account for parameter discreteness, the Self-Supervised Avatar Parameterization step incorporates a relaxed avatar vector formulation. This lenient approach encodes discrete parameters as continuous one-hot vectors, allowing for differentiability while training an imitator to behave like the non-differentiable engine. In the Avatar Vector Conversion step, all discrete parameters are converted to one-hot vectors, crossing the domain from the relaxed avatar vector space to the strict avatar vector space. The final avatars are then constructed, and rendering takes place using the strict avatar vector. A unique search technique is employed to produce superior outcomes compared to direct quantization, ensuring the preservation of personal uniqueness.
Through human preference research and comparison to baseline approaches, such as F2P and manual production, ByteDance AI Research demonstrates the effectiveness of their method in producing high-quality stylized 3D avatars. Their approach achieves significantly higher scores than baseline techniques and rivals hand creation.
To support their pipeline’s design decisions, the research team also provides an ablation study. In summary, their technical contributions include the development of a self-supervised learning framework for creating stylized 3D avatars with a combination of continuous and discrete parameters, a method for bridging the style domain gap with portrait stylization, and a cascaded relaxation and search pipeline to address convergence issues in discrete avatar parameter optimization.
ByteDance AI Research has released a video demonstration of their paper on their website, showcasing the capabilities and results of their innovative approach. This research opens the door to easier and more efficient creation of customized and visually appealing 3D avatars, revolutionizing the way users interact with the digital world.
