Conference Paper

We explore the previously untapped advantages of diffusion models over autoregressive (AR) methods in image-to-text generation. Through meticulous design of a latent-based diffusion model tailored for captioning, we achieve comparable performance with some strong AR baselines.

An inside project in Microsoft. We design a codec to disentangle each frame of talking face videos into motion and appearance representations and then curated a large-scale, high-quality dataset to train our diffusion-based GAIA model. The results demonstrate remarkable naturalness and scalability.

We found that powerful multimodal LLM like GPT4-Vision makes End-to-End embodied decision making more possible than ever. Moreover, we propose a new benchmark called PCA-EVAL and a multi-agent cooperation framework HOLMES for evaluation.