DiffusionGPT, Medusa, Depth Anything, VMamba, Self-Rewarding LLMS,

Week 4, January 2024

Greetings,

Welcome to the 42nd edition of the State of AI. In this milestone issue, we explore revolutionary developments including DiffusionGPT's text-to-image generation, Medusa's inference acceleration, VMamba's visual state space modeling, Depth Anything's use of large-scale unlabeled data, and the innovative concept of self-rewarding language models.

Each topic in this edition sheds light on the cutting-edge advancements shaping the future of AI, offering both excitement and deep insights. We are thrilled to present these transformative ideas and their implications for the world of artificial intelligence. Dive in for an enriching and enlightening journey!

Best regards,

State of AI

---

Contents

1. DiffusionGPT: LLM-Driven Text-to-Image Generation System

2. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

3. VMamba: Visual State Space Model

4. Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

5. Self-Rewarding Language Models

---

DiffusionGPT: LLM-Driven Text-to-Image Generation System

Authors: Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, Shilei Wen

Source and references: https://arxiv.org/abs/2401.10061

---

Introduction

Generating high-quality images from textual prompts has become an essential task in the intersection between machine learning and natural language processing. Popular models like DALLE-2 and Imagen have pioneered this area, but quite often, they fall short in handling diverse inputs or are limited to single models. DiffusionGPT aims to address these issues by leveraging Large Language Models (LLM) for a unified generation system capable of accommodating various types of prompts and integrating domain-expert models seamlessly. This summary will provide an insight into the inner workings of this exciting, new text-to-image generation system and highlight its achievements.

Key Concepts and Challenges

Diffusion models have significantly impacted image generation tasks, with models like Stable Diffusion gaining popularity due to their open-source availability and rapid advancements. Despite their strengths, these models face challenges in handling diverse input prompts and delivering exceptional performance across different domains.

The main challenges faced by diffusion models are:

1. Model Limitation – While models like Stable Diffusion exhibit adaptability to a wide range of prompts, they often struggle to perform well in specific domains. In contrast, domain-specific models excel in their respective sub-fields but lack versatility.

2. Prompt Constraint – Most generation models find it difficult to achieve optimal performance when presented with diverse input prompts, often limited to descriptive statements such as captions.

Recent research attempts to address these challenges by either improving the diffusion models themselves or by using prompt engineering techniques, but there's still a need for a more comprehensive solution.

Introducing DiffusionGPT

DiffusionGPT leverages Large Language Models (LLM) to create a one-for-all generation system that effortlessly integrates superior generative models and effectively parses diverse input prompts. The main components of the system are a Tree-of-Thought (ToT) that encompasses various generative models based on prior knowledge and human feedback and an Advantage Database that aligns the model selection process with human preferences.

The main contributions of DiffusionGPT include:

• New insight in employing LLMs to drive the text-to-image generation process

• An all-in-one system that is compatible with a wide range of diffusion models

• Efficiency and pioneering in its training-free nature

• High effectiveness by outperforming traditional stable diffusion models

The Working of DiffusionGPT

The DiffusionGPT pipeline consists of four steps: Prompt Parse, Tree-of-thought of Models, Model Selection, and Execution of Generation.

Prompt Parsing

The Prompt Parse Agent in DiffusionGPT recognizes and extracts the core textual information from input prompts. The agent interprets various input types, including prompt-based, instruction-based, inspiration-based, and hypothesis-based inputs. Identifying these forms ensures accurate recognition and representation of user intent, paving the way for selecting appropriate generative models.

Building and Searching the Model Tree

After parsing the input prompt, DifussionGPT addresses the challenge of finding the most suitable models from an extensive library. It does so by constructing a model tree that narrows down the candidate set of models and enhances the accuracy of the model selection process. This tree is generated using a Tree-of-Thought (ToT) of Models Building Agent. The built model tree is automatically filled by placing models in the appropriate position within the tree based on their attributes.

The Tree-of-Thought (ToT) of Models Searching Agent searches this model tree using a breadth-first approach. It iteratively evaluates the best subcategory at each node, comparing the categories against the input prompt to determine the category exhibiting the closest match.

Model Selection with Human Feedback

By employing Advantage Databases and human feedback, the Model Selection Agent aligns the model selection process with user preferences and ensures the generation of high-quality images. The model calculates scores for all generated images based on a reward model and stores this score information in an advantage database.

Execution of Generation

With the most suitable model selected, the final step involves generating the output image based on the parsed prompt and the chosen generative model.

Conclusion

DiffusionGPT provides a versatile, effective, and efficient approach to generating high-quality images from diverse input prompts. By leveraging Large Language Models, it introduces a unified generation system that overcomes limitations in domain-specific models and handles various types of prompts. As an all-in-one system, DiffusionGPT offers a convenient solution that paves the way for further advancements in the field of image generation.

---

MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Authors: Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao

Source and references: https://arxiv.org/abs/2401.10774

---

Introduction

With the continued growth of large language models (LLMs), their inference latency has become a significant challenge for real-world applications. Existing methods like speculative decoding have attempted to accelerate it but face difficulties in acquiring and maintaining a separate draft model. Enter MEDUSA, a novel technique designed to enhance LLM inference by adding extra decoding heads predicting multiple subsequent tokens in parallel. MEDUSA offers seamless integration, minimal overhead, and reduces the number of decoding steps.