WaveCoder, VideoPoet, StreamDiffusion, PowerInfer and RAG Survey

Week 4, December 2023

Greetings,

As the year draws to a close, we are thrilled to present the 38th edition of the State of AI, marking not only the final issue of 2023 but a milestone in our continuous journey through the ever-evolving landscape of artificial intelligence.

In this special edition, we explore the innovative WaveCoder, a tool reshaping instruction tuning with enhanced data generation. We delve into the realm of visual creativity with VideoPoet, a pioneering large language model for zero-shot video generation. Our focus then shifts to StreamDiffusion, a cutting-edge solution for real-time interactive generation, demonstrating remarkable progress in streamlining complex processes. We also highlight PowerInfer, showcasing the capabilities of consumer-grade GPUs in rapidly serving large language models. Lastly, we provide an in-depth survey of retrieval-augmented generation, a technique revolutionizing the way large language models access and utilize information.

Each article in this issue is a testament to the incredible advancements and diverse applications of AI, promising an enlightening and engaging read. We hope you find inspiration and insight in these pages as we bid farewell to 2023.

Happy reading and best wishes for the coming year!

Best regards,

State of AI

---

Contents

1. WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with Refined Data Generation

2. VideoPoet: A Large Language Model for Zero-Shot Video Generation

3. StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation

4. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

5. Retrieval-Augmented Generation for Large Language Models: A Survey

---

WaveCoder: Widespread and Versatile Enhanced Instruction Tuning with Refined Data Generation

Authors: Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, Qiufeng Yin

Source and references: https://arxiv.org/abs/2312.14187

---

Introduction

WaveCoder is a new research paper that aims to improve the generalization and performance of Code Language Models (LLMs) using what the authors call "instruction tuning." In recent years, LLMs have demonstrated impressive results in various code-related tasks. However, traditional methods for generating instruction data can often produce duplicate data and limit control over data quality. WaveCoder addresses these issues by proposing a new approach to generating high-quality instruction data from open-source code. The study also introduces CodeOcean, a dataset of 20,000 instruction instances across four universal code-related tasks.

The Four Core Tasks of CodeOcean

Instead of focusing on just one type of code-related task, WaveCoder addresses four core tasks:

1. Code Summarization: Given a piece of code, the LLM should generate a brief summary of what the code does.

2. Code Generation: Generate code based on a user's demand description. The model should simulate the user's demand discourse based on raw code and generate the appropriate output.

3. Code Translation: Given a piece of code, the LLM should convert it from one programming language to another.

4. Code Repair: Identify issues in the given code and provide solutions to fix them.

Covering these tasks helps demonstrate the versatility and enhanced generalization capabilities of WaveCoder.

LLM-based Generator-Discriminator Framework

To generate high-quality instruction data, the authors propose a novel LLM-based Generator-Discriminator framework. The framework leverages open-source code to generate supervised instruction data rather than relying only on the teacher LLM. By integrating open-source code, the diversity of the generated data can be improved.

The framework has two phases: a generation phase and a discriminator phase. In the generation phase, GPT-4 generates a task definition and the associated generation requirements for a given scenario. In the discriminator phase, the paper establishes several criteria and uses GPT-4 to assess the quality of each instruction instance. Good and bad examples are selected and reused in the subsequent generation phase.

Experiments and Results

WaveCoder was evaluated on three benchmarks: HumanEval, MBPP, and HumanEvalPack. The study uses StarCoder, CodeLLaMa, and DeepseekCoder as base models, and compares their performance to the newly proposed WaveCoder model.

The results demonstrate that WaveCoder outperforms the other open-source models in terms of generalization ability across different code-related tasks at the same fine-tuning scale. Additionally, WaveCoder demonstrates high efficiency in various code generation tasks.

Implications and Future Work

WaveCoder's LLM-based Generator-Discriminator framework and CodeOcean dataset show the potential of enhancing the generalization and performance of Code LLMs in various code-related tasks. The study not only contributes to the field of instruction data generation and fine-tuning models but also provides new insights and tools for enhancing performance in code-related tasks.

The authors suggest future work on LLM-based frameworks for instruction data generation to explore other code-related tasks and learning scenarios. As this field continues to evolve, there's potential for even more efficient and effective fine-tuned models that can better generalize and adapt to different tasks.

---

VideoPoet: A Large Language Model for Zero-Shot Video Generation

Authors: Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, and many others from Google Research

Source and references: https://arxiv.org/abs/2312.14125

---

Introduction

Welcome to the exciting world of video generation with large language models! Researchers from Google have developed a new model called VideoPoet that, as the name suggests, generates high-quality video content, including audio, using a large language model backbone. This recent research paper explores how the model adapts existing techniques while introducing new methods to create versatile video generation capabilities.