Kangaroo, LEGENT, PLLaVA, Replacing Judges with Juries & More

Week 5, April 2024

Welcome to the latest edition of the State of AI. This issue is packed with innovative developments and thought-provoking themes, from "Kangaroo," a breakthrough approach in self-speculative decoding, to "LEGENT," a new platform revolutionizing embodied AI agents. We also explore the transformative potential of LLMs in decision-making roles, delve into the advancements of "PLLaVA" for video dense captioning, and uncover techniques to enhance context utilization in LLMs. Each article sheds light on the incredible strides being made in AI, offering both deep insights and practical applications. Dive in for an enlightening journey through the frontiers of AI technology. Enjoy!

Best regards,

State of AI

---

Contents

1. Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting

2. LEGENT: Open Platform for Embodied Agents

3. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

4. PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

5. Make Your LLM Fully Utilize the Context

---

Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting

Authors: Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, Yunhe Wang

Source and references: https://arxiv.org/abs/2404.18911

---

Introduction

In the dynamic and ever-expanding world of Machine Learning, the quest for efficiency in processing large language models (LLMs) continues at a brisk pace. A new approach titled "Kangaroo" emerges from Huawei’s Noah’s Ark Lab, promising significant strides in speeding up the inference processes of these computational behemoths without sacrificing accuracy. Co-authored by a team led by Fangcheng Liu and Yunhe Wang, this paper delves into speculative decoding—a technique that could potentially recalibrate our expectations from future LLM architectures.

Accelerating Inference: The Crux of Modern LLMs

The brilliance of LLMs often comes at the cost of high latency, primarily due to the massive computational demands they impose. Traditional speculative decoding developed to tackle this issue typically relies on a draft model that predicts a sequence of tokens, which are then verified and corrected by a larger, more sophisticated model. However, this method isn't without its drawbacks—primarily, the need for separate training of a draft model, adding additional complexities and costs.

Kangaroo: A Novel Framework

Kangaroo introduces a self-contained approach that integrates early exiting—a concept where decisions are made at preliminary stages, thus saving computational resources. Instead of relying on a separate draft model, Kangaroo smartly employs a shallow sub-network within the main model. This sub-network acts as the draft model, which makes initial predictions that are refined by the subsequent deeper layers if necessary.

The Design of the Kangaroo Framework

At its core, the Kangaroo framework is designed around efficiency and parsimony. The shallow sub-network drafts the tokens with high confidence early, significantly reducing the processing load on the deeper, more complex layers of the network. An additional adapter module is trained to bridge the representational gap between the shallow and deep layers, ensuring that the quality of output is not compromised.

The striking part of this framework is its self-regulatory nature. The draft phase utilizes an innovative double early-exit mechanism that not only decides when to halt the shallow network based on confidence thresholds but also dynamically adjusts the speculative steps based on the difficulty of token prediction—essentially cutting down unnecessary calculations on challenging tokens.

Performance and Practicality

Kangaroo's benchmarks are promising. It beats the existing speculative decoding method Medusa by achieving higher inference speeds with significantly fewer parameters—about 88.7% fewer, to be exact. In practical terms, this means that Kangaroo can achieve up to 1.7x speedup on the Spec-Bench test suite compared to other leading methods.

The Impact of Kangaroo

The implications are manifold. For companies and developers working with LLMs, this could mean drastically reduced server costs and energy consumption. For users of technologies powered by LLMs, it could translate into quicker response times and more fluent interactions.

Toward a Greener AI

Beyond performance metrics, the reduced computational demand inherently suggests a move towards greener, more sustainable AI practices. By optimizing the inference process, Kangaroo not only accelerates computational tasks but also potentially reduces the carbon footprint associated with running large-scale models.

Conclusion: What Lies Ahead?

The introduction of the Kangaroo framework into the landscape of LLMs opens up new avenues for research and development. Its innovative approach challenges the traditional methodologies and paves the way for more efficient, sustainable, and cost-effective machine learning practices.

While the current implementation is focused on optimizing language models, the underlying principles could be adapted for other types of neural networks. As the digital world continues to evolve, so too does our approach to building the systems that support it. Kangaroo is not just a step forward in machine learning—it's a leap towards the future of AI.

---

LEGENT: Open Platform for Embodied Agents

Authors: Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun

Source and references: https://arxiv.org/abs/2404.18243

---

Introduction

The fascinating world of simulated, interactive environments where agents interact much like humans has been a cornerstone of AI research, pushing the boundaries of how machines understand and interact within complex environments. In the latest addition to this stream of exploration, researchers from Tsinghua University and Central South University have unveiled LEGENT—an open, scalable platform designed to refine the development and deployment of sophisticated embodied agents. Integrating Large Language Models (LLMs) and Large Multimodal Models (LMMs), LEGENT aims to bridge significant gaps in current technological capacities by offering a versatile playground for these agents to learn, operate, and generalize across various tasks within physical and simulated environs. Let’s dive into the details.