GAIA, Orca 2, PhysGaussian and more 🚀

Week 4, November 2023

Greetings,

Welcome to the 34th edition of the State of AI, a landmark issue where we explore cutting-edge advancements that are shaping the future of artificial intelligence.

In this edition, we unveil GAIA, a new benchmark for General AI Assistants, marking a significant leap in AI's capacity to assist in a broader range of tasks. We also explore breakthroughs in Exponentially Faster Language Modelling, demonstrating remarkable improvements in processing speed and efficiency.

Dive into the world of Orca 2, an innovative approach to teaching small language models how to reason, bringing us closer to more intuitive and intelligent AI. PhysGaussian introduces a novel integration of physics with 3D Gaussians, a breakthrough in generative dynamics. Lastly, we discuss System 2 Attention, a concept that might be just as vital for humans as it is for AI.

Each article in this issue offers a deep dive into these revolutionary concepts, promising a journey through the latest and most profound developments in AI. Prepare to be enlightened!

Best regards,

State of AI

---

Contents

1. GAIA: a benchmark for General AI Assistants

2. Exponentially Faster Language Modelling

3. Orca 2: Teaching Small Language Models How to Reason

4. PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

5. System 2 Attention (is something you might need too)

---

GAIA: A Benchmark for General AI Assistants

Authors: Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom

Source & References: https://arxiv.org/abs/2311.12983

---

Introduction

In the world of AI research, Large Language Models (LLMs) are paving the way for general-purpose systems. These models can significantly impact various industries, including law, chemistry, and more. However, evaluating their capabilities has become increasingly challenging due to their often complex and specialized nature. GAIA is a new benchmark designed to address these challenges by evaluating AI assistants on fundamental abilities rather than specific skills. In this paper, the authors introduce GAIA and detail its methodology, design choices, and question categories.

A Convenient Yet Challenging Benchmark

GAIA is a benchmark for AI systems that comprises 466 carefully curated questions, each having a single, factual answer. This benchmark focuses on elementary tasks that require a solid execution of complex sequences of actions. It aims to evaluate AI assistants on their fundamental abilities, such as reasoning, multi-modality handling, and tool-use proficiency. GAIA questions are simple for humans, with human respondents scoring 92%, while current AI systems like GPT-4 equipped with plugins achieve only a 15% success rate.

Design Choices

GAIA's design is grounded in four principles: real-world questions, interpretability, robustness against memorization, and ease of use. The benchmark covers various assistant use cases, such as personal tasks, science, and general knowledge. The questions are designed to be interpretable, unambiguous, and robust against data contamination or brute force attempts by AI systems. Evaluation in GAIA is automated, fast, and factual, using exact matches between the model's answers and the ground truth.

Categories of Questions

GAIA questions are categorized into three levels of difficulty:

1. Level 1 focuses on fundamental abilities, such as web browsing and multi-modality handling. These questions are relatively simple and do not require extensive reasoning.

2. Level 2 questions target more advanced capabilities, such as reasoning over multiple steps and making use of diverse tools. They necessitate more intricate strategies for answering.

3. Level 3 is the most complex level, where questions demand an AI assistant to master multiple tools, skills, and reasoning abilities simultaneously.

Crafting Questions and Challenges

Creating questions for GAIA involves focusing on real-world scenarios and increasing the required capabilities for each level. The authors provide guidelines to craft new questions and prevent data contamination, ensuring a fair evaluation of AI systems.

Evaluating AI Systems with GAIA

The authors evaluate several AI systems, including GPT-4, with varying levels of success. They analyze the successes and shortcomings of these systems, illustrating the potential of augmenting LLMs for improved performance. This evaluation helps pave the way for improvements in AI assistants, addressing issues such as tool usage safety and multi-modality handling.

Conclusion and Future Research

GAIA is a promising benchmark for evaluating and improving AI assistants, targeting both fundamental abilities and real-world use cases. By addressing the shortcomings of current benchmarks and offering guidelines to create new questions, GAIA can contribute to the development of more advanced and competent AI systems. The authors encourage the AI research community to engage with GAIA and extend it in ways that address emerging challenges in tool use safety, multi-modality, and other assistant capabilities. With the advent of Artificial General Intelligence (AGI), it is critical to evaluate AI systems on their ability to demonstrate human-like robustness and adaptability. GAIA serves as a step in this direction, providing a new benchmark that can help shape the next generation of AI assistants.

---

Exponentially Faster Language Modeling

Authors: Peter Belcak and Roger Wattenhofer

Source & References: https://arxiv.org/abs/2311.10770

---

Introduction

As artificial intelligence and natural language processing continue to advance rapidly, researchers strive to create models that can process language quickly, efficiently, and accurately. Enter UltraFastBERT, a groundbreaking new language model developed by Peter Belcak and Roger Wattenhofer. Making huge strides in both efficiency and speed, UltraFastBERT promises a complete game-changer in the world of language modeling. To ensure our tech-savvy audience stays updated, we're diving deep into this fascinating research paper, exploring its core concepts and shedding light on how it could revolutionize the field.