State of AI

Week 4, June 2023

Greetings,

Welcome to the 12th edition of the State of AI. In this issue we navigate through the interesting narrative of how AI is learning directly from textbooks, capturing the essence of human knowledge. Then, we explore the fascinating area of robotics, where our foundation agent, RoboCat, continually improves its manipulation skills.

Further into our journey, we dive into the intriguing interplay between natural language and cognitive thought models, demonstrating AI's growing aptitude for intricate translations. Next, we introduce you to AudioPaLM, a large language model capable of both speaking and listening, pushing the boundaries of interactive AI. Finally, we showcase rapid advancements in object segmentation, promising a faster and more accurate recognition of various entities.

Each of these papers offers a unique insight into the latest breakthroughs and exciting applications of AI, promising a read that is as insightful as it is thought-provoking. Enjoy!

Best regards,

State of AI

---

Contents

1. Textbooks Are All You Need

2. RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation

3. From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought

4. AudioPaLM: A Large Language Model That Can Speak and Listen

5. Fast Segment Anything

---

Textbooks Are All You Need

Authors: Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S´ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li

Source & References: https://arxiv.org/abs/2306.11644v1

---

Introduction

This research paper introduces phi-1, a new large language model optimized for code generation. The authors argue that focusing on "textbook-quality" training data can significantly improve model performance while requiring smaller dataset size and less training compute. Phi-1 achieves state-of-the-art results on programming tasks, surpassing previous models that rely on vastly larger datasets and more costly compute resources. The authors also examine emergent properties in phi-1 and how they’re influenced by the number of model parameters.

The Power of Textbook-Quality Data

Phi-1 is a 1.3 billion-parameter model trained on a total of 7 billion code and textual tokens. The authors hypothesize that training on high-quality data, which they refer to as “textbook-quality” data, will improve model performance while reducing both dataset size and training compute time. To test this hypothesis, the authors start by filtering publicly available Python code datasets and creating synthetic textbook and exercises datasets using GPT-3.5.

Unlike standard coding datasets, which typically contain vast amounts of noise and ambiguity, textbook-quality data is designed to be clear, self-contained, instructive, and balanced. By focusing on this type of data, the authors aim to help the model better reason and plan algorithmically, ultimately improving its performance on code-generation tasks.

Training Process and Data Sources

Phi-1 is based on Transformer architecture and uses FlashAttention and Rotary Position Embeddings to optimize multi-head attention calculations. The training process includes three main datasets:

1. A filtered code-language dataset comprised of roughly 6 billion tokens selected from The Stack and StackOverflow using a language model-based classifier.

2. A synthetic textbook dataset containing less than 1 billion tokens of GPT-3.5-generated Python textbooks.

3. A small synthetic exercises dataset containing around 180 million tokens of Python exercises and solutions.

By using these datasets in combination, the authors aim to help the model better align with its primary task of generating simple Python functions based on natural language instructions.

Emergent Properties

Despite using fewer tokens and parameters than previous models, phi-1 demonstrates impressive emergent properties. By comparing phi-1 with models that have both more and fewer parameters, the authors explore the impact of parameter count on emergence.

Emergent properties are unexpected capabilities that arise from training models on diverse tasks. In phi-1, these properties include code understanding and generation, as well as the ability to answer questions about code snippets.

Alternative Benchmarks and Data Contamination Study

To further validate phi-1’s performance, the authors examine the model's capabilities on alternative benchmarks—tasks that assess how well the model can generate code in different problem settings. These benchmarks help demonstrate that phi-1 can perform a wide range of code generation tasks, despite its relatively small size.

Additionally, the authors undertake a data contamination study to ensure that the examples used for evaluation were not part of the training set. This study helps confirm that the model's performance on its primary tasks is due to generalization and not data contamination.

Key Takeaways

Overall, the research demonstrates the power of focusing on textbook-quality data for training large language models. By using this approach, the authors show that:

1. Models can achieve state-of-the-art performance on code-generation tasks while requiring significantly smaller datasets and training compute resources.

2. Emergent properties can still arise in models trained on smaller datasets, suggesting that the number of parameters plays a crucial role in driving their emergence.

3. Alternative benchmarks and data contamination studies are essential tools for understanding the broader capabilities of these models and validating their performance.

In summary

The phi-1 language model offers a compelling glimpse into how optimizing training data can yield better-performing, more efficient, and more environmentally friendly models for code generation. This research paper serves as a valuable contribution to the rapidly evolving field of artificial neural network research, offering new and exciting possibilities for the future of program synthesis and, more broadly, the development of large language models.

---

RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation

Authors: Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, and Nicolas Heess

Source & References: https://arxiv.org/abs/2306.11706

---

Introduction

The researchers at Google DeepMind have created a new foundation agent for robotic manipulation called RoboCat. This agent aims to improve robotic learning by leveraging heterogeneous experiences across different robots and tasks. The study uses a transformer-based model to develop a highly adaptable, multi-task, multi-embodiment agent that can generalize to new tasks and robots with limited examples.