State of AI

Week 2, July 2023

Greetings,

Welcome to the 14th edition of the State of AI. This issue transports us into a fascinating exploration of AI and robotics, where we examine how large language models are lending their planning capabilities to robots and prompting them to ask for help. We then delve into the intriguing domain of AI psychology, exploring how personality traits are manifesting in large language models.

Continuing our journey, we look at scalability and marvel at how transformers are being scaled to handle an unprecedented 1,000,000,000 tokens. The safety and reliability of large language models come under our microscope next, as we study why and how LLM safety training sometimes fails. Finally, we turn our attention towards the intriguing interplay between AI and mental health, as we consider how models are being trained to generate, recognize, and reframe unhelpful thoughts.

Each of these topics illustrates the breadth and depth of AI's potential and provides a fascinating insight into the state of the field. We invite you to join us on this journey of discovery and hope you enjoy this issue.

Best regards,

State of AI

---

Contents

1. Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners

2. Personality Traits in Large Language Models

3. LongNet: Scaling Transformers to 1,000,000,000 Tokens

4. Jailbroken: How Does LLM Safety Training Fail?

5. Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts

---

Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners

Authors: Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar

Source & References: https://arxiv.org/abs/2307.01928

---

Introduction

Large language models (LLMs) have made tremendous advancements in recent years and continue to demonstrate remarkable capabilities, such as step-by-step planning and commonsense reasoning. These traits make LLMs an attractive solution for enabling robots to perform complex tasks. However, despite their potential, LLMs still suffer from a tendency to generate confidently wrong outputs, or "hallucinations." In this research paper, the authors present KNOW NO, a framework designed to align the uncertainty of LLM-based planners and help robots recognize when they don't know the solution, prompting them to ask for assistance when necessary.

Aligning Uncertainty with Conformal Prediction

To tackle the problem of hallucinations in LLMs, the authors of the paper utilize Conformal Prediction (CP), a method for aligning the uncertainty of LLM-based planners. CP provides statistical guarantees on task completion while minimizing the need for human intervention. This approach is built on the observation that language-instructed robot planning can be treated as a multiple-choice Q&A (MCQA) problem. By transforming the planning problem into MCQA and applying CP, the framework is able to provide calibrated confidence and minimize help from humans.

With CP, the robot generates a set of plausible next steps and their corresponding confidence scores. If there is more than one option in the set generated by CP, the robot asks for help, improving efficiency and autonomy compared to modern baselines that require extensive prompt tuning.

Theoretical Guarantees and Minimal Help

The authors prove theoretical guarantees on calibrated confidence for both single-step and multi-step planning problems. These guarantees ensure that, with a user-specified probability level (1-ϵ), the robot successfully performs tasks in which it seeks help when necessary. Additionally, CP minimizes the average size of prediction sets, helping to achieve the goal of minimal help.

Evaluating KNOW NO in Simulated and Real-World Environments

The researchers evaluated the KNOW NO framework across a variety of simulated and real-world robot setups, involving tasks containing different types of ambiguity, such as spatial or numeric uncertainties and human preferences. Their experiments demonstrated that KNOW NO significantly improves efficiency and autonomy compared to traditional baselines.

In these settings, the robot first generates multiple candidate plans for the next step and ranks them based on confidence scores. If there is more than one option and the robot is uncertain, it seeks help from a human, who can choose one of the options provided or halt the operation if necessary. This setup ensures that the robot's task completion rate aligns with the probability level specified by the user.

Addressing Ambiguous Instructions

In real-world scenarios, natural language instructions often contain inherent or unintentional ambiguity. Instead of following an incorrectly constructed plan that could lead to undesirable or unsafe actions, a robot using KNOW NO would ask for help or clarification when necessary. This ability to identify when it doesn't know the best course of action allows the robot to operate more safely and effectively.

Enabling Efficient and Autonomous Robots

Overall, KNOW NO represents a significant step toward enabling robots to make intelligent decisions when faced with ambiguous or uncertain situations. By aligning the uncertainty of LLM-based planners and providing calibrated confidence and minimal help, the framework allows robots to improve their efficiency and autonomy. With further development and integration, this approach has the potential to significantly advance the field of robotics and create safer, more reliable, and more capable machines.

---

LONGNET: Scaling Transformers to 1,000,000,000 Tokens

Authors: Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei

Source & References: https://arxiv.org/abs/2307.02486

---

The Pursuit of Scaling Sequence Length

A recent development in the world of machine learning and natural language processing comes in the form of LONGNET, a Transformer variant capable of scaling up to a staggering one billion tokens without sacrificing its performance on shorter sequences. Conceived by researchers at Microsoft Research, this accomplishment addresses a persistent challenge in the development of neural networks—balancing computational complexity and model expressivity, particularly when it comes to increasing sequence lengths. Scaling up sequence length introduces a host of benefits, such as enabling interaction with humans and exploring the limits of in-context learning.