Pioneering AI Innovations in Robotics, Chip Design, and Beyond

Week 1, November 2023

Greetings,

Welcome to the landmark 31st edition of the State of AI. In this edition, we explore revolutionary advancements that are reshaping the landscape of artificial intelligence. We dive into RoboGen's approach to automated robot learning through generative simulation, the integration of domain-adapted LLMs in chip design with ChipNeMo, and the cutting-edge developments in structural biology through the latest AlphaFold model.

Further, we present LLaVA-Interactive, a comprehensive demo encapsulating image chat, segmentation, generation, and editing. Lastly, we explore CapsFusion, a novel perspective on rethinking image-text data at scale.

Each article in this issue stands as a testament to the relentless progress and broadening horizons in AI. Prepare to be immersed in a world where technological boundaries are continuously being pushed. Enjoy your read!

Best regards,

State of AI

---

Contents

1. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

2. ChipNeMo: Domain-Adapted LLMs for Chip Design

3. Performance and structural coverage of the latest, in-development AlphaFold model

4. LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing

5. CapsFusion: Rethinking Image-Text Data at Scale

---

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

Authors: Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, Chuang Gan

Source & References: arxiv.org/abs/2311.01455

---

Introduction

Researchers have developed RoboGen, a generative robotic agent that uses the latest advancements in foundation and generative models to automatically learn diverse robotic skills in simulated environments. Leveraging these models, the system creates tasks, scenes, and training supervisions for a robot, allowing it to scale up robotic skill learning with minimal human involvement.

Goal and Motivation

The primary goal of RoboGen is to increase the diversity and scale of robotic skill learning by using generative simulation. This approach combines the advantages of simulated robotic skill learning with the latest progress in foundation and generative models, which provides access to privileged low-level states, unlimited exploration opportunities, and supports massively parallel computation.

One of the main challenges in robotic skill learning is creating realistic and diverse simulation environments. Manually designing tasks, selecting relevant assets, generating plausible scene layouts, and crafting training supervisions is cumbersome and time-consuming. RoboGen aims to automate this process, extracting the extensive knowledge embedded in large-scale models and transferring it to the field of robotics.

The RoboGen Pipeline

The RoboGen pipeline consists of four stages: Task Proposal, Scene Generation, Training Supervision Generation, and Skill Learning.

Task Proposal

RoboGen first generates meaningful, diverse, and high-level tasks for robots to learn. It initializes the system with a specific robot type and an object randomly selected from a pool. The provided robot and sampled object information serves as a basis upon which large language models (LLMs) like GPT-4 can condition and subsequently reason and extrapolate to generate a variety of tasks, taking into account both robot and object affordances.

Scene Generation

Once a task is proposed, the system generates a corresponding simulation scene for learning the skills to accomplish the task. The scene components and configurations include queries for relevant assets, their physical parameters, initial joint angles, and the overall spatial configurations of the assets. GPT-4 is used to generate a list of assets required for the proposed task, which is then used to populate the scene.

Training Supervision Generation

After generating the simulated environment, RoboGen moves to the stage of producing training supervisions such as reward or loss functions to train the robot to acquire the proposed skill. Depending on the task and robot setup, the system can employ conventional reinforcement learning, motion planning, or trajectory optimization methods.

Skill Learning

With generated tasks, scenes, and training supervisions in place, the robot learns the policies required to acquire the proposed skill. The knowledge gathered from this learning process is fed back into the RoboGen pipeline, allowing the system to generate more refined tasks, scenes, and supervisions as it learns.

Experimental Results and Implications

RoboGen has demonstrated the capability to deliver a continuous stream of diversified skill demonstrations for various tasks and environments, surpassing previous human-created robotic skill learning datasets with minimal human involvement. The skills generated by RoboGen include rigid and articulated object manipulation, deformable object manipulation, and legged locomotion.

Theoretically, by querying RoboGen endlessly, one could generate an unlimited amount of diversified demonstration data for robot learning. However, it's important to note that the current pipeline relies on the large language models like GPT-4, which have certain limitations. These models may not grasp a full understanding of physical interactions and dynamics, so extracting the knowledge needed for robots to effectively execute physical actions remains an open challenge.

Nevertheless, RoboGen marks a step towards fully automated, large-scale robotic skill training and the development of generalizable robotic systems. Its ability to transfer knowledge from large-scale models to the robotics domain highlights the potential for generative simulation in unlocking new possibilities for robotic skill learning.

Concluding Thoughts

RoboGen offers an exciting new approach to robotic skill learning by leveraging the power of generative simulation and foundation models. It demonstrates the capabilities of these models in automatically generating tasks, scenes, and training supervisions for robots to learn diverse skills at scale. While limitations exist, RoboGen shows promise in bridging the gap between robotics and large-scale models, potentially paving the way for robust, generalizable robotic systems in the future.

---

ChipNeMo: Domain-Adapted LLMs for Chip Design

Authors: Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, Kishor Kunal, Xiaowei Li, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Kaizhe Xu, Haoxing Ren

Source & References: arXiv:2311.00176v1

---

Introduction and Motivation

The research paper "ChipNeMo: Domain-Adapted LLMs for Chip Design" revolves around utilizing large language models (LLMs) to address some of the pressing issues in chip design, particularly tasks that involve natural language or programming languages. The goal is to improve chip design productivity by employing generative AI models to automate tasks like code generation, responses to engineering questions, analysis and report generation, and bug triage.

The domain-adaptation techniques used in this research include custom tokenizers, domain-adaptive continued pretraining, supervised fine-tuning (SFT) with domain-specific instructions, and domain-adapted retrieval models. The evaluation of these techniques was performed across three LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis.