Gemini, Mamba, AVID, EfficientSAM & Chain of Code

Week 2, December 2023

Greetings,

Welcome to the 36th edition of the State of AI. In this landmark issue, we're excited to introduce groundbreaking advancements shaping the future of artificial intelligence. Explore Gemini's multimodal models that redefine versatility, Mamba's linear-time sequence modeling, AVID's pioneering approach to any-length video inpainting, EfficientSAM's innovative strides in image pretraining for segmenting anything, and the Chain of Code's novel integration of language models with code emulation.

These developments not only demonstrate the rapid evolution of AI but also highlight the vast potential of these technologies in diverse applications. We invite you to immerse yourself in these exciting new frontiers and join us in envisioning a future transformed by AI.

Best regards,

State of AI

---

Contents

1. Gemini: A Family of Highly Capable Multimodal Models

2. Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3. AVID: Any-Length Video Inpainting with Diffusion Model

4. EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything

5. Chain of Code: Reasoning with a Language Model-Augmented Code Emulator

---

Gemini: A Family of Highly Capable Multimodal Models

Authors: Gemini Team, Google

Source & References: https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf

---

Introduction

Google recently introduced Gemini, a new family of highly capable multimodal models that exhibit remarkable performance across different modalities like text, images, audio, and video. With three sizes in the family – Ultra, Pro, and Nano – Gemini models can handle a wide range of applications, from complex reasoning tasks to on-device, memory-constrained use-cases. In an extensive series of evaluations, the most capable model, Gemini Ultra, has advanced the state of the art in 30 out of 32 benchmarks, including a groundbreaking achievement of human-expert performance on the well-studied MMLU exam benchmark.

Model Architecture

The Gemini models build on Transformer decoders equipped with enhancements for stable training at scale and optimized inference on Google's Tensor Processing Units (TPUs). These models are capable of accommodating textual input interleaved with various audio and visual inputs, making them natively multimodal.

The first version of Gemini, called Gemini 1.0, features three models tailored to different computational limitations and application requirements:

• Ultra: The most capable model, ideal for highly complex tasks, offers state-of-the-art performance across numerous reasoning and multimodal challenges.

• Pro: A performance-optimized model balancing cost and latency, providing strong reasoning performance and broad multimodal capabilities.

• Nano: The most efficient model, designed for on-device deployment, comes in two versions (Nano-1 and Nano-2) targeting low and high memory devices, respectively.

Training Infrastructure

Google trained the Gemini models using TPUv5e and TPUv4 accelerators, depending on their sizes and configurations. Training Gemini Ultra required numerous TPUv4 accelerators across multiple data centers, presenting unique infrastructure challenges. Lessons learned from scaling up included minimizing the rate of hardware failure, leveraging Google’s network latencies and bandwidths, and finding innovative ways to maintain high performance in recovering from hardware issues.

Evaluations

The Gemini models set new records across a broad range of text, image, audio, and video benchmarks. The evaluation includes well-studied benchmarks, human-preference evaluations, and assessment of English performance and multilingual capabilities.

Text-Based Academic Benchmarks

Gemini Pro and Ultra excel in a variety of text-based academic benchmarks covering reasoning, reading comprehension, STEM, and coding. Gemini Ultra is the first model to surpass human-expert performance on MMLU, a prominent benchmark testing knowledge and reasoning via a suite of exams. The model achieves a remarkable 90.04% accuracy on MMLU and sets new benchmarks in solving complex mathematical problems with grade-school math and competition-grade problem sets.

Gemini Ultra also excels in coding, making impressive advances on code-completion benchmarks like HumanEval and Natural2Code.

Multimodal Benchmarks

Gemini Ultra stands out in achieving the highest scores on recent MMMU benchmarks that test college-level subject knowledge and reasoning. This demonstrates Gemini’s potential for applications in education and across various other fields.

Efficiency and On-Device Deployment

Gemini Nano models raise the bar for efficiency in on-device tasks, like summarization and reading comprehension, by utilizing advancements in distillation and training algorithms.

Responsible Deployment

Google is taking a proactive approach to responsible deployment of Gemini models by performing impact assessments, developing model policies, and implementing evaluations and harm mitigation before deciding to deploy.

Broader Implications and Limitations

The new capabilities of Gemini models in cross-modal reasoning and language understanding open up a world of exciting applications, from educational settings to image and audio understanding. However, limitations exist, including dependence on the quality of input data and biases present in the training data. As we move forward, addressing these limitations will be crucial in unlocking the full potential of Gemini models.

In summary, Google's newly introduced Gemini is a groundbreaking family of multimodal models that push the boundaries of language, image, audio, and video understanding. With impressive performance across numerous benchmarks, Gemini has the potential to revolutionize applications in education, coding, and beyond. Balancing responsible deployment with the exploration of new applications will be crucial in leveraging the full potential of these highly capable models.

---

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Authors: Albert Gu, Tri Dao

Source & References: https://arxiv.org/abs/2204.04257

---

Introduction

The Mamba research paper presents a novel approach to sequence modeling, focusing on developing a more efficient and scalable alternative to the widely-used Transformer architecture in deep learning. Transformers have been successful across various domains such as language, images, speech, audio, time series, and genomics. However, they come with inherent limitations such as inability to handle very long sequences and quadratic scaling concerning the window length.

The authors introduce Mamba, a linear-time sequence model incorporating Selective State Space Models (SSMs), aiming to achieve the modeling power of Transformers while addressing their inefficiencies. The new approach overcomes the limitations of previous models by incorporating content-based reasoning and an improved selection mechanism that allows the model to selectively propagate or forget information depending on the current token.