Meteor, FIFO-Diffusion, ConvLLaVA, Linear Transformers & Intro to VLMs

Week 4, May 2024

Greetings,

Welcome to the latest edition of State of AI! In this issue, we embark on a journey exploring the expanding horizons and frontiers of artificial intelligence. We kick things off with a deep dive into "Meteor," a Mamba-based traversal framework that's transforming the capabilities of large language and vision models.

Next, prepare to be amazed by "FIFO-Diffusion," a groundbreaking technique for generating infinite videos from text without the need for extensive training. Then, we'll delve into "ConvLLaVA," which leverages hierarchical backbones as visual encoders for large multimodal models to push the limits of AI further.

Ever wondered if your transformer model could secretly be linear? We'll uncover this intriguing insight in "Your Transformer is Secretly Linear." Lastly, we provide a comprehensive introduction to vision-language modeling, highlighting the convergence of visual and textual understanding in AI.

Each of these topics showcases the remarkable innovations and dynamic advancements in the AI landscape, promising an engaging and enlightening read. Dive in and enjoy the cutting-edge developments presented in this issue!

Best regards,

State of AI

---

Contents

1. Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

2. FIFO-Diffusion: Generating Infinite Videos from Text without Training

3. ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

4. Your Transformer is Secretly Linear

5. An Introduction to Vision-Language Modeling

   ---

Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

Authors: Byung-Kwan Lee, Chae Won Kim, Beomchan Park, Yong Man Ro

Source and references: https://arxiv.org/abs/2405.15574v2

---

Introduction

Meet Meteor, your new LLVM (large language and vision model) buddy, designed to make strides in understanding and answering capabilities. This paper comes from the minds at KAIST and introduces Meteor—a clever model that handles lengthy and multifaceted rationales to significantly improve performance across various vision-language tasks. But what exactly makes Meteor stand out in the fast-evolving world of LLVMs? Let's dive in.

The Building Blocks

Traditionally, the development of LLVMs has been propelled by visual instruction tuning. Think of visual instruction tuning as feeding the model a diverse diet of images paired with rich textual descriptions and question-answer pairs, nourishing it with the ability to understand complex scenarios. The game's objective is to create LLVMs with the prowess of closed-source giants like GPT-4V or Qwen-VL-Plus, but via open-source pathways.

Many open-source models tried scaling their parameters sky-high or adding multiple vision encoders to bridge performance gaps. Meteor takes a detour from this approach, instead leveraging its 'Mamba' architecture to process rich, elaborate rationales efficiently. This technique allows it to excel without bloating model size or integrating additional vision encoders.

The Heart of Meteor: Mamba Architecture

So, what’s this Mamba architecture all about? Named presumably for its agility and efficiency, much like the venomous snake, Mamba can process sequential data with linear time complexity. This is crucial when dealing with multi-sentence rationales that provide deeper insights and step-by-step explanations required for solving complex questions or understanding diverse visual inputs.

Meteor Mamba (let's call it Meteor-Mamba for simplicity) is designed to embed lengthy rationales without breaking a sweat. Thanks to this architecture, Meteor can tackle multifaceted rationales and enrich the corresponding backbone multi-modal language model (MLM) with this wealth of embedded information.

Journey Through the Rationale

One standout feature of Meteor is the introduction of 'Traversal of Rationale'. Imagine a traveler navigating through a winding library filled with long-explanations for every image-question pair. Traversal of Rationale involves embedding these lengthy explanations into the Meteor model systematically. During training, sequences of these rationales are incorporated, enabling Meteor-MLM to generate well-informed answers.

What sets Meteor apart is this ability to handle long-form rationales efficiently, without requiring additional model parameters or external vision encoders during inference. It’s like giving Meteor an extensive encyclopedia instead of just a dictionary, allowing it to draw from a richer knowledge base.

Training with Rich Datasets

The authors gathered 2.1 million question-answer pairs from various meticulously curated visual instruction datasets. These datasets span across fundamental image understanding and broaden to include common-sense knowledge, charts, diagrams, symbols, signs, and even math problems. Incorporating such diverse information allows Meteor to excel in a plethora of tasks requiring different capabilities.

By leveraging Claude Haiku API and rigorous human review, they crafted detailed rationales for these question-answer pairs, ultimately yielding 1.1 million question-rationale-answer triples. This curation ordeal ensures the rationales bolster Meteor’s ability to tackle a wide range of vision-language queries.

Hands-On with Meteor

The Vision Encoder

Meteor employs CLIP-L/14 as its vision encoder. This component is like the visual cortex of the human brain, adept at drawing text-aligned interpretations from images. Alongside this, an MLP-based vision projector helps map these visual embeddings into a form digestible by the model.

The Rationale Processor

Enter the Mamba-130M architecture, notable for its efficiency in handling long sequences. Together with InternLM2-7B as the backbone LLM, it processes the richly embedded rationales. This ensures that each step of the complex reasoning is encoded seamlessly.

Traversal of Rationale Magic

During the training, the model embeds the rationales marked with special tokens (<tor>), enabling Meteor-MLM to incorporate this rich contextual knowledge directly. When it comes time to generate responses during inference, the enrichment provided by these rationales becomes evident in the enhanced quality of answers, even without external API calls.

Making Meteor Stand Out

The creators' most significant contribution with Meteor is proving that high-performance LLVMs don’t always need gargantuan model sizes or multitudinous vision encoders. Instead, by efficiently embedding multifaceted rationales, Meteor achieves stellar results on numerous benchmarks—be it common-sense reasoning, understanding non-object concepts, or complex mathematical problem-solving.

Embracing Multifaceted information

But it's more than just about answering questions correctly. Meteor’s training incorporated millions of data points not just to understand images but to build a semantic web of knowledge. Incorporating explanations, it learns to approach problems systematically, much like how a human might reason through multiple steps to arrive at a conclusion.

Conclusion: The Future with Meteor

Meteor’s multifaceted approach presents a compelling blueprint for future LLVMs. By emphasizing quality over sheer size and integrating rich, detailed rationales, this model exemplifies how efficiency can go hand in hand with performance. Its results advocate a shift towards smarter, more informative embeddings in model design, opening pathways to sophisticated yet accessible AI solutions.

This journey through Meteor provides insights not just into the technical craft of building LLVMs but hints at broader implications. Imagine conversational agents that not only understand but can explain, teach, and reason like never before. With advancements like Meteor, we’re stepping ever closer to this reality—balanced on the sophisticated dance between language, vision, and rational thought.

Looking forward, the future models can imitate and even extend this rationale-embedding approach, propelling AI into new realms of capability. So keep an eye on Meteor—it’s navigating skies laden with rich, complex rationales and lighting up paths for future innovations.

---

FIFO-Diffusion: Generating Infinite Videos from Text without Training

Authors: Jihwan Kim, Junoh Kang, Jinyoung Choi, Bohyung Han [@ Computer Vision Laboratory, Seoul National University]

Source and references: https://arxiv.org/abs/2405.11473v1

---

Introduction

Imagine being able to generate an infinite video sequence from a single text prompt. Picture describing a firework display over Sydney Harbour or an astronaut walking on the moon and watching these scenes unfold seamlessly and endlessly. This is no longer just a futuristic idea. In the paper titled “FIFO-Diffusion: Generating Infinite Videos from Text without Training," researchers Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han introduce an innovative method that harnesses the power of a pretrained diffusion model to generate unending videos while preserving high visual quality and scene dynamics. No extensive training required. Intrigued? Read on to discover how it works.