Gemini 1.5, Hacker Agents, Promptless CoT, Linear Transformers & 10M Haystack

Week 3, February 2024

Greetings,

Get ready to explore cutting-edge developments in artificial intelligence with the 46th edition of State of AI. This time, we dive into Gemini 1.5's innovative features, autonomous hacking abilities of LLM agents, Chain-of-Thought reasoning, recurrent memory prowess in finding hidden gems, and linear transformer breakthroughs—all designed to expand your knowledge and spark curiosity.

Enjoy the journey through AI innovation!

Best regards,

State of AI

---

Contents

1. Gemini 1.5

2. LLM Agents can Autonomously Hack Websites

3. Chain-of-Thought Reasoning Without Prompting

4. In Search of Needles in a 10M Haystack:
   Recurrent Memory Finds What LLMs Miss

5. Linear Transformers with Learnable Kernel Functions are Better
   In-Context Models

---

Unlocking Multimodal Understanding Across Millions of Tokens of Context with Gemini 1.5 Pro

Authors: Gemini Team at Google

Source and references: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf

---

Introduction

Welcome to a fascinating journey through the world of cutting-edge Machine Learning research! Tech enthusiasts and followers, brace yourselves as we explore Google's latest breakthrough: "Gemini 1.5 Pro", a highly compute-efficient multimodal model capable of processing millions of tokens of context, including text, video, and audio data.

In simple terms, this incredible model can recall and reason over massive amounts of information from different types of data sources. It's a major leap forward in the AI landscape that not only achieves near-perfect recall on long-context retrieval tasks but also improves the state-of-the-art in various applications like long-document question answering, video question answering, and long-context Automatic Speech Recognition.

Breaking the Limits

The main highlight of Gemini 1.5 Pro is its unprecedented ability to work with extremely long contexts - we're talking about 10 million tokens! This scale is quite impressive compared to the current leading language models, allowing the processing of entire collections of documents, multiple hours of video, and nearly a full day's worth of audio recordings.

With the ability to handle such large-scale inputs, Gemini 1.5 Pro brings an exciting range of potential applications and novel capabilities to the table.

Measuring Performance

So, how do we evaluate this model's long-context capabilities?

Google researchers conducted a comprehensive set of experiments to test the model on synthetic and real-world tasks. The results show that Gemini 1.5 Pro excels at "needle-in-a-haystack" tasks, obtaining near-perfect recall up to millions of tokens of context in different modalities like text, video, and audio.

In more realistic multimodal scenarios that require retrieval and reasoning, Gemini 1.5 Pro outperforms other models across all modalities, even those boosted with external retrieval methods!

Showcasing Novel Capabilities

To give you a sense of how powerful this model is, let's look at some intriguing examples across text, images, video, and code:

• Given an entire large codebase, Gemini 1.5 Pro can correctly identify the specific location of core automatic differentiation methods.

• When provided with an entire reference grammar book and a bilingual wordlist, the model demonstrates an impressive ability to translate from English to Kalamang (a language spoken by fewer than 200 people) on par with a human who learned from the same materials.

• With the complete text of "Les Misérables" in context, the model can accurately identify and locate a famous scene from a hand-drawn sketch.

• Gemini 1.5 Pro can watch a 45-minute movie and answer specific questions about it, finding precise timestamps and details down to the second.

These examples give us a glimpse into the promising potential of Gemini 1.5 Pro in a wide range of applications and settings.

Diagnostic Evaluation Results

To thoroughly assess the long-context abilities of Gemini 1.5 Pro, researchers performed a series of diagnostic-focused probing and realistic evaluations. Here, we summarize the key findings:

1. Perplexity analysis indicates that the model continues to improve in predictive performance as context length increases, up to 10 million tokens.

2. In needle-in-a-haystack retrieval tasks, Gemini 1.5 Pro achieves near-perfect recall at context lengths up to 10 million tokens, outperforming the competition by a large margin.

3. Real-world benchmarks, like long-document question-answering, demonstrate the model's superiority in retrieving and reasoning over multiple parts of long-file contexts.

4. Surprisingly, the model can learn in-context from entire long documents, showcasing its potential in learning new languages, software codebases, and more.

Core Capabilities

Gemini 1.5 Pro doesn't just excel in long-context tasks; it also maintains exceptional performance in other core multi-modality capabilities, such as Math, Science, Multilinguality, Video Understanding, and Code. In short, this model outperforms its predecessors (Gemini 1.0 Pro and 1.0 Ultra) on most benchmarks, breaking new ground in the realm of multimodal understanding.

Responsible Deployment

As AI advances and models become more capable, addressing potential risks and advocating for responsible deployment becomes increasingly important.

Google's Gemini team is committed to evaluating potential long-context implications, impact assessments, and harm mitigations to ensure safe, ethical, and responsible use of their models.

Conclusion

Gemini 1.5 Pro is an exceptional leap forward in the AI and Machine Learning landscape. The model's remarkable ability to process millions of tokens of context across different modalities is a game changer, surpassing state-of-the-art performance in various long-context tasks.

With its ground-breaking capabilities, this model is set to transform the way we interact with and make use of large-scale, complex information. The burgeoning tech community should be eagerly looking forward to the myriad of applications and advances that Gemini 1.5 Pro's innovative approach will bring to the field.

---

LLM Agents can Autonomously Hack Websites

Authors: Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Dainel Kang

Source and references: https://arxiv.org/abs/2402.06664v1

---

Introduction

Welcome to the exciting world of large language models (LLMs) and their ever-evolving potential to blend into today's digital ecosystem. In this particular research, the authors explore the offensive side of LLM agents and demonstrate how they can autonomously hack websites with an impressive success rate. Stick around, and let's dive into this fascinating study.