Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Contents

1. Instruction-Driven Game Engines on Large Language Models

2. How Diffusion Models Learn to Factorize and Compose

3. Enhancing Training Efficiency Using Packing with Flash Attention

4. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

5. Solving Robotics Problems in Zero-Shot with Vision-Language Models

6. Celtibero: Robust Layered Aggregation for Federated Learning

7. Reprogramming Foundational Large Language Models(LLMs) for Enterprise Adoption for Spatio-Temporal Forecasting Applications: Unveiling a New Era in Copilot-Guided Cross-Modal Time Series Representation Learning

8. LLM Pruning and Distillation in Practice: The Minitron Approach

9. Non-discrimination Criteria for Generative Language Models

10. PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods

11. GenRec: Unifying Video Generation and Recognition with Diffusion Models

12. Generative Verifiers: Reward Modeling as Next-Token Prediction

13. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

14. Can Unconfident LLM Annotations Be Used for Confident Conclusions?

Instruction-Driven Game Engines on Large Language Models

Authors: Hongqiu Wu, Yan Wang, Xingyuan Liu, Hai Zhao, Min Zhang

Source and references: https://arxiv.org/abs/2404.00276v4

---

Introduction

This research paper introduces the Instruction-Driven Game Engine (IDGE) project, which aims to democratize game development by enabling large language models (LLMs) to follow free-form game rules and autonomously generate game-play processes.

Key Points

• The IDGE allows users to create games by issuing simple natural language instructions, significantly lowering the barrier for game development.

• The learning process for IDGEs is approached as a Next State Prediction task, where the model autoregressively predicts in-game states given player actions.

• This is a challenging task because the computation of in-game states must be precise, as slight errors could disrupt the game-play.

• The researchers address this challenge by training the IDGE in a curriculum manner that progressively increases the model's exposure to complex scenarios.

• The initial progress of the IDGE project is in developing an engine for the universally cherished card game of Poker.

Methodology

The researchers train the IDGE model using a curriculum learning approach, where the model's exposure to complex scenarios is gradually increased to ensure precise computation of in-game states and robust game-play.

Results and Findings

The IDGE designed for Poker supports a wide range of poker variants and allows for high customization of rules through natural language inputs. Additionally, the engine favors rapid prototyping of new games from minimal samples, proposing an innovative paradigm in game development that relies on large language models.

Implications and Conclusions

The IDGE project represents a significant advancement in democratizing game development by leveraging the capabilities of large language models to enable users to create games through simple natural language instructions, potentially transforming the game development landscape.

---

How Diffusion Models Learn to Factorize and Compose

Authors: Qiyao Liang, Ziming Liu, Mitchell Ostrow, Ila Fiete

Source and references: https://arxiv.org/abs/2408.13256v1

---

Introduction

This paper examines how diffusion models learn to factorize and compose features in their internal representations, and how this impacts their ability to generalize beyond the training distribution.