Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Contents

1. TRANSAGENT: An LLM-Based Multi-Agent System for Code Translation

2. FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model

3. CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation

4. FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

5. Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space

6. Gemma 2: Improving Open Language Models at a Practical Size

7. DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

8. Grounded Answers for Multi-agent Decision-making Problem through Generative World Model

9. Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

10. Preble: Efficient Distributed Prompt Scheduling for LLM Serving

11. SIEVE: General Purpose Data Filtering System Matching GPT-4o Accuracy at 1% the Cost

12. Grounding Large Language Models In Embodied Environment With Imperfect World Models

---

TRANSAGENT: An LLM-Based Multi-Agent System for Code Translation

Authors: Zhiqiang Yuan, Weitong Chen, Hanlin Wang, Kai Yu, Xin Peng, Yiling Lou

Source and references: https://arxiv.org/abs/2409.19894v2

---

Introduction

This paper proposes TRANS AGENT, an LLM-based multi-agent system that enhances LLM-based code translation by fixing both syntax and semantic errors.

Key Points

• TRANS AGENT includes four LLM-based agents: Initial Code Translator, Syntax Error Fixer, Code Aligner, and Semantic Error Fixer.

• The key insight is to first localize the error code block in the target program based on the execution alignment between the target and source program, which can narrow down the fixing space.

• TRANS AGENT outperforms the latest LLM-based code translation technique UniTrans in both translation effectiveness and efficiency.

• The ablation study shows the contribution of each agent in TRANS AGENT and the complementarity between different fixing strategies.

• TRANS AGENT demonstrates generalization across different LLMs.

Methodology

TRANS AGENT is designed as a multi-agent system with four different LLM-based agents. The Initial Code Translator generates an initial version of the target program. The Syntax Error Fixer iteratively fixes syntax errors based on compilation or interpreting error messages. The Code Aligner maps semantically-equivalent code elements between the source and target programs. The Semantic Error Fixer localizes and fixes the error blocks in the target program by comparing runtime behaviors.

Results and Findings

TRANS AGENT outperforms state-of-the-art transpilers like TransCoder and UniTrans across various translation tasks. The ablation study shows that both Syntax Error Fixer and Semantic Error Fixer in TRANS AGENT substantially enhance translation performance, and their repair strategies are more effective than those used in UniTrans. The block-level code mapping approach in Code Aligner achieves better alignment than the purely LLM-based mapping in TransMap. TRANS AGENT also demonstrates generalization across different LLMs.

Implications and Conclusions

This work proposes a novel LLM-based multi-agent system, TRANS AGENT, that enhances LLM-based code translation by effectively fixing both syntax and semantic errors. The key insights, including the fine-grained error localization and the synergy of program analysis and LLMs, can be valuable for future research in LLM-based software engineering tasks.

---

FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model

Authors: Zidong Wang, Zeyu Lu, Di Huang, Cai Zhou, Wanli Ouyang, Lei Bai

Source and references: https://arxiv.org/abs/2402.12376v2

---

Introduction

This paper introduces the Flexible Vision Transformer (FiT), a novel architecture designed for generating images with unrestricted resolutions and aspect ratios. The key motivation is to conceptualize images as sequences of variable-length tokens, departing from the traditional perspective of static grids with fixed dimensions.