Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Contents

1. Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits

2. Wolf: Captioning Everything with a World Summarization Framework

3. Small Molecule Optimization with Large Language Models

4. Towards Effective and Efficient Continual Pre-training of Large Language Models

5. MUVO: A Multimodal World Model with Spatial Representations for Autonomous Driving

6. ByteCheckpoint: A Unified Checkpointing System for LLM Development

7. Matryoshka Multimodal Models

8. Long-form music generation with latent diffusion

9. AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLM

10. Theia: Distilling Diverse Vision Foundation Models for Robot Learning

11. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments

12. CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning

13. MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning

14. Fast Multipole Attention: A Divide-and-Conquer Attention Mechanism for Long Sequences

15. From Feature Importance to Natural Language Explanations Using LLMs with RAG

---

Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits

Authors: Jimin Mun, Liwei Jiang, Jenny Liang, Inyoung Cheong, Nicole DeCario, Yejin Choi, Tadayoshi Kohno, Maarten Sap

Source and references: https://arxiv.org/abs/2403.14791v2

---

Introduction

This paper introduces Particip-AI, a framework for lay users to speculate and assess AI use cases and their impacts. The framework allows for collecting diverse public opinions on AI through a four-step process: brainstorming use cases, rating harms and benefits under alternate scenarios (developing and not developing), and making a choice on the technology's development.

Key Points

• Particip-AI is a novel framework to assess AI use cases and their impact, developed with insights from AI, computer security, and philosophy.

• A survey with 295 diverse participants showcases the usability of Particip-AI and provides insights into public perceptions of AI.

• The results surface a wide array of anticipated use cases, highlighting common themes of interest in improving personal, everyday life and making societal impact.

• Participants surface a set of harms complementary to expert-generated taxonomies, such as issues of distrust in institutions and the need for regulation to protect mental health.

• The findings uncover benefits and harms associated with not developing AI, and highlight tensions around the value of human potential.

• Participants' responses on harms and benefits of not developing significantly predict their judgements on whether AI use cases should be developed.

Methodology

The authors designed the Particip-AI framework to effectively elicit lay users' opinions on potential harms and benefits across near-future and far-future AI applications. They recruited 295 demographically diverse participants in the US to complete a survey based on the Particip-AI framework. The survey prompted participants to imagine potential use cases, rate harms and benefits under alternate scenarios, and make a choice on the technology's development. The authors conducted a mixture of quantitative and qualitative analyses on the survey responses.

Results and Findings

The results show that participants' responses emphasize applications for personal life and society, contrasting with the business focus of most current AI development. Participants also surface diverse harms such as distrust in AI and institutions, complementing expert-defined taxonomies. The findings uncover benefits and harms associated with not developing AI, and highlight tensions around the value of human potential. Importantly, the authors find that perceived impact of not developing use cases significantly predicted participants' judgements on whether AI use cases should be developed.

Implications and Conclusions

This work highlights the promise and importance of including lay publics and diverse voices into the future of AI design and governance. The Particip-AI framework can further guide democratic AI development and governance by facilitating public assessment of potential real-world AI applications across domains. The results underscore the need to reflect diverse goals and needs, address regulatory gaps around intangible harms, and navigate tensions over the value of human work in the face of AI progress.

---

Wolf: Captioning Everything with a World Summarization Framework

Authors: Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Andrew Tao, Ming-Yu Liu, Sanja Fidler, Boris Ivanovic, Trevor Darrell, Jitendra Malik, Song Han, Marco Pavone

Source and references: https://arxiv.org/abs/2407.18908v1

---

Introduction

This paper proposes Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs).