Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

State of AI Pod • 015

Contents

1. Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

2. A Survey on Knowledge Organization Systems of Research Fields: Resources and Challenges

3. From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap

4. FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registers

5. 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

6. Return of the Encoder: Maximizing Parameter Efficiency for SLMs

7. Enhancing Brain Age Estimation with a Multimodal 3D CNN Approach Combining Structural MRI and AI-Synthesized Cerebral Blood Volume Data

8. Contextual Feedback Loops: Amplifying Deep Reasoning with Iterative Top-Down Feedback

9. Gaussian entropic optimal transport: Schrödinger bridges and the Sinkhorn algorithm

10. Selective Generation for Controllable Language Models

11. RAPID: Retrieval-Augmented Parallel Inference Drafting for Text-Based Video Event Retrieval

12. Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width

13. AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses

14. MIND-EEG: Multi-granularity Integration Network with Discrete Codebook for EEG-based Emotion Recognition

15. TelePreview: A User-Friendly Teleoperation System with Virtual Arm Assistance for Enhanced Effectiveness

Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

Authors: Amirreza Esmaeili, Iman Saberi, Fatemeh H. Fard

Source and references: https://arxiv.org/abs/2405.01553v2

---

Introduction

This paper evaluates the effectiveness of Parameter Efficient Fine-Tuning (PEFT) methods, including LoRA, Compacter, and IA3, when applied to Large Language Models (LLMs) for code summarization and generation tasks, with a focus on knowledge transfer to the unseen programming language R.

Key Points

• The study examines the performance of PEFT methods on code-specific LLMs (CodeT5 and CodeLlama) and general-purpose LLMs (T5 and Llama2).

• It assesses the capability of PEFT methods in transferring knowledge from natural language to code, and from code to the unseen language R.

• The research compares the resource efficiency and robustness of the PEFT methods, and analyzes the impact of the trainable parameter budget on their performance.

• The study includes qualitative analyses of the generated code summaries and styling quality.

Methodology

The authors conduct extensive experiments to evaluate the PEFT methods on code summarization and generation tasks, using both code-specific and general-purpose LLMs. They compare the performance of the PEFT methods to their fully fine-tuned counterparts and assess their effectiveness in transferring knowledge to the unseen language R.

Results and Findings

The results show that LoRA consistently outperforms Compacter and IA3 across all settings, while Compacter offers significant resource efficiency with minimal performance trade-offs. The number of trainable parameters has a greater influence on the functional accuracy of the generated code than the PEFT architecture. The qualitative analyses demonstrate that both LoRA and Compacter can generate code summaries and styling quality that are close to the ground truth.

Implications and Conclusions

The findings of this study can guide future research in developing code intelligence tasks for unseen languages, including R, and the choice of PEFT methods for knowledge transfer, especially when balancing computational cost and performance. The focus on the low-resource language of R can help in automating software engineering tasks for a wider community of developers.

---

A Survey on Knowledge Organization Systems of Research Fields: Resources and Challenges

Authors: Angelo Salatino, Tanay Aggarwal, Andrea Mannocci, Francesco Osborne, Enrico Motta

Source and references: https://arxiv.org/abs/2409.04432v2