Bi-Weekly AI Research Roundup

Latest research summaries in ML, Robotics, CV, NLP and AI

Due to a technical issue, there have been no publications during the past few weeks, but we're back on track and I'm excited to share with you the latest in frontier AI research in today's State of AI issue!

Grab this 30 day State of AI Premium and enjoy upcoming issues free of charge!

Contents

1. Natural Language to Verilog: Design of a Recurrent Spiking Neural Network using Large Language Models and ChatGPT

2. Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation

3. Interpretable Graph Neural Networks for Heterogeneous Tabular Data

4. VIRUS-NeRF -- Vision, InfraRed and UltraSonic based Neural Radiance Fields

5. Agent Instructs Large Language Models to be General Zero-Shot Reasoners

6. LLM4DSR: Leveraing Large Language Model for Denoising Sequential Recommendation

7. Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2

8. BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

9. P/D-Serve: Serving Disaggregated Large Language Model at Scale

10. Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models

11. Entendre, a Social Bot Detection Tool for Niche, Fringe, and Extreme Social Media

12. EXAONE 3.0 7.8B Instruction Tuned Language Model

13. Transformers Can Do Bayesian Inference

14. Imagen 3

15. Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models

---

Natural Language to Verilog: Design of a Recurrent Spiking Neural Network using Large Language Models and ChatGPT

Authors: Paola Vitolo, George Psaltakis, Michael Tomlinson, Gian Domenico Licciardo, Andreas G. Andreou

Source and references: https://arxiv.org/abs/2405.01419v2

Introduction

This paper investigates the use of Large Language Models (LLMs) and natural language prompts to generate hardware description code, namely Verilog. The authors employ OpenAI's ChatGPT4 and natural language prompts to synthesize an RTL Verilog module of a programmable recurrent spiking neural network, while also generating test benches to assess the system's correctness.

Key Points

• The authors demonstrate the use of natural language prompts and LLMs to generate hardware description code, namely Verilog, for a more complex hardware design architecture, a 3 layer, 3 neuron per layer Recurrent Spiking Neuron Network (RSNN).

• They show how to use natural language prompts and LLMs to support test-bed development and the verification process.

• The LLM generated Verilog code was validated on a Field-Programmable Gate Array (FPGA) and implemented in the SkyWater 130 nm technology, utilizing an open source EDA tool.

• The design was submitted to Efabless Tiny Tapeout 6 for fabrication.

• Three simple case studies of AI/ML tasks were employed to validate the design: exclusive OR, IRIS flower classification, and MNIST handwritten digit classification.

Methodology

The authors employ OpenAI's ChatGPT-4 and follow a modular approach with a bottom-up design methodology. They decompose the architecture of the complex RSNN system into a hierarchy of smaller, reusable submodules, which leads to speeding up the development process, improving manageability, scalability, reusability and facilitating early error detection.

Results and Findings

ChatGPT successfully generated the Verilog hardware description code for each module after a total of 117 iterations. The LIF Neuron module and RLIF Layer proved to be the most challenging, requiring 38 and 17 iterations respectively, across 2 separate conversations. The findings highlight the importance of a modular system design and the need for clear and well-defined requirements, supplemented by practical examples.

The FPGA prototyping results show an utilization of 1011 LUTs and 507 FFs, with a maximum allowed system clock frequency of 83 MHz. The power analysis revealed a total power consumption of 65 mW, of which 4 mW is dynamic power and 61 mW static power.

The ASIC implementation in SkyWater 130 nm technology shows an area occupation of 0.11 mm2 and 5187 total cells, distributed across various logic elements.

Implications and Conclusions

The findings of this research demonstrate the robustness of the hardware design assisted by ChatGPT and highlight its potential as a useful tool in the hardware design process. The successful validation of the design on FPGA and its subsequent synthesis in the SkyWater 130 nm technology using an open-source electronic design automation flow, as well as the submission to Efabless Tiny Tapeout 6, showcase the capabilities of LLMs in accelerating and streamlining the hardware design process.

---

Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation

Authors: Zhengyuan Yang, Jianfeng Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang

Source and references: https://arxiv.org/abs/2310.08541v2

Introduction

This paper introduces "Idea to Image" (Idea2Img), an agent system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation. The system aims to mimic human-like exploration to efficiently convert high-level generation ideas into effective text-to-image (T2I) prompts that can produce good images.