State of AI

Week 3, August 2023

Greetings,

Welcome to the 19th edition of the State of AI. In this issue, we embark on a thrilling exploration of the mind with image reconstruction from human brain signals, providing a novel view of visual perception. Journey further into the cutting-edge territory of real-time 3D field rendering, where Gaussian splatting elevates visual experiences to new heights.

Our exploration continues with "Shepherd," an innovative critic designed to guide language model generation, shaping the future of AI communication. Delve into the vibrant world of conversation with PIPPA, a groundbreaking dataset that opens doors to more human-like interactions. Finally, we present JEN-1, an advanced model for universal music generation, where text guides a symphony of sounds in mesmerizing harmony.

Each of these compelling subjects showcases the frontiers of AI research, promising an invigorating and enlightening read. Dive in and discover the exciting vistas of contemporary artificial intelligence!

Best regards,

State of AI

---

Contents

1. Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

2. 3D Gaussian Splatting for Real-Time Radiance Field Rendering

3. Shepherd: A Critic for Language Model Generation

4. PIPPA: A Partially Synthetic Conversational Dataset

5. JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models

---

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

Authors: Yu-Ting Lan, Kan Ren, Yansen Wang, Wei-Long Zheng, Dongsheng Li, Bao-Liang Lu, Lili Qiu

Source & References: https://arxiv.org/abs/2308.02510

---

Introduction

In a groundbreaking new study, a group of researchers from Shanghai Jiao Tong University and Microsoft Research have developed a method called NEUROIMAGEN that successfully reconstructs images based on visual perception data obtained from human brain signals. This exciting development provides valuable insights into the complex relationship between human visual perception and cognitive processes, with potentially far-reaching implications for various fields such as cognitive computing and neuroscience.

The Challenge of Visual Perception Reconstruction

Reconstructing visual stimuli from brain signals, such as electroencephalography (EEG) data, poses a significant challenge. One reason is that EEG signals are highly dynamic, noisy, and recorded in a time-series format, making it difficult to extract precise and relevant information. Another is that the relatively low spatial resolution of EEG data compared to functional magnetic resonance imaging (fMRI) data makes the task of image reconstruction even more arduous.

NEUROIMAGEN: A Comprehensive Pipeline

To tackle these challenges, the researchers developed a comprehensive pipeline called NEUROIMAGEN, which extracts multi-level semantics from EEG signals and then uses them to reconstruct the corresponding visual stimuli. The pipeline consists of two parts: a pixel-level semantics extractor that decodes fine-grained information like color and shape of the visual stimuli and a sample-level semantics extractor that retrieves coarse-grained information such as image categories or text captions. The extracted semantics are then fed into a pretrained latent diffusion model, which ultimately reconstructs the observed visual stimuli.

Pixel-Level Semantics Extraction

The first step in the NEUROIMAGEN pipeline is extracting pixel-level semantics – fine-grained information capturing the color, position, and shape details of the observed visual stimuli. This involves two components: contrastive feature learning, which obtains discriminative features of EEG signals, and estimation of the saliency map based on the learned features. The generated saliency map successfully captures the rough structure information of visual stimuli from noisy EEG signals, albeit with limited semantic accuracy and low image resolution.

Sample-Level Semantics Extraction

In addition to fine-grained pixel-level semantics, the researchers also leverage sample-level semantics extraction to derive coarse-grained information from EEG signals. This level of extraction focuses on aligning the information decoded from the input EEG signals with image captions generated by models like the Contrastive Language-Image Pretraining (CLIP) model, for example. Combining these two levels of semantic extraction yields a more robust and meaningful reconstruction of the observed visual stimuli.

Image Reconstruction Process

The final step of the NEUROIMAGEN pipeline involves integrating the coarse-grained and fine-grained semantics with a pretrained latent diffusion model to reconstruct the observed visual stimuli from EEG signals. Starting with the saliency map as the initial image, the sample-level semantics are used to refine the saliency map and ultimately reconstruct the final image. This approach allows for a high degree of similarity between the original visual stimuli and the reconstructed image, even in the presence of inherent noise in EEG signals.

Superior Performance

The NEUROIMAGEN method was evaluated against traditional image reconstruction solutions on an EEG-image dataset, demonstrating the superior quantitative and qualitative performance of this new approach. By effectively incorporating and leveraging multi-level semantics extraction, the NEUROIMAGEN pipeline is better able to navigate the challenges posed by noisy and complex EEG data, ultimately producing reconstructed images with a high degree of structural similarity and semantic accuracy.

Implications and Future Research

The successful development of the NEUROIMAGEN pipeline represents a significant leap forward in our understanding of the complex relationship between human visual perception and cognitive processes. The method's ability to reconstruct images from EEG data has a wide range of potential applications in cognitive computing, neuroscience, and other fields.

As this research continues to advance, it will be interesting to see how future iterations of the NEUROIMAGEN pipeline could improve image reconstruction accuracy and tackle new challenges related to decoding and analyzing EEG signals. With the increasing prevalence of machine learning and artificial intelligence technologies, the possibilities for understanding and leveraging human brain signals for various practical applications are virtually limitless.

---

3D Gaussian Splatting for Real-Time Radiance Field Rendering

Authors: Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis

Source & References: https://arxiv.org/abs/2308.04079

---

Introduction

This research paper presents a new method of rendering radiance fields for real-time novel-view synthesis at high-resolution (1080p) using 3D Gaussian scene representations. The authors demonstrate that their approach achieves state-of-the-art visual quality, competitive training times, and notably, real-time rendering speeds. The method combines the benefits of point-based rendering and continuous volumetric radiance field representations, providing a flexible and efficient solution for rendering complex scenes.