State of AI

Week 3, May 2023

Greetings,

Welcome to the seventh edition of the State of AI newsletter, featuring an exciting selection of cutting-edge research in AI and Machine Learning. This week, we delve into the fascinating realms of internet language models, machine learning advancements, generative adversarial networks, and the fusion of large language and world models.

We kick off with an exploration of language models' intriguing applications, focusing on the challenging aspects of the internet. Next, we provide a glimpse into the current technical advances reshaping the machine learning landscape. Our journey then shifts to the creative sphere of image generation and manipulation, highlighting AI's footprint in digital art.

Lastly, we delve into the intersection of large language and world models, discussing how AI's embodied experiences can enhance these models. This edition offers a snapshot of AI's versatile applications and future potentials. Enjoy the journey into these captivating domains of AI research.

Best regards,

State of AI

---

Contents

1. DarkBERT: A Language Model for the Dark Side of the Internet

2. PaLM 2: Technical Report

3. Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

4. Tree of Thoughts: Deliberate Problem Solving with Large Language Models

5. Language Models Meet World Models: Embodied Experiences Enhance Language Models

---

DarkBERT: A Language Model for the Dark Side of the Internet

Authors: Youngjin Jin, Eugene Jang, Jian Cui, Jin-Woo Chung, Yongjae Lee, Seungwon Shin

Source & References: https://arxiv.org/abs/2305.08596v2

---

Introduction

The Dark Web is a mysterious realm, a region of cyberspace only accessible via special software and rife with illicit activities. Given the unique linguistic characteristics in the Dark Web, natural language processing (NLP) models tailored for this domain are necessary to unravel its secrets. Enter DarkBERT – a powerful new language model that surpasses its counterparts in various tasks, such as Dark Web activity classification.

In this summary, we will dissect the construction of DarkBERT, show how it excels in identifying Dark Web activities, and explore its potential use cases in cybersecurity research. So grab a cup of coffee, strap in, and let's dive into the enigmatic world of DarkBERT!

Constructing DarkBERT

DarkBERT's construction begins with an essential first step: gathering text from Dark Web pages. Researchers collected a vast amount of text, ensuring that only English language data is included. The data underwent a thorough filtering process to remove pages with low information density, balance categories, and eliminate duplicates. Additionally, precautions were taken to exclude sensitive information, laying the groundwork for a solid foundation.

With data in hand, researchers pretrained two versions of DarkBERT on raw and preprocessed text using RoBERTa – a model chosen for its performance advantages over BERT. This pretrained model became the backbone of DarkBERT, paving the way for its prowess in understanding the unique language of the Dark Web.

Dark Web Activity Classification

To put DarkBERT to the test, researchers utilized two popular datasets – DUTA and CoDA. These datasets, containing various Dark Web activities, served as benchmarks to evaluate DarkBERT's performance against other language models, such as BERT and RoBERTa. Employing separate classifiers for each dataset, the team examined the models' performance on cased and uncased versions of the datasets.

And the verdict? DarkBERT outperformed its competitors across all tests. While the differences in language between the Surface and Dark Web played a role in the performance gap, it's evident that DarkBERT's prowess in representing the unique language of the Dark Web secures its position as a valuable tool in ongoing research efforts.

DarkBERT's Use Cases in Cybersecurity

Armed with its newfound power, DarkBERT shows immense potential in various cybersecurity applications. Let's explore three intriguing use cases that demonstrate its practical applications and advantages over existing models.

Ransomware Leak Site Detection

On the Dark Web, ransomware leak sites reveal private, confidential data of organizations, often demanding hefty ransoms. Detecting these sites is crucial in addressing potential threats. In this use case, DarkBERT shines by outpacing BERT and RoBERTa in detecting ransomware leak sites, proving its worth as a valuable tool in cybersecurity research.

Noteworthy Thread Detection

Researchers tested DarkBERT for its ability to detect threads containing malicious content or discussions. Once again, DarkBERT flexes its muscles, proving superior in detecting a higher number of noteworthy threads compared to other language models. With this competence, DarkBERT can help cybersecurity experts keep up to speed on the latest digital threats brewing in online forums.

Threat Keyword Inference

But DarkBERT's capabilities don't stop there! When it comes to inferring threat keywords from texts to understand threat types, DarkBERT demonstrated impressive performance. This skill can help cybersecurity researchers pinpoint and analyze specific threats lurking in the Dark Web.

Conclusion

DarkBERT is a groundbreaking model that sheds light on the elusive world of the Dark Web. Through its powerful performance in Dark Web activity classification and its potential use cases in cybersecurity research, researchers have unlocked a treasure trove of insights into this hidden domain.

As we venture further into the enigmatic landscape of the Dark Web, it's essential to remember that DarkBERT represents a beacon of knowledge and potential, illuminating a world that has traditionally remained shrouded in darkness. By harnessing the power of DarkBERT, cybersecurity researchers can now brave the depths of the Dark Web and uncover the truth behind the web's most hidden secrets.

So the next time you scratch your head, wondering about the mysteries of the Dark Web, remember that there's one powerful ally on your side: DarkBERT, an unsung hero making sense of the chaos beneath the surface of cyberspace.

---

Introducing PaLM 2: A More Efficient and Robust Multilingual Language Model

Authors: Google AI Language Team

Source & References: https://arxiv.org/abs/2305.10403v1

---

Meet the new state-of-the-art language model, PaLM 2, which boasts better multilingual capabilities, faster and more efficient production, and robust reasoning skills than its predecessor, PaLM. In this summary, we'll explore what makes PaLM 2 stand out, its architecture, and its performance on various language and reasoning tasks.