Graph-Based Agentic AI, Adaptive Speculative Decoding, and Physics-Based Digital Twins
This edition showcases a fascinating convergence of three major trends reshaping AI systems: the infrastructure and governance challenges of building reliable, auditable agentic workflows; the practical engineering required to make inference faster and more cost-efficient at scale; and the mechanics of grounding AI capabilities in the physical world through simulation and embodied reasoning. We‘re also seeing exciting progress in multimodal reasoning—from theory-of-mind in meetings to adaptive navigation—alongside deep dives into mechanistic interpretability, geometric deep learning, and novel optimization techniques for reasoning-heavy tasks.
Here‘s what caught our attention:
Graph-Based Agentic AI with LangGraph — A practitioner’s guide clarifying when stateful workflow orchestration is justified, complete with decision tables comparing it to simpler alternatives and three executable recipes for SQL repair, RAG, and human-in-the-loop processes.
CodeRescue: Budget-Calibrated Recovery Routing — Rather than defaulting to expensive models after failures, this work routes coding agents through three recovery actions (reflect, replan, escalate) using a learned policy calibrated to arbitrary budget constraints without retraining.
OmniReasoner: Thinking with Long Audio-Video — Teaches omnimodal models to strategically zoom into relevant video intervals using absolute timestamps and tool-use learning, with gains scaling dramatically on 10–30 minute videos.
Off-Context GRPO: Learning from Privileged Guidance — Solves the “learning cliff” where hard problems produce zero reward signal by correcting the distribution mismatch between training (guided) and deployment (unguided) conditions through importance weighting.
ARMOR: Stabilizing On-Policy LLM RL — Addresses over-optimization in reasoning RL by actively injecting off-policy correct samples as anchors rather than relying on passive KL penalties, achieving +5–11 point improvements on AIME.
Agentic Real2Sim: Physics-Based Digital Twin Creation — Automates real robot video-to-simulation conversion using vision-language agents for orchestration, working across rigid, deformable, and humanoid domains at <$3 per episode with open-weight models.
CircuitKIT: Mechanistic Interpretability Toolkit — Unifies circuit discovery, evaluation, and intervention through thirteen algorithms and six complementary diagnostic pillars, revealing that single-metric faithfulness scores reverse method rankings and fail to predict intervention outcomes.
MeetingToM: Theory-of-Mind in Multi-Party Meetings — A benchmark targeting pseudo-consensus—where participants verbally agree despite nonverbal dissent—with substantial gaps between human reasoning (75–86%) and state-of-the-art models (22–88%).
Let‘s get into it 👇
Bi-Weekly AI Research Roundup
Latest research summaries in ML, Robotics, CV, NLP and AI
Contents
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties
OmniReasoner: Thinking with Long Audio-Video via Native Tool Use
Appearance Pointers -- Multimodal Region Control of Diffusion Transformers
MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Provable diffusion-based posterior sampling for linear inverse problems via DDIM
CircuitKIT : Circuit Discovery, Evaluation, and Application Toolkit for Mechanistic Interpretability
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation
Graph-Based Agentic AI with LangGraph: Workflow Pathways for Long-Running Stateful Business Processes
Authors: Daniel Pearson, Sidney Shapiro, Emiliano Sebastian Gonzalez Venegas, Sanad Al-Khatib, Aurora Pinzón Arzola
Source and references: https://arxiv.org/abs/2607.19297v1
Introduction
This paper is a practitioner guide to LangGraph, a low-level orchestration framework for building long-running, stateful multi-step AI systems in business processes. Rather than positioning LangGraph as a universal solution, the authors present it as a specialized tool suited to specific workflow complexity requirements, with three executable recipes demonstrating when and how to use it effectively.
Key Points
LangGraph is a workflow-complexity fit, not a universal default: The framework excels for processes requiring durable state, human review gates, and audit trails, but simpler alternatives (plain SDK, schema-first tools, DSPy) are better for basic tool use, structured extraction, or prompt optimization.
Three core recipes address recurring business patterns: SQL analytics with repair loops, agentic retrieval-augmented generation (RAG) with evidence gating, and human-in-the-loop (HITL) policy review with interrupt and checkpoint recovery demonstrate how typed state, conditional routing, and deterministic tools work together.
Explicit governance and durability are first-class features: LangGraph makes workflow structure, route history, and state persistence transparent in the product contract rather than hidden in prompt logic, enabling inspection, debugging, and compliance auditing.
Decision criteria determine when LangGraph justifies the overhead: Use LangGraph when workflows require pause-and-resume capability, next steps depend on explicit state branches, failures need repair paths, teams need route auditing, or multiple tool calls must share durable state.
Control plane primitives replace informal application logic: Typed shared state, graph nodes and edges, conditional routing, durable checkpoints, and interrupt/resume capabilities turn multi-step LLM applications into inspectable process graphs.
Methodology
The paper provides a decision framework rather than empirical benchmarks. The authors organize workflow requirements against LangGraph mechanisms and simpler alternatives through decision tables and comparative analysis. They present three concrete executable recipes that illustrate common business-process patterns: SQL repair workflows, evidence-aware RAG systems, and checkpointed human-review processes. Code patterns are explained to show how typed state, node boundaries, conditional routing, retries, interrupts, and checkpoints integrate into cohesive systems.
Results and Findings
The paper delivers a structured decision guide (Table 1) mapping six workflow requirements to LangGraph mechanisms and simpler alternatives. Key findings include: LangGraph is justified when processes must pause and resume (checkpointers and interrupts preserve state), when next steps depend on explicit state like risk level or validation status (conditional edges make routes inspectable), when failures require repair paths (retry budgets and error nodes enable controlled recovery), when auditability is required (node boundaries create traceable decision records), and when multiple model/tool calls coordinate through shared state (typed state objects keep artifacts explicit across nodes). The comparative analysis (Table 2) shows that plain ReAct-style loops suffice for simple tool use without checkpoints, schema-first systems excel at structured extraction and validation, and DSPy better addresses prompt optimization problems.
Implications and Conclusions
This research provides practical guidance for engineering teams building production AI systems by clarifying when infrastructure complexity is justified by operational requirements rather than treated as a default choice. The framework‘s emphasis on making workflow structure, governance, and audit trails explicit product behavior rather than hidden implementation details has significant implications for building reliable, maintainable, and compliant AI systems in regulated business environments.
CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
Authors: Qijia He, Jiayi Cheng, Chenqian Le, Rui Wang, Xunmei Liu, Yixian Chen, Jie Mei, Zhihao Wang, Xupeng Chen, Yuhuan Chen, Tao Wang
Source and references: https://arxiv.org/abs/2607.19338v1
Introduction
CodeRescue addresses a critical inefficiency in cost-aware LLM deployment for coding agents: when a cheap model fails to solve a programming problem, the system must decide not just which model to use next, but which recovery action to take. Rather than defaulting to an expensive stronger model, the paper introduces budget-calibrated routing that intelligently chooses between three recovery actions—reflecting on feedback, replanning from scratch, or escalating to a stronger model—based on execution diagnostics.
Key Points
Post-failure recovery routing: Coding failures produce actionable execution feedback (error messages, failed tests, stderr traces). The paper formulates recovery as routing over three distinct actions—reflect (repair using feedback), replan (fresh attempt with cheap model), and escalate (defer to stronger model)—rather than a binary cheap-vs-strong decision.
Complementary success patterns: Empirical analysis across five benchmarks reveals that cheap recovery and escalation are not strictly ordered. Some failures can only be solved by expensive escalation (45%), some only by cheap recovery (28%), and some can be solved by either (27%), meaning no single fixed action is optimal across all instances.
Budget-controllable deployment via Conformal Risk Control (CRC): A single trained router is converted into a family of budgeted operating points by adding a cost-penalty hyperparameter λ. CRC calibration maps user budgets to appropriate λ values without retraining, providing marginal expected-cost control under exchangeability.
Learned router outperforms baselines: A supervised router trained on recovery rollouts significantly outperforms fixed policies (always-escalate, always-replan) and zero-shot prompt-based routers, achieving 81.7% solve rate versus 68.6% for always-escalate at lower cost.
Practical cost-quality frontier: At a medium budget of 2.56 m$ per example, the CRC frontier reaches 71.7% solve rate, exceeding a binary cascade baseline and approaching always-escalate’s 68.6% solve rate while using only 35% of its cost.
Methodology
The paper collects recovery rollouts from five coding benchmarks (APPS, TACO, BigCodeBench, LiveCodeBench, CodeContests) by executing cheap-model attempts and recording which recovery actions successfully solve each failure. A supervised recovery router is trained on ~4,656 examples by fine-tuning Qwen3.5-4B to score the three recovery actions, labeled with the cheapest successful action for each instance. The router outputs normalized action probabilities via softmax over token log-likelihoods. To enable budget control, a cost-regularized policy π_λ(x) = argmax_a {s_θ(a|x) − λc(a,x)} shifts action selection toward cheaper options as λ increases. CRC calibration selects an appropriate λ on a held-out calibration set to satisfy a user-specified mean-cost budget B, then deploys this fixed λ on a disjoint test set.
Results and Findings
On held-out GPT-5.4-NANO/GPT-5.4 test splits, the unconstrained learned router achieves 81.7% solve rate at 5.51 m$ mean cost per example, substantially outperforming always-escalate (68.6% at 7.22 m$) and always-replan (45.3% at 1.59 m$). The CRC-calibrated frontier provides discrete operating points: at tight budgets (~1.43 m$), solve rate reaches 48.6%; at medium budgets (~2.56 m$), it reaches 71.7%; at loose budgets (~5.51 m$), it reaches 81.7%. Benchmark-level analysis shows substantial variance in recovery patterns—BigCodeBench is dominated by cheap-only recoveries (78% of failures), while TACO (very hard) shows escalation-only recoveries in 35% of cases. Router ablations confirm that metadata prefixes (source, difficulty tags) improve performance from 0.656 to 0.697 solve rate, and full fine-tuning outperforms LoRA. Cross-model validation with Gemini-2.5-FLASH/Pro pairs confirms the approach generalizes beyond GPT models.
Implications and Conclusions
This work demonstrates that executable coding environments enable a richer post-failure decision space than binary model cascades. By recognizing that cheap recovery and escalation can be complementary rather than strictly ordered, and by providing deployment-time budget control without retraining, CodeRescue offers practical cost savings for production coding agents. The research suggests that future work should extend recovery routing to longer action sequences, incorporate richer execution traces, and explore how recovery patterns transfer across different model pairs and benchmarks.
Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties
Authors: Philipp J. Schneider, Lin Tian, Marian-Andrei Rizoiu
Source and references: https://arxiv.org/abs/2510.19299v2
Learning to Make Friends: Coaching LLM Agents toward Emergent Social Ties
Introduction
This paper investigates whether large language model (LLM) agents can replicate complex social dynamics observed in human online communities and what mechanisms enable authentic social behavior to emerge. The researchers present a multi-agent LLM simulation framework where agents repeatedly interact, evaluate each other, and adapt behavior through in-context learning guided by reward signals.
Key Points
Multi-Agent Simulation Framework: Introduces a platform supporting both public and private communication channels that enables agents to develop distinct strategies across different contexts and systematically study how channel choice shapes conversational dynamics.
Behaviorally Grounded Reward Functions: Formalizes reward structures capturing empirically observed user motivations—social interaction, information seeking, self-presentation, coordination, and emotional support—creating a principled bridge between behavioral theory and agent objectives.
Endogenous Tie Formation: Implements mechanisms allowing social ties to emerge organically from conversational interactions without relying on pre-defined network structures, enabling systematic investigation of how support, alignment, and homophily drive group formation.
Psychologically Coherent Personas: Develops agent personas through three-layer architecture incorporating Big Five personality traits, task-based motivations derived from gratification theory, and multi-component memory structures (conversation, relationship, and opinion memory) that enable path-dependent behavior.
Emergent Network Structures: Demonstrates that coached LLM agents develop stable interaction patterns that yield network structures mirroring properties of real online communities, including clustering, modularity, and sustained tie persistence.
Methodology
The framework comprises two main components: persona creation and social media simulation. Agents are initialized with content-grounded personas extracted from real-world online discussions, incorporating personality traits assessed through the Mini-IPIP inventory, task assignments reflecting user motivations, and lightweight memory structures tracking interactions and beliefs. The simulation operates through iterative rounds where each agent follows a plan-execute-reflect cycle, selecting from actions including posting, commenting, direct messaging, or remaining inactive. After each round, agents cast votes on publicly visible content and reweight relationship strengths using behavioral signals. Critically, the framework incorporates optional “coaching“—external guidance helping agents make strategic decisions—and defines specific reward functions that incentivize behaviors aligned with observed human motivations, enabling in-context learning without explicit reward training.
Results and Findings
The paper demonstrates that LLM agents equipped with behavioral rewards and coaching mechanisms develop stable interaction patterns that reproduce key properties of real social networks. The experiments reveal agents form emergent social ties through micro-level signals including approval, reciprocity, and response latency, which aggregate into macro-level structures exhibiting clustering, modularity, and tie persistence. The framework successfully captures diverse engagement levels—from passive observers to active creators—and agents exhibit context-dependent communication strategies, deploying different approaches across public versus private channels. Network structures that emerge exhibit homophily and community formation patterns similar to observed online communities, validating the framework‘s capacity to approximate human-like social behavior.
Implications and Conclusions
This research provides a rigorous, principled testbed for investigating collective dynamics in LLM populations and offers policymakers, researchers, and security analysts a tool for safely studying online social phenomena without compromising individual privacy. By demonstrating that LLM agents can authentically reproduce complex social dynamics through behavioral rewards and adaptive learning, the work advances the feasibility of creating digital twins of social media ecosystems for testing moderation strategies, studying opinion formation, and detecting coordinated influence operations.
OmniReasoner: Thinking with Long Audio-Video via Native Tool Use
Authors: Yu Chen, Caorui Li, Ziyu Xiong, Yidong Wang, Mingqi Gao, Shuman Liu, Biao Liu, Chunfeng Yang, Anxiang Zeng, Haibo Zhang, Chaofan Chen
Source and references: https://arxiv.org/abs/2607.19339v1
Introduction
OmniReasoner introduces a tool-use framework that enables omnimodal language models to reason more effectively over long audio-video content by learning when and where to request higher-fidelity evidence. Rather than processing entire videos uniformly, the model learns to strategically zoom into relevant temporal intervals—a capability grounded in absolute timestamps that remain consistent across changing sampling granularities.
Key Points
Adaptive Tool-Use Framework: OmniReasoner teaches models to make three-part decisions: whether to answer from a global low-fidelity preview, when to invoke a zoom-in tool, and where on the timeline that tool should inspect, using supervised fine-tuning followed by reinforcement learning.
TimeAnchor Mechanism: Introduces plain-text temporal markers that bind audio-video tokens to wall-clock time, enabling the model’s zoom-in tool arguments (e.g., “inspect seconds 32–38”) to remain valid when transitioning from sparse global observations to dense local clips, solving a critical grounding problem unique to multi-granularity audio-video processing.
Temporal Augmented Data Engine: Synthetically generates training data through video editing operations (multi-segment composition and anomaly insertion) that automatically label evidence intervals, eliminating the need for expensive manual annotation while providing coupled supervision for answers, intervals, and tool-use trajectories.
Significant Performance Gains on Long-Form Content: Achieves improvements of 5.5 points on OmniVideoBench and 3.4 points on LVOmniBench over the Qwen2.5-Omni-7B baseline, with gains scaling dramatically with video duration—9.9 points on 10–30 minute videos versus 3.2 points on 0–5 minute videos.
Verified Evidence Integration: Empirical validation confirms that retrieved zoom-in clips meaningfully contribute to final answers rather than serving as decorative reasoning traces, demonstrated through performance drops when clips are removed and through attention rollout analysis.
Methodology
OmniReasoner operates in two reasoning stages: first, the model constructs a global observation by consuming the full audio-video stream at low fidelity, then either answers directly or requests a zoom-in interval using TimeAnchor‘s absolute-time format. When zoom is invoked, a media sandbox materializes a higher-fidelity local clip for the requested interval, and the model generates its final answer conditioned on both global context and local evidence. The framework is trained with supervised fine-tuning on 25,839 curated examples, followed by reinforcement learning using Group Relative Policy Optimization (GRPO) on 2,731 difficulty-aware examples. The data comes from two sources: automatically constructed long audio-video tasks via the Temporal Augmented Data Engine (combining multi-segment composition and anomaly insertion), and auxiliary omnimodal datasets. A custom omnimodal tool-use RL implementation built on TRL optimizes the policy using combined accuracy and format rewards without isolating a separate localization objective.
Results and Findings
OmniReasoner demonstrates consistent improvements across multiple benchmarks. On audio-visual benchmarks, it improves OmniVideoBench from 29.3% to 34.8%, LVOmniBench from 32.0% to 35.4%, and achieves notably large gains on VideoHolmes (24.4% to 40.0%). On general video reasoning benchmarks, improvements are more modest but still positive: Daily-Omni gains 2.1 points (62.1% to 64.2%) and WorldSense gains 1.3 points (45.4% to 46.7%). Critically, gains scale dramatically with video duration: improvements increase from 3.2 points on short videos (0–5 minutes) to 9.9 points on longer videos (10–30 minutes), and to 13.2 points on LVOmniBench‘s 50–90 minute content. Tool-use call rates correspondingly increase with duration, rising from 33.3% on 0–1 minute videos to 84.8% on 10–30 minute videos. Ablations confirm that self-curated temporal data (−3.8 points without it), tool-use trajectories (−2.0 points), audio input (−3.9 points), and TimeAnchor (−2.5 points on OmniVideoBench) each contribute substantially. Temporal grounding improves particularly sharply: on Charades-STA, TimeAnchor raises IoU@0.3 from 58.8% to 64.9% and mIoU from 37.9% to 41.3%.
Implications and Conclusions
OmniReasoner establishes that adaptive, tool-guided evidence acquisition significantly outperforms uniform processing for long audio-video reasoning, especially as input duration increases. The work demonstrates that omnimodal models can learn to strategically allocate high-fidelity computation to sparse, evidence-bearing moments while maintaining global temporal context—a capability that may become increasingly important as video reasoning tasks scale to longer durations and more complex cross-modal dependencies. However, the authors acknowledge infrastructural limitations: existing reinforcement learning frameworks lack native support for audio-conditioned multi-turn agentic workflows, and scaling beyond two-step reasoning requires both larger base models (beyond Qwen2.5-Omni-7B‘s 32K context) and more mature omnimodal RL infrastructure. Future work integrating stronger pre-trained tool-aware models and improved training frameworks could extend this paradigm to multi-turn reasoning scenarios and richer tool vocabularies.
Appearance Pointers -- Multimodal Region Control of Diffusion Transformers
Authors: Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha
Source and references: https://arxiv.org/abs/2607.19344v1
Introduction
This paper introduces Appearance Pointers, a novel mechanism for achieving precise, multimodal region-specific control in Diffusion Transformers (DiTs) without requiring model retraining. The work addresses a critical limitation in current generative image models: the inability to reliably control where and how different appearance cues—whether from text or reference images—should influence specific spatial regions in generated outputs.
Key Points
Appearance Pointers as Routing Tokens: The paper introduces compact “pointer” tokens that guide DiTs toward correct appearance cues at specified spatial locations, functioning as a lightweight interface between user intent and model generation without architectural modification.
Modality-Agnostic Framework: Unlike existing approaches constrained to either text or image conditioning, Appearance Pointers support simultaneous multimodal control, enabling users to condition different regions with both text descriptions and reference images within a single generation pass.
Region Correspondence Network: A dedicated network produces appearance pointers by fusing text or image inputs with user-specified spatial masks, combined with a spatial aggregation mechanism that handles multiple regional descriptions efficiently without excessive token overhead.
Comprehensive Capability Support: The unified model supports fine and sparse control layouts, object insertion with material fidelity, pose-conditioned generation, and multi-region synthesis—capabilities previously requiring separate specialized methods.
New Dataset and Evaluation: The authors introduce Appearance Pointers-37K, a synthetic dataset with regional text descriptions and appearance images, enabling systematic evaluation across multiple metrics including region adherence and identity preservation.
Methodology
The approach leverages the native heterogeneous token capabilities of Diffusion Transformers by introducing a region correspondence network that processes spatial masks alongside text embeddings or image tokens. This network produces appearance pointers—compact tokens that effectively “point“ the DiT‘s attention toward the correct conditioning signals at appropriate spatial locations. A spatial aggregation mechanism then refines these pointers to handle multiple regions simultaneously, maintaining computational efficiency while preserving the original DiT architecture and avoiding the need for complete model retraining.
Results and Findings
Across six evaluation metrics, Appearance Pointers achieves best or competitive performance compared to specialized state-of-the-art baselines. When regions are described through text alone, the method ranks first or second on all metrics. For image-based regional descriptions, it surpasses existing methods (MSDiffusion and DreamRenderer) in both region adherence and identity preservation. Critically, the single unified model outperforms all prior work across capability dimensions—it‘s the only method supporting simultaneous fine and sparse control, insertion, generation, and true multimodal conditioning where individual regions can be described by both images and text together.
Implications and Conclusions
Appearance Pointers represents a significant advancement toward practical creative tools by enabling precise spatial control while maintaining architectural simplicity and computational efficiency. The work demonstrates that effective region-aware multimodal control doesn‘t require complete model redesign but rather elegant interface mechanisms that route existing DiT capabilities, establishing a scalable path for integrating generative models into professional creative workflows where spatial precision and heterogeneous conditioning are essential requirements.
MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings
Authors: Ziyi Wang, Yuhang Wu, Dongxu Piao, Xingyu Liu, Tianhui Zhou, Miao Liu
Source and references: https://arxiv.org/abs/2607.19235v1
MeetingToM: Theory of Mind Reasoning in Multimodal Meeting Analysis
Introduction
This paper introduces MeetingToM, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to reason about Theory of Mind (ToM)—inferring beliefs, intentions, and mental states—in naturalistic multi-party meeting scenarios. The research addresses a critical gap in existing multimodal AI benchmarks by focusing on complex social phenomena like pseudo-consensus, where participants verbally agree while nonverbal cues reveal private dissent under social pressure.
Key Points
Hierarchical Task Design: MeetingToM comprises three progressively complex task levels—subject-level mental state prediction, dyadic-level addressee understanding and attitude inference, and group-level consensus reasoning—requiring increasingly sophisticated social reasoning.
Pseudo-Consensus Focus: Unlike existing benchmarks emphasizing overt signals, MeetingToM specifically targets meeting-specific phenomena where surface-level agreement masks hidden disagreement due to conformity effects and social pressure, a nuanced challenge for current models.
Comprehensive Multimodal Integration: The benchmark requires models to integrate verbal cues (speech, discourse markers, semantic content) with non-verbal signals (facial expressions, gaze, body orientation, posture) across multiple synchronized video perspectives to accurately assess social states.
Large-Scale Annotation Effort: MeetingToM contains 1,800 clips from 60 meeting sessions across three task families (600 instances per task), with rigorous quality control including hybrid annotation pipelines, inter-annotator agreement metrics (κ ranging from 0.50-0.73), and adjudication procedures.
Consistent MLLM Limitations: Evaluation reveals substantial performance gaps between human capability (74.97-86.33% accuracy across tasks) and state-of-the-art models (Gemini-3 Pro: 22.13-88.52% macro accuracy), with particular struggles in detecting subtle nonverbal cues and resolving verbal-visual conflicts.
Methodology
MeetingToM builds on the AMI Meeting Corpus, extracting task-specific video clips (5-second segments for Task 1, ~50-second segments for Tasks 2-3) synchronized with word-level transcripts. The benchmark employs a hybrid annotation pipeline: automated candidate identification of socially relevant events, GPT-assisted question generation without answer determination, independent human labeling by trained annotators, and adjudication procedures for cases without majority agreement. Models receive only visual input, aligned transcripts, and task prompts at evaluation time, with gold labels and auxiliary metadata excluded. The research evaluates both proprietary models (GPT-4o, GPT-5, Gemini-3 Pro) and open-source alternatives (Qwen2.5/3-VL variants) using exact-match accuracy and macro-averaged metrics for imbalanced tasks.
Results and Findings
Overall Performance: Proprietary models substantially outperform open-source alternatives. Gemini-3 Pro achieves 59.00% accuracy on Task 1 and 88.52% on Task 3.2, while GPT-5 reaches 55.67% and 90.70% respectively. However, macro-averaged scores reveal significant class imbalance effects—Gemini-3 Pro drops from 59.00% to 30.39% macro on Task 1, indicating uneven reliability across minority mental states. Performance degrades across the task hierarchy, with group-level reasoning proving most challenging.
Modality Ablations: Utterance-only input outperforms multimodal combinations on Tasks 1, 2.2, 3.1, and 3.2, suggesting models rely heavily on verbal content. However, Task 2.1 (addressee identification) benefits from video-utterance fusion, indicating that gaze and body orientation provide distinct value for referential reasoning. Joint multimodal input inconsistently outperforms unimodal settings, suggesting current MLLMs struggle with evidence fusion when cues are ambiguous or contradictory.
Prompting Effects: Chain-of-Thought variants show task- and model-dependent benefits rather than uniform improvements. Task-aligned CoT improves group-level reasoning on Gemini-3 Pro but reduces dyadic performance. Emotional CoT helps some models on subject-level tasks but over-emphasizes facial affect while missing interactional context. This suggests thinking prompts reshape attention allocation rather than adding simple reasoning depth.
Contextual Priors: Adding structured metadata (participant roles, meeting phases, dialogue acts) yields inconsistent or harmful effects. Dialogue-act labels sharply reduce addressee and group-level performance, while role and phase information provide isolated improvements. This indicates that contextual labels become useful only when calibrated against local evidence, not as automatic constraints.
Discourse Marker Analysis: Removing hesitation markers and backchannels improves Task 1 accuracy but reduces dyadic and group-level performance, suggesting these markers sometimes distract local mental-state recognition while carrying useful interactional information for broader social reasoning.
Implications and Conclusions
MeetingToM establishes critical benchmarks for advancing meeting-grounded Theory of Mind in multimodal systems, revealing that current MLLMs persistently struggle with integrating non-verbal cues, inferring hidden attitudes, and distinguishing genuine from pseudo-consensus. The substantial gaps between human performance (>74%) and model capability across all tasks, coupled with counterintuitive findings about prompting and contextual information, highlight that meeting-grounded social reasoning fundamentally requires models to calibrate competing evidence streams under social ambiguity—a capability that remains underdeveloped in today‘s most advanced multimodal systems. This work provides a rigorous testbed for the research community to develop more socially-aware AI systems applicable to collaborative decision support and socially-aware assistants in professional settings.
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Authors: Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi
Source and references: https://arxiv.org/abs/2607.19313v1
Introduction
This paper addresses a critical limitation in reinforcement learning for large language models: when training on hard problems that produce no correct solutions, existing methods receive zero learning signal. The authors introduce Off-Context GRPO (OC-GRPO), which uses privileged guidance (like solution prefixes) during training while maintaining alignment with the unguided deployment objective through importance-weighted corrections.
Key Points
The Learning Cliff Problem: Standard GRPO fails on hard problems where all sampled responses are incorrect, resulting in zero reward variance and zero gradient updates, preventing any learning signal.
Off-Context Distribution Mismatch: Existing guided training methods sample from augmented prompts containing privileged information but compute gradients as if sampling from the original unguided prompt, optimizing a misaligned objective (J_guide instead of J).
Importance-Corrected Solution: OC-GRPO applies per-token importance ratios ρ^oc_i,t(θ) = π_θ(y_i,t|x, y_i,<t) / π_θ_old(y_i,t|g(x), y_i,<t) to correct for the distribution shift, ensuring the algorithm optimizes the original unguided objective while maintaining learning signal from guided rollouts.
Behavior-Aware Credit Assignment: The importance correction automatically adjusts how much credit each trajectory receives based on guidance dependence—trajectories relying heavily on privileged information receive suppressed credit, while failures persisting despite guidance receive amplified penalties, without requiring additional reward shaping.
Minimal Implementation: OC-GRPO requires only a small modification to the standard GRPO surrogate loss (Equation 9), replacing the standard importance ratio with the off-context corrected ratio while maintaining all other components.
Methodology
The authors frame guided RLVR training as an off-context sampling problem: rollouts are generated under a guidance-augmented prompt g(x) but should optimize for the original prompt x to match deployment conditions. They employ importance sampling theory to derive the corrected per-token ratio that reweights advantages from guided samples back toward the unguided objective. For implementation, they use solution prefix guidance at nested levels (20%, 40%, 60%, 80%, 100% of reference solutions) and present two variants: OC-GRPO-Fixed (selecting guidance levels once using the base model) and OC-GRPO-Adaptive (re-evaluating guidance levels during training).
Results and Findings
On Qwen2.5-7B-Instruct, OC-GRPO-Fixed achieves a 13.8% relative improvement over vanilla GRPO (31.7 vs 27.8 average Pass@1 across benchmarks), outperforming guided baselines POPE (+8.7%) and PrefixRL by 1.4 absolute percentage points. Performance gains are consistent across three major benchmarks: AIME, Gaokao2023, and OmniMath. Critically, at smaller model scales (3B and 1.5B parameters), guided baselines that optimize misaligned objectives degrade below vanilla GRPO (PrefixRL* drops 7.2% at 1.5B), while OC-GRPO maintains consistent gains of +7.2% (3B) and +10.2% (1.5B). This scale-dependent pattern suggests back-generalization—the assumption that guided updates transfer to unguided prompts—is capacity-dependent and breaks at smaller model sizes without explicit correction.
Implications and Conclusions
OC-GRPO demonstrates that properly accounting for training-time context distribution shifts is essential for reliable learning from privileged guidance, particularly as model capacity decreases. The framework generalizes beyond solution prefixes to any form of privileged information (hints, intermediate goals, tool results) and extends naturally to multi-turn and agentic settings where training-time context differs from deployment conditions. This work establishes importance correction as a foundational principle for guided RL in language models, ensuring that learning signals guide models toward genuine problem-solving capabilities rather than hint-following behaviors.
Provable diffusion-based posterior sampling for linear inverse problems via DDIM
Authors: Yuchen Jiao, Na Li, Changxiao Cai, Yuxin Chen, Gen Li
Source and references: https://arxiv.org/abs/2607.19333v1
Introduction
This paper introduces Posterior-DDIM, a theoretically principled and computationally efficient algorithm for solving linear inverse problems using diffusion models as priors. The method achieves provable posterior consistency while maintaining the simplicity and speed of standard DDIM samplers through adaptive, direction-wise updates based on signal-to-noise ratio comparisons.
Key Points
SNR-guided singular direction partitioning: The algorithm partitions measurement operator singular directions into three groups—measurement-dominated, prior-dominated observed, and unobserved—based on comparing observation SNR with diffusion SNR at each time step.
Lightweight coordinate-wise modifications: Rather than introducing complex projection steps or approximate posterior scores, Posterior-DDIM requires only simple, efficient modifications to standard DDIM updates tailored to each singular direction category.
Posterior consistency guarantees: Theorem 1 and Theorem 2 establish that under exact diffusion priors and sufficiently fine time discretization, the sampler converges to the true Bayesian posterior distribution as discretization vanishes.
Comprehensive empirical validation: Experiments on inpainting, super-resolution, Gaussian deblurring, and compressed sensing tasks demonstrate superior or competitive performance across PSNR, SSIM, LPIPS, and FID metrics compared to state-of-the-art methods.
Plug-and-play framework: The approach requires no task-specific retraining and readily transfers across sensing modalities, making it practical for real-world applications including medical imaging and scientific discovery.
Methodology
Posterior-DDIM exploits the structure of linear measurement operators through singular value decomposition. For each singular direction, the algorithm compares the effective observation SNR (determined by singular values and measurement noise) against the diffusion SNR (α_t/σ_t) to decide whether measurements or the learned prior should drive updates. The algorithm uses direct measurement injection with calibrated noise for high-SNR directions, standard DDIM dynamics for low-SNR observed directions, and stochastic DDIM updates with higher noise injection for unobserved directions. Theoretically, convergence is established through auxiliary sequence constructions and KL divergence decomposition arguments that track how discretization errors propagate through the sampling procedure.
Results and Findings
Across multiple image restoration tasks on CelebA and ImageNet datasets, Posterior-DDIM achieves best performance on at least two of four metrics in every task evaluated. For inpainting on CelebA, it achieves best results on all four metrics (PSNR: 33.23, SSIM: 0.91, LPIPS: 15.06, FID: 25.33). On compressed sensing tasks, it consistently outperforms DDNM+ across all metrics. The method demonstrates robustness: performance saturates around 80-100 function evaluations, providing favorable computational-quality trade-offs. Hyperparameter analysis shows η₁ (stochasticity for unobserved directions) should be sufficiently large, while η₀ (stochasticity for prior-dominated directions) performs best at zero, recovering deterministic DDIM updates.
Implications and Conclusions
This work bridges theory and practice in diffusion-based inverse problem solving by providing the first rigorous posterior consistency guarantees for DDIM-type samplers on linear problems while maintaining computational efficiency. The SNR-guided directional partitioning framework offers a principled foundation for integrating measurement information with learned priors, with significant implications for high-stakes applications requiring both statistical reliability and computational tractability. Future directions include extending to nonlinear inverse problems, characterizing convergence rates, and analyzing dependencies on noise levels and operator conditioning.
Riemannian Deep Learning:Modules, Networks, and Geometries
Authors: Chen Ziheng
Source and references: https://arxiv.org/abs/2607.19305v1
Introduction
This doctoral thesis develops a comprehensive framework for building reusable neural network modules that operate on manifold-valued data, addressing fundamental limitations in geometric deep learning where operations are often tied to specific manifolds or rely on Euclidean approximations that sacrifice numerical stability and computational efficiency.
Key Points
Generalized Batch Normalization: Introduces Lie Group Batch Normalization (LieBN) with theoretical guarantees on Riemannian sample means and variances, extended to pseudo-reductive gyrogroups for applicability beyond traditional Lie group structures.
Riemannian Classification Layers: Extends Multinomial Logistic Regression from Euclidean space to Symmetric Positive Definite (SPD) manifolds and general Riemannian manifolds using Riemannian trigonometry, providing geometric alternatives to standard softmax classifiers.
Hyperbolic Geometry Innovations: Develops Proper Velocity Neural Networks as an unconstrained representation of hyperbolic space with stable geometry, and introduces Hyperbolic Busemann Neural Networks for efficient classification and fully connected operations in hyperbolic space.
Specialized Matrix Manifold Networks: Constructs networks operating directly on full-rank correlation matrices with accurate gradient computation under two distinct correlation geometries, providing a normalized alternative to SPD matrices for specific applications.
Learnable Metrics for SPD Manifolds: Proposes adaptive log-Euclidean metrics through parameterized matrix logarithms and develops fast Cholesky-based metrics with closed-form operators, enabling metrics to adapt to data and network dynamics while maintaining numerical stability.
Methodology
The thesis employs a hierarchical approach to geometric deep learning. It begins with general constructions grounded in differential and Riemannian geometry, leveraging Lie group structures where applicable. For cases where general formulations cannot exploit useful structure, the work designs specialized networks for particular manifold representations. The framework combines theoretical analysis with practical implementation, incorporating matrix function theory for backpropagation through geometric operations. Empirical validation spans vision, signal processing, graph learning, and genomics applications, with experiments demonstrating both the theoretical soundness and practical utility of proposed methods.
Results and Findings
The research yields seven peer-reviewed publications at premier venues (ICLR, CVPR, NeurIPS, ICML) plus numerous collaborative works advancing the field. Key empirical contributions include: batch normalization methods demonstrating improved convergence on SPD manifolds; multinomial logistic regression variants showing competitive or superior classification accuracy on manifold-valued data; Proper Velocity networks achieving stable training in hyperbolic spaces without projection constraints; and Cholesky-based metrics reducing computational overhead while maintaining numerical precision. Applications demonstrate consistent improvements in EEG signal classification, skeleton-based action recognition, and brain imaging analysis—domains where manifold structure is inherent to the data representation.
Implications and Conclusions
This work significantly advances geometric deep learning by bridging the gap between theoretical differential geometry and practical neural network design, providing practitioners with reusable, theoretically-grounded modules for manifold-valued data. The framework‘s modularity and emphasis on both numerical stability and computational efficiency position it as a foundation for future research in applications where data naturally resides on non-Euclidean geometries, from medical imaging to graph-based learning systems.
CircuitKIT : Circuit Discovery, Evaluation, and Application Toolkit for Mechanistic Interpretability
Authors: Pratinav Seth, Hem Gosalia, Aditya Kasliwal, Vinay Kumar Sankarapu
Source and references: https://arxiv.org/abs/2607.19317v1
Introduction
CircuitKIT is a comprehensive, source-available library that unifies the mechanistic interpretability workflow for large language models by connecting circuit discovery, evaluation, and downstream interventions through a single typed artifact. The toolkit addresses the fragmentation in circuit analysis by providing standardized interfaces, multiple discovery algorithms, complementary evaluation diagnostics, and practical intervention modules for model compression, editing, and safety auditing.
Key Points
Unified Pipeline Architecture: CircuitKIT connects thirteen discovery algorithms across four backend families (gradient-attribution, search-based, information-bottleneck, and contextual decomposition) with six complementary evaluation pillars and seven downstream application modules through a single
CircuitScoresartifact, eliminating the need to stitch together separate implementations.Declarative Custom-Data Path: A template-driven interface maps structured datasets (CSV, JSONL, HuggingFace) into circuit-discovery tasks without hand-authoring contrastive prompts, with automatic token alignment and corruption synthesis supporting both paired and clean-only discovery routes.
Multi-Pillar Faithfulness Evaluation: Rather than relying on a single metric, the framework evaluates circuits across six complementary diagnostics (causal patching, ablation sufficiency, stability, robustness, baseline comparison, and generalization) plus an optional intervention-reliability pillar, acknowledging that faithfulness scores depend on ablation methodology and can reverse method rankings.
Three-Interface Accessibility: The same workflow operates through a stateful Pipeline class, a functional flat API, and a YAML-driven CLI, allowing circuits discovered in one interface to be evaluated or applied through another without format conversion.
Extensible Registry Pattern: Discovery algorithms, corruption strategies, selectors, and model architectures register through decorators, establishing a clear governance model where stable-tier methods carry backward-compatibility guarantees while research-tier contributions can be promoted as validation evidence accumulates.
Methodology
CircuitKIT operates on decoder-only transformers by decomposing the model into addressable components (attention heads, MLP sublayers, individual neurons) connected through a computational graph. Discovery locates causal subgraphs responsible for specific behaviors through causal intervention using either contrastive pairs (activation patching via EAP/EAP-IG/EAP-GP/ACDC) or clean inputs alone (IBCircuit, CD-T). The framework standardizes both node-level (entire components) and neuron-level (individual channels) granularities, with evaluation running across six configurable diagnostic pillars computed from a single CircuitScores object. Interventions (pruning, quantization, fine-tuning, editing, steering) consume this artifact through unified APIs, with results exported as reloadable HuggingFace checkpoints and benchmarked through lm-evaluation-harness integration.
Results and Findings
Algorithm Validation (E1): Six stable algorithms on GPT-2 Small IOI recover the canonical circuit with perfect causal-patching recovery (1.0 for EAP family) and Jaccard overlap of 0.43–0.50 with known components. CD-T achieves equal ablation faithfulness (1.0) despite zero canonical-head overlap, demonstrating that behavioral recovery and architectural overlap are distinct measures requiring panel-based reporting.
Cross-Family Discovery (E2): EAP-IG produces faithful, stable circuits across six model families (GPT-2, Pythia, Llama, Gemma, Qwen, Phi) at 124M–2.8B parameters with consistent patching recovery (P1 ≥ 0.91) and stability (Jaccard 0.80–0.92). Ablation sufficiency varies widely (P2: 0.24–1.00), illustrating that single-metric evaluation obscures performance variation.
Neuron-Level Discovery (E3): Neuron-granularity discovery preserves patching recovery (EAP-IG: 1.0) while retaining 70% of units at fixed sparsity, directly exposing the units intervention modules act on. Single-point-gradient EAP degrades and inverts under hard ablation (P2 = −0.35), whereas integrated-gradient interpolation stabilizes fine-grained attribution.
Custom-Data Path (E4): A 334-record jailbreak CSV reaches multi-pillar evaluation via both paired (EAP-IG) and clean-only (IBCircuit) routes with zero pairing code. The paired circuit recovers 85% of refusal under soft patching but inverts toward compliance under hard ablation (P2 = −2.61, raw = −1.55), illustrating that intervention safety cannot be inferred from intrinsic scores alone.
Circuit-Guided Pruning (E5): IBCircuit prunes competitively (98.6% accuracy retention), but patch faithfulness anti-correlates with retention (Spearman ρ = −0.78, p = 0.010), with EAP-IG achieving lowest perplexity (35.5) despite lowest accuracy retention (57.1%). This demonstrates that intrinsic faithfulness does not predict intervention performance.
Quantization (E6): Unlike pruning, mixed-precision quantization is insensitive to patch faithfulness (ρ = +0.23, p = 0.55) but correlates with ablation faithfulness (ρ = +0.73, p = 0.031), showing that intervention-selector relationships differ by application type and must be validated extrinsically.
Selective Fine-Tuning (E7): Circuit-guided fine-tuning does not separate from random-budget masking (mean Δ = −0.001 pp), with the 30% budget constraint itself protecting coherence rather than circuit importance determining which parameters to update.
Implications and Conclusions
CircuitKIT establishes that mechanistic interpretability requires standardized infrastructure supporting method comparison, custom-data integration, and honest downstream validation. The finding that single faithfulness metrics can reverse method rankings and fail to predict intervention outcomes argues for panel-based evaluation and extrinsic benchmarking rather than relying on intrinsic scores. The toolkit‘s demonstration that circuit-derived importance performs variably across interventions—decisively in pruning, negligibly in quantization, neutrally in fine-tuning—establishes that interpretability metrics must be validated per application and that comprehensive dashboards rather than summary statistics are necessary for practitioners making safety and efficiency decisions. By releasing CircuitKIT with extensible registries and source-available governance restricting commercial use that degrades model safety, the authors provide common infrastructure for the mechanistic-interpretability community while acknowledging the dual-use risk inherent in circuit localization and ablation techniques.
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Authors: Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou
Source and references: https://arxiv.org/abs/2607.10481v2
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Introduction
This paper addresses a critical instability in reinforcement learning for large language models: over-optimization, where models exploit training patterns that don‘t generalize to validation tasks. The authors demonstrate that standard reverse KL regularization is insufficient for preventing this collapse and propose ARMOR, a framework that combines active sample stabilization with adaptive exploration to achieve sustained performance improvements during extended training.
Key Points
Over-optimization Problem: Models exhibit a “reward-validation gap” where training rewards improve while validation performance degrades—a generalization failure distinct from classical reward hacking that persists even with verifiable reward signals.
KL Regularization Limitations: Standard reverse KL has two fundamental flaws: (1) its mode-seeking nature allows collapse onto narrow “shortcut” patterns without penalty, and (2) uniform penalties suppress both harmful degradation and beneficial exploration, creating a stability-exploration dilemma.
Anchor Rollout Component: Actively injects off-policy correct samples from the reference policy into training batches to explicitly prevent drift from known good solutions, replacing passive loss penalties with active distribution stabilization.
Mixed Optimization Component: Reformulates the policy objective to optimize a mixture policy α·πθ + (1−α)·πref, which constructs an adaptive trust region that permits controlled exploration without uniform suppression effects.
Broad Empirical Validation: Demonstrates consistent improvements across multiple base models (Qwen2.5-Math-7B, Qwen3-8B-Base) and RL algorithms (DAPO, QAE), with gains of +5–8 points on mathematical reasoning benchmarks without sacrificing general capabilities.
Methodology
ARMOR operates in two phases executed iteratively. During Anchor Rollout, the framework constructs hybrid response groups by sampling on-policy responses from the current policy and augmenting them with rejection-sampled correct responses from the reference policy. In the Mixed Optimization phase, the framework replaces standard importance sampling ratios with mixed variants that account for the reference policy‘s contribution to the data distribution, then performs policy updates using the underlying RL algorithm (DAPO or QAE). Periodically, the reference policy is reset to the current best checkpoint to prevent the anchor from becoming a bottleneck.
Results and Findings
Experiments on AIME24, AIME25, and AMC benchmarks show substantial improvements. With DAPO on Qwen2.5-Math-7B, ARMOR achieves +5.91 points on AIME24 (37.13→43.04) and +6.74 on average math reasoning (40.58→45.77). On Qwen3-8B-Base, gains reach +11.15 points on AIME24 (36.98→48.13). QAE integration validates robustness (+1.79 points on AIME24), though with slight general capability trade-offs attributed to the algorithm‘s advantage masking mechanism. Pass@k evaluations confirm that gains reflect genuine reasoning improvements rather than accuracy-diversity trade-offs. Ablation studies conclusively demonstrate the necessity of both components: Mixed Optimization alone eventually collapses without Anchor Rollout‘s stability floor, while Anchor Rollout without Mixed Optimization plateaus at lower performance ceilings. Analysis of KL divergence reveals that the adaptive trust region mechanism from Mixed Optimization selectively reinforces verified correct actions while permitting stronger penalization of reference biases.
Implications and Conclusions
This work reveals a fundamental structural limitation in standard regularization approaches for long-horizon RL training in reasoning tasks, proposing a principled alternative that decouples stability concerns from exploration constraints. By shifting from passive penalty-based regularization to active sample-based stabilization combined with adaptive trust regions, ARMOR provides a practical framework for sustained reasoning capability improvement in large language models, with immediate applicability to production systems scaling RL for mathematical reasoning and other verifiable tasks.
AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
Authors: Yu-Yang Qian, Hao-Cong Wu, Chen Chen, Jiacheng Sun, Zhenhua Dong, Peng Zhao, Zhi-Hua Zhou
Source and references: https://arxiv.org/abs/2607.19223v1
Introduction
This paper presents ADAFlash, a framework for accelerating large language model inference through adaptive speculative decoding using diffusion-based draft models. The work identifies and addresses critical variance issues that emerge when using bidirectional-attention diffusion models as drafters during deployment.
Key Points
Identifies high-variance problem: Diffusion drafters exhibit substantially higher variance than autoregressive drafters at both domain-level (acceptance rates fluctuate 2.1× across domains vs. 1.2× for AR models) and token-level (per-position acceptance probability varies significantly within sequences).
On-Policy Distillation (OPD) algorithm: Introduces a specialized distillation approach using reverse-KL divergence with entry-wise clipping, designed specifically for diffusion drafters’ high-entropy outputs, reducing domain-level variance during deployment.
Adaptive Length Head: Proposes a lightweight prediction module that dynamically adjusts verification sequence length based on predicted acceptance rates, addressing token-level variance and eliminating wasteful computation on low-quality tokens.
Infrastructure for online adaptation: Implements asynchronous training-inference pipelines and adaptive request scheduling within a serving engine, enabling continuous drafter updates without blocking inference.
Substantial performance gains: Achieves up to 5.3× speedup over standard autoregressive decoding and 66% higher throughput than prior methods under high-concurrency scenarios (concurrency=128).
Methodology
ADAFlash combines two complementary mechanisms. First, it performs on-policy distillation during deployment by collecting feedback from the target model‘s distributions at each verification step, then updating the drafter using a mixture loss combining reverse-KL divergence (which encourages mode-seeking behavior) and hard-label cross-entropy on top-1 tokens, with entry-wise clipping to prevent outlier gradients from dominating updates. Second, it attaches a lightweight neural head to the drafter that predicts overall acceptance rates and dynamically determines verification length, with this head continuously updated via MSE loss against ground-truth acceptance rates obtained from the verification outcomes.
Results and Findings
Experiments across eight datasets and three foundation models (including dense and mixture-of-experts architectures) demonstrate consistent improvements. At single concurrency (C=1), ADAFlash achieves 4.06× average speedup on Qwen3-8B compared to 3.53× for DFlash and 3.95× for OSD. Critically, at high concurrency (C=128), ADAFlash maintains 1.15× speedup while competing methods degrade to 0.76-0.83×, falling below baseline performance. Ablation studies confirm both components contribute meaningfully: divergence clipping and mixture OPD improve domain-level acceptance consistency, while the adaptive length head is essential for high-concurrency performance. Analysis shows ADAFlash narrows the domain-level variance distribution substantially and improves per-position acceptance probabilities, particularly at later token positions where baseline drafters struggle.
Implications and Conclusions
This work reveals that bidirectional-attention mechanisms in diffusion drafters create practical deployment challenges beyond their well-known benefits, and demonstrates that adaptive online learning can effectively mitigate these issues. The framework‘s superior performance under high concurrency has significant implications for production LLM serving environments, where systems must handle multiple simultaneous requests and resource constraints become acute bottlenecks.
Masked Visual Actions for Unified World Modeling
Authors: Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala, Gordon Wetzstein, Li Fei-Fei, Yilun Du, Jiajun Wu, Jia-Bin Huang
Source and references: https://arxiv.org/abs/2607.19343v1
Masked Visual Actions for Unified World Modeling
Introduction
This paper introduces Masked Visual Actions, a novel approach that enables pretrained video models to serve as unified robotic world models by representing actions as pixel-space masked trajectories. The method allows a single model checkpoint to function simultaneously as a forward dynamics model (predicting scene responses to robot actions) and an inverse model (recovering robot behavior from desired object motions), all while generalizing across different robot embodiments.
Key Points
Pixel-aligned action conditioning: Actions are expressed as partially revealed spatiotemporal patterns in video frames rather than low-dimensional commands, creating a representation naturally aligned with video models’ learned priors about visual interaction and motion.
Unified forward and inverse modeling: By varying which entities are revealed in masked videos—robot motion versus desired object motion—the same model can function as either a forward dynamics predictor or inverse model without separate training, a capability unique to this spatial masking approach.
Embodiment-agnostic generalization: The method demonstrates strong zero-shot transfer to unseen robot morphologies (including bimanual systems), significantly outperforming baselines that condition on skeleton poses or end-effector positions, which collapse or hallucinate when encountering novel embodiments.
Efficient training and practical applications: Using only 15 hours of robot interaction data (combined from real DROID videos and Robocasa simulation), the model achieves state-of-the-art visual fidelity while supporting three downstream robotics applications: policy evaluation, model-based planning, and action extraction.
Superior visual fidelity: Quantitative metrics (LPIPS, SSIM, PSNR) demonstrate substantial improvements over prior work like Ctrl-World, with particularly pronounced advantages on out-of-distribution embodiments (LPIPS of 0.123 vs. 0.196 on BEHAVIOR dataset).
Methodology
The authors finetune a pretrained video diffusion model (Wai-Fun-Control 2.2 14B) using LoRA adaptation (rank 256) on masked video sequences. Training data construction employs two complementary approaches: segmentation-based masking using SAM to isolate robots from real DROID videos, and rendering-based masking using robot URDFs from recorded states in both real and simulated environments. The model learns conditional video generation by receiving masked input videos concatenated spatially with reference frames, predicting unmasked regions through standard diffusion training over ~10,000 steps on 8 H200 GPUs.
Results and Findings
Controllable generation: The model matches and exceeds Ctrl-World‘s performance on seen embodiments (LPIPS 0.0945 vs. 0.362) while gracefully generalizing to unseen bimanual robots where baselines fail completely. Ablations confirm masked visual actions substantially outperform sparse conditioning signals (end-effector positions, skeletons) on out-of-distribution data.
Model-based planning: Best-of-N planning using the video model to evaluate trajectory rollouts improves task success by 7-26% across six manipulation tasks (close microwave, open drawer, etc.), with gains increasing with the number of evaluated candidates.
Policy evaluation: Simulated rollout success rates exhibit strong correlation (r = 0.982) with ground-truth environment outcomes in simulation. Real-world validation shows per-trial progress distributions in generated videos closely match actual execution, though with a consistent positive bias toward task success.
Action extraction: Zero-shot inverse modeling—generating robot trajectories from desired object motion—recovers the highest success rate (90%) on the COFFEESERVEMUG task compared to imitation learning baselines (Diffusion Policy, ACT, SmolVLA), despite the video model never seeing task-specific training data.
Implications and Conclusions
Masked Visual Actions establishes a fundamentally new paradigm for robotic world modeling by leveraging video models‘ rich visual priors through pixel-aligned action conditioning rather than embodiment-specific control signals. This work demonstrates that spatial masking enables a single unified model to bridge forward and inverse reasoning while achieving unprecedented generalization across robot morphologies, with clear practical benefits for planning, evaluation, and action synthesis in robotic manipulation tasks.
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
Authors: Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang, Bole Ma, Justin Qian, Ziyi Jiao, Bingyang Zhou, Luoxin Ye, Kaifeng Zhang, Kunyi Wang, Weijia Zeng, Yunuo Chen, Pengzhi Yang, Ziqiu Zeng, Huamin Wang, Chao Liu, Alan Yuille, Fan Shi, Changxi Zheng, Yunzhu Li, Chenfanfu Jiang, Peter Yichen Chen
Source and references: https://arxiv.org/abs/2607.19190v1
Agentic Real2Sim: Automating Physics-Based Digital Twin Creation from Robot Videos
Introduction
This paper introduces Agentic Real2Sim, a framework that automatically converts real-world robot interaction videos into physically simulatable digital twins. Rather than relying on manual tuning and brittle workflows, the system uses vision-language agents to orchestrate the complete conversion pipeline—from scene reconstruction to physical parameter inference—enabling scalable transformation of robot demonstrations into simulation-ready artifacts.
Key Points
Unified Episode Conversion Pipeline: The framework converts recorded robot-object interactions into simulatable twins that preserve observations, geometries, trajectories, physical parameters, and actor states through four linked agents: visual processing, physical-prior inference, scene preparation, and simulator-in-the-loop grasp optimization.
VLM-Agnostic Architecture with Cost Efficiency: Agentic Real2Sim supports interchangeable vision-language model backends. An open-weight 31B model (Gemma) achieves comparable 48/100 replay-success outcomes to proprietary models while reducing costs by up to 31.4× compared to frontier models, with total conversion costs as low as $2.62 per 100 episodes.
Generalization Across Domains: The same core conversion contract extends beyond rigid-body manipulation to deformable-object interactions (rope, cloth, soft materials) and humanoid locomotion, demonstrating that the framework handles diverse physical interaction types without domain-specific rewrites.
Deliberate Separation of Concerns: The architecture cleanly separates deterministic visual and simulation tools from agentic decision-making. VLMs make bounded, schema-constrained choices about object discovery, keyframe selection, and refinement strategies, while specialized perception and physics components handle geometry recovery, pose tracking, and grasp optimization.
Structured Evaluation Methodology: Episodes are evaluated using a VLM-based replay-success metric that compares real and simulated keyframes across four dimensions—target object identity, final location, action similarity, and gripper location—with three independent judges scoring each candidate.
Methodology
The framework processes DROID-style robot demonstrations through a multi-stage agentic pipeline. The visual processing agent extracts segmentation masks using SAM 3, recovers 3D geometry with SAM 3D, performs depth estimation via FoundationStereo, and tracks object poses using FoundationPose. The physical-prior inference agent infers material classes and mass properties from visual evidence. Scene preparation then calibrates camera extrinsics, optimizes robot base pose alignment, estimates ground planes, and loads the scene into MuJoCo. Finally, a simulator-in-the-loop grasp optimization stage evaluates candidate object placements to identify configurations enabling successful grasping. VLM queries are scoped to high-level decisions with bounded retry budgets rather than geometric computation, enabling backend interchangeability. The system outputs a standardized episode folder containing meshes, pose tracks, robot trajectories, camera metadata, and task semantics.
Results and Findings
On DROID-100 (100 diverse manipulation episodes), Gemma 31B achieved 48 successful replays, 8 partial successes, and 44 failures at a model cost of $2.62. Across four VLM backends tested—Gemma 4 (31B), Qwen (35B), GPT-5.4, and Claude Haiku—replay-success rates ranged from 37 to 48 episodes, but model costs varied dramatically: Gemma required $2.62, Claude $9.09, Qwen $13.00, and GPT-5.4 $82.30. The similar success rates across backends despite 31.4× cost differences suggest that remaining performance headroom lies primarily in visual and simulation components rather than VLM capability alone. Qualitative results demonstrate successful conversions across rigid manipulation, deformable materials (cloth, rope, soft packages), and humanoid motion, with representative visual comparisons showing real-to-simulated alignment. The framework‘s deterministic tools enable reliable geometry and physics computation while VLM orchestration handles ambiguous perceptual decisions.
Implications and Conclusions
Agentic Real2Sim addresses a critical bottleneck in robotics research by automating the labor-intensive conversion of real demonstrations into simulation assets suitable for policy learning and evaluation. By achieving comparable results with open-weight models at minimal cost while maintaining modularity across rigid, deformable, and humanoid domains, this work establishes a scalable foundation for leveraging large real-world robot datasets in simulation-based learning pipelines.
No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation
Authors: Feinan Cheng, Dongliang Xu, Wenli Nong, Zhiheng Zhang, Ang Liu, Tianyu Wang, Yue Yao
Source and references: https://arxiv.org/abs/2607.19288v1
Test-Time Scaled VLMs for UAV Navigation: Summary
Introduction
This paper introduces a test-time scaling approach to improve Vision-Language Model (VLM) performance for unmanned aerial vehicle (UAV) navigation without requiring additional model training. Rather than relying on a single inference pass, the authors propose an iterative refinement process that enhances navigation reasoning through parallel exploration, serial self-correction, and multi-criteria evaluation.
Key Points
Three-Stage Pipeline: The method implements an “Explore–Refine–Select” framework that generates multiple candidate trajectories in parallel, refines each through iterative self-correction, and selects the optimal path using a multi-criteria scoring function.
No Model Retraining Required: By operating entirely at inference time, the approach improves performance on frozen, pre-trained VLMs without modifying any parameters or requiring additional training data.
State-of-the-Art Performance: The method achieves superior results across all test sets, with 2.02% improvement in success rate (SR) on seen environments and 1.28-0.94% improvements on unseen objects and maps compared to baseline approaches.
Safety-First Evaluation: The scoring function prioritizes collision avoidance (50% weight) while balancing goal alignment (30%) and forward progress (20%), ensuring UAVs make deliberate, reliable decisions in complex environments.
Computational Budget Trade-off: Performance scales positively with inference-time token consumption, demonstrating that allocating more computational resources during inference translates directly to better navigation accuracy.
Methodology
The approach formulates UAV navigation as a three-stage process applied at test time. First, the model generates N distinct candidate coordinates through parallel inference calls, creating a diverse set of initial hypotheses. Second, each candidate undergoes M rounds of iterative self-correction via a self-reflective prompt strategy that explicitly instructs the model to reconsider its initial plan. Finally, a weighted multi-criteria scoring function evaluates all refined candidates across three dimensions—Safety (based on depth sensor data), Goal-Alignment (cosine similarity to target direction), and Forward-Progress (Euclidean distance with tanh normalization)—selecting the highest-scoring trajectory for execution.
Results and Findings
Experiments on the TRAVEL-UAV dataset demonstrate consistent improvements across multiple evaluation metrics. On the Test-Seen (TS) subset, the method improves success rate to 24.96% (+2.02%), overall success rate to 47.39% (+2.47%), and success-weighted path length to 20.93 (+1.43%), while reducing navigation error to 106.32 meters (-4.67%). On more challenging Unseen Object (UO) and Unseen Map (UM) subsets, SR improvements of 1.28% and 0.94% respectively demonstrate generalization capability despite distribution shifts. Ablation studies confirm that combining parallel exploration (Par=3) with serial refinement (Ser=2) yields superior performance compared to single-dimension scaling strategies. Token analysis reveals a clear positive correlation between inference-time computational budget (ranging from 6,705 to 38,315 tokens) and navigation success rates.
Implications and Conclusions
This work establishes test-time scaling as a practical paradigm for improving VLM-based UAV navigation without architectural modifications or retraining, offering a complementary approach to existing training-focused optimization methods. The framework‘s demonstration that increased inference-time computation directly translates to safer, more reliable autonomous navigation opens pathways for deploying robust aerial navigation systems in real-world applications including logistics, infrastructure inspection, agriculture, and emergency response without incurring significant training overhead.