Computation and Language · Cross-listed from cs.CV
Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent
Yihong Tang, Kehai Chen, Xuefeng Bai et al. (+1 more)
The advancement of
Multimodal Large
Language Models (MLLMs) has expanded Role-Playing
Agents (RPAs) into visually grounded environments. However, human
vision is inherently subjective and identity-driven, whereas existing MLLMs extract objective, character-agnostic features for general tasks. In RPAs, this generic visual noise overpowers fragile character traits, causing Modality-Role Interference (MRI), where
agents struggle to integrate visual grounding and character consistency. To address this, we introduce the training-free Character-Aware Visual Intervention (CAVI) framework, enabling
agents to perceive the world through the lens of character. CAVI systematically targets MRI: macroscopically, Character-Guided
Token Pruning (CTP) restricts the visual receptive field to role-relevant entities; microscopically, Orthogonal Feature Modulation (OFM) projects
tokens onto a character-context subspace to extract aligned facts; and during decoding, Modality-Adaptive Role Steering (MARS) dynamically optimizes steering intensity based on visual reliance. Extensive experiments show CAVI effectively alleviates MRI, significantly enhancing character-consistent
multimodal interactions.
Read the full abstract ▾
All authors Yihong Tang, Kehai Chen, Xuefeng Bai, Min Zhang
Computation and Language
Key Coverage Matters: Semi-Structured Extraction of OCR Clinical Reports
Yu Wang, Yingyun Li, Ying Qin et al. (+1 more)
Clinical reports are often fragmented across healthcare institutions because
privacy regulations and data silos limit direct information sharing. When patients seek care at a different hospital, they often carry paper or scanned reports from prior visits. This hinders EHR integration and longitudinal review, and downstream applications that depend on more complete patient records, such as patient management, follow-up care, real-world studies, and clinical-trial matching. Although OCR can digitize such reports, reliable extraction remains challenging because clinical documents are heterogeneous, OCR text is noisy, and many healthcare settings require low-cost on-premise deployment. We formulate this problem as canonical key-conditioned extractive question answering over OCR-derived clinical reports. Because the key fields are neither fixed nor known in advance, the key space is open. We maintain a canonical key inventory through iterative key mining, normalization, clustering, and lightweight human verification, and introduce key coverage as a metric to quantify inventory completeness. Using a 0.2B BERT-based model, experiments on real-world reports from more than 20 hospitals show performance improves monotonically with key coverage. The model achieves F1 scores of 0.839 and 0.893 under exact match and boundary-tolerant matching, respectively, once the Top-90 canonical keys are covered. These results show that key coverage is a dominant factor for end-to-end performance. At Top-90 coverage, our model outperforms a fine-tuned Qwen3-0.6B baseline under exact match. Although our annotated corpus is Chinese, the method relies on the
language-agnostic key-value organization of semi-structured clinical reports and can be adapted to other settings given an appropriate canonical key inventory and alias mapping.
Read the full abstract ▾
All authors Yu Wang, Yingyun Li, Ying Qin, Haiyang Qian
Computation and Language
PumpSense: Real-Time Detection and Target Extraction of Crypto Pump-and-Dumps on Telegram
Ahmed Mahrous and Roberto Di Pietro
Cryptocurrency pump-and-dump schemes coordinated via Telegram threaten
market integrity. However, existing research addressing this specific threat has not yet produced solutions that combine reliable results with fast response. This is in part due to the absence of publicly available, message-level labeled data, as well as design choices. In this paper, we address both issues. In particular, we introduce a corpus of over 280,000 Telegram posts from 39 pump-organizing groups, all manually reviewed to identify 2,246 pump announcements and their targeted cryptocurrency and exchange. Leveraging this
dataset, we define two tasks: real-time pump-announcement detection and target cryptocurrency/exchange extraction. For detection, we compare two machine-learning models: a lightweight tree-based LightGBM classifier (F1=0.79, latency=9.4 s/sample) and a
transformer-based BGE-M3 (F1=0.83, latency=50 ms/sample). With our proposed approach, we show that message analysis can achieve near-instant pump detection at the level of individual Telegram message windows. Unlike prior work that relies purely on
market data and typically detects pumps tens of seconds after abnormal trading activity is observed, our method operates directly on the coordination messages themselves and can be evaluated in microseconds per window on commodity hardware. To our knowledge, we also establish the first
benchmark for manipulated coin and exchange extraction. We demonstrate that traditional rule-based extraction methods, widely relied upon in prior literature, are ineffective due to ticker ambiguity. In contrast, LLMs achieve the highest accuracy with a score of 0.91.
Read the full abstract ▾
Computation and Language
Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs
Jiafeng Liang, Zhihao Zhu, Zihan Zhang et al. (+7 more)
Although Large
Multimodal Models (LMMs) have achieved strong performance on general
video understanding, their susceptibility to textual prior shortcuts during causal discovery has been recognized as a critical deficit. The underlying mechanisms of this phenomenon remain incompletely understood, as existing
benchmarks only measure response accuracy without revealing the sources and extent of the deficit. We introduce ProCauEval, a perturbation-based evaluation protocol that shifts from outcome assessment to mechanism diagnosis, probing causal discovery through five controlled configurations that systematically manipulate visual and textual modalities to decompose their respective contributions to model behavior and dissect the failure modes. Evaluating 17 mainstream LMMs, we find that models faithfully perceive
video content yet systematically underexploit it during causal
reasoning. We further observe that stronger post-training amplifies rather than mitigates textual prior reliance, and that higher baseline performance correlates with greater fragility under perturbation. To address these, we propose Anti-
Distillation Policy Optimization (ADPO), a reinforcement learning framework built on negative teacher
alignment, which augments GRPO by explicitly pushing the policy away from a prior-only counterfactual teacher induced by visual corruption. Specifically, ADPO maximizes the divergence between the policy distributions conditioned on the original and visually corrupted inputs, thereby forcing the model to ground its
reasoning in visual evidence rather than textual shortcuts. Extensive experiments show that ADPO improves visual engagement without sacrificing fundamental comprehension, thus offering a preliminary step toward reliable causal discovery.
Read the full abstract ▾
All authors Jiafeng Liang, Zhihao Zhu, Zihan Zhang, Baoqi Ren, Shixin Jiang, Runxuan Liu, Tao Ren, Ming Liu, See-Kiong Ng, Bing Qin
Computation and Language
Cross-Cultural Transfer of Emoji Semantics and Sentiment in Financial Social Media
Ahmed Mahrous and Roberto Di Pietro
Emojis are widely used in online financial communication, but it is unclear whether they provide transferable sentiment signals across
languages, platforms, and asset communities. This study examines the extent to which emoji usage, semantics, and sentiment polarity remain stable across financial communities, and how these layers influence zero-shot sentiment transfer. Using large corpora of Twitter and StockTwits posts in four
languages, we measure cross-community divergence and evaluate sentiment models trained under emoji-only, text-only, and text+emoji inputs. We find that emoji frequencies differ across communities, especially across
languages, but their semantics and sentiment polarity are largely stable. Cross-asset transferability shows minimal degradation, while cross-
language transfer remains the most challenging. Including emojis consistently reduces transfer gaps relative to text-only models. These results indicate that financial communication exhibits a partially shared “emoji code,” and that emojis provide compact,
language-independent sentiment cues that improve model generalization across
markets and platforms.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Let the Target Select for Itself: Data Selection via Target-Aligned Paths
Huitao Yang, Hengzhi He, Tung Sum Thomas Kwok et al. (+1 more)
Targeted data selection seeks training examples from a candidate pool that improve downstream task performance. While trajectory-based selectors effectively guide this process, existing approaches typically construct reference states by warming up on the candidate pool itself, thereby inheriting pool-dependent distributional biases. To address this coupling, we introduce Target-Aligned Candidate Selection (TACS), which constructs a short, capacity-constrained reference path using a compact target-validation proxy. Candidates are ranked by their normalized loss reduction along this trajectory, requiring only forward passes and allowing the path to be reused across distinct pools for a fixed model-target pair. Empirically, TACS achieves competitive downstream performance across controlled logistic,
vision, and NLP
benchmarks, while slashing selection compute by up to 69% and candidate storage to less than 10 MB in large-scale instruction tuning.
Read the full abstract ▾
All authors Huitao Yang, Hengzhi He, Tung Sum Thomas Kwok, Guang Cheng
Computation and Language · Cross-listed from cs.CV
EduStory: A Unified Framework for Pedagogically-Consistent Multi-Shot STEM Instructional Video Generation
Xinyi Wu, Jayant Teotia, Shuai Zhao et al. (+1 more)
Long-horizon
video generation has advanced in visual quality, yet existing methods still struggle to maintain knowledge consistency and coherent pedagogical narratives across multi-shot instructional
videos, especially in STEM domains. To address these challenges, we propose EduStory, a unified framework for reliable instructional
video generation. EduStory integrates pedagogical state modeling to track persistent knowledge states, script-guided structured control to organize multi-shot narratives, and learning-oriented evaluation metrics to assess knowledge fidelity and constraint satisfaction. To support rigorous evaluation, we further introduce EduVideoBench, a diagnostic
benchmark with multi-granularity annotations, including pedagogical storyboards, shot-level semantics, and knowledge state transitions, together with baseline tasks for controllable instructional
video generation. Extensive experiments demonstrate that domain-aware state modeling and structured control substantially reduce narrative breakdown and improve
alignment with instructional intent. These results highlight the significance of domain-specific structural constraints and tailored
benchmarks for advancing reliable, controllable, and also trustworthy long-horizon
video generation.
Read the full abstract ▾
All authors Xinyi Wu, Jayant Teotia, Shuai Zhao, Erik Cambria
Computation and Language · Cross-listed from cs.AI
Position: Avoid Overstretching LLMs for every Enterprise Task
Kuldeep Singh, Anson Bastos and Isaiah Onando Mulang'
Enterprise workloads are dominated by deterministic, structured, and knowledge-dependent tasks operating under strict cost, latency, and reliability constraints. While these are often addressed through large
language model (LLM) deployment or
distillation into smaller models, we argue this is inefficient, unreliable, and misaligned with enterprise task structures. Instead, AI systems should treat
language models as interfaces rather than monolithic engines, externalizing knowledge and computation into dedicated components for greater reliability, scalability, and transparency. Our theoretical evidences show that finite-capacity models cannot fully capture the breadth of knowledge required for enterprise tasks, creating inherent limits to efficiency and interpretability. Building on this, we take the position that
language models should primarily be used for structured extraction in deterministic enterprise workflows, while computation and storage are delegated to knowledge bases and symbolic procedures. We formally demonstrate that such modular architectures are more reliable and maintainable than monolithic frameworks, offering a sustainable foundation for enterprise tasks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code
Zhenghan Song, Yulong Liu, Cheng Wan et al. (+4 more)
Execution-based evaluation of LLM-generated code implicitly treats successful execution as a proxy for correctness. In scientific simulation, this proxy is insufficient: a generated input file can run, mesh, and converge while encoding governing equations that differ from the user's intent. We call this mismatch between intended physics and generated code the comprehension-generation gap. We instantiate this in MOOSE, where Kernel and BC objects map compositionally to weak-form residual terms, enabling deterministic reconstruction of the encoded PDE and comparison against an intended contract. We formalize this comparison as the Intent Fidelity Score (IFS), a structural metric covering governing terms, BCs, ICs, coefficients, and time scheme. Building on IFS, we develop a PDE-grounded refinement loop that uses deterministic violation reports to correct generated code iteratively. We evaluate on MooseBench, a 220-case multiphysics
benchmark with PDE-level ground truth released with this work. On this
benchmark, our method consistently improves mean IFS over direct generation, with gains concentrated on hard cases. On the subset where direct generation falls below IFS 0.7, refinement adds +0.22 to +0.41 absolute IFS. In the deployment audit, execution-only repair improves execution success while leaving 39-40% of all 220 cases runnable but still solving the wrong physics across the three main deployment-audit models, exposing executability and intent fidelity as separable failure modes. Static proof-of-concept experiments on four PDE-oriented DSLs (UFL/FEniCS, FreeFEM, FiPy, and Devito) suggest that the reconstruction-and-comparison pattern extends beyond MOOSE. These findings reinforce that executable simulation code should be verified against the mathematical structure it is intended to encode, not accepted on execution alone.
Read the full abstract ▾
All authors Zhenghan Song, Yulong Liu, Cheng Wan, Chenjun Li, Lingfu Liu, Yunyi Li, Congcong Yuan
Computation and Language
HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities
Shusaku Egami, Aoi Ohta, Tomoki Tsujimura et al. (+5 more)
Large
Language Models (LLMs) provide flexible natural
language processing capabilities, while knowledge graphs (KGs) offer explicit and structured knowledge. Integrating these two in a complementary manner enables the development of reliable and verifiable AI systems. In particular, knowledge graph question answering (KGQA) has attracted
attention as a means to reduce LLM
hallucinations and to leverage knowledge beyond the training data. However, existing KGQA
benchmark datasets are biased toward encyclopedic knowledge, limited to a single modality, and lack fine-grained spatiotemporal data, which limits their applicability to real-world scenarios targeted by
Embodied AI. We introduce HOME-KGQA, a novel KGQA
benchmark dataset built on a
multimodal KG of daily household activities. HOME-KGQA consists of complex, multi-hop natural
language questions paired with graph database query
languages. Compared to existing
benchmarks, it includes more challenging questions that involve multi-level spatiotemporal
reasoning,
multimodal grounding, and aggregate functions. Experimental results show that the LLM-based KGQA methods fail to achieve performance comparable to that on existing
datasets when evaluated on HOME-KGQA. This highlights significant challenges that should be addressed for the real-world deployment of KGQA systems. Our
dataset is available at https://github.com/aistairc/home-kgqa
Read the full abstract ▾
All authors Shusaku Egami, Aoi Ohta, Tomoki Tsujimura, Masaki Asada, Tatsuya Ishigaki, Ken Fukuda, Masahiro Hamasaki, Hiroya Takamura
Computation and Language
RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors From Multi-Step to One-Step
Xiaocheng Luo, Kang Wang, Zaifu Zhan et al. (+2 more)
The Chain-of-Thought (CoT) paradigm, while enhancing the interpretability of Large
Language Models (LLMs), is constrained by the inefficiencies and expressive limits of natural
language. Latent Chain-of-Thought (latent CoT)
reasoning, which operates in a continuous latent space, offers a promising alternative but faces challenges from structural complexities in existing multi-step or multi-model paradigms, such as error propagation and coordination overhead. In this paper, we introduce One-Model One-Step, a novel compression framework for Latent
Reasoning with Rule-Based Priors(RuPLaR) to address this challenge. Our method trains an LLM to autonomously generate latent
reasoning tokens in a single training stage, guided by rule-based prior probability distributions, thereby eliminating cascaded processes and inter-model dependencies. To ensure
reasoning quality, we design a joint training objective that enforces answer consistency via cross-
entropy, aligns soft
token distributions with rule-based priors via KL divergence (the Soft Thinking constraint), and adds a problem-thought semantic
alignment constraint in the representation space. Extensive experiments show that our compression framework not only improves accuracy by 11.03% over existing latent CoT methods on GSM8k-Aug but also achieves this with minimal
token usage, underscoring its effectiveness and extensibility. Code: https://github.com/xiaocen-luo/RuPLaR.
Read the full abstract ▾
All authors Xiaocheng Luo, Kang Wang, Zaifu Zhan, Yuechi Zhou, Xiangyu Duan
Computation and Language · Cross-listed from cs.MA
SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent System
Shuai Pan, Yixiang Liu, Jiaye Gao et al. (+7 more)
Large
language model (LLM)
agent systems are increasingly expected to improve after deployment, but existing work often decouples two adaptation targets: skill
evolution and multi-
agent system (MAS) restructuring. This separation can create organization bottlenecks, context pressure, and mis-specialization. We present SkillMAS, a non-parametric framework for adaptive specialization in multi-
agent systems that couples skill
evolution with MAS restructuring. SkillMAS uses Utility Learning to assign credit from verified execution traces, bounded skill
evolution to refine reusable procedures without unfiltered library growth, and evidence-gated MAS restructuring when retained failures and Executor Utility indicate a structural mismatch. Across
embodied manipulation, command-line execution, and retail workflows, SkillMAS is competitive under the reported harnesses while clarifying how post-deployment specialization is attributed, updated, and applied.
Read the full abstract ▾
All authors Shuai Pan, Yixiang Liu, Jiaye Gao, Te Gao, Weiwen Liu, Jianghao Lin, Zhihui Fu, Jun Wang, Weinan Zhang, Yong Yu
Computation and Language
Test-Time Speculation
Avinash Kumar, Sujay Sanghavi and Poulami Das
Speculative decoding accelerates LLM inference by using a fast draft model to generate
tokens and a more accurate target model to verify them. Its performance depends on the $\textit{acceptance length}$, or number of draft
tokens accepted by the target. Our studies show that the acceptance length of even state-of-the-art speculators, like DFlash, EAGLE-3 and PARD degrade with generation length, reaching values close to 1 (i.e. no speedup) within just a few thousand output
tokens, making speculators ineffective for long-response tasks. Acceptance lengths decline because most speculators are trained offline on short sequences, but are forced to match the target model on much longer outputs at inference, well beyond their training distribution. To address this issue, we propose $\textit{Test-Time Speculation (TTS)}$, an online
distillation approach that continuously adapts the speculator at test-time. TTS leverages the key insight that the
token verification step already invokes the target model for each draft
token, providing the training signal needed to adapt the draft at no additional cost. Treating the draft as the student and the target as a teacher, TTS adjusts the draft over several speculation rounds, with each update improving the draft's accuracy as generation proceeds. Our results across multiple models from the Qwen-3, Qwen-3.5, and Llama3.1 families show that TTS improves acceptance lengths over state-of-the-art speculators by up to $72\%$ and $41\%$ on average, with the benefits scaling with increased generation lengths.
Read the full abstract ▾
Computation and Language
Mem-W: Latent Memory-Native GUI Agents
Guibin Zhang, Yaohui Ling, Fanci Meng et al. (+2 more)
GUI
agents are beginning to operate the web, mobile, and desktop as interactive worlds, where successful control depends on carrying forward visual, procedural, and task-level evidence beyond the fleeting present screen. Yet most
agents still treat
memory as an external, human-readable artifact: histories are summarized, categorized, retrieved, and reinserted as text or structured records before being encoded again by the policy. This creates a mismatch between the representational form in which experience is stored and the latent
embedding sequence over which modern GUI policies actually act. We introduce Mem-W, a series of latent-
memory-native GUI
agents that treat
memory as part of the
agent's continuous context rather than as an auxiliary symbolic scaffold. Mem-W weaves both historical trajectories (as experiential
memory) and in-session segments (as working
memory) into compact
memory tokens through a shared trajectory-to-latent compressor. These
tokens are woven with the current GUI observation and local context into one continuous
embedding sequence, allowing the
agent to read successes, failures, and unfinished progress through the same machine-native interface. Mem-W is trained with self-
distillation and outcome-aware supervision to preserve decision-relevant state while filtering
memory toward evidence that truly supports task success. Across four web and mobile
navigation benchmarks, Mem-W consistently improves diverse backbones and
memory-enhanced baselines, with gains of up to $+30.0$, suggesting that latent-context-native
memory can serve as a scalable foundation for long-horizon GUI agency.
Read the full abstract ▾
All authors Guibin Zhang, Yaohui Ling, Fanci Meng, Kun Wang, Shuicheng Yan
Computation and Language · Cross-listed from cs.AI
Do Self-Evolving Agents Forget? Capability Degradation and Preservation in Lifelong LLM Agent Adaptation
Ye Yu, Xiaopeng Yuan, Haibo Jin et al. (+3 more)
Recent advances in LLM
agents enable systems that autonomously refine workflows, accumulate reusable skills, self-train their underlying models, and maintain persistent
memory. However, we show that such self-
evolution is often non-monotonic: adapting to new task distributions can progressively degrade previously acquired capabilities across all major
evolution channels. We identify this phenomenon as capability erosion under self-
evolution and show that it consistently emerges across workflow, skill, model, and
memory evolution. To mitigate this issue, we propose Capability-Preserving
Evolution (CPE), a general stabilization principle that constrains destructive capability drift during continual adaptation. Across all four
evolution dimensions, CPE consistently improves retained capability stability while preserving adaptation performance. For example, in workflow
evolution, CPE improves retained simple-task performance from 41.8% to 52.8% under GPT-5.1 optimization while simultaneously achieving stronger complex-task adaptation. Our findings suggest that stable long-horizon self-evolving
agents require not only acquiring new capabilities, but also explicitly preserving previously learned ones during continual adaptation.
Read the full abstract ▾
All authors Ye Yu, Xiaopeng Yuan, Haibo Jin, Heming Liu, Yaoning Yu, Haohan Wang
Computation and Language
LEAF-SQL: Level-wise Exploration with Adaptive Fine-graining for Text-to-SQL Skeleton Prediction
Zhao Tan, Xiping Liu, Qing Shu et al. (+3 more)
Text-to-SQL translates natural
language questions into executable SQL queries, enabling intuitive database access for non-experts. While large
language models achieve strong performance on Text-to-SQL with prompting, they still struggle with complex queries that involve deeply nested logic or multiple clauses. A widely used approach employs SQL skeletons–intermediate representations of query logic–to streamline generation, but existing methods are limited by their reliance on a single structural hypothesis and lack of progressive
reasoning. To overcome these limitations, we propose LEAF-SQL, a novel framework that reframes skeleton prediction as a coarse-to-fine tree search process. LEAF-SQL enables systematic exploration of diverse structural hypotheses with adaptive refinement. Several key techniques are employed in LEAF-SQL: (1) a three-level skeleton hierarchy to guide the search, (2) a Skeleton Formulation
Agent to generate diverse candidates, and (3) a Skeleton Evaluation
Agent to efficiently prune the search space. This integrated design yields skeleton candidates that are both structurally diverse and granularity-adaptive, providing a stronger foundation for the SQL generation. Extensive experiments show that LEAF-SQL consistently improves the performance of various LLM backbones. On the official hidden test set of the challenging BIRD
benchmark, our method achieves 71.6 execution accuracy, which outperforms leading search-based and skeleton-based methods, affirming its effectiveness for complex queries.
Read the full abstract ▾
All authors Zhao Tan, Xiping Liu, Qing Shu, Qizhi Wan, Dexi Liu, Changxuan Wan
Computation and Language
BetaEdit: Null-Space Constrained Sequential Model Editing
Bingqing Liu, Wei Liu and Yuhua Li
Null-space-based methods have garnered considerable
attention in model editing by constraining updates to the null space of the pre-existing knowledge representation, thereby preserving the model's original behavior. However, in practice these methods rely on an approximate null space–leading to knowledge leakage–and further suffer from severe performance degradation during sequential editing. Recent work shows that history-aware editing strategies can empirically mitigate this decline, yet the underlying reason remains unclear. In this paper, we first expose the knowledge leakage inherent in existing null-space approaches and then analyze why history-aware updates effectively preserve both editing performance and general capabilities during long-horizon editing. Building on these insights, we propose BetaEdit, a refined framework that effectively controls the knowledge leakage and integrates history-aware updates into the null-space paradigm. Extensive experiments on three large
language models across two standard
benchmarks show that BetaEdit consistently outperforms prior methods in the challenging regime of massive-scale sequential editing. Code is available at: https://github.com/lbq8942/BetaEdit.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
A Prompt-Aware Structuring Framework for Reliable Reuse of AI-Generated Content in the Agentic Web
Shusaku Egami and Masahiro Hamasaki
The
evolution of Large
Language Models (LLMs) and the software
agents built on them (AI
agents) marks a turning point in the transition from a human-centric Web to an “Agentic Web” driven by AI
agents. However, for AI-Generated Content (AIGC), which is expected to dominate the Web, there is currently no mechanism for
agents to verify its reliability, reproducibility, or license compliance during generation. This lack of transparency risks causing chained
hallucinations and compliance violations through the reuse of AIGC. Consequently, a framework to manage the provenance and generation conditions of AIGC is essential. In this paper, we present a framework that automatically attaches structured metadata to AIGC at generation time, including modularized
prompts, contexts, thoughts, model information, hyperparameters, and confidence. The metadata is enveloped together with verifiable credentials to support the reliable assessment and reuse of AIGC. This framework enables efficient curation of structured AIGC and facilitates its safe use for applications such as fine-tuning and knowledge
distillation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Towards Conversational Medical AI with Eyes, Ears and a Voice
Meet Shah, Jason Gusdorf, Anil Palepu et al. (+50 more)
The practice of medicine relies not only upon skillful dialogue but also on the nuanced exchange and interpretation of rich auditory and visual cues between doctors and patients. Building on the low-latency voice and
video processing capabilities of Gemini, we introduce AI co-clinician, a first-of-its-kind conversational AI system utilizing continuous streams of audio-visual data from live patient conversations to inform real-time clinical decisions. Its dual-
agent architecture balances deep clinical
reasoning with the low latency required for natural dialogue. To assess this system, we implemented a
video-based interface emulating telemedicine consultations. We crafted 20 standardized outpatient scenarios requiring proactive real-time auditory and visual
reasoning and designed "TelePACES" evaluation criteria alongside case-specific rubrics. In a randomized, interface-blinded, crossover simulation study (n = 120 encounters) with 10 internal medicine residents as patient actors, we compared AI co-clinician with primary care physicians (PCPs), GPT-Realtime, and a baseline
agent. AI co-clinician approached PCPs in key TelePACES dimensions, including management plans and differential diagnosis, while significantly outperforming GPT-Realtime across all general criteria. While our
agent demonstrated parity with PCPs in case-specific triage measures, physicians maintained superior overall performance in case-specific assessments. Although AI co-clinician marks a significant advance in real-time telemedical AI, gaps remain in physical examination and disease-specific
reasoning. Our work shows that text-only approaches fail to capture the true challenges of medical consultation and suggests that high-stakes real-time diagnostic AI is most safely advanced in collaborative, triadic models where AI can be a supportive co-clinician for doctors and patients.
Read the full abstract ▾
All authors Meet Shah, Jason Gusdorf, Anil Palepu, Chunjong Park, Jack W. O'Sullivan, Vishnu Ravi, Tim Strother, Pavel Dubov, Aliya Rysbek, Toshiyuki Fukuzawa, Yana Lunts, Jan Freyberg, Michael B. Chang, Aniruddh Raghu, David Stutz, Devora Berlowitz, Eliseo Papa, Taylan Cemgil, JD Velasquez, Jack Chen, Arthur Chen, Doug Fritz, Charlie Taylor, Katya Tregubova, Jing Rong Lim, Richard Green, Sara Mahdavi, Mahvish Nagda, Jihyeon Lee, Craig Schiff, Liviu Panait, Sukhdeep Singh, Valentin Liévin, David G.T. Barrett, Hannah Gladman, Anna Cupani, Francesca Pietra, Uchechi Okereke, Katherine Tong, Clemens Meyer, Erwan Rolland, Mili Sanwalka, Michael D. Howell, Shixiang Shane Gu, Bibo Xu, Euan A. Ashley, S. M. Ali Eslami, Gregory Wayne, Pushmeet Kohli, Vivek Natarajan, Adam Rodman, Alan Karthikesalingam, Ryutaro Tanno
Computation and Language
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
Rui Liu, Dian Yu, Zhenwen Liang et al. (+6 more)
Aligning
Multimodal Large
Language Models (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting
language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to
multimodal tasks is bottlenecked by the complexity of visual
reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce $\textbf{DeltaRubric}$, an approach that reformulates
multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a $\textit{Disagreement Planner}$, the model generates a neutral, instance-specific verification checklist. Transitioning into a $\textit{Checklist Verifier}$, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by $\textbf{+22.6}$ (4B) and $\textbf{+18.8}$ (8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable
multimodal reward modeling.
Read the full abstract ▾
All authors Rui Liu, Dian Yu, Zhenwen Liang, Yucheng Shi, Tong Zheng, Runpeng Dai, Haitao Mi, Pratap Tokekar, Leoweiliang
Computation and Language
Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
Aditya Sinha, Harald Steck, Vito Ostuni et al. (+1 more)
Users interacting with Large
Language Models (LLMs) in a multi-turn conversation routinely refine their requests or pivot to new topics. LLMs, however, often miss these topic shifts and carry over irrelevant context from previous turns, leading to inaccurate responses. In this paper, we stress-test the multi-turn understanding of LLMs and study the following two sub-tasks: (1) detecting whether the user pivots or refines in the current turn, and (2) shortlisting relevant context from previous turns. To this end, we construct synthetic
benchmarks based on real-world
datasets from varied domains, as to simulate context shifts of different levels of difficulty. We then evaluate the zero-shot performance of ten LLMs (open-weight, closed-source and
reasoning), and demonstrate that only some
reasoning and strongly instructed LLMs are accurate in detecting pivots; open-weight LLMs struggle with the task and frequently carry stale context even with explicit cues; and all models suffer from a position bias. Based on the results, we discuss key takeaways for improving long-term robustness in multi-turn capabilities for LLMs.
Read the full abstract ▾
All authors Aditya Sinha, Harald Steck, Vito Ostuni, Matteo Rinaldi
Computation and Language · Cross-listed from cs.CV
Reinforcing Multimodal Reasoning Against Visual Degradation
Rui Liu, Dian Yu, Haolin Liu et al. (+6 more)
Reinforcement Learning has significantly advanced the
reasoning capabilities of
Multimodal Large
Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from
vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing
reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce
reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a
token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven
multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.
Read the full abstract ▾
All authors Rui Liu, Dian Yu, Haolin Liu, Yucheng Shi, Tong Zheng, Runpeng Dai, Haitao Mi, Pratap Tokekar, Leoweiliang
Computation and Language
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta et al. (+2 more)
While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical
tokens disproportionately drives
reasoning gains, an analogous
token-level understanding of On-Policy
Distillation (OPD) remains largely unexplored. In this work, we investigate high-loss
tokens, a
token type that–as the most direct signal of student-teacher mismatch under OPD's per-
token KL objective–should progressively diminish as training converges according to existing studies; however, our empirical analysis shows otherwise. Even after OPD training reaches apparent saturation, a substantial subset of
tokens continues to exhibit persistently high loss; these
tokens, which we term Rock
Tokens, can account for up to 18% of the
tokens in generated outputs. Our investigation reveals two startling paradoxes. First, despite their high occurrence frequency providing a disproportionately large share of total gradient norms, Rock
Tokens themselves remain stagnant throughout training, resisting teacher-driven corrections. Second, through causal intervention, we find that these
tokens provide negligible functional contribution to the model's actual
reasoning performance. These findings suggest that a vast amount of optimization bandwidth is spent on structural and discourse residuals that the student model cannot or need not internalize. By deconstructing these dynamics, we demonstrate that strategically bypassing these “stumbling blocks” can significantly streamline the
alignment process, challenging the necessity of uniform
token weighting and offering a more efficient paradigm for large-scale model
distillation.
Read the full abstract ▾
All authors Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang
Computation and Language
LLM Agents Already Know When to Call Tools – Even Without Reasoning
Chung-En Sun, Linbo Liu, Ge Yan et al. (+2 more)
Tool-augmented LLM
agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and latency, yet no existing
benchmark systematically studies when a tool call is actually needed. We propose When2Tool, a
benchmark of 18 environments (15 single-hop, 3 multi-hop) spanning three categories of tool necessity – computational scale, knowledge boundaries, and execution reliability – each with controlled difficulty levels that create a clear decision boundary between tool-necessary and tool-unnecessary tasks. We evaluate two families of training-free baselines:
Prompt-only (varying the
prompt to discourage unnecessary calls) and Reason-then-Act (requiring the model to reason about tool necessity before acting). Both provide limited control:
Prompt-only suppresses necessary calls alongside unnecessary ones, and Reason-then-Act still incurs a disproportionate accuracy cost on hard tasks. To understand why these baselines fail, we probe the models' hidden states and find that tool necessity is linearly decodable from the pre-generation representation with AUROC 0.89–0.96 across six models, substantially exceeding the model's own verbalized
reasoning. This reveals that models already know when tools are needed, but fail to act on this knowledge during generation. Building on this finding, we propose Probe&Prefill, which uses a lightweight linear probe to read the hidden-state signal and prefills the model's response with a steering sentence. Across all models tested, Probe&Prefill reduces tool calls by 38% with only 1.7% accuracy loss, while the best baseline at comparable accuracy only reduces 6% of tool calls, or achieves a similar tool call reduction but incurs a 5$\times$ higher accuracy loss. Our code is available at https://github.com/Trustworthy-ML-Lab/when2tool
Read the full abstract ▾
All authors Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, Tsui-Wei Weng
Computation and Language
Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs
Sohan Venkatesh
Large
language models fail at counting how many times a word repeats in a list, even though they perform well on far harder
reasoning tasks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-
embedding layer and they do so even at the exact layers where the wrong answer crystallizes in the output.
Attention patterns show no evidence of collapse over repeated
tokens and tokenization artifacts account for none of the failure. Instead, a multi-layer perceptron (MLP) block at roughly 85–93% network depth overwrites the correctly-encoded count with a fixed wrong answer. Ablating this block changes the wrong output and establishes it as causally responsible for the failure. The block fires on the space-separated repeated-word format and is absent for repeated digit-
tokens. The pattern holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. The count is represented correctly and a specific computation prevents it from reaching the output, so representation failures and routing failures require different interventions.
Read the full abstract ▾
Computation and Language
Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke
Yu Wu, Ananth Mahadevan, Filip Ginter et al. (+2 more)
While digitized corpora have transformed the study of intellectual transmission, current methods rely heavily on lexical text reuse detection, capturing verbatim quotations but fundamentally missing paraphrases and complex implicit engagement. This paper evaluates semantic search in 18th-century intellectual history through the reception of John Locke's foundational work. Using expert annotation grounded in a semantic taxonomy, we examine whether an off-the-shelf semantic search pipeline can surface meaning-level correspondences overlooked by lexical methods. Our results demonstrate that semantic search retrieves substantially more implicit receptions than lexical baselines. However, linguistic diagnostics also reveal a "lexical gatekeeping" effect, where
retrieval remains partially constrained by surface vocabulary overlap. These findings highlight both the potential and the limitations of semantic
retrieval for analyzing the circulation of ideas in large historical corpora. The data is available at https://github.com/COMHIS/locke-sim-data.
Read the full abstract ▾
All authors Yu Wu, Ananth Mahadevan, Filip Ginter, Michael Mathioudakis, Mikko Tolonen