Computation and Language · Cross-listed from cs.AI
Directional Hallucinations: Ideological Drift in News-Grounded LLM Question Answering
Chendi Wang, Liam Cunningham, Tom Yishay et al. (+1 more)
Large
language models (LLMs) are increasingly used to answer questions about political information, including in election-adjacent information settings where factual errors and ideological distortions are high-stakes. We present a reproducible measurement framework that treats
hallucinations, unsupported statements in document-grounded QA, as diagnostic signals of ideological drift. Using 21,727 expert-labeled U.S. political news articles from QBias spanning left, center, and right sources, we (i) generate an article-specific question, (ii) elicit document-grounded answers from three open-weight LLMs and one proprietary model, (iii) detect sentence-level
hallucinations via reference-based comparison, (iv) classify the ideological valence of hallucinated sentences with a fine-tuned stance classifier, and (v) probe output logits to relate
token-level uncertainty to
hallucination and drift.
Hallucination rates vary substantially across models and concentrate in contentious topics, while source-ideology differences in
hallucination frequency are modest. In contrast,
hallucination content exhibits robust leftward drift: a majority of hallucinated sentences are classified as left-leaning, including among
hallucinations generated from right-leaning sources. Logit-level analysis shows
hallucinations arise in high-
entropy generation contexts, and in some models uncertainty also predicts leftward drift, consistent with an "uncertainty to guessing" mechanism. We discuss implications for
auditing AI-mediated political information and for designing safeguards in election-relevant deployments.
Read the full abstract ▾
All authors Chendi Wang, Liam Cunningham, Tom Yishay, Jieying Chen
Computation and Language · Cross-listed from cs.AI
Expectation Alignment of Language Models for Real-World User Expectations
Miaomiao Li, Yang Wang, Bin Liang et al. (+3 more)
Large
language models (LLMs) have demonstrated remarkable performance on standard
benchmarks, yet it remains largely unexplored whether they truly meet user expectations. Existing evaluation approaches, relying on model heuristics, expert rubrics, or user simulation, fail to capture the diversity and subtlety of real human expectations, causing models to appear competent while misaligning with what users actually seek. We present the first systematic study of user expectations in real-world LLM interactions, proposing a principled procedure to extract semantically rich expectations and introducing ExpectBench, a
benchmark grounded in real user expectations. Analyses reveal that current LLMs struggle to satisfy and anticipate what users hope to obtain, highlighting a fundamental source of misalignment. Building on these observations, we propose LENS, a lightweight latent expectation-aware response generation framework. LENS enables models to internalize user expectations and generate better-aligned responses, consistently improving expectation satisfaction and underscoring the importance of explicitly modeling user expectations for realistic human-AI
alignment.
Read the full abstract ▾
All authors Miaomiao Li, Yang Wang, Bin Liang, Shudong Liu, Zhiwei Zhang, Kam-Fai Wong
Computation and Language · Cross-listed from cs.AI
Trajectory-Aware Clinical Risk Prediction via Severity-Grounded Knowledge Graphs and Retrieval-Augmented Generation
Kyunghoon Jeon, Youmin Ko, Woohwan Jung et al. (+1 more)
While Electronic Health Records (EHRs) offer a wealth of clinical data, effectively augmenting a patient's records with heterogeneous external knowledge to predict the patient's clinical risk remains a significant challenge. Existing methods fail to capture disease severity, treatment responses, and nuanced clinical progression, due to data sparsity and the underutilization of unstructured clinical notes. To address these challenges, we propose TRACER (a trajectory-aware and clinically grounded prediction framework) that (1) constructs a medical knowledge graph enriched with severity information from medical literature, (2) retrieves clinically relevant, severity-weighted paths of a patient's progression from the knowledge graph, (3) extracts clinically relevant events from unstructured clinical notes, and (4) augments patient context with similar peer cases. Experiments on the MIMIC-III and MIMIC-IV
datasets demonstrate large gains over state-of-the-art baselines, with up to 28.5% increase in Macro F1 score for the mortality prediction task, and 19.7% increase for the readmission prediction task.
Read the full abstract ▾
All authors Kyunghoon Jeon, Youmin Ko, Woohwan Jung, Hyunjoon Kim
Computation and Language · Cross-listed from cs.CY
Structuring the Space of Sociotechnical Alignment
Esra Dönmez and Agnieszka Falenska
Sociotechnical
alignment concerns the social desirability of AI behavior and is thus inherently normative, not merely technical. While NLP research increasingly addresses its technical aspects, it often leaves underspecified what such "social desirability" entails. We argue that this reflects a fundamental gap: the absence of a systematic way to specify how sociotechnical
alignment defines, justifies, and evaluates socially desirable AI behavior. To address this gap, we introduce a human-centered framework for specifying sociotechnical
alignment. We draw on social-scientific accounts of sociobehavioral desirability to ground the basis for behavioral desirability judgments and use this framework to analyze how
alignment is specified in practice. Our systematic literature review identifies recurring patterns: normative concepts grounding desirability judgments are often unspecified or conflated with
alignment targets for (desired) system behavior, target populations are underdefined, and design choices are rarely theoretically justified. These findings point to a lack of conceptual specificity that limits cumulative progress. We therefore offer recommendations that link social-scientific frameworks to
alignment design choices, supporting more conceptually precise approaches to sociotechnical
alignment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
Auditing LLM-Governed Social Robots with Culture-Specific Moral Gradients
Carmen Ng and Gjergji Kasneci
LLM-governed social
robots increasingly decide who receives real-world assistance first. As prioritization norms vary across cultures by age, status, and group size, failure to calibrate pluralistically can scale into unequal access. Yet LLM moral audits remain English-centered, rarely test
embodied contexts, leaving pluralistic calibration as an urgent diagnostic gap amid intensifying LLM-
robot deployment. We introduce a gradient-based audit framework for
multilingual evaluation of LLM moral trade-off behavior against cultural preference gradients. Grounded in nine cross-domain social robotics reviews (>8,000 papers), we derive
symmetry-controlled scenarios across care, education, and services, translating the Moral Machine Experiment's "whom to spare" into "whom to assist first" dilemmas with preserved identity trade-offs (many vs. few; young vs. old; higher vs. lower status). We audit four LLMs across four country-
language pairs in four prompting regimes (57,600 decisions), benchmarked against country-specific MME preference gradients. Ordinal concordance tests whether models differentiate cultural contexts; a governance typology maps vulnerabilities in gradient differentiation, directional tendency, and deliberation. We find persistent, culturally asymmetric gradient tracking failures that prompting alone cannot reliably correct: quality calibration is nearly twice as strong for Western-
language decisions as for Chinese and Japanese; high determinism in majority-first trade-offs often erases cross-cultural gradients; partial sensitivity to age- and status-based norms risks sidelining minorities. Prompting effects are uneven; only contrastive exemplars yield consistent gains, while
reasoning-only
prompts can worsen tracking. Our results motivate
multilingual, pluralistic audits as an LLM-
robot pre-deployment gate and suggest model factors are a more robust lever than prompting alone.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SkillHarness: Harnessing Safe Skills for Computer-Use Agents
Yurun Chen, Biao Yi, Keting Yin et al. (+1 more)
Computer-Use
Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from
adversarial interactions (e.g.,
prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
Read the full abstract ▾
All authors Yurun Chen, Biao Yi, Keting Yin, Shengyu Zhang
Computation and Language · Cross-listed from cs.CV
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Ruipeng Zhang, Zhihao Li, C. L. Philip Chen et al. (+1 more)
Large
vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet
hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects
token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical
tokens, and can cause instability. To address these limitations, we propose
Token-Level Visual-Sensitivity Steering (TLVS) for
hallucination mitigation. Our approach first extracts
token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse
vision-
language models. It modulates the steering strength at each decoding step, selectively suppressing
hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several
benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.
Read the full abstract ▾
All authors Ruipeng Zhang, Zhihao Li, C. L. Philip Chen, Tong Zhang
Computation and Language
Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings
Linsen Li, Aron Culotta and Nicholas Mattei
Online reviews provide valuable insights into the perceived quality of facets of a product or service. While aspect-based sentiment analysis has focused on extracting these facets from reviews, there is less work understanding the impact of each aspect on overall perception. This is particularly challenging given correlations among aspects, making it difficult to isolate the effects of each. This paper introduces a methodology based on recent advances in text-based causal analysis, specifically CausalBERT, to disentangle the effect of each factor on overall review ratings. We enhance CausalBERT with three key improvements: temperature scaling for better calibrated treatment assignment estimates; hyperparameter optimization to reduce confound overadjustment; and interpretability methods to characterize discovered confounds. In this work, we treat the textual mentions in reviews as proxies for real-world attributes. We validate our approach on real and semi-synthetic data from over 600K reviews of U.S. K-12 schools. We find that the proposed enhancements result in more reliable estimates, and that perception of school administration and performance on
benchmarks are significant drivers of overall school ratings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Yifan Wang, Jinyi Mu, Mayank Jobanputra et al. (+5 more)
Preference modeling aims to capture heterogeneous human values and increasingly relies on learning latent components to represent this variation without fine-grained supervision. What these components actually represent, however, is rarely examined. We audit Mixture-of-Experts (MoE) reward models along three rungs—semantic coherence, functional correspondence, and composability—in controlled and real-world settings, holding the architecture and training data fixed so that differences isolate the effect of the training objective. Our diagnosis is that the components conventional MoE reward models learn are entangled, which limits not only interpretability but also the personalization such models are built for. The
entanglement is not intrinsic to the architecture but a property of the training objective: off-the-shelf sparsity and diversity regularizers already substantially reduce it. The resulting configuration, which we call sparse MoE, recovers all underlying semantic categories without domain supervision in a controlled setting, reaching 84.8% expert purity against 60.8% for conventional MoE; on real-world data its experts admit human-interpretable task- and domain-level descriptions aligned with their functional strengths. Targeted interventions on routing weights produce predictable preference changes, showing that these interpretations are behaviorally grounded rather than merely descriptive. Crucially, the disentanglement induced by this objective also makes the experts composable: lightweight router updates yield a 25.81-point improvement in few-shot personalization with 50 examples, roughly 2.7 times the gain of the identical architecture and adaptation procedure applied to entangled experts. Together, these results show that MoE reward models can learn interpretable expert structure, and that the same structure can substantially improve personalization.
Read the full abstract ▾
All authors Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
Computation and Language
Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit
JooYoung Lee, Lin Tian, Angela Brillantes et al. (+2 more)
As large
language models (LLMs) become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims (environment, health, immigration), labelled as belief (propagates the claim), fact-check (corrects it), or other. We compare nine models across three paradigms – BART-MNLI, three Llama variants, three commercial frontier LLMs (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6), and fine-tuned DistilBERT and RoBERTa – under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F_1$ against a best zero-shot result of 0.50 (Claude Haiku 4.5), at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-
alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F_1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.
Read the full abstract ▾
All authors JooYoung Lee, Lin Tian, Angela Brillantes, Adriana-Simona Mihăiţă, Marian-Andrei Rizoiu
Computation and Language
Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA
Maroof Kousar, Jinsung Han, Mohit Chandra et al. (+2 more)
Large
language models (LLMs) are increasingly used to answer everyday health questions online, including whether someone can take another dose of an over-the-counter (OTC) medication. Yet this common safety-relevant setting remains underexplored in medical QA. Answering these questions requires checking previous doses, timing, and product-specific dosing limits, while recognizing when important information is missing. We introduce DoseBench, a focused
benchmark of 230 consumer-style acetaminophen and ibuprofen dosing questions with human-annotated YES, NO, and AMBIGUOUS labels, and evaluate eight LLMs with five responses per question. Overall accuracy ranges from 44.8% to 62.2%, while models incorrectly approve 14.0% to 60.5% of prohibited doses. Models with similar accuracy can therefore make very different dosing decisions. Additional analyses show that these failures are not easily fixed by giving models more guidance. Asking models to check dose, timing, and missing information makes them more cautious, while providing drug facts improves accuracy but can make them overlook missing information. These findings highlight a broader challenge in health QA: applying rules correctly and recognizing when information is missing.
Read the full abstract ▾
All authors Maroof Kousar, Jinsung Han, Mohit Chandra, Munmun De Choudhury, Yibo Hu
Computation and Language · Cross-listed from cs.AI
Can Generalist Agents Automate Data Curation?
Feiyang Kang, Hanze Li, Adam Nguyen et al. (+5 more)
Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy
benchmark feedback. We ask whether generalist coding
agents can automate this data-curation loop. We introduce *Curation-Bench*, an
agent-centric
benchmark that fixes the model, training recipe, and evaluation suite while giving
agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a
vision-
language instruction-tuning instantiation, out-of-the-box
agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*:
agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift
agents toward method-guided exploration. The scaffolded
agent autonomously composes – without human design input – a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current
agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and
benchmark are open-sourced.
Read the full abstract ▾
All authors Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia
Computation and Language · Cross-listed from cs.AI
StepPRM-RTL: Stepwise Process-Reward Guided LLM Fine-Tuning for Enhanced RTL Synthesis
Prashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi et al. (+2 more)
Automatic generation of RTL code for digital hardware designs remains challenging due to long-horizon
reasoning, multi-step dependencies, and strict correctness constraints in Verilog and VHDL. We present StepPRM-RTL, a novel framework that combines stepwise trajectory modeling, process-reward modeling (PRM), and
retrieval-augmented fine-tuning (RAFT) to enhance both the functional correctness and
reasoning fidelity of LLM-based RTL code generation. StepPRM-RTL constructs stepwise
reasoning trajectories from canonical solutions, where each step contains a rationale and incremental code modification. A Process Reward Model (PRM) evaluates intermediate steps, providing dense feedback that guides reinforcement-style updates during RAFT fine-tuning. Monte Carlo Tree Search (MCTS) explores alternative
reasoning paths, enriching the training
dataset with high-quality trajectories. This integration of stepwise and outcome-aware rewards allows the model to learn both how and why to construct correct RTL, improving long-horizon
reasoning beyond standard supervised or outcome-based training. Experimental evaluation on
benchmark Verilog and VHDL
datasets demonstrates that StepPRM-RTL outperforms the best prior methods by over 10% in functional correctness and
reasoning fidelity metrics. Ablation studies confirm that the combination of PRM-guided rewards and stepwise trajectory exploration is key to its performance. StepPRM-RTL generalizes across RTL
languages and provides a scalable framework for high-fidelity, interpretable code generation, establishing a new standard for LLM-assisted hardware design automation.
Read the full abstract ▾
All authors Prashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi, Ehsan Degan, Vandana Mukherjee
Computation and Language · Cross-listed from cs.AI
VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark
Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari et al. (+5 more)
Multimodal large
language models are increasingly capable of complex
reasoning, yet their performance often degrades when they must externalize a problem through a tool and then reason over the tool's output, specifically when they rely on visual aids. This gap is especially important because real engineering and scientific workflows often rely on visualization tools for analysis, validation, and decision-making. To study this discrepancy, we introduce VAMPS (Visual-Assisted Mathematical Problem Solving), a
benchmark for graph-assisted mathematics. VAMPS contains 1,168
multimodal, bilingual multiple-choice question-answer pairs drawn from Iranian University Entrance Exam
algebra and calculus problems and expanded with human-reviewed LLM-generated synthetic variants, all selected so that plotting provides a natural solution strategy by revealing intersections, extrema, asymptotes, etc. Designed for both benchmarking and diagnosis, VAMPS goes beyond prior
multimodal benchmarks that primarily evaluate
reasoning over fixed visual inputs by testing whether a model can benefit from constructing a useful graph and grounding its answer in the resulting visualization. Overall, we found that across a diverse set of models, direct analytical solving surprisingly outperforms tool-enabled visual solving, even on problems where plotting is a natural strategy.
Read the full abstract ▾
All authors Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari, Yasamin Medghalchi, Ilker Hacihaliloglu, Mesrob Ohannessian, Lele Wang, Giuseppe Carenini
Computation and Language · Cross-listed from cs.CV
Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)
Jingbiao Mei
Retrieval over visually-rich documents, pages that interleave text with figures, tables, and charts, is essential for
multimodal retrieval-augmented generation, yet most retrievers still discard the visual channel. The
Multimodal Document
Retrieval Challenge, Track~1 of the MIR Challenge at the first EReL@MIR workshop, co-located with The Web Conference 2025, asks participants to build a single
retrieval system that handles two complementary regimes: closed-set document page
retrieval within long documents from a text query (MMDocIR), and open-domain
retrieval of Wikipedia-style passages from an image or image-plus-text query (M2KR). Systems are ranked by the macro-average of mean Recall@$\{1,3,5\}$ over the two tasks. The challenge drew 455 entrants and 586 submissions across 22 teams. This report describes the challenge design,
datasets, and evaluation protocol; reports the final standings; and analyses the three winning teams' systems. All three build on decoder-based
Multimodal-LLM embedders from the Qwen2-VL family rather than on CLIP-style encoders, and differ chiefly in whether they reach the top through fine-tuned ensembles, training-free multi-route fusion with a strong
vision-
language re-ranker, or zero-shot late interaction. The training-free system finished within $0.1$ point of the fine-tuned winner.
Read the full abstract ▾
Computation and Language
Supportive Token Revealing for Fast Diffusion Language Model Decoding
Giries Abu Ayoub, Mario Barbara, Lluís Pastor-Pérez et al. (+4 more)
Discrete
diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent
tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which
tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain
tokens may depend on masked
tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for
diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked
tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which
tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked
tokens that uncertain positions attend to, using
attention, uncertainty, and confidence signals. Experiments on
reasoning and code-generation
benchmarks across multiple
diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
Read the full abstract ▾
All authors Giries Abu Ayoub, Mario Barbara, Lluís Pastor-Pérez, Tanja Bien, Aneesh Barthakur, Alaa Maalouf, Loay Mualem
Computation and Language
MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&A
Hanoz Bhathena, Parin Rajesh Jhaveri, Rohan Mittal et al. (+7 more)
Recent advances in
multimodal retrieval-augmented generation (MM-RAG) have shifted toward minimal parsing, relying on page-level images for producing retriever
embeddings and for answer generation. While efficient, this trend often neglects explicit handling of the rich, structured information in complex enterprise documents, instead depending on pre-trained
embeddings or
vision-
language models to implicitly capture such structure. In this work, we take a more direct approach: MM-BizRAG proactively extracts and represents document structure via a document structure-aware split that dynamically routes documents through orientation-specific ingestion pipelines, applying explicit layout-aware parsing for vertically structured documents (e.g., reports) and holistic page-level representations for horizontally structured documents (e.g., slide decks). A unified LLM-driven artifact transformation pipeline with placeholder-based positional
alignment preserves natural reading order, while inference-time
multimodal assembly decouples
retrieval representations from generation context, enabling richer, more grounded answers without any finetuning requirement. Through experiments on a large, heterogeneous enterprise
dataset and two public
benchmarks (SlideVQA and FinRAGBench-V), MM-BizRAG consistently outperforms state-of-the-art
vision-centric baselines by up to 32% points, with especially strong gains on report-style layouts. Furthermore, we introduce FastRAGEval, a single-call LLM Judge metric for fine-grained generative recall that halves RAGChecker's cost while achieving stronger human
alignment.
Read the full abstract ▾
All authors Hanoz Bhathena, Parin Rajesh Jhaveri, Rohan Mittal, Prateek Singh, Aymen Kallala, Rachneet Kaur, Yiqiao Jin, Zhen Zeng, Adwait Ratnaparkhi, Denis Kochedykov
Computation and Language · Cross-listed from cs.MM
DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities
Sajad Ebrahimi, Nima Jamali, Bardia Shirsalimian et al. (+8 more)
The growing popularity and capacity of generative models have eroded the distinction between human and machine-generated content, motivating a growing body of work on detection across text, images, and audio. Most available detectors are either commercial software or, if open-source, come with incompatible codebases with bespoke preprocessing, evaluation protocols, and evaluation metrics, which make their adoption, fair comparison, and reproduction quite difficult. To address this critical gap, we introduce DetectZoo, a first-of-its-kind, extensible toolkit designed to provide a unified interface for AI-generated content detection across text, audio, and image modalities. DetectZoo standardizes the complete empirical pipeline, from data ingestion and preprocessing to model assessment, offering researchers a cohesive framework to
benchmark state-of-the-art detectors systematically. By integrating diverse public
datasets and baseline detection
algorithms under a single, unified API, our toolkit facilitates rigorous and reproducible evaluation. DetectZoo provides reference implementations of 61 detectors, native loaders for 22
benchmark datasets, and a standardized evaluation pipeline that reports multiple metrics through a common interface. Each detector is self-contained yet accessible through the same interface, automatically caches pretrained weights, and reproduces the original published results. DetectZoo lowers the barrier to entry for multi-modal AI forensics, enabling researchers to identify performance gaps across domains and accelerating the development of robust, generalizable detection techniques. The open-source repository and comprehensive documentation are publicly available at https://github.com/sadjadeb/DetectZoo, and the package can be installed via pip install detectzoo.
Read the full abstract ▾
All authors Sajad Ebrahimi, Nima Jamali, Bardia Shirsalimian, Kelly McConvey, Wentao Zhang, Jalehsadat Mahdavimoghaddam, Maksym Taranukhin, Maura Grossman, Vered Shwartz, Yuntian Deng, Ebrahim Bagheri
Computation and Language
Cross-Prompt Generalization in Detecting AI-Generated Fake News Using Interpretable Linguistic Features
Aya Vera-Jimenez, Samuel Jaeger, Calvin Ibenye et al. (+1 more)
The increasing use of large
language models has raised concerns about the spread of AI-generated fake news, particularly under varying prompting strategies. Most existing detection models are trained and evaluated under a single generation setting, leaving their ability to generalize across unseen
prompts unclear. In this study, we investigate cross-
prompt generalization in fake news detection using three
datasets of AI-generated articles produced under distinct
prompts, combined with real news articles. We extract interpretable linguistic features capturing lexical diversity, readability, and
emotion-based characteristics and evaluate a random forest classifier under a cross-
prompt framework, where models trained on one
prompt are tested on another. Across all six train-test combinations, performance remains consistently high, with AUC values ranging from 0.988 to 1.000. Analysis of feature distributions shows that AI-generated text exhibits increased lexical diversity, reduced readability, and substantially lower emotional intensity compared to the overall
dataset, with variations across
prompts. Despite these distributional shifts, the classifier maintains strong performance, indicating that these features capture stable properties of AI-generated text that generalize across prompting strategies. These findings suggest that feature-based approaches can provide robust detection of AI-generated fake news under
prompt variability.
Read the full abstract ▾
All authors Aya Vera-Jimenez, Samuel Jaeger, Calvin Ibenye, Dhrubajyoti Ghosh
Computation and Language · Cross-listed from cs.MA
Exploring the Topology and Memory of Consensus: How LLM Agents Agree, Fragment, or Settle When Forming Conventions
Aliakbar Mehdizadeh and Martin Hilbert
How much should an LLM
agent remember, and how should multi-
agent systems be connected when trying to reach consensus? We show these two design choices interact in a way that flips the sign of
memory's effect on coordination. Across 432 simulation runs of a networked Naming Game on eight fixed 16-
agent topologies, we vary
memory depth and network structure. Longer
memory slows the time to reach steady state in decentralized networks but accelerates it in centralized ones; the same parameter pushes the system in opposite directions depending on
topology. Critically, "faster settling" in centralized networks means locking in to a fragmented plateau more quickly, not reaching system-wide consensus, which can be used to generate diverging opinions. We further document a
memory-mediated speed-unity trade-off: centralized networks consistently preserve more competing conventions than decentralized networks, but their settling speed depends sharply on
memory. At the
agent level, within-network analyses show that high-betweenness bridges suffer a brokerage penalty while
agents in locally clustered neighborhoods achieve higher coordination success. Finally, in search of analytically tractable generative mechanisms, we find that
agents' choices are well captured by Fictitious Play, indicating belief-based rather than reward-based adaptation. The practical implication:
memory depth and communication
topology should be co-designed, not optimized in isolation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Training-Free Lexical-Dense Fusion for Conversational-Memory Retrieval
Christian Lysenstøen
Retrieving the few past turns that answer a new query across long multi-session histories is the
retrieval bottleneck behind long-term conversational
memory (LoCoMo, LongMemEval). Recent concurrent work, Nano-
Memory, shows that scoring a session by the maximum query-turn similarity (late interaction, "Turn Isolation
Retrieval") beats mean-pooled session
embeddings. We do not claim that effect; we replicate it and ask what a training-free, CPU-only
retrieval stage should add around it. We report four findings. (1) Fuse: score-level fusion of the late-interaction dense score with BM25, under a single leave-one-conversation-out weight, adds +8.8 to +17.2 points of LoCoMo Hit@1 over late interaction alone across six encoders (all p<1e-4), reaching Hit@1 0.752 / NDCG@5 0.829 (e5-large-v2), +11.2 pp over BM25. (2) An off-the-shelf web-search cross-encoder reranker over the fused top-10 hurts here, degrading Hit@1 by 6.9 pp (one reranker, one configuration). (3) A pooling-operator ablation shows top-k late interaction matches max-similarity, but a naive smooth-max (log-sum-exp) collapses for half the encoders. (4) The late-minus-early gap is large for all six encoders and tends to be larger for larger ones, while the marginal fusion gain shrinks; on LongMemEval-S, a lexical regime where BM25 saturates, the net fusion gain over BM25 is small and not significant. A per-category analysis frames the gain as a division of labor: dense late interaction helps most on multi-hop and temporal questions but trails BM25 on
adversarial ones. The contribution is a controlled, reproducible account of a strong training-free
retrieval recipe, not the late-interaction retriever itself (Nano-
Memory's). We make no claim to a complete
memory architecture; this is a
retrieval-stage study.
Read the full abstract ▾
Computation and Language
ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation
Ana-Maria Luisa Mocanu, Ciprian-Octavian Truica and Elena-Simona Apostol
Aspect-Based Sentiment Analysis (ABSA) requires high-quality
datasets to train reliable models. However, existing annotation tools treat output as flat files, leaving researchers to manually consolidate multi-annotator data, reconstruct relational structures, and compute reliability metrics through custom scripts. This paper introduces ACAT (Aspect-based sentiment analysis Collaborative Annotation Tool), a web-based platform natively supporting four ABSA workflows: (1) Aspect-Category Sentiment Analysis, (2) Clause-Level
Segmentation, (3) Aspect-Term Sentiment Analysis with character-level position tracking, and (4) Aspect Sentiment Triplet Extraction with dual span offset preservation. Its core contribution is an automated Extract, Transform, Load (ETL) pipeline that aligns collaborative annotations and computes Inter-Annotator Agreement (IAA) metrics directly at export, yielding training-ready
datasets. In a preliminary validation on 1,002 restaurant reviews with two annotators of differing expertise, ACAT achieves a median annotation time of 31.58 seconds and a raw IAA ranging from 0.78 to 0.86 across all tasks.
Read the full abstract ▾
Computation and Language
A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models
Yassir El Attar, Esra Dönmez, Maximilian Maurer et al. (+1 more)
Interpretable linguistic features offer a promising approach for explaining why a given text appears machine-generated, particularly for non-expert users. However, existing findings on which features reliably indicate LLM-generated text remain fragmented across feature sets, models, and text domains. To address this gap, we conduct a large-scale empirical study assessing the robustness of linguistic signals for characterizing AI-generated text. Our analysis covers 284 interpretable linguistic features across outputs from 27 LLMs and ten text domains under cross-model and cross-domain generalization settings. We show that classifiers based solely on linguistic features can reliably distinguish AI-generated from human-written text. However, many previously proposed indicators prove strongly context-dependent, with the exception of measures of lexical richness, which remain robust signals across model families and text domains. These results demonstrate which linguistic signals generalize across contexts and provide a foundation for more reliable, interpretable analyses of AI-generated
language.
Read the full abstract ▾
All authors Yassir El Attar, Esra Dönmez, Maximilian Maurer, Agnieszka Falenska
Computation and Language
Decomposing Refusal Steering in Mixture-of-Experts Models
Anna C. Marbut, Daniel R. Olson and Travis J. Wheeler
Safety
alignment in instruction-tuned large
language models (LLMs) depends on a model's ability to reliably refuse harmful or disallowed requests. Recent work has shown that a steering vector can be applied to a dense LLM during inference to suppress refusal behavior and induce responses to harmful requests. We extend this refusal steering method to three open-source Mixture-of-Experts (MoE) LLMs to decompose and better understand how refusal mechanisms operate across MoE components. We find that steering performance is uninhibited by the complex routing patterns inherent to the MoE architecture, and that a single expert recovers 78% of the full steering effect on average when free to select its own location. However, expert-level steering only recovers 54% on average when constrained to the location used by full-layer steering, a gap that widens further under safety-related system
prompts. Our results also show that refusal signals captured by these steering methods differ from those captured by expert routing behavior. Together, these results suggest that refusal behavior is distributed unevenly across the MoE LLM architecture rather than governed by a single unified mechanism.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.HC
SocialCoach: Personalized Social Skill Learning with Agentic Tutoring and Practice
Tianfu Wang, Max Xiong, Jianxun Lian et al. (+9 more)
Social skills such as negotiation and leadership are crucial for personal and professional success in today's interconnected world. However, scalable and effective training remains a significant challenge due to the scarcity of expert coaching. In this work, we introduce SocialCoach, an LLM-powered agentic tutoring system for personalized social skill learning. SocialCoach constructs a theory-to-practice corpus of traceable strategies, cases, and practice scenarios, and uses this corpus for both scheduling and reflective tutoring. We formulate social practice personalization as cold-start,
retrieval-constrained sequential practice scheduling. Given a learner profile, simulated proficiency state, and observed practice history, a policy produces structured prescriptions that are realized through corpus
retrieval. To enhance scheduling effectiveness, we optimize complete pathways with trajectory-level GRPO using rubric-judge based pairwise preferences. Additionally, we instantiate the scheduling approach in a deployed platform with goal-driven practice and knowledge-grounded reflective tutoring. Finally, in the synthetic cold-start setting, experiment results show that SocialCoach achieves higher pathway-quality ratings than baselines in scheduling and tutoring quality. We also conduct human studies to demonstrate its usefulness for real-world social skill learning.
Read the full abstract ▾
All authors Tianfu Wang, Max Xiong, Jianxun Lian, Hongyuan Zhu, Zhengyu Hu, Yuxuan Lei, Linxiao Gong, Dapeng Hu, Xiaofang Li, Peiting Tsai, Nicholas Jing Yuan, Qi Zhang
Computation and Language
When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
Erfan Nourbakhsh, Rocky Slavin, Ke Yang et al. (+1 more)
Medical question answering is a high-stakes setting where factual errors can have serious consequences.
Retrieval-augmented generation (RAG) is widely viewed as a promising solution, and prior work has reported substantial gains for large medical QA models. We revisit this assumption across a broad range of open-weight instruction-tuned models spanning 7B to 72B parameters. Across five models, ten biomedical QA
datasets, four
retrieval methods, and four
retrieval corpora, we find that
retrieval yields only small and inconsistent improvements over a no-
retrieval baseline, typically within 1-2 points. In contrast, the choice of backbone model has a much larger effect than the choice of retriever or corpus, and expert and layman
retrieval sources perform similarly in most settings. These results suggest that the main bottleneck is not
retrieval quality alone, but the model's limited ability to use retrieved evidence effectively.
Read the full abstract ▾
All authors Erfan Nourbakhsh, Rocky Slavin, Ke Yang, Anthony Rios
Computation and Language
SaliMory: Orchestrating Cognitive Memory for Conversational Agents
Kai Zhang, Xinyuan Zhang, Hongda Jiang et al. (+11 more)
Conversational
agents that serve as lifelong companions must maintain persistent
memory across all interactions. However, simply expanding context windows with raw
retrieval degrades
reasoning quality, while training
memory agents via standard reinforcement learning creates a severe credit assignment bottleneck in a multi-stage pipeline. To solve this, we introduce SALIMORY, a framework that trains a single
language model to manage a cognitively-structured
memory-spanning user facts, preferences, and working
memory. By introducing a hierarchical stage-wise process reward and reward-decomposed contrastive refinement, SALIMORY provides isolated supervision for distinct
memory operations (selective filtering, consolidation, and cue-driven recall) end-to-end. SALIMORY cuts
memory-attributed failures by one-third, outperforms the state-of-the-art by over 10% in end-to-end accuracy, and more than doubles the Good Personalization rate.
Read the full abstract ▾
All authors Kai Zhang, Xinyuan Zhang, Hongda Jiang, Shiun-Zu Kuo, Hyokun Yun, Ejaz Ahmed, Shereen Oraby, Ziyun Li, Sanat Sharma, Ann Lee, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Xin Luna Dong
Computation and Language
Computational conceptual history of scientific concepts: From early digital methods to LLMs
Michael Zichert and Arno Simons
This article situates large
language models (LLMs) within the longer history of computational approaches to concept analysis in the history, philosophy, and sociology of science (HPSS). We examine what LLMs add to existing methods, how they inherit longstanding problems, and review recent case studies that employ them. In the first part, we reconstruct computational conceptual history before LLMs by bringing together three strands of work: early digital methods in HPSS, distributional approaches from digital history and related research, and lexical semantic change detection. We provide an overview of the main challenges and opportunities, focusing on corpus construction, operationalization and modelling choices, and evaluation and interpretation. In the second part, we turn to the era of LLMs, starting with a short introduction to LLMs before reviewing LLM-based work on lexical semantic change detection and relevant case studies in HPSS. We then revisit the earlier methodological questions, showing how issues of corpus construction, model choice and training data, operationalization trade-offs, and evaluation and interpretation play out in LLM-based workflows.
Read the full abstract ▾
Computation and Language
Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models
Jianguo Zhu, Xiangmei Li and Wenjie Liu
Context-augmented
language model systems often wrap supplied content with labels such as Reference:, Evidence:, Instruction:, Note:, or Example:, but the effect of these labels on reader-model behavior remains underexplored. We introduce a paired fixed-content probe over 500 MMLU-Pro items: each item receives the same misleading answer-bearing assertion under different discourse-role labels, and adoption is measured by whether the model outputs the injected wrong option. Across GPT-5.5, DeepSeek V4 Pro, Llama-3-8B-Instruct, and Qwen2.5-7B-Instruct, Misleading Adoption Rate shifts by 56-84 percentage points. Binding or source-like labels such as Instruction: and Reference: produce high adoption, whereas Example: consistently suppresses it. Paired tests, bootstrap intervals, final-instruction ablations, and Qwen final-step log-probability probes support a label-conditioned candidate preference. Boundary probes show where the effect weakens or persists: arithmetic tasks reduce adoption, passage-shaped external context preserves smaller label gaps, short-answer evaluation rules out option-letter copying, and nested-label conflicts suggest that illustrative framing can delimit adoption scope. A 200-case single-author manual audit confirms that the short-answer contrasts are stable under conservative adjudication. The resulting claim is bounded but practical: context-utilization and reader-side RAG
benchmarks should report and control wrapper labels, because presentation choices can change measured reliance on supplied context.
Read the full abstract ▾
Computation and Language
POLARIS: Guiding Small Models to Write Long Stories
Rishanth Rajendhran, Jenna Russell, Mohit Iyyer et al. (+1 more)
Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases, especially when compared to frontier models. We present POLARIS (Policy Optimization with LLM-as-a-judge rewards and Anchored-Reference Injection for Storywriting), a lower-compute GRPO recipe with two key ingredients: a frontier LLM judge with a structured Story Quality rubric as the online reward, and human-reference injection (HRI), where a teacher-forced human-written story serves as a high-reward anchor within each GRPO group. By applying our training recipe to Qwen3.5-9B, using a
dataset of approximately 1.4K
prompt-story pairs derived from 100 short-story anthologies and 4 A100 GPUs, we obtain POLARIS-9B. Across five
benchmarks spanning in-distribution and out-of-distribution
prompts and rubrics, POLARIS-9B is competitive with much larger open-weight models while following length instructions more closely. A blinded human evaluation confirms that POLARIS-9B is preferred to the base Qwen3.5-9B and on par with Qwen3.5-27B. Despite training only on stories up to 4k words, POLARIS-9B preserves quality on
prompts requesting stories up to 3 times the training length, a regime where most open-weight models degrade substantially in quality, length adherence, or both. More broadly, our results suggest that length generalization is a meaningful stress test for creative-writing models and a useful lens for distinguishing otherwise close models.
Read the full abstract ▾
All authors Rishanth Rajendhran, Jenna Russell, Mohit Iyyer, John Frederick Wieting