Computation and Language · Cross-listed from cs.SD
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
Yonggang Zhu, Liting Gao, Aidong Men et al. (+1 more)
Contrastive
Language-Audio Pretraining (CLAP) models are widely used for audio understanding and support modality-agnostic condition swapping in many zero-shot applications. However, their performance is heavily affected by the modality gap between audio and text
embeddings. Existing explanations mainly attribute this gap to the cone effect, treating it as a shift between mean
embeddings, yet correcting the mean alone yields only limited improvements. Alternative hypotheses, such as information imbalance and dimensionality collapse, have also been proposed, but they remain insufficiently verified and have not been thoroughly studied in the audio domain. Meanwhile, several works attempt to decompose
multimodal contrastive
embeddings into interpretable concepts, but none explicitly analyze the modality gap from the perspective of concept decomposition. In this work, we introduce COMET (Concept space Organization and Modality gap Explanation with PLS-SVD Transformation), a novel partial least squares singular value decomposition (PLS-SVD) framework for CLAP that unveils a broader perspective of the modality gap. Our framework reveals that only a small, interpretable subset of axes, which captures shared concepts, contributes substantially to similarity computation, and that the mean component represents only partially the modality gap. Building on this insight, we propose a simple spectral truncation method that mitigates the modality gap in a training-free manner. The method enables zero-shot audio captioning with condition swapping to approach fully supervised performance, without requiring large auxiliary
memory banks or expensive computation. At the same time, it achieves substantial
embedding dimensionality reduction while preserving strong performance on
retrieval and audio captioning tasks.
Read the full abstract ▾
All authors Yonggang Zhu, Liting Gao, Aidong Men, Wenwu Wang
Computation and Language
DLM-SWAI: Steering Diffusion Language Models Before They Unmask
Hyeseon An and Yo-Sub Han
Steering
language model generation toward desired textual properties is essential for practical deployment, and inference-time methods are particularly appealing because they enable controllable generation without retraining. Recent work has also highlighted
diffusion language models as an emerging generation paradigm with distinct decoding properties. However, most existing steering approaches either rely on auxiliary models or are designed for autoregressive next-
token decoding, making them difficult to apply to
diffusion language models DLMs, which generate text through iterative denoising of partially masked sequences. Therefore, we propose DLM-SWAI, a simple training-free steering method that biases the
token distribution at each denoising step using pre-computed
token-level style scores. Experiments on style and safety control tasks show that DLM-SWAI effectively steers
diffusion language models while preserving generation quality and requiring minimal computational overhead. Ablations further reveal a controllable trade-off between steering strength and fluency, and our analysis links class-wise steerability to the strength of
token-level attribute cues.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?
Linhao Zhang, Aiwei Liu, Yuan Liu et al. (+1 more)
Vision-
language models (VLMs) have made strong progress on high-level image-text
alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI
agents and design tools. We introduce DiffSpot, a code-driven
benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The
benchmark contains 4{,}400 pairs, including 3{,}900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for
hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only $40.7\%$ of true changes, with Hard-tier Recall below $23\%$ for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.
Read the full abstract ▾
All authors Linhao Zhang, Aiwei Liu, Yuan Liu, Xiao Zhou
Computation and Language · Cross-listed from cs.MA
CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems
Ziyang Ma, Dingyi Zhang, Sichu Liang et al. (+4 more)
Although large
language model (LLM) based multi-
agent systems (MAS) show their capability to solve complex tasks and achieve higher performance over single
agent systems, they lead to huge computational overheads because of heavy communication between
agents. Previous research has made efforts to train a sparse multi-
agent graph or fine-tune a planner to orchestrate the workflow better. However, such extra training processes introduce computational costs and limit MAS to specific domains, therefore compromising their generalizability. In this paper, we propose CONCAT, a training-free multi-
agent collaboration framework based on CONsensus and Confidence-driven Ad hoc Teaming to efficiently organize
agent interactions. Specifically,
agents are clustered based on their initial answers, and leaders of each cluster are selected based on the
agents' confidence. Then, a heuristic function based on the Theory of Mind is designed to predict the collaboration benefits between every two leaders according to their answers and confidence. Finally, an ad hoc multi-
agent network is organized after evicting a percentage of communications based on the predicted benefits. Experiments across three LLMs and three
benchmarks show that CONCAT achieves up to 2.02x higher efficiency (accuracy/latency ratio) than LLM-Debate and outperforms training-aware methods such as AgentDropout, while reducing average latency by 50.1% on Qwen2.5-14B-Instruct, without any task-specific training.
Read the full abstract ▾
All authors Ziyang Ma, Dingyi Zhang, Sichu Liang, Jiajia Chu, Pengfei Xia, Hui Zang, Deyu Zhou
Computation and Language
Training Deliberative Monitors for Black-Box Scheming Detection
Aditya Sinha, Akshat Naik, Victor Gillioz et al. (+5 more)
As autonomous
agents become more capable of performing real-world tasks, distinguishing scheming behavior from benign task pursuit may become a central AI control problem. Existing monitors often rely on chain-of-thought access or internal activations, or use prompted frontier models, all of which can be unavailable, unreliable or expensive in deployment. In this work, we study action-only deliberative monitors: smaller open-weight models trained to detect scheming and sabotage from agentic trajectories without accessing the monitored
agent's
reasoning or model internals. Our method, inspired by deliberative
alignment, uses a scheming specification to elicit structured rationales from a frontier teacher, filters them with a separate judge, and distills the highest-quality rationales into open-weight monitors with supervised fine-tuning and reinforcement learning. We train on five
datasets, and evaluate across six out-of-distribution agentic misalignment
benchmarks. We show that applying our method to Qwen3.5-27B yields higher performance than all low-cost frontier models as prompted monitors (Gemini 3.1 Flash-Lite, GPT-5.4 Nano, and Claude Haiku 4.5) and than Gemini 2.5 Pro, while also achieving lower marginal inference cost (
token-metered USD per 1,000 evaluations). Stronger prompted frontier monitors (Gemini 3.1 Pro, GPT-5.4, Claude Sonnet 4.6, and Claude Opus 4.6) achieve higher performance but at roughly $16$–$34\times$ higher marginal inference cost. Several of our trained monitors are positioned on the empirical cost–performance Pareto frontier among the monitors we evaluate, providing practical low-cost, low-FPR alternatives to prompted frontier models.
Read the full abstract ▾
All authors Aditya Sinha, Akshat Naik, Victor Gillioz, Simon Storf, Kilian Merkelbach, Rich Barton-Cooper, Axel Højmark, Marius Hobbhahn
Computation and Language
World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
Emmanuelle Bourigault
Vision-
language models (VLMs) are increasingly used to answer questions about physical scenes, yet most evaluations reduce performance to a final answer. This hides whether the model perceived the right objects, represented the right physical state, predicted a plausible transition, or merely selected the right option for the wrong reasons. We introduce \wmw, an evaluation framework for
auditing the
language-expressed physical commitments of VLMs. Instead of scoring only $I,q\mapsto a$, we ask models to produce a typed trace $I,q\mapsto(s_0,\Delta s,s_1,a)$: an initial state, a state transition, a resulting state, and an answer. A hybrid verifier then checks schema validity, state grounding, transition consistency, and answer-trace compatibility, yielding typed error labels such as object, relation, force, transition, temporal, unit/scale, and faithfulness errors. We release \tracebank, a controlled trace resource with \nSeed schema- and recomputation-validated synthetic scenarios across \nFamilies physics families, \nPairs minimally perturbed contrastive preference pairs, verifier code, audit guidelines, and model outputs. We evaluate \nModels VLMs on both controlled and external physical-
reasoning examples. \wmw reveals failures that answer-only evaluation misses: 35% of correct answers from mid-tier models are backed by physically invalid traces. Verifier-guided reranking recovers up to 7 percentage points of trace validity without sacrificing answer accuracy, and trace-level preference tuning reduces hidden inconsistency by 41% relative. The contribution is not another final-answer physics
benchmark, but a reusable protocol for measuring whether a VLM's stated physical world can be true at the same time as its answer.
Read the full abstract ▾
Computation and Language
GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
Xin Sun, Jianan Xie, Zhongqi Chen et al. (+6 more)
Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA
benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy
Distillation framework that adds dense
token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose
token distribution is distilled back to the ordinary student policy over generated action-
token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.
Read the full abstract ▾
All authors Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Bowen Song, Weiqiang Wang, Zilei Wang, Liang Wang
Computation and Language · Cross-listed from cs.LG
PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
Qikai Chang, Zhenrong Zhang, Linbo Chen et al. (+4 more)
Large
Language Models (LLMs) show strong potential as educational tutors. Existing approaches typically train them to solve problems and provide correct answers, but this problem-solving-centered paradigm overlooks key requirements of effective tutoring: progressive guidance and the coordination of multiple pedagogical objectives across multi-turn interactions. Developing such tutors remains challenging because student behavior varies substantially with individual knowledge states, pedagogical effectiveness depends on multiple factors beyond final-answer correctness, and coordinating these objectives over tutor-student interactions is inherently difficult. To address these challenges, we propose PEARL, a PEdagogically Aligned Reinforcement Learning framework for training Socratic tutoring
agents. First, we introduce a controllable student simulator that disentangles latent cognitive states from response generation, enabling simulation of diverse abilities and misconceptions. Second, we develop a pedagogically aligned reward model that jointly assesses pedagogical quality and objective correctness. Finally, we propose a stable multi-objective reinforcement learning approach that balances competing pedagogical objectives during tutor training. Experiments across multiple
benchmarks show that PEARL performs competitively against tutoring-specific open-source systems and leading proprietary LLMs.
Read the full abstract ▾
All authors Qikai Chang, Zhenrong Zhang, Linbo Chen, Pengfei Hu, Jianshu Zhang, Youhui Guo, Jun Du
Computation and Language
LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agents
Xiaoxuan Peng, Kaiqi Zhang, Xinyu Lu et al. (+5 more)
Mastering terminal environments requires
language agents capable of multi-step planning, feedback-grounded execution, and dynamic state adaptation. However, training such
agents is currently bottlenecked by a reliance on scraped external repositories, which limits domain diversity, environment controllability, and the targeting of specific capability deficits. We introduce LiteCoder-Terminal-Gen, a zero-dependency synthesis pipeline that autonomously generates executable and verifiable terminal training environments directly from domain specifications. Using this framework, we construct two large-scale resources: LiteCoder-Terminal-SFT, comprising 11,255 expert trajectories across 10 domains, and LiteCoder-Terminal-RL, featuring 602 verifiable environments for trajectory-level preference optimization. Supervised fine-tuning of Qwen-family models on our SFT
dataset yields
agents that significantly outperform their base counterparts. Notably, our 32B variant achieves 29.06%, 18.54%, and 34.00% pass@1 on Terminal Bench 1.0, 2.0, and Pro, respectively. Furthermore, applying Direct Multi-turn Preference Optimization (DMPO) on our RL environments yields additional performance gains. These results systematically demonstrate that fully synthetic, executable environments offer a scalable and verifiable supervision signal for mastering complex, real-world command-line workflows.
Read the full abstract ▾
All authors Xiaoxuan Peng, Kaiqi Zhang, Xinyu Lu, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
Computation and Language
From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals
Yeyong Yu, Wenya Hu, Xing Wu et al. (+1 more)
As candidate generation and high-throughput experimentation advance, the primary bottleneck in materials discovery is shifting from property prediction to making reliable evaluations among massive candidate sets. We propose a Knowledge-Augmented Preference Signals Framework, MaterEval, that automatically produces, for the same candidate, two evaluations: an informed judgment that follows expert rules and provides supporting evidence, and a rule-removed blind guess. By pairing the two evaluations as preference data, we guide general-purpose large
language models (LLMs), originally lacking materials-specific criteria, from intuitive judgment toward reliable evaluation supported by explicit evidence. To balance throughput, cost, and reliability, we further introduce a fast-slow
reasoning scheme that decouples large-scale rapid screening from in-depth review on a small subset. Using high-
entropy alloy (HEA) assessment as a case study, we show that, without external
retrieval and relying solely on internalized capabilities, small open-source LLMs achieve substantial gains in accuracy, conclusion consistency, and evidence discrimination, approaching the performance of rule-based closed-source LLMs. These results demonstrate that expert rules can be systematically transformed into learnable preference signals, enabling a low-cost and deployable evaluation module for autonomous materials discovery loops.
Read the full abstract ▾
All authors Yeyong Yu, Wenya Hu, Xing Wu, Quan Qian
Computation and Language · Cross-listed from cs.LG
SCOPE: A Lightweight-training LLM Framework for Air Traffic Control Readback Monitoring
Qihan Deng, Minghua Zhang, Yang Yang et al. (+1 more)
Pilot readback of Air Traffic Control (ATC) voice instructions is a primary safeguard against miscommunication in air transportation. However, readback anomalies remain implicated in approximately 80% of aviation incidents. This vulnerability is further exacerbated by rising traffic volume and elevated cognitive workload, thereby motivating automated readback monitoring by machine. Traditional rule-based and machine learning approaches struggle to generalize across the highly variable and evolving phraseology of air traffic controller-pilot communications. While Large
Language Models (LLMs) have opened a new avenue through their strong
reasoning and generalization capabilities, existing approaches still face deployment and computational barriers in practice. In this work, we propose Semantic
reasoning for Communication via Open-set Plug-in with Examples (SCOPE), a novel lightweight-training LLM framework that advances both the efficiency and accuracy of machine-based ATC readback monitoring. The core idea is to couple a plug-in open-set classifier with a carefully designed in-context learning mechanism on top of a frozen LLM. Extensive experiments on the semi-synthetic communication
dataset show that SCOPE attains superior accuracy while delivering the low-latency response required for operational environments. Under a few-shot setting, SCOPE achieves 91.05% accuracy in open-set detection and corrects 96.63% of anomalous readbacks, thereby outperforming the strongest available baselines while providing explanations for its decisions. These findings demonstrate the potential of our framework as a practical pathway toward interpretable and controllable ATC readback monitoring.
Read the full abstract ▾
All authors Qihan Deng, Minghua Zhang, Yang Yang, Zhenyu Gao
Computation and Language · Cross-listed from cs.MA
DynaGraph: Lightweight Multi-Model Interaction Framework via Dynamic Topological Reconfiguration
Yanxing Guo, Zihao Zheng, Fangzhou Wu et al. (+4 more)
Tackling complex
reasoning tasks typically relies on massive monolithic LLMs, which suffer from severe computational redundancy. While task decomposition through structured pipelines or multi-
agent collaborations offers an alternative, these approaches inevitably fall into a critical dilemma: predefined static
topologies are highly vulnerable to cascading errors, whereas unconstrained dynamic
agents suffer from trajectory divergence and unpredictable
memory bloat. To address this, we present DynaGraph, a lightweight multi-model framework driven by dynamic topological reconfiguration. At the execution level, DynaGraph multiplexes time-division PEFT adapters over a shared base model, enabling both full system training and inference deployment on a single consumer-grade GPU. At the routing level, the Evaluator continuously monitors execution confidence to trigger hierarchical self-healing: Fine-grained Patching for localized data gaps and Subgraph Reconstruction for severe logical ruptures. Experiments on StrategyQA, MATH, and FinQA demonstrate our 8B model closely approximates the
reasoning capabilities of a 72B monolithic model (e.g., 87.6% on StrategyQA, 82.7% on MATH). Furthermore, it reduces latency by up to 68.1% and
token consumption by 68.6% compared to unconstrained dynamic architectures.
Read the full abstract ▾
All authors Yanxing Guo, Zihao Zheng, Fangzhou Wu, Ling Liang, Lin Bao, Zongwei Wang, Yimao Cai
Computation and Language
Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generation
Zeli Su, Ziyin Zhang, Zewei Pan et al. (+8 more)
Low-resource target-
language generation is often limited by scarce parallel data, while high-resource source-
language monolingual data is abundant but difficult to use with standard supervised fine-tuning. We propose Source-Grounded Semantic Reinforcement Learning (SG-SRL), a resource-utilization framework that converts source-
language monolingual data into cross-lingual semantic supervision for target-
language generation. SG-SRL performs reference-free reinforcement learning (RL) on source-
language data using a cross-lingual semantic reward model, instantiated by a cross-lingual reranker that scores the semantic relevance between the source input and the target-
language generation. While this induces severe verbosity-based reward hacking, a lightweight recovery stage using a small parallel corpus restores fluency, conciseness, and task format while preserving the semantic gains. Experiments on Chinese-to-Thai generation show that SG-SRL improves semantic grounding and factual coverage over cold-start SFT. Additional analyses on long-form transfer and Tibetan
embedding-based rewards clarify the generalization behavior of SG-SRL and show that an encoder-based semantic reward can substitute for an LLM-based reranker in a realistic low-resource
language setting.
Read the full abstract ▾
All authors Zeli Su, Ziyin Zhang, Zewei Pan, Zhou Liu, Dingcheng Huang, Dehan Li, Zhankai Xu, Longfei Zheng, Xiaolu Zhang, Jun Zhou, Wentao Zhang
Computation and Language
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
Runze Xu, Arpit Garg, Hemanth Saratchandran et al. (+1 more)
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large
language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and
alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the models original training or
alignment distributions. The challenge is amplified in practical settings, where the original training and
alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth
token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base models relative preferences among alternative
tokens without directly opposing the cross-
entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base models original training or
alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.
Read the full abstract ▾
All authors Runze Xu, Arpit Garg, Hemanth Saratchandran, Simon Lucey
Computation and Language
On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training
Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang et al. (+1 more)
Post-training has greatly improved
reasoning in frontier
vision-
language models, yet its gains for perception remain comparatively limited, creating a bottleneck for end-to-end visual
reasoning. To investigate this gap, we introduce a controlled diagnostic framework with two synthetic tasks that disentangle perception from
reasoning. Our analysis reveals a consistent perception-
reasoning asymmetry: post-training improves
reasoning more substantially than perception, though the underlying mechanism differs across training paradigms. For supervised fine-tuning (SFT), this asymmetry stems from
token imbalance, with perception occupying a smaller fraction of
tokens in chain-of-thought supervision. Reweighting the loss boosts end-to-end performance by up to 18.2 points. For reinforcement learning (RL), the asymmetry instead arises from reward coupling, as outcome rewards correlate more strongly with
reasoning than perception. Adding a perception-aware reward improves end-to-end accuracy by up to 6.0 points; when ground-truth perception rewards are unavailable, a reliable surrogate provides useful signal, yielding gains of 2.2 points. Beyond the controlled setting, these strategies also improve real-world visual
reasoning, with gains of up to 3.3 points across three
benchmarks. Overall, we diagnose the causes of asymmetric optimization and provide actionable guidance that benefits both synthetic and realistic settings.
Read the full abstract ▾
All authors Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng
Computation and Language
PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents
Yuxuan Liu, Xin Lai, Junyi Li et al. (+21 more)
Real applications provide the training setting closest to phone-
agent deployment, but are difficult to reset, scale safely, and verify programmatically. Static screenshots and interaction trajectories preserve realistic evidence but cannot generate new experience. We introduce PhoneWorld, a trace-grounded framework that converts such evidence into runnable, resettable, and verifiable Android environments. PhoneWorld induces a usage-weighted interaction skeleton from observed pages, transitions, and state-changing operations; translates it into a behavior-grounded app specification; realizes the specification through an autonomous build–inspect–repair loop; and synthesizes executable tasks with programmatic verifiers. The resulting suite spans 34 consumer-facing apps across 16 domains and supports an audited online
benchmark, verified trajectory generation, and online RL through common reset and verification interfaces. Evaluations with diverse general and open-source GUI
agents show that PhoneWorld supports reliable end-to-end online interaction and exposes capabilities complementary to AndroidWorld. Controlled SFT experiments further show that PhoneWorld trajectories complement AndroidWorld supervision, transfer across online and offline
benchmarks, and become more effective as data volume and app coverage increase. Under a matched RL budget, combining PhoneWorld mock-app rollouts with real-app rollouts improves performance over real-app RL alone on both real-phone tasks and AndroidWorld. Together, these results demonstrate that trace-grounded executable abstraction can bridge realistic mobile behavior and scalable
agent learning, turning limited real-app evidence into a growing supply of controllable and verifiable environments for training and evaluation.
Read the full abstract ▾
All authors Yuxuan Liu, Xin Lai, Junyi Li, Pengyuan Lyu, Jason, Yiduo Guo, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Huawen Shen, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Chengquan Zhang, Han Hu, Benyou Wang, Ji-Rong Wen, Rui Yan, Zhengyang Tang
Computation and Language
Comparative Evaluation of Machine Translation Systems on Images with Text
Blai Puchol, Sergio Gómez González, Miguel Domingo et al. (+1 more)
This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the intersection of computer
vision and natural
language processing. The study compares three main paradigms: modular pipelines that separate text detection, recognition, and translation; multi-modal large
language models (MLLMs) capable of processing both image and text jointly; and an end-to-end model, Translatotron-V, which directly generates translated images. The modular systems employ state-of-the-art OCR (docTR) combined with
multilingual LLMs such as Llama and EuroLLM, while the evaluated MLLMs include different configurations of Gemini 2.5. Experiments were conducted on parallel
multilingual datasets covering multiple
language pairs, with evaluation based on BLEU, chrF, and TER metrics. The results show that modular pipelines outperform the end-to-end approach, while MLLMs achieve the best overall performance, demonstrating superior flexibility and contextual understanding. These findings underscore the effectiveness of multi-modal
reasoning for image-to-text translation and provide a solid foundation for future research on integrating visual understanding and
language generation in
multilingual settings.
Read the full abstract ▾
All authors Blai Puchol, Sergio Gómez González, Miguel Domingo, Francisco Casacuberta
Computation and Language
MOOSE-Copilot: A Web-Based Interactive Assistant for Unified Exploratory and Fine-Grained Scientific Hypothesis Discovery
Hongran An and Zonglin Yang
Large
language models (LLMs) show remarkable potential in scientific hypothesis discovery. However, existing approaches face two critical limitations: they treat divergent exploratory search and convergent fine-grained refinement as isolated tasks, and they operate autonomously with little to no human guidance. We present MOOSE-Copilot, the first unified framework to bridge this abstraction gap through a formalized human-AI interaction (HAII) protocol. Our system empowers scientists to steer the generative process via three explicit signals: initial blueprints, inter-stage routing, and intra-stage feedback. Using an oracle-simulated evaluation in which an LLM provides idealized expert signals, we show that injecting these structured signals significantly outperforms purely autonomous baselines, characterizing the gains achievable under high-quality guidance. Furthermore, we build a web-based interface that turns the framework into a no-code workflow: researchers pose a question, watch the hypothesis search unfold as an interactive tree, and steer it by selecting hypotheses, routing between stages, and injecting feedback-no command-line
agents required. This makes end-to-end hypothesis discovery directly accessible to interdisciplinary researchers.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.HC
Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles
Drishti Goel, Agam Goyal, Veda Duddu et al. (+8 more)
Language models are increasingly being deployed for conversational support in informal caregiving contexts, where interactions often extend beyond information-seeking: caregivers seek emotional reassurance, guidance, and help, while navigating uncertain, relationally complex care decisions. Yet most safety evaluations assess model behavior under generic
prompts, leaving a critical question unexamined: does a model's safety profile change with its support role? We study this by operationalizing four expert-reviewed support roles grounded in social support theory: Inform, Coach, Relate, and Listen, and comparing them against two baseline controls: a basic prompting condition and a
retrieval-augmented generation (RAG) condition. We evaluate across three
language models (GPT-4o-mini, Llama-3.1-8B-Instruct, and MedGemma-1.5-4b-it) on 5,000 real-world queries from online Alzheimer's Disease and Related Dementias (ADRD) communities. We find that the LLM's support role systematically shapes both the prevalence and composition of interactional risks. Furthermore, a human evaluation study reveals a perceived quality–safety tension: more directive, information-oriented roles are rated as more helpful and trustworthy despite exhibiting elevated interactional risk profiles. We release ~90,000 support role-conditioned model responses with risk annotations as an ecologically grounded resource for research on safer LLM-mediated conversational support.
Read the full abstract ▾
All authors Drishti Goel, Agam Goyal, Veda Duddu, Olivia Pal, Jeongah Lee, Qiuyue Joy Zhong, Violeta J. Rodriguez, Daniel S. Brown, Dong Whi Yoo, Ravi Karkar, Koustuv Saha
Computation and Language
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
Rohan Shravan
Large
language models route every input through a learned
embedding table of shape |V| x d_model, consuming hundreds of millions to billions of trainable parameters at frontier scale. We introduce Kronecker
Embeddings, a deterministic byte-level character-position factorization that replaces this table with a fixed encoder and a single learned projection, compatible with standard BPE tokenizers, eliminating 91–94% of input-side trainable parameters at frontier scale. We provide five contributions. First, a cross-model probe across six LMs (135M-671B parameters) shows trained input
embeddings cluster typographic variants of the probe word far more than morphological relatives; Kronecker escapes this clustering at the
embedding layer. Second, a controlled three-seed comparison on nanoGPT GPT-2 124M over 2.5B
tokens of FineWeb-Edu shows Kronecker reaching 2.5 +- 0.2% lower validation loss than the BPE-tied baseline (gap 0.083 +- 0.007 nats, ~9% lower perplexity), needing ~1.43x fewer steps to reach BPE's converged loss. Third, a spelling-robustness probe over 110 clean/typo pairs shows Kronecker preserves the top-1 prediction on 55.5% of pairs vs. 47.3% for BPE (+8.2 pp) and lowers KL by 7.6%, winning or tying in 10 of 11 categories; a generation probe shows Kronecker echoes byte-novel strings and typos through generation where BPE forgets them. Fourth, BPE
embedding norm drifts during training while Kronecker projection norm stays near 1.0, consistent with a stable representational target. Fifth, an on-the-fly runtime variant reconstructs
embeddings from a 4.5 MB byte buffer rather than a 2.15 GB table at vocabulary 131,072, with 0.01–0.24% step-time overhead. Byte-level locality has a tradeoff: byte-similar but semantically distant pairs (compute/commute, nation/notion) cluster together, shifting disambiguation to early
attention layers.
Read the full abstract ▾
Computation and Language
What a One-Hour Interview Does Not Tell an LLM About You: Interviews Elicit Traits, Not Decision Policies
Ruoxi Su, Yuhan Liu and Jingyu Hu
Large
language models are increasingly asked to stand in for particular people, and interviewing the person is the natural way to tell a model who they are. We test what a one-hour LLM-led adaptive interview tells a model about an individual's value-laden decisions. 46 adults completed such an interview and then answered 25 dilemma questions, which are the ground truth. We predict their answers from different parts of the transcript, compare adaptive follow-up questions with fixed ones on the same participants, and
benchmark every variant against a model given no information about the person and against the majority answer. No part of the interview is reliably better than either baseline, and adaptive follow-ups are no better than fixed ones. Two audits of the transcripts show why. For 85% of the errors an auditor finds no evidence that distinguishes the person's choice from the alternatives, and where a transcript does contain related evidence, the option it favours matches the person's own answer less often than the majority answer does. The same transcripts do improve recovery of the person's MBTI type. Interviews of this kind elicit traits, not decision policies. We discuss what decision-aware elicitation would require and why persona
agents should not be delegated value-laden decisions, and we release the consenting participants' transcripts with all predictions.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
Tianpeng Bu, Xin Liu, Qihua Chen et al. (+7 more)
While GUI
agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our
dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI
benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.
Read the full abstract ▾
All authors Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu, Bin Yang, Minying Zhang
Computation and Language
SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents
Wentao Hu, Zhendong Chu, Yiming Zhang et al. (+6 more)
Retrieval-augmented LLM
agents increasingly rely on curated skill banks: collections of reusable textual principles that guide decision making on complex tasks. Existing approaches typically expand these banks in an append-only fashion, continuously adding new skills without removing redundant, outdated, or harmful ones, resulting in inefficient and poorly curated repositories. In this paper, we formulate the skill bank curation as a constrained multi-objective problem: a desirable bank must be useful for the
agent, diverse in its content, and provide good coverage of the query distribution. To this end, we introduce SkillBrew, a multi-objective curation framework that formalizes skill bank curation as Pareto-aware optimization under a utility constraint, and solves it via a bi-level propose-then-verify loop. We evaluate our approach on two public
benchmarks. Our findings suggest that treating skill banks as objects of principled curation, rather than ever-growing append-only logs, is an important step toward building self-improving LLM
agents.
Read the full abstract ▾
All authors Wentao Hu, Zhendong Chu, Yiming Zhang, Junda Wu, Ming Jin, Xiangyu Zhao, Yilei Shao, Yanfeng Wang, Qingsong Wen
Computation and Language · Cross-listed from cs.CR
AliMark: Enhancing Robustness of Sentence-Level Watermarking Against Text Paraphrasing
Yuexin Li, Wenjie Qu, Linyu Wu et al. (+5 more)
Existing sentence-level watermarking methods enhance robustness to paraphrasing by anchoring watermarks in sentence semantics. However, their prefix-based designs remain vulnerable to structural perturbations, such as sentence splitting and merging, which commonly arise under strong paraphrasers like DIPPER and GPT-3.5. To mitigate this issue, we propose AliMark, a framework that reformulates sentence-level watermarking as a bit sequence encoding and
alignment problem between a potentially watermarked text and a secret bit sequence. Notably, our approach adopts a two-stage detection strategy: we generate multiple restructured text variants and adaptively align their extracted bit sequences with the secret bit sequence to minimize
alignment cost. This multi-candidate
alignment design naturally improves robustness to sentence merges and splits. Extensive experiments demonstrate that AliMark substantially outperforms state-of-the-art baselines under diverse paraphrasing attacks.
Read the full abstract ▾
All authors Yuexin Li, Wenjie Qu, Linyu Wu, Yulin Chen, Yufei He, Tri Cao, Bryan Hooi, Jiaheng Zhang
Computation and Language · Cross-listed from cs.AI
Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation
Zixuan Jiang, Yanqiao Zhu, Peng Wang et al. (+8 more)
Automatic
speech recognition (ASR) is a core component of human–computer interaction and an increasingly important front-end for LLM-based assistants and
agents. However, most current ASR systems still follow a single-pass paradigm, which is poorly aligned with human communication, where misunderstandings are resolved through iterative clarification and refinement. This mismatch makes it difficult to correct meaning-critical errors once they occur. Meanwhile,
token-level metrics such as WER or CER cannot adequately reflect such a problem. To address these limitations, we formulate Interactive ASR as a multi-turn refinement task and propose Agentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and
reasoning-based editing. We further introduce the Sentence-level Semantic Error Rate ($S^2ER$), an LLM-based semantic evaluation metric, together with an Interactive Simulation System for scalable and reproducible benchmarking. Experiments on
multilingual, named-entity-intensive, and code-switching
benchmarks show that iterative interaction consistently reduces semantic errors, with much larger gains in $S^2ER$ than in conventional
token-level metrics. Human–AI
alignment and ablation studies further validate the reliability of the semantic judge and the robustness of the proposed framework. The code is available at: https://interactiveasr.github.io/ and the live demo is available at https://i-asr.sjtuxlance.com/
Read the full abstract ▾
All authors Zixuan Jiang, Yanqiao Zhu, Peng Wang, Qinyuan Chen, Xinjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao, Xiangang Li, Kai Yu, Xie Chen
Computation and Language
FinGuard: Detecting Financial Regulatory Non-Compliance in LLM Interactions
Huaixia Dou, Jie Zhu, Minghao Wu et al. (+5 more)
As large
language models (LLMs) are increasingly deployed in financial services, a single non-compliant interaction can expose institutions to regulatory penalties and direct consumer harm. Existing guard models are built around general harm taxonomies and overlook violations grounded in specific financial regulations. We address this gap with a regulation-driven pipeline that operates directly on regulatory documents, inducing a financial compliance risk taxonomy and synthesizing grounded training data without any predefined violation categories. Instantiating the pipeline on Chinese financial regulations, we release FinGuard-Bench, to our knowledge the first
benchmark for financial regulatory compliance detection, with expert-annotated labels at both the query and response levels. We further train FinGuard, a financial compliance detection model built on Qwen3-8B and trained on the regulation-grounded data via supervised fine-tuning and self-play reinforcement learning. On FinGuard-Bench, FinGuard substantially outperforms all baselines, including dedicated guard models and much larger general-purpose LLMs such as Qwen3.5-397B-A17B and GPT-5.1. Furthermore, FinGuard also preserves general safety capabilities and adapts to unseen institution-specific policies using policy documents alone. We will publicly release the code,
prompts, and resources used in this work on GitHub.
Read the full abstract ▾
All authors Huaixia Dou, Jie Zhu, Minghao Wu, Shuo Jiang, Junhui Li, Lifan Guo, Feng Chen, Chi Zhang
Computation and Language
Learning Design Skills as Memory Policies for Agentic Photonic Inverse Design
Shengchao Chen, Ting Shu and Sufen Ren
Photonic
crystal fiber (PCF) inverse design remains challenging because candidate geometries must satisfy coupled optical targets under expensive electromagnetic simulation. Existing pipelines improve surrogate prediction or one-shot parameter recommendation, but they do not accumulate reusable design knowledge across iterative trials. We formulate PCF inverse design as a
memory-policy learning problem and propose SkillPCF, a closed-loop
agent framework that combines a physics-guided
memory skill bank, reinforcement-learned skill selection, and simulator-grounded skill
evolution. We further construct a real-world
dataset with 479 expert interaction traces (2,507 spans) and 553
memory-dependent evaluation queries covering dispersion engineering, loss optimization, and multi-objective design. Experiments across multiple LLM backbones and classical baselines show that SkillPCF achieves stronger design-quality and efficiency trade-offs under practical simulation budgets, demonstrating the effectiveness of our proposed
memory-skill learning paradigm for physics-aware PCF inverse design.
Read the full abstract ▾
Computation and Language
Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning
Shunta Asano, Jeonghun Baek and Toshihiko Yamasaki
Recent studies have shown that code-switching data (CSD), in which multiple
languages are mixed within the same context, can improve cross-lingual transfer and
multilingual alignment in large
language models (LLMs). However, existing studies primarily focus on bilingual transfer between English and a target
language, leaving
multilingual settings involving three or more
languages largely unexplored. In this work, we investigate
multilingual code-switching instruction tuning across four
languages: English, Japanese, Korean, and Chinese. We evaluate
multilingual understanding on Belebele. Our experiments show that simple sentence-level
multilingual CSD consistently improves average
multilingual performance across all four
languages, indicating that
multilingual code-switching can be effective beyond bilingual transfer settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
Rahul Bissa, Abhishek Vyas and Yash Jain
We
benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Action, Rationale), a 12,929-tuple corpus of screen-anchored behavioural rationales curated from public app-store reviews, Pew American Trends Panel demographics, and the OPeRA shopper traces. Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline. Two findings. First, frontier zero-shot baselines (Claude Opus 4.7 and GPT-5.5) reach sem_sim 0.459 and 0.482 respectively; a fine-tuned Qwen3-VL-8B-Instruct reaches 0.783 and clears sem_sim >= 0.7 on 79% of rows, against 1-2% for either frontier baseline, a gap of 0.30 absolute on the same test set. Second, the same training data and recipe on Gemma-4-26B-A4B-IT scores only 0.441, in the same band as the frontier zero-shot baselines rather than the fine-tuned Qwen. We read this as a recipe-vs-model mismatch: the
reasoning-tuned high-parameter model resists displacement and would likely need either more data or a stronger fine-tuning method.
Read the full abstract ▾
Computation and Language
Revisiting Observation Reduction for Web Agents: Comprehensive Evaluation with a Lightweight Framework
Masafumi Enomoto, Ryoma Obara, Haochen Zhang et al. (+1 more)
HTML observations in LLM-based web
agents are extremely long, and while many reduction methods have been proposed, it remains unclear which methods reduce overall
agent latency while maintaining performance. The main obstacle is the high cost of end-to-end evaluation: in our experiments, evaluating 11 methods across 32 configurations on 33 tasks of WorkArena L1 required 232.4 cumulative hours. To address this, we propose a lightweight evaluation framework based on the Minimal Failure Set (MFS), the minimal set of HTML elements whose removal causes task failure. We define coverage as the fraction of instances in which a reduction method fully retains the MFS, which serves as a proxy metric that requires neither web access nor LLM inference. We validate that coverage strongly correlates with end-to-end success rate, with over 100$\times$ speedup in cumulative evaluation time on both
benchmarks. Using this framework, we find that extractive HTML reduction methods require either high computation cost or domain-specific optimization to reduce
agent latency while maintaining performance. Building on this, we optimize a
pruning program on MFS training data, achieving 2.2$\times$ faster per-step latency on WorkArena L1 while retaining 84% of the original success rate, and 3.1$\times$ faster on WebLinx while retaining 89%.
Read the full abstract ▾
All authors Masafumi Enomoto, Ryoma Obara, Haochen Zhang, Masafumi Oyamada