Computation and Language
Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verification
Minseok Kang, Dongwook Choi, Gyeom Hwangbo et al. (+3 more)
Accurately synthesizing 3D scenes from user-provided text descriptions is crucial for developing
embodied agents. Despite the importance of scene-description
alignment, existing evaluation methods for such text-guided 3D scene synthesis either capture only coarse similarity between the synthesized scene and the user description, or ignore the spatial
reasoning for verifying object placement. None of them addressed the fine-grained constraints (e.g., X needs to be in the scene in a Y manner) implied by the description from users. To address this, we introduce LEGO, a
benchmark dataset that pairs each user description with human-annotated constraints and a reference scene, and LEGO-Eval, an evaluation framework that decomposes a description into atomic constraints and verifies each one using tools that ground textual references to 3D objects and reason about their spatial relationships. We show that (i) LEGO-Eval evaluates misalignment far more accurately than existing methods and (ii) current scene synthesis approaches achieve at most 10% success rate in LEGO-Eval.
Read the full abstract ▾
All authors Minseok Kang, Dongwook Choi, Gyeom Hwangbo, Seungwon Lim, Kai Tzu-iunn Ong, Jinyoung Yeo
Computation and Language
Automatic Machine Translation Detection Using a Surrogate Multilingual Translation Model
Cristian García-Romero, Miquel Esplà-Gomis and Felipe Sánchez-Martínez
Modern machine translation (MT) systems depend on large parallel corpora, often collected from the Internet. However, recent evidence indicates that (i) a substantial portion of these texts are machine-generated translations, and (ii) an overreliance on such synthetic content in training data can significantly degrade translation quality. As a result, filtering out non-human translations is becoming an essential pre-processing step in building high-quality MT systems. In this work, we propose a novel approach that directly exploits the internal representations of a surrogate
multilingual MT model to distinguish between human and machine-translated sentences. Experimental results show that our method outperforms current state-of-the-art techniques, particularly for non-English
language pairs, achieving gains of at least 5 percentage points of accuracy.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Zero-shot data citation function classification using transformer-based large language models (LLMs)
Neil Byers, Ali Zaidi, Valerie Skye et al. (+2 more)
Efforts have increased in recent years to identify associations between specific
datasets and the scientific literature that incorporates them. Knowing that a given publication cites a given
dataset, the next logical step is to explore how or why that data was used. Advances in recent years with pretrained,
transformer-based large
language models (LLMs) offer potential means for scaling the description of data use cases in the published literature. This avoids expensive manual labeling and the development of training
datasets for classical machine-learning (ML) systems. In this work we apply an open-source LLM, Llama 3.1-405B, to generate structured data use case labels for publications known to incorporate specific genomic
datasets. We also introduce a novel evaluation framework for determining the efficacy of our methods. Our results demonstrate that the stock model can achieve an F1 score of .674 on a zero-shot data citation classification task with no previously defined categories. While promising, our results are qualified by barriers related to data availability,
prompt overfitting, computational infrastructure, and the expense required to conduct responsible performance evaluation.
Read the full abstract ▾
All authors Neil Byers, Ali Zaidi, Valerie Skye, Chris Beecroft, Kjiersten Fagnan
Computation and Language
Cache Mechanism for Agent RAG Systems
Shuhang Lin, Zhencan Peng, Lingyao Li et al. (+3 more)
Recent advances in Large
Language Model (LLM)-based
agents have been propelled by
Retrieval-Augmented Generation (RAG), which grants the models access to vast external knowledge bases. Despite RAG's success in improving
agent performance,
agent-level cache management, particularly constructing, maintaining, and updating a compact, relevant corpus dynamically tailored to each
agent's need, remains underexplored. Therefore, we introduce ARC (
Agent RAG Cache Mechanism), a novel, annotation-free caching framework that dynamically manages small, high-value corpora for each
agent. By synthesizing historical query distribution patterns with the intrinsic geometry of cached items in the
embedding space, ARC automatically maintains a high-relevance cache. With comprehensive experiments on three
retrieval datasets, our experimental results demonstrate that ARC reduces storage requirements to 0.015% of the original corpus while offering up to 79.8% has-answer rate and reducing average
retrieval latency by 80%. Our results demonstrate that ARC can drastically enhance efficiency and effectiveness in RAG-powered LLM
agents.
Read the full abstract ▾
All authors Shuhang Lin, Zhencan Peng, Lingyao Li, Xiao Lin, Xi Zhu, Yongfeng Zhang
Computation and Language · Cross-listed from cs.AI
Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
Huawei Lin, Yunzhi Shi, Tong Geng et al. (+3 more)
Multimodal large
language models (MLLMs) have shown strong capabilities but remain limited to fixed modality pairs and require costly fine-tuning with large aligned
datasets. Building fully omni-capable models that can integrate text, images, audio, and
video remains impractical and lacks robust
reasoning support. In this paper, we propose an
Agent-Omni framework that coordinates existing foundation models through a master-
agent system, enabling flexible
multimodal reasoning without retraining. The master
agent interprets user intent, delegates subtasks to modality-specific
agents, and integrates their outputs into coherent responses. Extensive experiments across text, image, audio,
video, and omni
benchmarks show that
Agent-Omni consistently achieves state-of-the-art performance, particularly on tasks requiring complex cross-modal
reasoning. Its
agent-based design enables seamless integration of specialized foundation models, ensuring adaptability to diverse inputs while maintaining transparency and interpretability. In addition, the framework is modular and easily extensible, allowing future improvements as stronger models become available.
Read the full abstract ▾
All authors Huawei Lin, Yunzhi Shi, Tong Geng, Weijie Zhao, Wei Wang, Ravender Pal Singh
Computation and Language · Cross-listed from cs.LG
In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi et al. (+2 more)
Knowledge
distillation is an efficient strategy to use data generated by large "teacher"
language models to train smaller capable "student" models, but selecting the optimal teacher for a specific student-task combination requires expensive trial-and-error. We propose a lightweight score called GRACE to quantify how effective a teacher will be for post-training a student model. GRACE measures distributional properties of the student's gradients without access to a verifier, teacher logits, teacher internals, or test data. From an information-theoretic perspective, GRACE connects to leave-one-out stability of gradient-based
algorithms, which controls the generalization performance of the distilled students. On GSM8K and MATH, GRACE correlates strongly (up to 86% Spearman correlation) with the performance of the distilled LLaMA and OLMo students. In particular, training a student using the GRACE-selected teacher can improve the performance by up to 7.4% over naively using the best-performing teacher. Further, GRACE can provide guidance on crucial design choices in
distillation, including (1) the best temperature to use when generating from the teacher, (2) the best teacher to use given a size constraint, and (3) the best teacher to use within a specific model family. Altogether, our findings demonstrate that GRACE can efficiently and effectively identify a strongly compatible teacher for a given student and provide fine-grained guidance on how to perform
distillation.
Read the full abstract ▾
All authors Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham Kakade, Surbhi Goel
Computation and Language
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Amanda Bertsch, Adithya Pratapa, Teruko Mitamura et al. (+2 more)
As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on
retrieval from one or more sections of the context, which allows nearly all of the context
tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a
benchmark of long-context
reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the
reasoning problem; and Oolong-real, a downstream setting which requires
reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.
Read the full abstract ▾
All authors Amanda Bertsch, Adithya Pratapa, Teruko Mitamura, Graham Neubig, Matthew R. Gormley
Computation and Language
MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning
Qianhao Yuan, Jie Lou, Zichao Li et al. (+6 more)
LLM-based search
agents often concatenate the full interaction history into the context, producing long and noisy inputs, and increasing compute cost and GPU
memory overhead. To address this issue, we propose MemSearcher, an
agent framework that maintains a compact
memory during multi-turn interactions, retaining only question-relevant information and thereby keeping the context length stable across turns. Training MemSearcher is challenging because each trajectory spans multiple turns under different LLM contexts, making each turn an independent optimization target in reinforcement learning. We introduce multi-context GRPO, which propagates trajectory-level advantages to all turns for end-to-end optimization. Experiments demonstrate that MemSearcher outperforms strong history-concatenation (ReAct-style) baselines on a range of public
datasets while maintaining nearly constant
token counts across multi-turn interactions. The code and models will be publicly available at https://github.com/icip-cas/MemSearcher
Read the full abstract ▾
All authors Qianhao Yuan, Jie Lou, Zichao Li, Jiawei Chen, Yaojie Lu, Hongyu Lin, Le Sun, Debing Zhang, Xianpei Han
Computation and Language · Cross-listed from cs.LG
Can LLMs subtract numbers?
Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta et al. (+7 more)
We present a systematic study of subtraction in large
language models (LLMs). While prior
benchmarks emphasize addition and multiplication, subtraction has received comparatively little
attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a<b$). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.
Read the full abstract ▾
All authors Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg
Computation and Language · Cross-listed from cs.CV
VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation
Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran et al. (+5 more)
Code has emerged as a precise and executable medium for
reasoning and action in the
agent era. Yet, progress has largely focused on
language-centric tasks such as program synthesis and debugging, leaving visual-centric coding underexplored. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a
benchmark that reframes
multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream
reasoning. VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVGs; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between
language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues such as objects, shapes, and text beyond the model's intrinsic capacity. Across
benchmarks, frontier VLMs with strong
reasoning capabilities score well overall yet remain limited in professional knowledge and 3D
reasoning. VCoder delivers a 12.3-point overall gain over the top-performing Claude-4-Opus. Human studies show that both humans and VLMs perform worse on rendered SVGs, their consistency reveals the promise of symbolic visual representation. The
benchmark and code are available at https://github.com/CSU-JPG/VCode.
Read the full abstract ▾
All authors Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran, Dantong Zhu, Dongxing Mao, Linjie Li, Philip Torr, Alex Jinpeng Wang
Computation and Language
Beyond Single Embeddings: Capturing Diverse Targets with Multi-Query Retrieval
Hung-Ting Chen, Xiang Liu, Shauli Ravfogel et al. (+1 more)
Most text retrievers generate one query vector to retrieve relevant documents. Yet, the conditional distribution of relevant documents for the query may be
multimodal, e.g., representing different interpretations of the query. We first quantify the limitations of existing retrievers. All retrievers we evaluate struggle more as the distance between target document
embeddings grows. To address this limitation, we develop a new retriever architecture, Autoregressive Multi-
Embedding Retriever (AMER). Our model autoregressively generates multiple query vectors, and all the predicted query vectors are used to retrieve documents from the corpus. We show that on the synthetic vectorized data, the proposed method could capture multiple target distributions perfectly, showing 4x better performance than single
embedding model. We also fine-tune our model on real-world multi-answer
retrieval datasets and evaluate in-domain. AMER presents 4 and 21% relative gains over single-
embedding baselines on two
datasets we evaluate on. Furthermore, we consistently observe larger gains on the subset of
dataset where the
embeddings of the target documents are less similar to each other. We demonstrate the potential of using a multi-query vector retriever and open up a new direction for future work.
Read the full abstract ▾
All authors Hung-Ting Chen, Xiang Liu, Shauli Ravfogel, Eunsol Choi
Computation and Language
Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning
Bowen Jin, TJ Collins, Donghan Yu et al. (+10 more)
Large
language models (LLMs) exhibit complementary strengths across domains and come with varying inference costs, motivating the design of multi-
agent LLM systems where specialized models collaborate efficiently. Existing approaches predominantly rely on decentralized frameworks, which invoke multiple LLMs for every input and thus lead to substantial and uncontrolled inference costs. In this work, we introduce a centralized multi-LLM framework, where a controller LLM selectively coordinates a pool of expert models in a cost-efficient and cost-controllable manner. We formulate this coordination problem as reinforcement learning with dual objectives: maximizing task performance while minimizing the overall inference cost. In addition, we expect the multi-
agent system to have adapted behavior with different budget conditions during inference. To this end, we propose CoRL, a reinforcement learning framework that optimizes the performance cost trade-off in a controllable multi-budget setting. Experiments on four diverse
benchmarks demonstrate that CoRL enables a single system to surpass the best expert LLM under high-budget settings, while maintaining strong performance in more economical low-budget modes, highlighting the effectiveness of centralized coordination for scalable and cost-efficient multi-
agent LLM systems.
Read the full abstract ▾
All authors Bowen Jin, TJ Collins, Donghan Yu, Mert Cemri, Shenao Zhang, Mengyu Li, Jay Tang, Tian Qin, Zhiyang Xu, Jiarui Lu, Guoli Yin, Jiawei Han, Zirui Wang
Computation and Language
Amit Misra, Syed Waqas Zamir, Wassim Hamidouche et al. (+2 more)
Artificial intelligence (AI) is diffusing globally at unprecedented speed, but adoption remains uneven. Frontier Large
Language Models (LLMs) are known to perform poorly on low-resource
languages due to data scarcity. We hypothesize that this performance deficit reduces the utility of AI, thereby slowing adoption in Low-Resource
Language Countries (LRLCs). To test this, we use a weighted regression model to isolate the
language effect from socioeconomic and demographic factors, finding that LRLCs have a share of AI users that is approximately 20% lower relative to their baseline. These results indicate that linguistic accessibility is a significant, independent barrier to equitable AI
diffusion.
Read the full abstract ▾
All authors Amit Misra, Syed Waqas Zamir, Wassim Hamidouche, Inbal Becker-Reshef, Juan Lavista Ferres
Computation and Language · Cross-listed from cs.AI
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
Jiayu Liu, Cheng Qian, Zhaochen Su et al. (+4 more)
Current evaluations of Large
Language Model (LLM)
agents primarily emphasize task completion, often overlooking resource efficiency and adaptability. This neglects a crucial capability:
agents' ability to devise and adjust cost-optimal plans in response to changing environments. To bridge this gap, we introduce CostBench, a scalable, cost-centric
benchmark designed to evaluate
agents' economic
reasoning and replanning abilities. Situated in the travel-planning domain, CostBench comprises tasks solvable via multiple sequences of atomic and composite tools with diverse, customizable costs. It also supports four types of dynamic blocking events, such as tool failures and cost changes, to simulate real-world unpredictability and necessitate
agents to adapt in real time. Evaluating leading open-sourced and proprietary models on CostBench reveals a substantial gap in cost-aware planning:
agents frequently fail to identify cost-optimal solutions in static settings, with even GPT-5 achieving less than 75% exact match rate on the hardest tasks, and performance further dropping by around 40% under dynamic conditions. By diagnosing these weaknesses, CostBench lays the groundwork for developing future
agents that are both economically rational and robust.
Read the full abstract ▾
All authors Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung
Computation and Language
PETra: A Multilingual Corpus of Pragmatic Explicitation in Translation
Doreen Osmelak, Koel Dutta Chowdhury, Uliana Sentsova et al. (+2 more)
Translators often enrich texts with background details that make implicit cultural meanings explicit for new audiences. This phenomenon, known as pragmatic explicitation, has been widely discussed in translation theory but rarely modeled computationally. We introduce PragExTra, the first
multilingual corpus and detection framework for pragmatic explicitation. The corpus covers eight
language pairs from TED-Multi and Europarl and includes additions such as entity descriptions, measurement conversions, and translator remarks. We identify candidate explicitation cases through null
alignments and refined using active learning with human annotation. Our results show that entity and system-level explicitations are most frequent, and that active learning improves classifier accuracy by 7-8 percentage points, achieving up to 0.88 accuracy and 0.82 F1 across
languages. PragExTra establishes pragmatic explicitation as a measurable, cross-linguistic phenomenon and takes a step towards building culturally aware machine translation. Keywords: translation, multilingualism, explicitation
Read the full abstract ▾
All authors Doreen Osmelak, Koel Dutta Chowdhury, Uliana Sentsova, Cristina España-Bonet, Josef van Genabith
Computation and Language · Cross-listed from cs.AI
The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation
Tim R. Davidson, Adam Fourney, Saleema Amershi et al. (+3 more)
The trajectory of AI development suggests that we will increasingly rely on
agent-based systems powered by
language models, composed of independently developed
agents with different information, privileges, and tools. The success of these systems will depend critically on effective collaboration among these heterogeneous
agents, even under partial observability. Despite intense interest, the literature lacks empirical studies evaluating agentic collaboration without relying on fixed communication protocols, limiting insights for open-world deployments. We propose an illustrative collaborative maze-solving
benchmark that (i) isolates collaborative capabilities, (ii) modulates problem complexity, (iii) enables scalable automated grading, and (iv) imposes no output-format constraints, benchmarking unguided, natural communication. Using this
benchmark, we evaluate 32 leading open- and closed-source models in solo, homogeneous, and heterogeneous pairings. Our results reveal a surprising collaboration gap: models that perform well solo often degrade substantially when required to collaborate. We identify mitigations that show remarkable influence. For example, a small nudge via a relay inference approach, where a stronger
agent leads before handing off to a weaker one, closes much of the gap. Our findings argue for (1) collaboration-aware evaluation, (2) training strategies to enhance collaborative capabilities, and (3) deliberate interaction design to elicit
agents' collaboration skills, principles relevant to AI-AI and human-AI settings where
agents must establish common ground.
Read the full abstract ▾
All authors Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar
Computation and Language
Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimes
Mohammadsajad Alipour and Mohammad Mohammadi Amiri
Large
language models (LLMs) are increasingly prevalent across diverse applications. However, their enormous size limits storage and processing capabilities to a few well-resourced stakeholders. As a result, most applications rely on pre-trained LLMs, fine-tuned for specific tasks. However, even storing the fine-tuned versions of these models remains a significant challenge due to the wide range of tasks they address. Recently, studies show that fine-tuning these models primarily affects a small fraction of parameters, highlighting the need for more efficient storage of fine-tuned models. This paper focuses on efficient storage of parameter updates in pre-trained models after fine-tuning. To address this challenge, we leverage the observation that fine-tuning updates are both low-rank and sparse, which can be utilized for storage efficiency. However, using only low-rank approximation or sparsification may discard critical singular components that enhance model expressivity. We first observe that given the same
memory budget, sparsified low-rank approximations with larger ranks outperform standard low-rank approximations with smaller ranks. Building on this, we propose our method, optimal singular damage, that selectively sparsifies low-rank approximated updates by leveraging the interleaved importance of singular vectors, ensuring that the most impactful components are retained. We demonstrate through extensive experiments that our proposed methods lead to significant storage efficiency and superior accuracy within the same
memory budget compared to employing the low-rank approximation or sparsification individually.
Read the full abstract ▾
Computation and Language
Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretation
Renfei Dang, Peng Hu, Zhejian Lai et al. (+3 more)
Prior works have shown that fine-tuning on new knowledge can induce factual
hallucinations in large
language models (LLMs), leading to incorrect outputs when evaluated on previously known information. However, the specific manifestations of such
hallucination and its underlying mechanisms remain insufficiently understood. Our work addresses this gap by designing a controlled
dataset Biography-
Reasoning, and conducting a fine-grained analysis across multiple knowledge types and two task types, including knowledge question answering (QA) and knowledge
reasoning tasks. We find that
hallucinations not only severely affect tasks involving newly introduced knowledge, but also propagate to other evaluation tasks. Moreover, when fine-tuning on a
dataset in which a specific knowledge type consists entirely of new knowledge, LLMs exhibit elevated
hallucination tendencies. This suggests that the degree of unfamiliarity within a particular knowledge type, rather than the overall proportion of new knowledge, is a stronger driver of
hallucinations. Through interpretability analysis, we show that learning new knowledge weakens the model's
attention to key entities in the input question, leading to an over-reliance on surrounding context and a higher risk of
hallucination. Conversely, reintroducing a small amount of known knowledge during the later stages of training restores
attention to key entities and substantially mitigates
hallucination behavior. Finally, we demonstrate that disrupted
attention patterns can propagate across lexically similar contexts, facilitating the spread of
hallucinations beyond the original task.
Read the full abstract ▾
All authors Renfei Dang, Peng Hu, Zhejian Lai, Changjiang Gao, Min Zhang, Shujian Huang
Computation and Language
The Realignment Problem: When Right becomes Wrong in LLMs
Aakash Sen Sharma, Debdeep Sanyal, Manodeep Ray et al. (+3 more)
Post-training
alignment of large
language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static
alignment increasingly brittle. As policies evolve, deployed models can diverge from current
alignment objectives, creating an
Alignment-Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale. We introduce TRACE (Triage and Re-align by
Alignment Conflict Evaluation), a framework that transforms realignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on
alignment conflicts; (2) computing an
alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression. Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust realignment on synthetic
benchmarks and the PKU-SafeRLHF
dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and
alignment guidelines. We release our code: https://respailab.github.io/TRACE/
Read the full abstract ▾
All authors Aakash Sen Sharma, Debdeep Sanyal, Manodeep Ray, Vivek Srivastava, Shirish Karande, Murari Mandal
Computation and Language · Cross-listed from cs.CV
UniChange: Unifying Change Detection with Multimodal Large Language Model
Xu Zhang, Danyang Li, Xiaohang Dong et al. (+5 more)
Change detection (CD) is a fundamental task for monitoring and analyzing land cover dynamics. While recent high performance models and high quality
datasets have significantly advanced the field, a critical limitation persists. Current models typically acquire limited knowledge from single-type annotated data and cannot concurrently leverage diverse binary change detection (BCD) and semantic change detection (SCD)
datasets. This constraint leads to poor generalization and limited versatility. The recent advancements in
Multimodal Large
Language Models (MLLMs) introduce new possibilities for a unified CD framework. We leverage the
language priors and unification capabilities of MLLMs to develop UniChange, the first MLLM-based unified change detection model. UniChange integrates generative
language abilities with specialized CD functionalities. Our model successfully unifies both BCD and SCD tasks through the introduction of three special
tokens: [T1], [T2], and [CHANGE]. Furthermore, UniChange utilizes text
prompts to guide the identification of change categories, eliminating the reliance on predefined classification heads. This design allows UniChange to effectively acquire knowledge from multi-source
datasets, even when their class definitions conflict. Experiments on four public
benchmarks (WHU-CD, S2Looking, LEVIR-CD+, and SECOND) demonstrate SOTA performance, achieving IoU scores of 90.41, 53.04, 78.87, and 57.62, respectively, surpassing all previous methods. The code is available at https://github.com/Erxucomeon/UniChange.
Read the full abstract ▾
All authors Xu Zhang, Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, Xiang Li
Computation and Language
CGES: Confidence-Guided Early Stopping for Efficient and Accurate Self-Consistency
Ehsan Aghazadeh, Ahmad Ghasemi, Hedyeh Beyhaghi et al. (+1 more)
Large
language models (LLMs) are often queried multiple times at test time, with predictions aggregated by majority vote. While effective, this self-consistency (Wang et al., 2023) strategy requires a fixed number of calls and fails when the correct answer is infrequent. We introduce Confidence-Guided Early Stopping (CGES), a
Bayesian framework that forms posteriors over candidate answers and adaptively halts sampling once one answer accumulates enough posterior mass. We prove guarantees in both an ideal calibrated regime and a realistic noisy-confidence regime under a directional drift condition. Averaged over five
reasoning benchmarks, CGES reduces the average number of calls by 58% on average (from 16.0 to 6.7) while matching its accuracy within 0.4 percentage points of self-consistency.
Read the full abstract ▾
All authors Ehsan Aghazadeh, Ahmad Ghasemi, Hedyeh Beyhaghi, Hossein Pishro-Nik
Computation and Language
Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour
Max Norris, Kobi Gal and Sahan Bulathwela
Modelling student knowledge is a key challenge when leveraging AI in education, with major implications for personalised learning. The Knowledge Tracing (KT) task aims to predict how students will respond to educational questions in learning environments, based on their prior interactions. Existing KT models typically use response correctness along with metadata like skill tags and timestamps, often overlooking the question text, which is an important source of pedagogical insight. This omission poses a lost opportunity while limiting predictive performance. We propose Next
Token Knowledge Tracing (NTKT), a novel approach that reframes KT as a next-
token prediction task using pretrained Large
Language Models (LLMs). NTKT represents both student histories and question content as sequences of text, allowing LLMs to learn patterns in both behaviour and
language. Our series of experiments significantly improves performance over state-of-the-art neural KT models and generalises much better to cold-start questions and users. These findings highlight the importance of question content in KT and demonstrate the benefits of leveraging pretrained representations of LLMs to model student learning more effectively.
Read the full abstract ▾
Computation and Language
The Analysis of Lexical Errors in Machine Translation from English into Romanian
Angela Stamatie
The research explores error analysis in the performance of translating by Machine Translation from English into Romanian, and it focuses on lexical errors found in texts which include official information, provided by the World Health Organization (WHO), the Gavi Organization, by the patient information leaflet (the information about the active ingredients of the vaccines or the medication, the indications, the dosage instructions, the storage instructions, the side effects and warning, etc.). All of these texts are related to Covid-19 and have been translated by Google Translate, a
multilingual Machine Translation that was created by Google. In the last decades, Google has actively worked to develop a more accurate and fluent automatic translation system. This research, specifically focused on improving Google Translate, aims to enhance the overall quality of Machine Translation by achieving better lexical selection and by reducing errors. The investigation involves a comprehensive analysis of 230 texts that have been translated from English into Romanian.
Read the full abstract ▾
Computation and Language
Smart-Hiring: An Explainable end-to-end Pipeline for CV Information Extraction and Job Matching
Kenza Khelkhal and Dihia Lanasri
Hiring processes often involve the manual screening of hundreds of resumes for each job, a task that is time and effort consuming, error-prone, and subject to human bias. This paper presents Smart-Hiring, an end-to-end Natural
Language Processing (NLP) pipeline de- signed to automatically extract structured information from unstructured resumes and to semantically match candidates with job descriptions. The proposed system combines document parsing, named-entity recognition, and contextual text
embedding techniques to capture skills, experience, and qualifications. Using advanced NLP technics, Smart-Hiring encodes both resumes and job descriptions in a shared vector space to compute similarity scores between candidates and job postings. The pipeline is modular and explainable, allowing users to inspect extracted entities and matching rationales. Experiments were conducted on a real-world
dataset of resumes and job descriptions spanning multiple professional domains, demonstrating the robustness and feasibility of the proposed approach. The system achieves competitive matching accuracy while preserving a high degree of interpretability and transparency in its decision process. This work introduces a scalable and practical NLP frame- work for recruitment analytics and outlines promising directions for bias mitigation,
fairness-aware modeling, and large-scale deployment of data-driven hiring solutions.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding
Zixuan Liu, Siavash H. Khajavi and Guangkai Jiang
Recent advances in multi-modal models have demonstrated strong performance in tasks such as image generation and
reasoning. However, applying these models to the fire domain remains challenging due to the lack of publicly available
datasets with high-quality fire domain annotations. To address this gap, we introduce DetectiumFire, a large-scale, multi-modal
dataset comprising of 22.5k high-resolution fire-related images and 2.5k real-world fire-related
videos covering a wide range of fire types, environments, and risk levels. The data are annotated with both traditional computer
vision labels (e.g., bounding boxes) and detailed textual
prompts describing the scene, enabling applications such as synthetic data generation and fire risk
reasoning. DetectiumFire offers clear advantages over existing
benchmarks in scale, diversity, and data quality, significantly reducing redundancy and enhancing coverage of real-world scenarios. We validate the utility of DetectiumFire across multiple tasks, including object detection,
diffusion-based image generation, and
vision-
language reasoning. Our results highlight the potential of this
dataset to advance fire-related research and support the development of intelligent safety systems. We release DetectiumFire to promote broader exploration of fire understanding in the AI community. The
dataset is available at https://kaggle.com/
datasets/38b79c344bdfc55d1eed3d22fbaa9c31fad45e27edbbe9e3c529d6e5c4f93890
Read the full abstract ▾
Computation and Language
Prompting for Policy: Forecasting Macroeconomic Scenarios with Synthetic LLM Personas
Giulia Iadisernia and Carolina Camassa
We evaluate whether persona-based prompting improves Large
Language Model (LLM) performance on macroeconomic
forecasting tasks. Using 2,368 economics-related personas from the PersonaHub corpus, we
prompt GPT-4o to replicate the ECB Survey of Professional Forecasters across 50 quarterly rounds (2013-2025). We compare the persona-prompted forecasts against the human experts panel, across four target variables (HICP, core HICP, GDP growth, unemployment) and four forecast horizons. We also compare the results against 100 baseline forecasts without persona descriptions to isolate its effect. We report two main findings. Firstly, GPT-4o and human forecasters achieve remarkably similar accuracy levels, with differences that are statistically significant yet practically modest. Our out-of-sample evaluation on 2024-2025 data demonstrates that GPT-4o can maintain competitive
forecasting performance on unseen events, though with notable differences compared to the in-sample period. Secondly, our ablation experiment reveals no measurable
forecasting advantage from persona descriptions, suggesting these
prompt components can be omitted to reduce computational costs without sacrificing accuracy. Our results provide evidence that GPT-4o can achieve competitive
forecasting accuracy even on out-of-sample macroeconomic events, if provided with relevant context data, while revealing that diverse
prompts produce remarkably homogeneous forecasts compared to human panels.
Read the full abstract ▾
Computation and Language
Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance
Kentaro Ueda, François Portet, Hirohiko Suwa et al. (+1 more)
While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical
reasoning, and
multilingual processing. Merging domain-specific Continual Pre-training (CPT) "experts" offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial
benchmark curated from 18 tasks across 8 established
datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.
Read the full abstract ▾
All authors Kentaro Ueda, François Portet, Hirohiko Suwa, Keiichi Yasumoto
Computation and Language
AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
Aashray Reddy, Andrew Zagula and Nicholas Saban
Large
Language Models (LLMs) remain vulnerable to jailbreaking attacks where
adversarial prompts elicit harmful outputs. Yet most evaluations focus on single-turn interactions while real-world attacks unfold through adaptive multi-turn conversations. We present AutoAdv, a training-free framework for automated multi-turn jailbreaking that achieves an attack success rate of up to 95% on Llama-3.1-8B within six turns, a 24% improvement over single-turn baselines. AutoAdv uniquely combines three adaptive mechanisms: a pattern manager that learns from successful attacks to enhance future
prompts, a temperature manager that dynamically adjusts sampling parameters based on failure modes, and a two-phase rewriting strategy that disguises harmful requests and then iteratively refines them. Extensive evaluation across commercial and open-source models (Llama-3.1-8B, GPT-4o mini, Qwen3-235B, Mistral-7B) reveals persistent vulnerabilities in current safety mechanisms, with multi-turn attacks consistently outperforming single-turn approaches. These findings demonstrate that
alignment strategies optimized for single-turn interactions fail to maintain robustness across extended conversations, highlighting an urgent need for multi-turn-aware defenses.
Read the full abstract ▾
Computation and Language
AyurParam: A State-of-the-Art Bilingual Language Model for Ayurveda
Sravan Gorugantu, Mohd Nauman, Vijay Devane et al. (+7 more)
Current large
language models excel at broad, general-purpose tasks, but consistently underperform when exposed to highly specialized domains that require deep cultural, linguistic, and subject-matter expertise. In particular, traditional medical systems such as Ayurveda embody centuries of nuanced textual and clinical knowledge that mainstream LLMs fail to accurately interpret or apply. We introduce AyurParam-2.9B, a domain-specialized, bilingual
language model fine-tuned from Param-1-2.9B using an extensive, expertly curated Ayurveda
dataset spanning classical texts and clinical guidance. AyurParam's
dataset incorporates context-aware,
reasoning, and objective-style Q&A in both English and Hindi, with rigorous annotation protocols for factual precision and instructional clarity. Benchmarked on BhashaBench-Ayur, AyurParam not only surpasses all open-source instruction-tuned models in its size class (1.5–3B parameters), but also demonstrates competitive or superior performance compared to much larger models. The results from AyurParam highlight the necessity for authentic domain adaptation and high-quality supervision in delivering reliable, culturally congruent AI for specialized medical knowledge.
Read the full abstract ▾
All authors Sravan Gorugantu, Mohd Nauman, Vijay Devane, Shyam Pawar, Viraj Thakur, Kundeshwar Pundalik, Piyush Sawarkar, Rohit Saluja, Maunendra Desarkar, Ganesh Ramakrishnan
Computation and Language
LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications
Yudong Li, Peiru Yang, Feng Huang et al. (+18 more)
We introduce LiveSecBench, a continuously updated safety
benchmark specifically for Chinese-
language LLM application scenarios. LiveSecBench constructs a high-quality and unique
dataset through a pipeline that combines automated generation with human verification. By periodically releasing new versions to expand the
dataset and update evaluation metrics, LiveSecBench provides a robust and up-to-date standard for AI safety. In this report, we introduce our second release v251215, which evaluates across five dimensions (Public Safety,
Fairness & Bias,
Privacy, Truthfulness, and Mental Health Safety.) We evaluate 57 representative LLMs using an ELO rating system, offering a leaderboard of the current state of Chinese LLM safety. The result is available at https://livesecbench.intokentech.cn/.
Read the full abstract ▾
All authors Yudong Li, Peiru Yang, Feng Huang, Zhongliang Yang, Kecheng Wang, Haitian Li, Baocheng Chen, Xingyu An, Ziyu Liu, Youdan Yang, Kejiang Chen, Sifang Wan, Xu Wang, Yufei Sun, Liyan Wu, Ruiqi Zhou, Wenya Wen, Xingchi Gu, Tianxin Zhang, Yue Gao, Yongfeng Huang