Computation and Language · Cross-listed from cs.MA
A Novel Hierarchical Multi-Agent System for Payments Using LLMs
Joon Kiat Chua, Donghao Huang and Zhaoxia Wang
Large
language model (LLM)
agents, such as OpenAI's Operator and Claude's Computer Use, can automate workflows but unable to handle payment tasks. Existing agentic solutions have gained significant
attention; however, even the latest approaches face challenges in implementing end-to-end agentic payment workflows. To address this gap, this research proposes the Hierarchical Multi-
Agent System for Payments (HMASP), which provides an end-to-end agentic method for completing payment workflows. The proposed HMASP leverages either open-weight or proprietary LLMs and employs a modular architecture consisting of the Conversational Payment
Agent (CPA - first
agent level), Supervisor
agents (second
agent level), Routing
agents (third
agent level), and the Process summary
agent (fourth
agent level). The CPA serves as the central entry point, handling all external requests and coordinating subsequent tasks across hierarchical levels. HMASP incorporates architectural patterns that enable modular task execution across
agents and levels for payment operations, including shared state variables, decoupled message states, and structured handoff protocols that facilitate coordination across
agents and workflows. Experimental results demonstrate the feasibility of the proposed HMASP. To our knowledge, HMASP is the first LLM-based multi-
agent system to implement end-to-end agentic payment workflows. This work lays a foundation for extending agentic capabilities into the payment domain.
Read the full abstract ▾
Computation and Language
Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis
Donghao Huang and Zhaoxia Wang
Large
language models (LLMs) with
reasoning capabilities have fueled a compelling narrative that
reasoning universally improves performance across
language tasks. We test this claim through a comprehensive evaluation of 504 configurations across seven model families–including adaptive, conditional, and reinforcement learning-based
reasoning architectures–on sentiment analysis
datasets of varying granularity (binary, five-class, and 27-class
emotion). Our findings reveal that
reasoning effectiveness is strongly task-dependent, challenging prevailing assumptions: (1)
Reasoning shows task-complexity dependence–binary classification degrades up to -19.9 F1 percentage points (pp), while 27-class
emotion recognition gains up to +16.0pp; (2) Distilled
reasoning variants underperform base models by 3-18 pp on simpler tasks, though few-shot prompting enables partial recovery; (3) Few-shot learning improves over zero-shot in most cases regardless of model type, with gains varying by architecture and task complexity; (4) Pareto frontier analysis shows base models dominate efficiency-performance trade-offs, with
reasoning justified only for complex
emotion recognition despite 2.1x-54x computational overhead. We complement these quantitative findings with qualitative error analysis revealing that
reasoning degrades simpler tasks through systematic over-deliberation, offering mechanistic insight beyond the high-level overthinking hypothesis.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DC
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
Ferran Agullo, Joan Oliveras, Chen Wang et al. (+5 more)
Large
Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently. While prior work has largely focused on latency and throughput optimization, minimizing GPU resource requirements through near-peak utilization remains largely underexplored. This paper presents a data-driven pipeline that, for a given workload, computes an adapter placement that serves the workload with the minimum number of GPUs while avoiding request starvation and GPU
memory errors. To that end, the approach identifies the maximum feasible throughput attainable on each GPU by leveraging accurate performance predictions learned from real serving behavior. The proposed pipeline integrates three components: (i) a Digital Twin (DT) tailored to LLM-adapter serving, (ii) a distilled machine learning (ML) model trained on DT-generated data, and (iii) a greedy placement
algorithm that exploits ML-based performance estimates to maximize GPU efficiency. The DT emulates real system dynamics with high fidelity, achieving below 5% throughput estimation error while executing up to 90x faster than full LLM benchmarking across both predictable and unpredictable workloads. The learned ML models further accelerate performance estimation with marginal accuracy degradation, enabling scalable optimization. Experimental results demonstrate that the pipeline substantially improves GPU efficiency, reducing the number of GPUs required to sustain target workloads by 60% on average across the evaluated scenarios. Beyond GPU efficiency, the pipeline can be adapted to alternative objectives, such as latency minimization, highlighting its versatility for future large-scale LLM serving infrastructures.
Read the full abstract ▾
All authors Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral
Computation and Language · Cross-listed from cs.LG
RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
Daniel Yang, Samuel Stante, Florian Redhardt et al. (+5 more)
Reward models are central to aligning large
language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.
Read the full abstract ▾
All authors Daniel Yang, Samuel Stante, Florian Redhardt, Lena Libon, Parnian Kassraie, Ido Hakimi, Barna Pásztor, Andreas Krause
Computation and Language · Cross-listed from cs.CR
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking
Zhicheng Fang, Jingjie Zheng, Chenxu Fu et al. (+1 more)
Jailbreak techniques for large
language models (LLMs) evolve faster than
benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in
datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-
agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-
agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living
benchmarks that keep pace with the rapidly shifting security landscape.
Read the full abstract ▾
All authors Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu
Computation and Language
Dialect and Gender Bias in YouTube's Spanish Captioning System
Iris Dania Jimenez and Christoph Kern
Spanish is the official
language of twenty-one countries and is spoken by over 441 million people. Naturally, there are many variations in how Spanish is spoken across these countries. Media platforms such as YouTube rely on automatic
speech recognition systems to make their content accessible to different groups of users. However, YouTube offers only one option for automatically generating captions in Spanish. This raises the question: could this captioning system be biased against certain Spanish dialects? This study examines the potential biases in YouTube's automatic captioning system by analyzing its performance across various Spanish dialects. By comparing the quality of captions for female and male speakers from different regions, we identify systematic disparities which can be attributed to specific dialects. Our study provides further evidence that algorithmic technologies deployed on digital platforms need to be calibrated to the diverse needs and experiences of their user populations.
Read the full abstract ▾
Computation and Language
The GRADIEND Python Package: An End-to-End System for Gradient-Based Feature Learning
Jonathan Drechsel and Steffen Herbold
We present gradiend, an open-source Python package that operationalizes the GRADIEND method for learning feature directions from factual-counterfactual MLM and CLM gradients in
language models. The package provides a unified workflow for feature-related data creation, training, evaluation, visualization, persistent model rewriting via controlled weight updates, and multi-feature comparison. We demonstrate gradiend through an English pronoun running example, a semantic sentiment use case that evaluates lexical generalization to held-out target words, and a large-scale feature comparison.
Read the full abstract ▾
Computation and Language
Peng Liu, Zhen Tao, Jihao Zhao et al. (+5 more)
Memory systems address the challenge of context loss in Large
Language Model during prolonged interactions. However, compared to human cognition, the efficacy of these systems in processing
emotion-related information remains inconclusive. To address this gap, we propose an
emotion-enhanced
memory evaluation
benchmark to assess the performance of mainstream and state-of-the-art
memory systems in handling affective information. We developed the Human-Like
Memory Emotion (HLME)
dataset, which evaluates
memory systems across three dimensions: emotional information extraction, emotional
memory updating, and emotional
memory question answering. Experimental results indicate that none of the evaluated systems achieve robust performance across all three tasks. Our findings provide an objective perspective on the current deficiencies of
memory systems in processing emotional
memories and suggest a new trajectory for future research and system optimization.
Read the full abstract ▾
All authors Peng Liu, Zhen Tao, Jihao Zhao, Ding Chen, Yansong Zhang, Cuiping Li, Zhiyu Li, Hong Chen
Computation and Language
EDDA-Coordinata: An Annotated Dataset of Historical Geographic Coordinates
Ludovic Moncla, Pierre Nugues, Thierry Joliveau et al. (+1 more)
This paper introduces a
dataset of enriched geographic coordinates retrieved from Diderot and d'Alembert's eighteenth-century Encyclopedie. Automatically recovering geographic coordinates from historical texts is a complex task, as they are expressed in a variety of ways and with varying levels of precision. To improve
retrieval of coordinates from similar digitized early modern texts, we have created a gold standard
dataset, trained models, published the resulting inferred and normalized coordinate data, and experimented applying these models to new texts. From 74,000 total articles in each of the digitized versions of the Encyclopedie from ARTFL and ENCCRE, we examined 15,278 geographical entries, manually identifying 4,798 containing coordinates, and 10,480 with descriptive but non-numerical references. Leveraging our gold standard annotations, we trained
transformer-based models to retrieve and normalize coordinates. The pipeline presented here combines a classifier to identify coordinate-bearing entries and a second model for
retrieval, tested across encoder-decoder and decoder architectures. Cross-validation yielded an 86% EM score. On an out-of-domain eighteenth-century Trevoux dictionary (also in French), our fine-tuned model had a 61% EM score, while for the nineteenth-century, 7th edition of the Encyclopaedia Britannica in English, the EM was 77%. These findings highlight the gold standard
dataset's usefulness as training data, and our two-step method's cross-lingual, cross-domain generalizability.
Read the full abstract ▾
All authors Ludovic Moncla, Pierre Nugues, Thierry Joliveau, Katherine McDonough
Computation and Language
Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language
Nischal Karki, Bipesh Subedi, Prakash Poudyal et al. (+2 more)
Transformer-based models such as BERT have significantly advanced Natural
Language Processing (NLP) across many
languages. However, Nepali, a low-resource
language written in Devanagari script, remains relatively underexplored. This study
benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification. Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali
dataset containing 25,006 sentences across five conceptual domains and the performance was evaluated using accuracy, weighted precision, recall, F1-score, and AUROC metrics. The results reveal that Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming
multilingual and monolingual models. NepBERTa also performed competitively with an F1-score of 88.26%. Overall, these findings establish a robust baseline for future document-level classification and broader Nepali NLP applications.
Read the full abstract ▾
All authors Nischal Karki, Bipesh Subedi, Prakash Poudyal, Rupak Raj Ghimire, Bal Krishna Bal
Computation and Language
The Astonishing Ability of Large Language Models to Parse Jabberwockified Language
Gary Lupyan and Senyi Yang
We show that large
language models (LLMs) have an astonishing ability to recover meaning from severely degraded English texts. Texts in which content words have been randomly substituted by nonsense strings, e.g., "At the ghybe of the swuint, we are haiveed to Wourge Phrear-gwurr, who sproles into an ghitch flount with his crurp", can be translated to conventional English that is, in many cases, close to the original text, e.g., "At the start of the story, we meet a man, Chow, who moves into an apartment building with his wife." These results show that structural cues (e.g., morphosyntax, closed-class words) constrain lexical meaning to a much larger degree than imagined. Although the abilities of LLMs to make sense of "Jabberwockified" English are clearly superhuman, they are highly relevant to understanding linguistic structure and suggest that efficient
language processing either in biological or artificial systems likely benefits from very tight integration between syntax, lexical semantics, and general world knowledge.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
Qihua Dong, Kuo Yang, Lin Ju et al. (+6 more)
Referring Expression Comprehension (REC) links
language to region level visual perception. Standard
benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with
multimodal LLMs but remain weak tests of visual
reasoning and grounding: (i) many expressions are very short, leaving little
reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual
reasoning. We introduce Ref-Adv, a modern REC
benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The
dataset contains referring expressions on real images, curated with hard distractors and annotated with
reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires
reasoning beyond simple cues, and we evaluate a broad suite of contemporary
multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual
reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual
reasoning and grounding in MLLMs.
Read the full abstract ▾
All authors Qihua Dong, Kuo Yang, Lin Ju, Handong Zhao, Yitian Zhang, Yizhou Wang, Huimin Zeng, Jianglin Lu, Yun Fu
Computation and Language · Cross-listed from cs.LG
LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding
Alexander Samarin, Sergei Krutikov, Anton Shevtsov et al. (+3 more)
Speculative decoding accelerates autoregressive large
language model (LLM) inference by using a lightweight draft model to propose candidate
tokens that are then verified in parallel by the target model. The speedup is significantly determined by the acceptance rate, yet standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective. While KL divergence and acceptance rate share the same global optimum, small draft models, having limited capacity, typically converge to suboptimal solutions where minimizing KL does not guarantee maximizing acceptance rate. To address this issue, we propose LK losses, special training objectives that directly target acceptance rate. Comprehensive experiments across four draft architectures and six target models, ranging from 8B to 685B parameters, demonstrate consistent improvements in acceptance metrics across all configurations compared to the standard KL-based training. We evaluate our approach on general, coding and math domains and report gains of up to 8-10% in average acceptance length. LK losses are easy to implement, introduce no computational overhead and can be directly integrated into any existing speculator training framework, making them a compelling alternative to the existing draft training objectives.
Read the full abstract ▾
All authors Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev
Computation and Language · Cross-listed from cs.SE
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov et al. (+1 more)
Software engineering
agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of
benchmarks have emerged,
datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource
language ecosystems. We introduce SWE-rebench V2, a
language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup
agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a
dataset of 32,079 tasks spanning 20
languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming
languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the
datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE
agents across diverse
languages and repositories.
Read the full abstract ▾
All authors Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander Golubev
Computation and Language · Cross-listed from cs.CV
NAU-QMUL: Utilizing BERT and CLIP for Multi-modal AI-Generated Image Detection
Xiaoyu Guo and Arkaitz Zubiaga
With the aim of detecting AI-generated images and identifying the specific models responsible for their generation, we propose a multi-modal multi-task model. The model leverages pre-trained BERT and CLIP
Vision encoders for text and image feature extraction, respectively, and employs cross-modal feature fusion with a tailored multi-task loss function. Additionally, a pseudo-labeling-based data augmentation strategy was utilized to expand the training
dataset with high-confidence samples. The model achieved fifth place in both Tasks A and B of the `CT2: AI-Generated Image Detection' competition, with F1 scores of 83.16% and 48.88%, respectively. These findings highlight the effectiveness of the proposed architecture and its potential for advancing AI-generated content detection in real-world scenarios. The source code for our method is published on https://github.com/xxxxxxxxy/AIGeneratedImageDetection.
Read the full abstract ▾
Computation and Language
CLFEC: A New Task for Unified Linguistic and Factual Error Correction in paragraph-level Chinese Professional Writing
Jian Kai, Zidong Zhang, Jiwen Chen et al. (+5 more)
Chinese text correction has traditionally focused on spelling and grammar, while factual error correction is usually treated separately. However, in paragraph-level Chinese professional writing, linguistic (word/grammar/punctuation) and factual errors frequently co-occur and interact, while many draft-level errors are sparsely observable in published texts after editorial review, making unified correction both necessary and controlled
benchmark construction essential. This paper introduces CLFEC (Chinese Linguistic & Factual Error Correction), a new task for joint linguistic and factual correction. We construct a mixed, multi-domain Chinese professional writing
dataset spanning current affairs, finance, law, and medicine. We then conduct a systematic study of LLM-based correction paradigms, from prompting to
retrieval-augmented generation (RAG) and agentic workflows. The analysis reveals practical challenges, including limited generalization of specialized correction models, the need for evidence grounding for factual repair, the difficulty of mixed-error paragraphs, and over-correction on clean inputs. Results further show that handling linguistic and factual errors within the same context outperforms decoupled pipelines, and that agentic workflows can be effective with suitable backbone models. Overall, CLFEC provides a new
benchmark for Chinese text correction research and practical guidance for proofreading systems.
Read the full abstract ▾
All authors Jian Kai, Zidong Zhang, Jiwen Chen, Zhengxiang Wu, Songtao Sun, Fuyang Li, Yang Cao, Qiang Liu
Computation and Language
Sebastian Gerstner, Leonor Veloso and Hinrich Schütze
We present **GLUScope**, an open-source resource for analyzing
neurons in
Transformer-based
language models (LMs), intended for interpretability researchers. We focus on the now-standard *GLU-based activation functions* such as SwiGLU, used in virtually all recent open-weights LMs. This introduces a new challenge: understanding positive activations is not enough. Instead, both the *gate* and the *in* activation of a
neuron can be positive or negative, leading to four different possible sign combinations (*activation quadrants*) that in some cases have quite different functionalities (e.g. promoting vs. suppressing a specific set of
tokens). Accordingly, for any
neuron, our tool shows text examples for each of the four sign combinations, and indicates how often each combination occurs. Our resource can lead to novel insights: in particular, thanks to analyzing all four activation quadrants, it can surface interpretable behavior that other tools would miss. We also release the code used to create the resource, thus enabling researchers to build on it by adding more models, text corpora, or
neuron pages.
Read the full abstract ▾
Computation and Language
Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding
Xiangzhong Luo, Yilin An, Zhicheng Yu et al. (+2 more)
Diffusion-based large
language models (dLLMs) have shown promising performance across various
reasoning tasks, establishing themselves as an alternative to autoregressive large
language models (LLMs). Unlike autoregressive LLMs that generate one
token per step based on all previous
tokens, dLLMs theoretically enable parallel generation of multiple
tokens at each decoding step. However, recent dLLMs still favor one-
token-per-step generation in practice, as directly decoding multiple masked
tokens often leads to degraded generation quality and stability. This reveals a substantial gap between the theoretical parallelism and practical performance of dLLMs. To bridge this gap, we introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm to unleash the inherent parallelism of dLLMs. During the Divide phase, DiCo first explores the input masked sequence and identifies masked
tokens as seed
tokens, which are then expanded to construct a set of local clusters. During the Conquer phase, DiCo performs parallel decoding across different local clusters constructed in the Divide phase. The divide-and-conquer process repeatedly alternates between the Divide and Conquer phases until convergence. During the Finalize phase, DiCo decodes the remaining few masked
tokens using an effective fine-grained compound decoding scheme to finalize the generation. Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.
Read the full abstract ▾
All authors Xiangzhong Luo, Yilin An, Zhicheng Yu, Weichen Liu, Xu Yang
Computation and Language
Structured Prompt Optimization for Few-Shot Text Classification via Semantic Alignment in Latent Space
Jiasen Zheng, Zijun Zhou, Huajun Zhang et al. (+3 more)
This study addresses the issues of semantic
entanglement, unclear label structure, and insufficient feature representation in few-shot text classification, and proposes an optimization framework based on structured
prompts to enhance semantic understanding and task adaptation under low-resource conditions. The framework first uses a pretrained
language model to encode the input text and obtain basic semantic representations. It then introduces structured
prompts composed of multi-dimensional semantic factors and integrates them with text features through a learnable combination mechanism, which forms task-related representations with clear boundaries in the latent space. To further strengthen the consistency between text representations and label semantics, the method constructs a structured label
embedding matrix and employs a cross-space
alignment mechanism to ensure stable matching between textual features and label attributes. In addition, the model applies
prompt orthogonality constraints and a joint optimization objective to maintain independence across different semantic factors in the
prompts, allowing the structured
prompts to provide transparent and controllable guidance for classification decisions. Three types of sensitivity experiments, including learning rate sensitivity,
prompt length sensitivity, and data scale sensitivity, are designed to evaluate the stability and robustness of the framework under different conditions. Experimental results show that the proposed structured
prompt optimization framework effectively alleviates semantic conflicts and label ambiguity in few-shot text classification. It significantly improves performance on accuracy, precision, recall, and AUC, and demonstrates strong cross-task applicability.
Read the full abstract ▾
All authors Jiasen Zheng, Zijun Zhou, Huajun Zhang, Junjiang Lin, Jingyun Jia, Qi Wang
Computation and Language · Cross-listed from cs.CV
UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking
Hao Wu, Xudong Wang, Jialiang Zhang et al. (+5 more)
One-stream
Transformer-based trackers achieve advanced performance in visual object tracking but suffer from significant computational overhead that hinders real-time deployment. While
token pruning offers a path to efficiency, existing methods are fragmented. They typically prune the search region, dynamic template, and static template in isolation, overlooking critical inter-component dependencies, which yields suboptimal
pruning and degraded accuracy. To address this, we introduce UTPTrack, a simple and Unified
Token Pruning framework that, for the first time, jointly compresses all three components. UTPTrack employs an
attention-guided,
token type-aware strategy to holistically model redundancy, a design that seamlessly supports unified tracking across
multimodal and
language-guided tasks within a single model. Extensive evaluations on 10
benchmarks demonstrate that UTPTrack achieves a new state-of-the-art in the accuracy-efficiency trade-off for
pruning-based trackers,
pruning 65.4% of
vision tokens in RGB-based tracking and 67.5% in unified tracking while preserving 99.7% and 100.5% of baseline performance, respectively. This strong performance across both RGB and
multimodal scenarios underlines its potential as a robust foundation for future research in efficient visual tracking. Code will be released at https://github.com/EIT-NLP/UTPTrack.
Read the full abstract ▾
All authors Hao Wu, Xudong Wang, Jialiang Zhang, Junlong Tong, Xinghao Chen, Junyan Lin, Yunpu Ma, Xiaoyu Shen
Computation and Language
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Seungdong Yoa, Sanghyu Yoon, Suhee Yoon et al. (+4 more)
The evaluation of large
language models (LLMs) has predominantly relied on static
datasets, which offer limited scalability and fail to capture the evolving
reasoning capabilities of recent models. To overcome these limitations, we propose an
agent-centric benchmarking paradigm that moves beyond static
datasets by introducing a dynamic protocol in which autonomous
agents iteratively generate, validate, and solve problems. Within this protocol, a teacher
agent generates candidate problems, an orchestrator
agent rigorously verifies their validity and guards against
adversarial attacks, and a student
agent attempts to solve the validated problems. An invalid problem is revised by the teacher
agent until it passes validation. If the student correctly solves the problem, the orchestrator
prompts the teacher to generate more challenging variants. Consequently, the
benchmark scales in difficulty automatically as more capable
agents are substituted into any role, enabling progressive evaluation of large
language models without manually curated
datasets. Adopting text anomaly detection as our primary evaluation format, which demands cross-sentence logical inference and resists pattern-matching shortcuts, we demonstrate that this protocol systematically exposes corner-case
reasoning errors that conventional
benchmarks fail to reveal. We further advocate evaluating systems along several complementary axes including cross-model pairwise performance and progress between the initial and orchestrator-finalized problems. By shifting the focus from fixed
datasets to dynamic protocols, our approach offers a sustainable direction for evaluating ever-evolving
language models and introduces a research agenda centered on the co-
evolution of
agent-centric
benchmarks.
Read the full abstract ▾
All authors Seungdong Yoa, Sanghyu Yoon, Suhee Yoon, Dongmin Kim, Ye Seul Sim, Junhyun Lee, Woohyung Lim
Computation and Language · Cross-listed from cs.CV
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Hao Wu, Yingqi Fan, Jinyang Dai et al. (+3 more)
The quadratic computational cost of processing
vision tokens in
Multimodal Large
Language Models (MLLMs) hinders their widespread adoption. While progressive
vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns
token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual
tokens exactly where active fusion begins; and (2) Concave Pyramid
Pruning with an Early Exit mechanism to dynamically adjust
pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible
token selection, and parallel decoupling of
vision computation to eliminate hidden overhead associated with dynamic
token reduction. Extensive experiments show that HiDrop compresses about 90% visual
tokens while matching the original performance and accelerating training by 1.72 times. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of
multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.
Read the full abstract ▾
All authors Hao Wu, Yingqi Fan, Jinyang Dai, Junlong Tong, Yunpu Ma, Xiaoyu Shen
Computation and Language
TRIZ-RAGNER: A Retrieval-Augmented Large Language Model for TRIZ-Aware Named Entity Recognition in Patent-Based Contradiction Mining
Zitong Xu, Yuqing Wu and Yue Zhao
TRIZ-based contradiction mining is a fundamental task in patent analysis and systematic innovation, as it enables the identification of improving and worsening technical parameters that drive inventive problem solving. However, existing approaches largely rely on rule-based systems or traditional machine learning models, which struggle with semantic ambiguity, domain dependency, and limited generalization when processing complex patent
language. Recently, large
language models (LLMs) have shown strong semantic understanding capabilities, yet their direct application to TRIZ parameter extraction remains challenging due to
hallucination and insufficient grounding in structured TRIZ knowledge. To address these limitations, this paper proposes TRIZ-RAGNER, a
retrieval-augmented large
language model framework for TRIZ-aware named entity recognition in patent-based contradiction mining. TRIZ-RAGNER reformulates contradiction mining as a semantic-level NER task and integrates dense
retrieval over a TRIZ knowledge base, cross-encoder reranking for context refinement, and structured LLM prompting to extract improving and worsening parameters from patent sentences. By injecting domain-specific TRIZ knowledge into the LLM
reasoning process, the proposed framework effectively reduces semantic noise and improves extraction consistency. Experiments on the PaTRIZ
dataset demonstrate that TRIZ-RAGNER consistently outperforms traditional sequence labeling models and LLM-based baselines. The proposed framework achieves a precision of 85.6%, a recall of 82.9%, and an F1-score of 84.2% in TRIZ contradiction pair identification. Compared with the strongest baseline using
prompt-enhanced GPT, TRIZ-RAGNER yields an absolute F1-score improvement of 7.3 percentage points, confirming the effectiveness of
retrieval-augmented TRIZ knowledge grounding for robust and accurate patent-based contradiction mining.
Read the full abstract ▾
Computation and Language
LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning
Yu Zhu and Kai Yang
The
reasoning capability of large
language models (LLMs), defined as their ability to analyze, infer, and make decisions based on input information, is essential for building intelligent task-oriented dialogue systems. However, existing
benchmarks do not sufficiently reflect the complexity of real-world scenarios, which limits their effectiveness in evaluating and enhancing LLM
reasoning in practical contexts. Many current
reasoning datasets are overly simplistic and abstract, often disconnected from realistic task flows, domain constraints, and operational rules, making it difficult to effectively evaluate LLMs' logical
reasoning ability. In addition, data contamination from pretraining corpora undermines the reliability of evaluation results, and traditional crowdsourcing methods for
dataset construction are labor-intensive and difficult to scale. To address these challenges, we propose a LLM-driven framework for synthesizing multi-turn, task-oriented dialogues grounded in realistic
reasoning scenarios, leveraging trilevel optimization to enhance dialogue quality. Our method generates dialogues grounded in authentic task scenarios, enriched with real-world information, and exhibiting strong contextual coherence. Corresponding
reasoning tasks are carefully designed around these dialogues and iteratively refined to continuously improve the tasks' quality and challenge. The resulting
dataset serves as a valuable
benchmark for assessing and advancing the realistic logical
reasoning capabilities of LLMs. Experimental results show that our synthetic data-based
reasoning tasks introduce non-trivial
reasoning challenges and provide meaningful support for improving the
reasoning capabilities of LLMs.
Read the full abstract ▾
Computation and Language
LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Rafid Ishrak Jahan, Fahmid Shahriar Iqbal and Sagnik Ray Choudhury
Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale
dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to
adversarial perturbations. Overall, this work provides one of the largest public LFQA preference
datasets and a rubric-driven framework for transparent and reliable evaluation.
Read the full abstract ▾
Computation and Language
BRIDGE the Gap: Mitigating Bias Amplification in Automated Scoring of English Language Learners via Inter-group Data Augmentation
Yun Wang, Xuansheng Wu, Jingyuan Huang et al. (+3 more)
In the field of educational assessment, automated scoring systems increasingly rely on deep learning and large
language models (LLMs). However, these systems face significant risks of bias amplification, where model prediction gaps between student groups become larger than those observed in training data. This issue is especially severe for underrepresented groups such as English
Language Learners (ELLs), as models may inherit and further magnify existing disparities in the data. We identify that this issue is closely tied to representation bias: the scarcity of minority (high-scoring ELL) samples makes models trained with empirical risk minimization favor majority (non-ELL) linguistic patterns. Consequently, models tend to under-predict ELL students who even demonstrate comparable domain knowledge but use different linguistic patterns, thereby undermining the
fairness of automated scoring outcomes. To mitigate this, we propose BRIDGE, a Bias-Reducing Inter-group Data GEneration framework designed for low-resource assessment settings. Instead of relying on the limited minority samples, BRIDGE synthesizes high-scoring ELL samples by "pasting" construct-relevant (i.e., rubric-aligned knowledge and evidence) content from abundant high-scoring non-ELL samples into authentic ELL linguistic patterns. We further introduce a discriminator model to ensure the quality of synthetic samples. Experiments on California Science Test (CAST)
datasets demonstrate that BRIDGE effectively reduces prediction bias for high-scoring ELL students while maintaining overall scoring performance. Notably, our method achieves
fairness gains comparable to using additional real human data, offering a cost-effective solution for ensuring equitable scoring in large-scale assessments.
Read the full abstract ▾
All authors Yun Wang, Xuansheng Wu, Jingyuan Huang, Lei Liu, Xiaoming Zhai, Ninghao Liu
Computation and Language
Multi-Agent Causal Reasoning for Suicide Ideation Detection Through Online Conversations
Jun Li, Xiangmeng Wang, Haoyang Li et al. (+6 more)
Suicide remains a pressing global public health concern. While social media platforms offer opportunities for early risk detection through online conversation trees, existing approaches face two major limitations: (1) They rely on predefined rules (e.g., quotes or relies) to log conversations that capture only a narrow spectrum of user interactions, and (2) They overlook hidden influences such as user conformity and suicide copycat behavior, which can significantly affect suicidal expression and propagation in online communities. To address these limitations, we propose a Multi-
Agent Causal
Reasoning (MACR) framework that collaboratively employs a
Reasoning Agent to scale user interactions and a Bias-aware Decision-Making
Agent to mitigate harmful biases arising from hidden influences. The
Reasoning Agent integrates cognitive appraisal theory to generate counterfactual user reactions to posts, thereby scaling user interactions. It analyses these reactions through structured dimensions, i.e., cognitive, emotional, and behavioral patterns, with a dedicated sub-
agent responsible for each dimension. The Bias-aware Decision-Making
Agent mitigates hidden biases through a front-door adjustment strategy, leveraging the counterfactual user reactions produced by the
Reasoning Agent. Through the collaboration of
reasoning and bias-aware decision making, the proposed MACR framework not only alleviates hidden biases, but also enriches contextual information of user interactions with counterfactual knowledge. Extensive experiments on real-world conversational
datasets demonstrate the effectiveness and robustness of MACR in identifying suicide risk.
Read the full abstract ▾
All authors Jun Li, Xiangmeng Wang, Haoyang Li, Yifei Yan, Shijie Zhang, Hong Va Leong, Ling Feng, Nancy Xiaonan Yu, Qing Li