Computation and Language
Can Agent Conquer Web? Exploring the Frontiers of ChatGPT Atlas Agent in Web Games
Jingran Zhang, Ning Li and Justin Cui
OpenAI's ChatGPT Atlas introduces new capabilities for web interaction, enabling the model to analyze webpages, process user intents, and execute cursor and keyboard inputs directly within the browser. While its capacity for information
retrieval tasks has been demonstrated, its performance in dynamic, interactive environments remains less explored. In this study, we conduct an early evaluation of Atlas's web interaction capabilities using browser-based games as test scenarios, including Google's T-Rex Runner, Sudoku, Flappy Bird, and Stein.world. We employ in-game performance scores as quantitative metrics to assess performance across different task types. Our results show that Atlas performs strongly in logical
reasoning tasks like Sudoku, completing puzzles significantly faster than human baselines, but struggles substantially in real-time games requiring precise timing and motor control, often failing to progress beyond initial obstacles. These findings suggest that while Atlas demonstrates capable analytical processing, there remain notable limitations in dynamic web environments requiring real-time interaction. The website of our project can be found at https://atlas-game-eval.github.io.
Read the full abstract ▾
Computation and Language
Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
Michal Štefánik, Timothee Mickus, Marek Kadlčík et al. (+7 more)
Prior work has shown that large
language models (LLMs) often converge to accurate input
embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors.
Read the full abstract ▾
All authors Michal Štefánik, Timothee Mickus, Marek Kadlčík, Bertram Højer, Michal Spiegel, Raúl Vázquez, Aman Sinha, Josef Kuchař, Philipp Mondorf, Pontus Stenetorp
Computation and Language
Do LLMs Signal When They're Right? Evidence from Neuron Agreement
Kang Chen, Yaoning Wang, Kai Xiong et al. (+4 more)
Large
language models (LLMs) commonly boost
reasoning via sample-evaluate-ensemble decoders, achieving label free gains without ground truth. However, prevailing strategies score candidates using only external outputs such as
token probabilities,
entropies, or self evaluations, and these signals can be poorly calibrated after post training. We instead analyze internal behavior based on
neuron activations and uncover three findings: (1) external signals are low dimensional projections of richer internal dynamics; (2) correct responses activate substantially fewer unique
neurons than incorrect ones throughout generation; and (3) activations from correct responses exhibit stronger cross sample agreement, whereas incorrect ones diverge. Motivated by these observations, we propose
Neuron Agreement Decoding (NAD), an unsupervised best-of-N method that selects candidates using activation sparsity and cross sample
neuron agreement, operating solely on internal signals and without requiring comparable textual outputs. NAD enables early correctness prediction within the first 32 generated
tokens and supports aggressive early stopping. Across math and science
benchmarks with verifiable answers, NAD matches majority voting; on open ended coding
benchmarks where majority voting is inapplicable, NAD consistently outperforms Avg@64. By
pruning unpromising trajectories early, NAD reduces
token usage by 99% with minimal loss in generation quality, showing that internal signals provide reliable, scalable, and efficient guidance for label free ensemble decoding.
Read the full abstract ▾
All authors Kang Chen, Yaoning Wang, Kai Xiong, Zhuoka Feng, Wenhe Sun, Haotian Chen, Yixin Cao
Computation and Language · Cross-listed from cs.CR
PVMark: Enabling Public Verifiability for LLM Watermarking Schemes
Haohua Duan, Liyao Xiang and Xin Zhang
Watermarking schemes for large
language models (LLMs) have been proposed to identify the source of the generated text, mitigating the potential threats emerged from model theft. However, current watermarking solutions hardly resolve the trust issue: the non-public watermark detection cannot prove itself faithfully conducting the detection. We observe that it is attributed to the secret key mostly used in the watermark detection – it cannot be public, or the adversary may launch removal attacks provided the key; nor can it be private, or the watermarking detection is opaque to the public. To resolve the dilemma, we propose PVMark, a plugin based on zero-knowledge proof (ZKP), enabling the watermark detection process to be publicly verifiable by third parties without disclosing any secret key. PVMark hinges upon the proof of `correct execution' of watermark detection on which a set of ZKP constraints are built, including mapping, random number generation, comparison, and summation. We implement multiple variants of PVMark in Python, Rust and Circom, covering combinations of three watermarking schemes, three hash functions, and four ZKP protocols, to show our approach effectively works under a variety of circumstances. By experimental results, PVMark efficiently enables public verifiability on the state-of-the-art LLM watermarking schemes yet without compromising the watermarking performance, promising to be deployed in practice.
Read the full abstract ▾
Computation and Language
Sukrit Sriratanawilai, Jhayahgrit Thongwat, Romrawin Chumpu et al. (+3 more)
Vision-
language models (VLMs) exhibit uneven performance across
languages, a problem that is often exacerbated when the model size is reduced. While Knowledge
distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five
distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five
distillation formulations across CLIP and SigLIP2, and evaluate them on in-domain
retrieval and out-of-domain visual QA. We find that some configurations preserve or even improve
multilingual retrieval robustness despite halving model size, but others fail to maintain cross-task stability, exposing design-sensitive trade-offs that aggregate accuracy alone does not reveal.
Read the full abstract ▾
All authors Sukrit Sriratanawilai, Jhayahgrit Thongwat, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, Peerat Limkonchotiwat
Computation and Language
Are Language Models Borrowing-Blind? A Multilingual Evaluation of Loanword Identification across 10 Languages
Mérilin Sousa Silva and Sina Ahmadi
Throughout
language history, words are borrowed from one
language to another and gradually become integrated into the recipient's lexicon. Speakers can often differentiate these loanwords from native vocabulary, particularly in bilingual communities where a dominant
language continuously imposes lexical items on a minority
language. This paper investigates whether pretrained
language models, including large
language models, possess similar capabilities for loanword identification. We evaluate multiple models across 10
languages. Despite explicit instructions and contextual information, our results show that models perform poorly in distinguishing loanwords from native ones. These findings corroborate previous evidence that modern NLP systems exhibit a bias toward loanwords rather than native equivalents. Our work has implications for developing NLP tools for minority
languages and supporting
language preservation in communities under lexical pressure from dominant
languages.
Read the full abstract ▾
Computation and Language
Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs
Takuma Sato, Seiya Kawano and Koichiro Yoshino
The ability to accurately interpret implied meanings plays a crucial role in human communication and
language use, and
language models are also expected to possess this capability. This study demonstrates that providing
language models with pragmatic theories as
prompts is an effective in-context learning approach for tasks to understand implied meanings. Specifically, we propose an approach in which an overview of pragmatic theories, such as Gricean pragmatics and Relevance Theory, is presented as a
prompt to the
language model, guiding it through a step-by-step
reasoning process to derive a final interpretation. Experimental results showed that, compared to the baseline, which
prompts intermediate
reasoning without presenting pragmatic theories (0-shot Chain-of-Thought), our methods enabled
language models to achieve up to 9.6% higher scores on pragmatic
reasoning tasks. Furthermore, we show that even without explaining the details of pragmatic theories, merely mentioning their names in the
prompt leads to a certain performance improvement (around 1-3%) in larger models compared to the baseline.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models
Shiho Matta, Lis Kanashiro Pereira, Peitao Han et al. (+2 more)
Modern
vision-
language models (VLMs) excel at many
multimodal tasks, yet their grasp of temporal information in
video remains weak and has not been adequately evaluated. We probe this gap with a deceptively simple but revealing challenge: judging the arrow of time (AoT)-whether a short clip is played forward or backward. We introduce AoT-PsyPhyBENCH, a psychophysically validated
benchmark that tests whether VLMs can infer temporal direction in natural
videos using the same stimuli and behavioral baselines established for humans. Our comprehensive evaluation of open-weight and proprietary,
reasoning and non-
reasoning VLMs reveals that most models perform near chance, and even the best model lags far behind human accuracy on physically irreversible processes (e.g., free fall,
diffusion/explosion) and causal manual actions (division/addition) that humans recognize almost instantly. These results highlight a fundamental gap in current
multimodal systems: while they capture rich visual-semantic correlations, they lack the inductive biases required for temporal continuity and causal understanding. We release the code and data for AoT-PsyPhyBENCH to encourage further progress in the physical and temporal
reasoning capabilities of VLMs.
Read the full abstract ▾
All authors Shiho Matta, Lis Kanashiro Pereira, Peitao Han, Fei Cheng, Shigeru Kitazawa
Computation and Language
Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning
Qi Luo, Xiaonan Li, Tingshuo Fan et al. (+2 more)
Retrieval-augmented generation (RAG) has emerged as a leading approach to reducing
hallucinations in large
language models (LLMs). Current RAG evaluation
benchmarks primarily focus on what we call local RAG: retrieving relevant chunks from a small subset of documents to answer queries that require only localized understanding within specific text chunks. However, many real-world applications require a fundamentally different capability – global RAG – which involves aggregating and analyzing information across entire document collections to derive corpus-level insights (for example, "What are the top 10 most cited papers in 2023?"). In this paper, we introduce GlobalQA – the first
benchmark specifically designed to evaluate global RAG capabilities, covering four core task types: counting, extremum queries, sorting, and top-k extraction. Through systematic evaluation across different models and baselines, we find that existing RAG methods perform poorly on global tasks, with the strongest baseline achieving only 1.51 F1 score. To address these challenges, we propose GlobalRAG, a multi-tool collaborative framework that preserves structural coherence through chunk-level
retrieval, incorporates LLM-driven intelligent filters to eliminate noisy documents, and integrates aggregation modules for precise symbolic computation. On the Qwen2.5-14B model, GlobalRAG achieves 6.63 F1 compared to the strongest baseline's 1.51 F1, validating the effectiveness of our method.
Read the full abstract ▾
All authors Qi Luo, Xiaonan Li, Tingshuo Fan, Xinchi Chen, Xipeng Qiu
Computation and Language
What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
Rajiv Movva, Smitha Milli, Sewon Min et al. (+1 more)
Human feedback can alter
language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a
dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7
datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of
dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community
Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.
Read the full abstract ▾
All authors Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson
Computation and Language
Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocation
Woojin Kim and Jaeyoung Do
While
diffusion language models (DLMs) enable fine-grained refinement, their practical controllability remains fragile. We identify and formally characterize a central failure mode called update forgetting, in which uniform and context agnostic updates induce
token level fluctuations across timesteps, erasing earlier semantic edits and disrupting the cumulative refinement process, thereby degrading fluency and coherence. As this failure originates in uniform and context agnostic updates, effective control demands explicit
token ordering. We propose
Token Timestep Allocation (TTA), which realizes soft and semantic
token ordering via per
token timestep schedules: critical
tokens are frozen early, while uncertain
tokens receive continued refinement. This timestep based ordering can be instantiated as either a fixed policy or an adaptive policy driven by task signals, thereby supporting a broad spectrum of refinement strategies. Because it operates purely at inference time, it applies uniformly across various DLMs and naturally extends to diverse supervision sources. Empirically, TTA improves controllability and fluency: on sentiment control, it yields more than 20 percent higher accuracy and nearly halves perplexity using less than one fifth the steps; in detoxification, it lowers maximum toxicity (12.2 versus 14.5) and perplexity (26.0 versus 32.0). Together, these results demonstrate that softened ordering via timestep allocation is the critical lever for mitigating update forgetting and achieving stable and controllable
diffusion text generation.
Read the full abstract ▾
Computation and Language
RCScore: Quantifying Response Consistency in Large Language Models
Dongjun Jang, Youngchae Ahn and Hyopil Shin
Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming
benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four
reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level
Hitomi Jin Ling Tee, Chaoren Wang, Zijie Zhang et al. (+1 more)
The evaluation of intelligibility for TTS has reached a bottleneck, as existing assessments heavily rely on word-by-word accuracy metrics such as WER, which fail to capture the complexity of real-world
speech or reflect human comprehension needs. To address this, we propose Spoken-Passage Multiple-Choice Question Answering, a novel subjective approach evaluating the accuracy of key information in synthesized
speech, and release SP-MCQA-Eval, an 8.76-hour news-style
benchmark dataset for SP-MCQA evaluation. Our experiments reveal that low WER does not necessarily guarantee high key-information accuracy, exposing a gap between traditional metrics and practical intelligibility. SP-MCQA shows that even state-of-the-art (SOTA) models still lack robust text normalization and phonetic accuracy. This work underscores the urgent need for high-level, more life-like evaluation criteria now that many systems already excel at WER yet may fall short on real-world intelligibility.
Read the full abstract ▾
All authors Hitomi Jin Ling Tee, Chaoren Wang, Zijie Zhang, Zhizheng Wu
Computation and Language
Similarity-Distance-Magnitude Language Models
Allen Schmaltz
We introduce Similarity-Distance-Magnitude (SDM)
language models (LMs), which are sequence prediction models fine-tuned to maximize the proportion of generations in the well-calibrated, high-probability region partitioned by a final-layer SDM activation layer used for binary classification of instruction-following. We demonstrate that existing pre-trained decoder-only
Transformer LMs can be readily converted into SDM LMs via supervised fine-tuning, using the final-layer SDM activation layer during training to estimate a change-of-base for a supervised next-
token loss over a contrastive input encoding scheme, with additional hard negative examples generated online during training. This results in reduced abstentions (i.e., improved statistical efficiency) compared to strong supervised baselines.
Read the full abstract ▾
Computation and Language
MossNet: Mixture of State-Space Experts is a Multi-Head Attention
Shikhar Tuli, James Seale Smith, Haris Jeelani et al. (+5 more)
Large
language models (LLMs) have significantly advanced generative applications in natural
language processing (NLP). Recent trends in model architectures revolve around efficient variants of
transformers or state-space/gated-recurrent models (SSMs, GRMs). However, prevailing SSM/GRM-based methods often emulate only a single
attention head, potentially limiting their expressiveness. In this work, we propose MossNet, a novel mixture-of-state-space-experts architecture that emulates a linear multi-head
attention (MHA). MossNet leverages a mixture-of-experts (MoE) implementation not only in channel-mixing multi-layered perceptron (MLP) blocks but also in the time-mixing SSM kernels to realize multiple "
attention heads." Extensive experiments on
language modeling and downstream evaluations show that MossNet outperforms both
transformer- and SSM-based architectures of similar model size and data budgets. Larger variants of MossNet, trained on trillions of
tokens, further confirm its scalability and superior performance. In addition, real-device profiling on a Samsung
Galaxy S24 Ultra and an Nvidia A100 GPU demonstrate favorable runtime speed and resource usage compared to similarly sized baselines. Our results suggest that MossNet is a compelling new direction for efficient, high-performing recurrent LLM architectures.
Read the full abstract ▾
All authors Shikhar Tuli, James Seale Smith, Haris Jeelani, Chi-Heng Lin, Abhishek Patel, Vasili Ramanishka, Yen-Chang Hsu, Hongxia Jin
Computation and Language · Cross-listed from cs.AI
ToolRM: Towards Agentic Tool-Use Reward Modeling
Renhao Li, Jianhong Tu, Yang Su et al. (+6 more)
Reward models (RMs) play a critical role in aligning large
language models (LLMs) with human preferences. Yet in the domain of tool learning, the lack of RMs specifically designed for function-calling tasks has limited progress toward more capable agentic AI. We introduce ToolRM, a family of lightweight reward models tailored for general tool-use scenarios. To build these models, we propose a novel pipeline that constructs high-quality pairwise preference data using rule-based scoring and multidimensional sampling. This yields ToolPref-Pairwise-30K, a diverse, balanced, and challenging preference
dataset that supports both generative and discriminative reward modeling. We also introduce TRBench$_{BFCL}$, a
benchmark built on the
agent evaluation suite BFCL to evaluate RMs on tool calling tasks. Trained on our constructed data, models from the Qwen3-4B/8B series achieve up to 17.94% higher accuracy, substantially outperforming frontier LLMs and RMs in pairwise reward judgments. Beyond training objectives, generative ToolRM generalizes to broader critique tasks, including Best-of-N sampling and self-correction. Experiments on ACEBench highlight its effectiveness and efficiency, enabling inference-time scaling while reducing output
token usage by over 66%. Its support for downstream RL training further validates its practical utility. We release data to facilitate future research.
Read the full abstract ▾
All authors Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F. Wong, Junyang Lin, Min Yang
Computation and Language · Cross-listed from cs.AI
Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
Bo Pang, Deqian Kong, Silvio Savarese et al. (+2 more)
Reinforcement learning (RL) can elicit strong
reasoning in large
language models (LLMs), yet most open efforts focus on math and code. We propose
Reasoning Curriculum, a simple two-stage curriculum that first elicits
reasoning skills in pretraining-aligned domains such as math, then adapts and refines these skills across other domains via joint RL. Stage 1 performs a brief cold start and then math-only RL with verifiable rewards to develop
reasoning skills. Stage 2 runs joint RL on mixed-domain data to transfer and consolidate these skills. The curriculum is minimal and backbone-agnostic, requiring no specialized reward models beyond standard verifiability checks. Evaluated on Qwen3-4B and Llama-3.1-8B over a multi-domain suite,
reasoning curriculum yields consistent gains. Ablations and a cognitive-skill analysis indicate that both stages are necessary and that math-first elicitation increases cognitive behaviors important for solving complex problems.
Reasoning Curriculum provides a compact, easy-to-adopt recipe for general
reasoning.
Read the full abstract ▾
All authors Bo Pang, Deqian Kong, Silvio Savarese, Caiming Xiong, Yingbo Zhou
Computation and Language
On the Influence of Discourse Relations in Persuasive Texts
Nawar Turk, Sevag Kaspar and Leila Kosseim
This paper investigates the relationship between Persuasion Techniques (PTs) and Discourse Relations (DRs) by leveraging Large
Language Models (LLMs) and
prompt engineering. Since no
dataset annotated with both PTs and DRs exists, we took the SemEval 2023 Task 3
dataset labelled with 19 PTs as a starting point and developed LLM-based classifiers to label each instance of the
dataset with one of the 22 PDTB 3.0 level-2 DRs. In total, four LLMs were evaluated using 10 different
prompts, resulting in 40 unique DR classifiers. Ensemble models using different majority-pooling strategies were used to create 5 silver
datasets of instances labelled with both persuasion techniques and level-2 PDTB senses. The silver
dataset sizes vary from 1,281 instances to 204 instances, depending on the majority pooling technique used. Statistical analysis of these silver
datasets shows that six discourse relations (namely Cause, Purpose, Contrast, Cause+Belief, Concession, and Condition) play a crucial role in persuasive texts, especially in the use of Loaded
Language, Exaggeration/Minimisation, Repetition and to cast Doubt. This insight can contribute to detecting online propaganda and misinformation, as well as to our general understanding of effective communication.
Read the full abstract ▾
Computation and Language
Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
Feng Ju, Zeyu Qin, Rui Min et al. (+3 more)
While Test-Time Scaling (TTS) has proven effective in improving the
reasoning ability of large
language models (LLMs), low diversity in model outputs often becomes a bottleneck; this is partly caused by the common "one problem, one solution" (1P1S) training practice, which provides a single canonical answer and can push models toward a narrow set of
reasoning paths. This homogenization not only limits sampling effectiveness but also restricts the exploration space for subsequent Reinforcement Learning (RL) stages. To address this, we propose a "one problem, multiple solutions" (1PNS) training paradigm that exposes the model to a variety of valid
reasoning trajectories and thus increases inference diversity. A core challenge for 1PNS is reliably measuring semantic differences between multi-step chains of thought, so we introduce
Reasoning Path Divergence (RPD), a step-level metric that aligns and scores Long Chain-of-Thought solutions to capture differences in intermediate
reasoning. Using RPD, we curate maximally diverse solution sets per problem and fine-tune Qwen3-4B-Base. Experiments show that RPD-selected training yields more varied outputs and higher pass@k, with an average +2.80% gain in pass@16 over a strong 1P1S baseline and a +4.99% gain on AIME24, demonstrating that 1PNS further amplifies the effectiveness of TTS. Our code is available at https://github.com/fengjujf/
Reasoning-Path-Divergence .
Read the full abstract ▾
All authors Feng Ju, Zeyu Qin, Rui Min, Zhitao He, Lingpeng Kong, Yi R. Fung
Computation and Language
QCoder Benchmark: Bridging Language Generation and Quantum Hardware through Simulator-Based Feedback
Taku Mikuriya, Tatsuya Ishigaki, Masayuki Kawarada et al. (+9 more)
Large
language models (LLMs) have increasingly been applied to automatic programming code generation. This task can be viewed as a
language generation task that bridges natural
language, human knowledge, and programming logic. However, it remains underexplored in domains that require interaction with hardware devices, such as
quantum programming, where human coders write Python code that is executed on a
quantum computer. To address this gap, we introduce QCoder
Benchmark, an evaluation framework that assesses LLMs on
quantum programming with feedback from simulated hardware devices. Our
benchmark offers two key features. First, it supports evaluation using a
quantum simulator environment beyond conventional Python execution, allowing feedback of domain-specific metrics such as
circuit depth, execution time, and error classification, which can be used to guide better generation. Second, it incorporates human-written code submissions collected from real programming contests, enabling both quantitative comparisons and qualitative analyses of LLM outputs against human-written codes. Our experiments reveal that even advanced models like GPT-4o achieve only around 18.97% accuracy, highlighting the difficulty of the
benchmark. In contrast,
reasoning-based models such as o3 reach up to 78% accuracy, outperforming averaged success rates of human-written codes (39.98%). We release the QCoder
Benchmark dataset and public evaluation API to support further research. (Codes and
datasets are available at https://qcoder-bench.github.io/ )
Read the full abstract ▾
All authors Taku Mikuriya, Tatsuya Ishigaki, Masayuki Kawarada, Shunya Minami, Tadashi Kadowaki, Yohichi Suzuki, Soshun Naito, Shunya Takata, Takumi Kato, Tamotsu Basseda, Reo Yamada, Hiroya Takamura
Computation and Language · Cross-listed from cs.IR
ORBIT – Open Recommendation Benchmark for Reproducible Research with Hidden Tests
Jingyuan He, Jiongnan Liu, Vishan Vishesh Oberoi et al. (+7 more)
Recommender systems are among the most impactful AI applications, interacting with billions of users every day, guiding them to relevant products, services, or information tailored to their preferences. However, the research and development of recommender systems are hindered by existing
datasets that fail to capture realistic user behaviors and inconsistent evaluation settings that lead to ambiguous conclusions. This paper introduces the Open Recommendation
Benchmark for Reproducible Research with HIdden Tests (ORBIT), a unified
benchmark for consistent and realistic evaluation of recommendation models. ORBIT offers a standardized evaluation framework of public
datasets with reproducible splits and transparent settings for its public leaderboard. Additionally, ORBIT introduces a new webpage recommendation task, ClueWeb-Reco, featuring web browsing sequences from 87 million public, high-quality webpages. ClueWeb-Reco is a synthetic
dataset derived from real, user-consented, and
privacy-guaranteed browsing data. It aligns with modern recommendation scenarios and is reserved as the hidden test part of our leaderboard to challenge recommendation models' generalization ability. ORBIT measures 12 representative recommendation models on its public
benchmark and introduces a prompted LLM baseline on the ClueWeb-Reco hidden test. Our
benchmark results reflect general improvements of recommender systems on the public
datasets, with variable individual performances. The results on the hidden test reveal the limitations of existing approaches in large-scale webpage recommendation and highlight the potential for improvements with LLM integrations. ORBIT
benchmark, leaderboard, and codebase are available at https://www.open-reco-bench.ai.
Read the full abstract ▾
All authors Jingyuan He, Jiongnan Liu, Vishan Vishesh Oberoi, Bolin Wu, Mahima Jagadeesh Patel, Kangrui Mao, Chuning Shi, I-Ta Lee, Arnold Overwijk, Chenyan Xiong
Computation and Language · Cross-listed from cs.LG
Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods
Jiali Cheng, Chirag Agarwal and Hadi Amiri
Knowledge
distillation (KD) is an effective method for model compression and transferring knowledge between models. However, its effect on model's robustness against spurious correlations that degrade performance on out-of-distribution data remains underexplored. This study investigates the effect of knowledge
distillation on the transferability of “debiasing” capabilities from teacher models to student models on natural
language inference (NLI) and image classification tasks. Through extensive experiments, we illustrate several key findings: (i) overall the debiasing capability of a model is undermined post-KD; (ii) training a debiased model does not benefit from injecting teacher knowledge; (iii) although the overall robustness of a model may remain stable post-
distillation, significant variations can occur across different types of biases; and (iv) we pin-point the internal
attention pattern and
circuit that causes the distinct behavior post-KD. Given the above findings, we propose three effective solutions to improve the distillability of debiasing methods: developing high quality data for augmentation, implementing iterative knowledge
distillation, and initializing student models with weights obtained from teacher models. To the best of our knowledge, this is the first study on the effect of KD on debiasing and its interenal mechanism at scale. Our findings provide understandings on how KD works and how to design better debiasing methods.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar et al. (+1 more)
The ability of LLM
agents to plan and invoke tools exposes them to new safety risks, making a comprehensive red-teaming system crucial for discovering vulnerabilities and ensuring their safe deployment. We present SIRAJ: a generic red-teaming framework for arbitrary black-box LLM
agents. We employ a dynamic two-step process that starts with an
agent definition and generates diverse seed test cases that cover various risk outcomes, tool-use trajectories, and risk sources. Then, it iteratively constructs and refines model-based
adversarial attacks based on the execution trajectories of former attempts. To optimize the red-teaming cost, we present a model
distillation approach that leverages structured forms of a teacher model's
reasoning to train smaller models that are equally effective. Across diverse evaluation
agent settings, our seed test case generation approach yields 2 – 2.5x boost to the coverage of risk outcomes and tool-calling trajectories. Our distilled 8B red-teamer model improves attack success rate by 100%, surpassing the 671B Deepseek-R1 model. Our ablations and analyses validate the effectiveness of the iterative framework, structured
reasoning, and the generalization of our red-teamer models.
Read the full abstract ▾
All authors Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar, Amin Saied
Computation and Language
Artificial Intelligence-Enabled Analysis of Radiology Reports: Epidemiology and Consequences of Incidental Thyroid Findings
Felipe Larios, Mariana Borras-Osorio, Yuqi Wu et al. (+18 more)
Importance Incidental thyroid findings (ITFs) are increasingly detected on imaging performed for non-thyroid indications. Their prevalence, features, and clinical consequences remain undefined. Objective To develop, validate, and deploy a natural
language processing (NLP) pipeline to identify ITFs in radiology reports and assess their prevalence, features, and clinical outcomes. Design, Setting, and Participants Retrospective cohort of adults without prior thyroid disease undergoing thyroid-capturing imaging at Mayo Clinic sites from July 1, 2017, to September 30, 2023. A
transformer-based NLP pipeline identified ITFs and extracted nodule characteristics from image reports from multiple modalities and body regions. Main Outcomes and Measures Prevalence of ITFs, downstream thyroid ultrasound, biopsy, thyroidectomy, and thyroid cancer diagnosis. Logistic regression identified demographic and imaging-related factors. Results Among 115,683 patients (mean age, 56.8 [SD 17.2] years; 52.9% women), 9,077 (7.8%) had an ITF, of which 92.9% were nodules. ITFs were more likely in women, older adults, those with higher BMI, and when imaging was ordered by oncology or internal medicine. Compared with chest CT, ITFs were more likely via neck CT, PET, and
nuclear medicine scans. Nodule characteristics were poorly documented, with size reported in 44% and other features in fewer than 15% (e.g. calcifications). Compared with patients without ITFs, those with ITFs had higher odds of thyroid nodule diagnosis, biopsy, thyroidectomy and thyroid cancer diagnosis. Most cancers were papillary, and larger when detected after ITFs vs no ITF. Conclusions ITFs were common and strongly associated with cascades leading to the detection of small, low-risk cancers. These findings underscore the role of ITFs in thyroid cancer overdiagnosis and the need for standardized reporting and more selective follow-up.
Read the full abstract ▾
All authors Felipe Larios, Mariana Borras-Osorio, Yuqi Wu, Ana Gabriela Claros, David Toro-Tobon, Esteban Cabezas, Ricardo Loor-Torres, Maria Mateo Chavez, Kerly Guevara Maldonado, Luis Vilatuna Andrango, Maria Lizarazo Jimenez, Ivan Mateo Alzamora, Misk Al Zahidy, Marcelo Montero, Ana Cristina Proano, Cristian Soto Jacome, Jungwei W. Fan, Oscar J. Ponce-Ponte, Megan E. Branda, Naykky Singh Ospina, Juan P. Brito