Computation and Language · Cross-listed from cs.AI
GraphBit: A Graph-based Agentic Framework for Non-Linear Agent Orchestration
Yeahia Sarker, Md Rahmat Ullah, Musa Molla et al. (+1 more)
Agentic LLM frameworks that rely on prompted orchestration, where the model itself determines workflow transitions, often suffer from hallucinated routing, infinite loops, and non-reproducible execution. We introduce GraphBit, an engine-orchestrated framework that defines workflows explicitly and deterministically as a directed acyclic graph (DAG). Unlike prompted orchestration,
agents in GraphBit operate as typed functions, while a Rust-based engine governs routing, state transitions, and tool invocation, ensuring reproducibility and auditability. The engine supports parallel branch execution, conditional control flow over structured state predicates, and configurable error recovery. A three-tier
memory architecture consisting of ephemeral scratch space, structured state, and external connectors isolates context across stages, preventing cascading context bloat that degrades
reasoning in long-running pipelines. Across
GAIA benchmark tasks spanning zero-tool, document-augmented, and web-enabled workflows, GraphBit outperforms six existing frameworks, achieving the highest accuracy (67.6 percent), zero framework-induced
hallucinations, the lowest latency (11.9 ms overhead), and the highest throughput. Ablation studies demonstrate that each
memory tier contributes measurably to performance, with deterministic execution providing the greatest gains on tool-intensive tasks representative of real-world deployments.
Read the full abstract ▾
All authors Yeahia Sarker, Md Rahmat Ullah, Musa Molla, Shafiq Joty
Computation and Language · Cross-listed from cs.HC
Evaluating LLMs as Human Surrogates in Controlled Experiments
Adnan Hoq and Tim Weninger
Large
language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured
prompt, and models generate a single 0–10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.
Read the full abstract ▾
Computation and Language
Benchmark for Assessing Olfactory Perception of Large Language Models
Eftychia Makri, Nikolaos Nakis, Laura Sisson et al. (+4 more)
Here we introduce the Olfactory Perception (OP)
benchmark, designed to assess the capability of large
language models (LLMs) to reason about smell. The
benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two
prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name
prompts consistently outperform isomeric SMILES, with gains ranging from +2.4 to +18.9 percentage points (mean approx +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular
reasoning. The best-performing model reaches 64.4% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory
reasoning. We further evaluate a subset of the OP across 21
languages and find that aggregating predictions across
languages improves olfactory prediction, with AUROC = 0.86 for the best performing
language ensemble model. LLMs should be able to handle olfactory and not just visual or aural information.
Read the full abstract ▾
All authors Eftychia Makri, Nikolaos Nakis, Laura Sisson, Gigi Minsky, Leandros Tassiulas, Vahid Satarifard, Nicholas A. Christakis
Computation and Language · Cross-listed from cs.LG
Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
Fangxin Wang, Peyman Baghershahi, Langzhou He et al. (+3 more)
Gradient-based data selection offers a principled framework for estimating sample utility in large
language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the current optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight
algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized
matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
Read the full abstract ▾
All authors Fangxin Wang, Peyman Baghershahi, Langzhou He, Henry Peng Zou, Sourav Medya, Philip S. Yu
Computation and Language
Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings
Abass Oguntade
This paper describes my submission to the Polarization Shared Task at SemEval-2025, which addresses polarization detection and classification in social media text. I develop
Transformer-based systems for English and Swahili across three subtasks: binary polarization detection, multi-label target type classification, and multi-label manifestation identification. The approach leverages
multilingual and African
language-specialized models (mDeBERTa-v3-base, SwahBERT, AfriBERTa-large), class-weighted loss functions, iterative stratified data splitting, and per-label threshold tuning to handle severe class imbalance. The best configuration, mDeBERTa-v3-base, achieves 0.8032 macro-F1 on validation for binary detection, with competitive performance on multi-label tasks (up to 0.556 macro-F1). Error analysis reveals persistent challenges with implicit polarization, code-switching, and distinguishing heated political discourse from genuine polarization.
Read the full abstract ▾
Computation and Language
Upgrade the Retriever First: What an LLM Call per Chunk Buys in RAG
Bhavik Mangla
Many RAG pipelines pay one LLM call per chunk at indexing time to add titles, summaries, keywords and questions (enrichment) or document context (contextual
retrieval, CR). We ask what that call buys over free structure and a better retriever. Our advice: upgrade the retriever first, chunk at headers, and enrich mainly for lexical
retrieval. Under a plan made public before any index was built, 7B enrichment beats a free section-path prefix under a hybrid retriever by +5.5 recall points on Qasper; on FreshStack its gain (+2.7) does not survive Holm adjustment. It is equivalent to CR, which sends 5.8-8.7 times the input
tokens, and Qasper answer F1 (7B reader) is equivalent for both contrasts. Header-led chunks beat 512-character windows; on FreshStack they lose to overlapping 256-
token windows. Under qwen3-
embedding and a reranker, the two strongest of eight retrievers (named as secondary before any outcome was computed), the CIs of 7B enrichment against the free prefix fall within the equivalence margins; under BM25 it gains +8.7. Exploratory analyses attribute the hybrid gain to document expansion and to repairing an untuned rank fusion; the free prefix under either strong retriever beats hybrid enrichment by +3.4 to +6.3 points, though the best 120B-indexed configuration leads the best free one by +2.5 on Qasper. Add-ons built on the call do no better than LLM-free controls, and late chunking lowers Qasper recall. We release the plan, harness and per-question results.
Read the full abstract ▾
Computation and Language
Generating Hierarchical JSON Representations of Scientific Sentences Using LLMs
Satya Sri Rajiteswari Nimmagadda, Ethan Young, Niladri Sengupta et al. (+2 more)
This paper investigates whether structured representations can preserve the meaning of scientific sentences. To test this, a lightweight LLM is fine-tuned using a novel structural loss function to generate hierarchical JSON structures from sentences collected from scientific articles. These JSONs are then used by a generative model to reconstruct the original text. Comparing the original and reconstructed sentences using semantic and lexical similarity we show that hierarchical formats are capable of retaining information of scientific texts effectively.
Read the full abstract ▾
All authors Satya Sri Rajiteswari Nimmagadda, Ethan Young, Niladri Sengupta, Ananya Jana, Aniruddha Maiti
Computation and Language · Cross-listed from q-fin.ST
Generalized Stock Price Prediction for Multiple Stocks Combined with News Fusion
Pei-Jun Liao, Hung-Shin Lee, Yao-Fei Cheng et al. (+3 more)
Predicting stock prices presents challenges in financial
forecasting. While traditional approaches such as ARIMA and RNNs are prevalent, recent developments in Large
Language Models (LLMs) offer alternative methodologies. This paper introduces an approach that integrates LLMs with daily financial news for stock price prediction. To address the challenge of processing news data and identifying relevant content, we utilize stock name
embeddings within
attention mechanisms. Specifically, we encode news articles using a pre-trained LLM and implement three
attention-based pooling techniques – self-attentive, cross-attentive, and position-aware self-attentive pooling – to filter news based on stock relevance. The filtered news
embeddings, combined with historical stock prices, serve as inputs to the prediction model. Unlike prior studies that focus on individual stocks, our method trains a single generalized model applicable across multiple stocks. Experimental results demonstrate a 7.11% reduction in Mean Absolute Error (MAE) compared to the baseline, indicating the utility of stock name
embeddings for news filtering and price
forecasting within a generalized framework.
Read the full abstract ▾
All authors Pei-Jun Liao, Hung-Shin Lee, Yao-Fei Cheng, Li-Wei Chen, Hung-yi Lee, Hsin-Min Wang
Computation and Language · Cross-listed from cs.AI
Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agents
Madhava Gaikwad
Memory-augmented
agents maintain multiple specialized stores, yet most systems retrieve from all stores for every query, increasing cost and introducing irrelevant context. We formulate
memory retrieval as a store-routing problem and evaluate it using coverage, exact match, and
token efficiency metrics. On downstream question answering, an oracle router achieves higher accuracy while using substantially fewer context
tokens compared to uniform
retrieval, demonstrating that selective
retrieval improves both efficiency and performance. Our results show that routing decisions are a first-class component of
memory-augmented
agent design and motivate learned routing mechanisms for scalable multi-store systems. We additionally formalize store selection as a cost-sensitive decision problem that trades answer accuracy against
retrieval cost, providing a principled interpretation of routing policies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Context-Enriched Natural Language Descriptions of Vessel Trajectories
Kostas Patroumpas, Alexandros Troupiotis-Kapeliaris, Giannis Spiliopoulos et al. (+4 more)
We address the problem of transforming raw vessel trajectory data collected from AIS into structured and semantically enriched representations interpretable by humans and directly usable by machine
reasoning systems. We propose a context-aware trajectory abstraction framework that segments noisy AIS sequences into distinct trips each consisting of clean, mobility-annotated episodes. Each episode is further enriched with multi-source contextual information, such as nearby geographic entities, offshore
navigation features, and weather conditions. Crucially, such representations can support generation of controlled natural
language descriptions using LLMs. We empirically examine the quality of such descriptions generated using several LLMs over AIS data along with open contextual features. By increasing semantic density and reducing spatiotemporal complexity, this abstraction can facilitate downstream analytics and enable integration with LLMs for higher-level maritime
reasoning tasks.
Read the full abstract ▾
All authors Kostas Patroumpas, Alexandros Troupiotis-Kapeliaris, Giannis Spiliopoulos, Panagiotis Betchavas, Dimitrios Skoutas, Dimitris Zissis, Nikos Bikakis
Computation and Language · Cross-listed from cs.AI
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
David Gringras
Safety
benchmarks usually test "bare" models that receive
prompts and output responses, but real-world deployments "wrap" those models in complex scaffolds. How much do these scaffolds affect model safety as measured by
benchmarks? We test six leading models on four pre-registered safety
benchmarks with a direct API and three scaffolds: ReAct, multi-
agent, and map-reduce. We conducted 60,112 scored evaluations. On average, how safety is measured matters more than scaffolding does: we find that using a multiple choice vs. open-ended format for otherwise-identical
benchmark items changes measured safety by about 5-20 percentage points (pp). The two formats are scored with different methods (answer extraction and an LLM judge), so the gap is due to measurement rather than differences in latent safety. Using a heuristic to classify model refusals would have led to different findings in four of five cases.
Benchmark choice explains 15.1% of the variation in outcomes; scaffold architecture explains 0.5%, about 33x less. We find that map-reduce scaffolds, a form of structure-destroying delegation that strips answer options by decomposing
prompts, reduce pooled measured safety by 7.3 pp (95% CI: 6.4 to 8.1). The pooled effects for ReAct and multi-
agent scaffolds are within our pre-registered +/-2 pp margin of equivalence. However, there are large differences across models for specific
benchmarks and scaffolds that are hidden by pooled estimates: for example, on the same sycophancy
benchmark items, Opus 4.6 has 16.8 pp lower measured safety with a map-reduce scaffold, while Llama 4 has 18.8 pp higher measured safety. Composite reliability is G = 0.251 (95% CI: [0.000, 0.879]). This wide confidence interval, which spans "of little use" to "very good", does not support using a single composite measure of model safety as the basis for go/no-go decisions about model deployment.
Read the full abstract ▾
Computation and Language
An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
Trinh Pham, Thanh Tam Nguyen, Viet Huynh et al. (+2 more)
Recent advances in large
language models have strengthened Text2SQL systems that translate natural
language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled
dataset when no verified answers are available. This situation arises frequently because database content and structure evolve,
privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled
datasets. It analyzes patterns in the system's own outputs to characterize how the target
dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.
Read the full abstract ▾
All authors Trinh Pham, Thanh Tam Nguyen, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen
Computation and Language
AI Steerability 360: A Toolkit for Steering Large Language Models
Erik Miehling, Karthikeyan Natesan Ramamurthy, Praveen Venkateswaran et al. (+10 more)
The AI Steerability 360 toolkit is an extensible, open-source Python library for steering LLMs. Steering abstractions are designed around four model control surfaces: input (modification of the
prompt), structural (modification of the model's weights or architecture), state (modification of the model's activations and attentions), and output (modification of the decoding or generation process). Steering methods exert control on the model through a common interface, termed a steering pipeline, which additionally allows for the composition of multiple steering methods. Comprehensive evaluation and comparison of steering methods/pipelines is facilitated by use case classes (for defining tasks) and a
benchmark class (for performance comparison on a given task). The functionality provided by the toolkit significantly lowers the barrier to developing and comprehensively evaluating steering methods. The toolkit is Hugging Face native and is released under an Apache 2.0 license at https://github.com/IBM/AISteer360.
Read the full abstract ▾
All authors Erik Miehling, Karthikeyan Natesan Ramamurthy, Praveen Venkateswaran, Irene Ko, Pierre Dognin, Moninder Singh, Tejaswini Pedapati, Avinash Balakrishnan, Matthew Riemer, Dennis Wei, Inge Vejsbjerg, Elizabeth M. Daly, Kush R. Varshney
Computation and Language · Cross-listed from cs.CR
DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation
Bo Jiang
Knowledge
distillation from proprietary LLM APIs poses a growing threat to model providers, yet defenses against this attack remain fragmented and unevaluated. We present DistillGuard, a framework for systematically evaluating output-level defenses against LLM knowledge
distillation. We introduce a taxonomy of three defense categories – output perturbation, data poisoning, and information throttling – and evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and Qwen2.5-7B-Instruct as student across three
benchmarks (MATH-500, HumanEval+, MT-Bench). Our results reveal that, in a same-family
distillation setting against a naive attacker, most output-level defenses are surprisingly ineffective: paraphrasing-based perturbation barely degrades distilled student quality, and data poisoning primarily impairs conversational fluency while leaving task-specific capabilities intact. Only chain-of-thought removal substantially impairs mathematical
reasoning (31.4% vs.\ 67.8% baseline), though code generation remains unaffected. These findings demonstrate that the effectiveness of
distillation defenses is highly task-dependent and that current output-level approaches are insufficient to broadly prevent knowledge theft.
Read the full abstract ▾
Computation and Language
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
David Beauchemin and Richard Khoury
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large
Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard
benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and
retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time
reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context
retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.
Read the full abstract ▾
Computation and Language
Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context
Ashish Pandey and Tek Raj Chhetri
Large
language models (LLMs) increasingly influence global digital ecosystems, yet their potential to perpetuate social and cultural biases remains poorly understood in underrepresented contexts. This study presents a systematic analysis of representational biases in seven state-of-the-art LLMs: GPT-4o-mini, Claude-3-Sonnet, Claude-4-Sonnet, Gemini-2.0-Flash, Gemini-2.0-Lite, Llama-3-70B, and Mistral-Nemo in the Nepali cultural context. Using Croissant-compliant
dataset of 2400+ stereotypical and anti-stereotypical sentence pairs on gender roles across social domains, we implement an evaluation framework, Dual-Metric Bias Assessment (DMBA), combining two metrics: (1) agreement with biased statements and (2) stereotypical completion tendencies. Results show models exhibit measurable explicit agreement bias, with mean bias agreement ranging from 0.36 to 0.43 across decoding configurations, and an implicit completion bias rate of 0.740-0.755. Importantly, implicit completion bias follows a non-linear, U-shaped relationship with temperature, peaking at moderate stochasticity (T=0.3) and declining slightly at higher temperatures. Correlation analysis under different decoding settings revealed that explicit agreement strongly aligns with stereotypical sentence agreement but is a weak and often negative predictor of implicit completion bias, indicating generative bias is poorly captured by agreement metrics. Sensitivity analysis shows increasing top-p amplifies explicit bias, while implicit generative bias remains largely stable. Domain-level analysis shows implicit bias is strongest for race and sociocultural stereotypes, while explicit agreement bias is similar across gender and sociocultural categories, with race showing the lowest explicit agreement. These findings highlight the need for culturally grounded
datasets and debiasing strategies for LLMs in underrepresented societies.
Read the full abstract ▾
Computation and Language
Scaling Data Difficulty: Improving Coding Models via Reinforcement Learning on Fresh and Challenging Problems
Zongqian Li, Tengchao Lv, Shaohan Huang et al. (+8 more)
Training next-generation code generation models requires high-quality
datasets, yet existing
datasets face difficulty imbalance, format inconsistency, and data quality problems. We address these challenges through systematic data processing and difficulty scaling. We introduce a four-stage Data Processing Framework encompassing collection, processing, filtering, and verification, incorporating Automatic Difficulty Filtering via an LLM-based predict-calibrate-select framework that leverages multi-dimensional difficulty metrics across five weighted dimensions to retain challenging problems while removing simplistic ones. The resulting MicroCoder
dataset comprises tens of thousands of curated real competitive programming problems from diverse platforms, emphasizing recency and difficulty. Evaluations on strictly unseen LiveCodeBench demonstrate that MicroCoder achieves 3x larger performance gains within 300 training steps compared to widely-used baseline
datasets of comparable size, with consistent advantages under both GRPO and its variant training
algorithms. The MicroCoder
dataset delivers obvious improvements on medium and hard problems across different model sizes, achieving up to 17.2% relative gains in overall performance where model capabilities are most stretched. These results validate that difficulty-aware data curation improves model performance on challenging tasks, providing multiple insights for
dataset creation in code generation.
Read the full abstract ▾
All authors Zongqian Li, Tengchao Lv, Shaohan Huang, Yixuan Su, Qinzheng Sun, Qiufeng Yin, Ying Xin, Scarlett Li, Lei Cui, Nigel Collier, Furu Wei
Computation and Language · Cross-listed from cs.LG
Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models
Zongqian Li, Shaohan Huang, Zewen Chi et al. (+5 more)
Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies,
algorithms, and
datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with three innovations: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-
Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream
datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework with approximately 25% improved evaluation accuracy and around 40% faster execution. Through comprehensive analysis across more than thirty controlled experiments, we reveal 34 training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.
Read the full abstract ▾
All authors Zongqian Li, Shaohan Huang, Zewen Chi, Yixuan Su, Lexin Zhou, Li Dong, Nigel Collier, Furu Wei
Computation and Language · Cross-listed from cs.DC
ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
Yuzhuang Xu, Xu Han, Yuxuan Li et al. (+1 more)
Although existing frameworks for large
language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and
memory. Current frameworks largely overlook the substantial overhead of cross-NUMA
memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient
memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node
memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.
Read the full abstract ▾
All authors Yuzhuang Xu, Xu Han, Yuxuan Li, Wanxiang Che
Computation and Language
QuadAI at SemEval-2026 Task 3: Ensemble Learning of Hybrid RoBERTa and LLMs for Dimensional Aspect-Based Sentiment Analysis
A.J.W. de Vink, Filippos Karolos Ventirozos, Natalia Amat-Lefort et al. (+1 more)
We present our system for SemEval-2026 Task 3 on dimensional aspect-based sentiment regression. Our approach combines a hybrid RoBERTa encoder, which jointly predicts sentiment using regression and discretized classification heads, with large
language models (LLMs) via prediction-level ensemble learning. The hybrid encoder improves prediction stability by combining continuous and discretized sentiment representations. We further explore in-context learning with LLMs and ridge-regression stacking to combine encoder and LLM predictions. Experimental results on the development set show that ensemble learning significantly improves performance over individual models, achieving substantial reductions in RMSE and improvements in correlation scores. Our findings demonstrate the complementary strengths of encoder-based and LLM-based approaches for dimensional sentiment analysis. Our development code and resources will be shared at https://github.com/aaronlifenghan/ABSentiment
Read the full abstract ▾
All authors A.J.W. de Vink, Filippos Karolos Ventirozos, Natalia Amat-Lefort, Lifeng Han
Computation and Language
Whitening Reveals Cluster Commitment as the Geometric Separator of Hallucination Types
Matic Korun
A
geometric hallucination taxonomy distinguishes three failure types – center-drift (Type~1), wrong-well convergence (Type~2), and coverage gaps (Type~3) – by their signatures in
embedding cluster space. Prior work found Types~1 and~2 indistinguishable in full-dimensional contextual measurement. We address this through PCA-whitening and eigenspectrum decomposition on GPT-2-small, using multi-run stability analysis (20 seeds) with
prompt-level aggregation. Whitening transforms the micro-signal regime into a space where peak cluster
alignment (max\_sim) separates Type~2 from Type~3 at Holm-corrected significance, with condition means following the taxonomy's predicted ordering: Type~2 (highest commitment) $>$ Type~1 (intermediate) $>$ Type~3 (lowest). A first directionally stable but underpowered hint of Type~1/2 separation emerges via the same metric, generating a capacity prediction for larger models.
Prompt diversification from 15 to 30
prompts per group eliminates a false positive in whitened
entropy that appeared robust at the smaller set, demonstrating
prompt-set sensitivity in the micro-signal regime. Eigenspectrum decomposition localizes this artifact to the dominant principal components and confirms that Type~1/2 separation does not emerge in any spectral band, rejecting the spectral mixing hypothesis. The contribution is threefold: whitening as preprocessing that reveals cluster commitment as the theoretically correct separating metric, evidence that the Type~1/2 boundary is a capacity limitation rather than a measurement artifact, and a methodological finding about
prompt-set fragility in near-saturated representation spaces.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
Shaoxiong Zhan, Yanlin Lai, Zheng Liu et al. (+6 more)
Current Large
Language Models have achieved Olympiad-level logic, yet
Vision-
Language Models paradoxically falter on elementary spatial tasks like block counting. This capability mismatch reveals a critical “spatial intelligence gap,” where models fail to construct coherent 3D mental representations from 2D observations. We uncover this gap via diagnostic analyses showing the bottleneck is a missing view-consistent spatial interface rather than insufficient visual features or weak
reasoning. To bridge this, we introduce 3ViewSense, a framework that grounds spatial
reasoning in Orthographic Views. Drawing on engineering cognition, we propose a “Simulate-and-Reason” mechanism that decomposes complex scenes into canonical orthographic projections to resolve
geometric ambiguities. By aligning
egocentric perceptions with these allocentric references, our method facilitates explicit mental rotation and reconstruction. Empirical results on spatial
reasoning benchmarks demonstrate that our method significantly outperforms existing baselines, with consistent gains on occlusion-heavy counting and view-consistent spatial
reasoning. The framework also improves the stability and consistency of spatial descriptions, offering a scalable path toward stronger spatial intelligence in
multimodal systems.~\footnote{https://github.com/Jasaxion/3ViewSense}
Read the full abstract ▾
All authors Shaoxiong Zhan, Yanlin Lai, Zheng Liu, Hai Lin, Shen Li, Xiaodong Cai, Zijian Lin, Wen Huang, Hai-Tao Zheng
Computation and Language · Cross-listed from cs.AI
Large Language Model for Discrete Optimization Problems: Evaluation and Step-by-step Reasoning
Tianhao Qian, Guilin Qi, Z.Y. Wu et al. (+3 more)
This work investigated the capabilities of different models, including the Llama-3 series of models and CHATGPT, with different forms of expression in solving discrete optimization problems by testing natural
language datasets. In contrast to formal
datasets with a limited scope of parameters, our
dataset included a variety of problem types in discrete optimization problems and featured a wide range of parameter magnitudes, including instances with large parameter sets, integrated with augmented data. It aimed to (1) provide an overview of LLMs' ability in large-scale problems, (2) offer suggestions to those who want to solve discrete optimization problems automatically, and (3) regard the performance as a
benchmark for future research. These
datasets included original, expanded and augmented
datasets. Among these three
datasets, the original and augmented ones aimed for evaluation while the expanded one may help finetune a new model. In the experiment, comparisons were made between strong and week models, CoT methods and No-CoT methods on various
datasets. The result showed that stronger model performed better reasonably. Contrary to general agreement, it also showed that CoT technique was not always effective regarding the capability of models and disordered
datasets improved performance of models on easy to-understand problems, even though they were sometimes with high variance, a manifestation of instability. Therefore, for those who seek to enhance the automatic resolution of discrete optimization problems, it is recommended to consult the results, including the line charts presented in the Appendix, as well as the conclusions drawn in this study for relevant suggestions.
Read the full abstract ▾
All authors Tianhao Qian, Guilin Qi, Z.Y. Wu, Ran Gu, Xuanyi Liu, Canchen Lyu
Computation and Language · Cross-listed from cs.DC
Scalable Training of Mixture-of-Experts Models with Megatron Core
Zijie Yan, Hongxiao Bai, Xin Yao et al. (+42 more)
Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each
token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-
token computation, creating coupled constraints across
memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning
memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core.
Read the full abstract ▾
All authors Zijie Yan, Hongxiao Bai, Xin Yao, Dennis Liu, Tong Liu, Hongbin Liu, Pingtian Li, Evan Wu, Shiqing Fan, Li Tao, Robin Zhang, Yuzhong Wang, Shifang Xu, Jack Chang, Xuwen Chen, Kunlun Li, Yan Bai, Gao Deng, Nan Zheng, Vijay Anand Korthikanti, Abhinav Khattar, Ethan He, Soham Govande, Sangkug Lym, Zhongbo Zhu, Qi Zhang, Haochen Yuan, Xiaowei Ren, Deyu Fu, Tailai Ma, Shunkang Zhang, Jiang Shao, Ray Wang, Vasudevan Rengasamy, Rachit Garg, Santosh Bhavani, Xipeng Li, Chandler Zhou, David Wu, Yingcan Wei, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Jiajie Yao, June Yang
Computation and Language
KohakuRAG: A simple RAG framework with hierarchical document indexing
Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang et al. (+1 more)
Retrieval-augmented generation (RAG) systems that answer questions from document collections face compounding difficulties when high-precision citations are required: flat chunking strategies sacrifice document structure, single-query formulations miss relevant passages through vocabulary mismatch, and single-pass inference produces
stochastic answers that vary in both content and citation selection. We present KohakuRAG, a hierarchical RAG framework that preserves document structure through a four-level tree representation (document $\rightarrow$ section $\rightarrow$ paragraph $\rightarrow$ sentence) with bottom-up
embedding aggregation, improves
retrieval coverage through an LLM-powered query planner with cross-query reranking, and stabilizes answers through ensemble inference with abstention-aware voting. We evaluate on the WattBot 2025 Challenge, a
benchmark requiring systems to answer technical questions from 32 documents with $\pm$0.1% numeric tolerance and exact source attribution. KohakuRAG achieves first place on both public and private leaderboards (final score 0.861), as the only team to maintain the top position across both evaluation partitions. Ablation studies reveal that
prompt ordering (+80% relative), retry mechanisms (+69%), and ensemble voting with blank filtering (+1.2pp) each contribute substantially, while hierarchical dense
retrieval alone matches hybrid sparse-dense approaches (BM25 adds only +3.1pp). We release KohakuRAG as open-source software at https://github.com/KohakuBlueleaf/KohakuRAG.
Read the full abstract ▾
All authors Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang, Buu-Khang Tu
Computation and Language
StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control
Haishu Zhao, Aokai Hao, Yuan Ge et al. (+3 more)
Speech language models (SLMs) have significantly extended the interactive capability of text-based Large
Language Models (LLMs) by incorporating paralinguistic information. For more realistic interactive experience with customized styles, current SLMs have managed to interpret and control speaking style intensity from user
prompts during the dialogue process. However, there remains a lack of systematic
benchmarks that quantifies and evaluates the style intensity control ability in conversations. In this paper, we propose StyleBench, a multi-turn dialogue
benchmark for comprehensively evaluating the style intensity control ability across four dimensions:
emotion, speed, volume, and pitch. Our results reveal the performance gaps between leading SLMs and omni
language models (OLMs), suggesting the underlying reasons and promising approaches for future exploration.
Read the full abstract ▾
All authors Haishu Zhao, Aokai Hao, Yuan Ge, Zhenqiang Hong, Tong Xiao, Jingbo Zhu
Computation and Language · Cross-listed from cs.SE
KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generation
Jiazhen Kang, Yuchen Lu, Chen Jiang et al. (+6 more)
Code
evolution is inevitable in modern software development. Changes to third-party APIs frequently break existing code and complicate maintenance, posing practical challenges for developers. While large
language models (LLMs) have shown promise in code generation, they struggle to reason without a structured representation of these evolving relationships, often leading them to produce outdated APIs or invalid outputs. In this work, we propose a knowledge graph-augmented framework that decomposes the migration task into two synergistic stages:
evolution path
retrieval and path-informed code generation. Our approach constructs static and dynamic API graphs to model intra-version structures and cross-version transitions, enabling structured
reasoning over API
evolution. Both modules are trained with synthetic supervision automatically derived from real-world API diffs, ensuring scalability and minimal human effort. Extensive experiments across single-package and multi-package
benchmarks demonstrate that our framework significantly improves migration accuracy, controllability, and execution success over standard LLM baselines. The source code and
datasets are available at: https://github.com/kangjz1203/KCoEvo.
Read the full abstract ▾
All authors Jiazhen Kang, Yuchen Lu, Chen Jiang, Jinrui Liu, Tianhao Zhang, Bo Jiang, Ningyuan Sun, Tongtong Wu, Guilin Qi
Computation and Language
Nwāchā Munā: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR
Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel et al. (+4 more)
Nepal Bhasha (Newari), an endangered
language of the Kathmandu Valley, remains digitally marginalized due to the severe scarcity of annotated
speech resources. In this work, we introduce Nwāchā Munā, a newly curated 5.39-hour manually transcribed Devanagari
speech corpus for Nepal Bhasha, and establish the first
benchmark using script-preserving acoustic modeling. We investigate whether proximal cross-lingual transfer from a geographically and linguistically adjacent
language (Nepali) can rival large-scale
multilingual pretraining in an ultra-low-resource Automatic
Speech Recognition (ASR) setting. Fine-tuning a Nepali Conformer model reduces the Character Error Rate (CER) from a 52.54% zero-shot baseline to 17.59% with data augmentation, effectively matching the performance of the
multilingual Whisper-Small model despite utilizing significantly fewer parameters. Our findings demonstrate that proximal transfer from Nepali
language serves as a computationally efficient alternative to massive
multilingual models. We openly release the
dataset and
benchmarks to digitally enable the Newari community and foster further research in Nepal Bhasha.
Read the full abstract ▾
All authors Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel, Rupak Tiwari, Arju Shrestha, Rupak Raj Ghimire, Bal Krishna Bal
Computation and Language
Learning-free L2-Accented Speech Generation using Phonological Rules
Thanathai Lertpetchpun, Yoonjeong Lee, Jihwan Lee et al. (+3 more)
Accent plays a crucial role in speaker identity and inclusivity in
speech technologies. Existing accented text-to-
speech (TTS) systems either require large-scale accented
datasets or lack fine-grained phoneme-level controllability. We propose a accented TTS framework that combines phonological rules with a
multilingual TTS model. The rules are applied to phoneme sequences to transform accent at the phoneme level while preserving intelligibility. The method requires no accented training data and enables explicit phoneme-level accent manipulation. We design rule sets for Spanish- and Indian-accented English, modeling systematic differences in consonants, vowels, and syllable structure arising from phonotactic constraints. We analyze the trade-off between phoneme-level duration
alignment and accent as realized in
speech timing. Experimental results demonstrate effective accent shift while maintaining
speech quality.
Read the full abstract ▾
All authors Thanathai Lertpetchpun, Yoonjeong Lee, Jihwan Lee, Tiantian Feng, Dani Byrd, Shrikanth Narayanan
Computation and Language
Abdessalam Bouchekif, Shahd Gaben, Samer Rashwani et al. (+5 more)
Islamic inheritance law is challenging for large
language models because solving inheritance cases requires complex, structured, multi-step
reasoning and the correct application of juristic rules to compute heirs' shares. We introduce MAWARITH, a large-scale annotated
dataset of 12,500 Arabic inheritance cases for training and evaluating models on the full
reasoning chain: (i) identifying eligible heirs, (ii) applying blocking (\textitḥajb}) and allocation rules, and (iii) computing exact inheritance shares. To the best of our knowledge, MAWARITH is the first Arabic corpus and
benchmark designed for end-to-end Islamic inheritance
reasoning. Unlike prior
datasets that restrict inheritance case solving to multiple-choice questions, MAWARITH supports the full
reasoning chain and provides step-by-step solutions with justifications grounded in classical juristic sources and established inheritance rules, as well as exact share calculations. This enables models to learn how to generate detailed, step-by-step responses to user queries that reflect real-world Islamic inheritance cases. To evaluate models beyond final-answer accuracy, we propose MIR-E (Mawarith Inheritance
Reasoning Evaluation), a weighted multi-stage metric that scores key
reasoning stages and captures error propagation across the pipeline. We evaluate six large
language models in a zero-shot setting. A commercial model achieves about 90%, whereas all evaluated open-source models remain below 50%. Our error analysis identifies recurring failure patterns, including scenario misinterpretation, errors in heir identification, errors in share allocation, and missing or incorrect application of key inheritance rules such as \textquotesingle awl and radd. The MAWARITH
dataset is publicly available at https://gitlab.com/nlpresearcher/mawarith.
Read the full abstract ▾
All authors Abdessalam Bouchekif, Shahd Gaben, Samer Rashwani, Somaya Eltanbouly, Mutaz Al-Khatib, Heba Sbahi, Mohammed Ghaly, Emad Mohamed