Artificial Intelligence
How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Ali Ansari, Haoran Sun, Andy Zeyi Liu et al. (+48 more)
Low reported scores on leading physics
benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier
language models still struggle with advanced physics, a demanding test of their scientific
reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics
benchmarks and
auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model errors from grader errors, incorrect reference solutions, and ambiguous or underspecified questions. Most audited cases initially evaluated as incorrect reflect these benchmarking issues rather than errors in the models' physics
reasoning. We then ask experts to address these benchmarking issues by correcting erroneous reference solutions and repairing or excluding flawed questions. We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-
Benchmark, while its corrected pass@4 reaches 94.4% on the 54 retained CritPt challenges. Corrected scores are computed on the retained evaluation subsets following expert review. Scores on the audited subsets of UGPhysics, PRISM-Physics, and PHYBench also rise substantially after correction. These findings suggest that current
benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.
Read the full abstract ▾
All authors Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous
Artificial Intelligence · Cross-listed from cs.CL
Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie et al. (+2 more)
Large
language models (LLMs) have achieved strong performance on a wide range of natural
language tasks, and recent
benchmarks suggest that they are increasingly adept at multi-hop
reasoning. However, these
benchmarks are typically short-horizon, requiring only a small number of
retrieval or inference steps, and provide limited evidence of reliability on real-world tasks that involve following manuals spanning hundreds of pages with complex, interdependent guidelines. In this paper, we introduce Tasks over Application Manuals (TAM), a
benchmark for evaluating long-horizon procedural
reasoning. We construct TAM by curating real-world tasks from two domains: ICD-10-CM clinical coding (mapping medical conditions to diagnostic codes) and U.S. federal sentencing (computing crime sentencing guideline outcomes, specifically offense levels), with human-validated labels. Each task requires following an authoritative manual with tens of thousands of rules and executing a sequence of interdependent steps across different sections to produce an exact answer. We evaluate general-purpose prompting approaches, including
retrieval-augmented generation, ReAct-style prompting, and an
agent-harness baseline on GPT-5, and find that the best exact-match performance remains extremely low: 1% on ICD-10-CM coding and 15.5% on sentencing tasks. These results show that current
benchmarks may overestimate LLM
reasoning ability and miss a key challenge: reliably following long, rule-based procedures. The complete TAM data and code are publicly available.
Read the full abstract ▾
All authors Utkarsh Soni, Syed Shariyar Murtaza, Yifan Nie, Sachin Chandrasekhar, Eugene Wen
Artificial Intelligence · Cross-listed from cs.OS
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Zihan Wang, Yuqi Wang, Lei Gong et al. (+5 more)
Mixture-of-Experts (MoE) creates a structural advantage for offloading: only a small fraction of activated experts need to reside in device
memory, and if they can be loaded in time for computation, offloading can in principle approach full-load performance, where all model weights reside in device
memory. Yet translating MoE's structural advantage into practical offloading gains remains challenging. We propose SeqMoE to bridge this gap. To maximize expert hits, we build predictive
memory management: (i) Sequence-to-sequence prediction. We are the first to recast expert activation prediction as sequence modeling, enabling accurate multi-step, multi-layer forecasts that provide a long and reliable window for downstream decisions. (ii) Joint prefetch scheduling. We formulate prefetch scheduling as Job Sequencing with Deadlines to maximize expected expert hits and improve bandwidth efficiency. (iii) Forecast-driven caching. Leveraging the recursive nature of sequence modeling, we introduce a probabilistic Belady policy for future-aware eviction. To eliminate execution bottleneck, we develop (iv) Graph-compatible offloading runtime. We derive general runtime principles encompassing compute-transparent expert placement and synchronization-free orchestration disciplines for end-to-end graph capture. With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance, advancing the state of the art in MoE offloading.
Read the full abstract ▾
All authors Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, Xuehai Zhou
Artificial Intelligence · Cross-listed from cs.CV
Generative Retrieval for Unsupervised Text-Based Person Search
Mang Ye, Yucheng Ji, Yang Bai et al. (+4 more)
Text-based person search (TBPS) aims to retrieve images of a target person from a large image gallery based on a given natural
language description. Most existing methods rely on supervised learning with manually annotated image-text pairs. In this paper, we explore unsupervised TBPS, with only unlabeled images. We propose GTR+, a two-stage generation-then-
retrieval framework. In the generation stage, we introduce a tiered description generation framework designed to produce fine-grained and stylistically diverse textual descriptions through a three-tier sequential process. The base tier leverages an automated question-and-answer mechanism to generate basic visual attribute descriptions; the intermediate tier enhances fine-grained detail using an inter-sample contrastive mechanism; the advanced tier further enriches textual diversity via a stylized expansion mechanism. In the
retrieval stage, to mitigate the impact of noisy pseudo texts, we develop an adaptive confidence-weighted
retrieval learning framework. We model image-text pairs as clean or noisy using a Gaussian Mixture Model, calibrated by real-time image-text similarity and static text generation probability from the prior stage, yielding adaptive sample weights during training. Beyond that, we also contribute LargeFine-Person, a large-scale TBPS
dataset with high-quality, fine-grained, and diverse textual annotations, enabling a practical and generalizable TBPS pre-training
benchmark under unsupervised setting. Experiments on multiple TBPS
benchmarks demonstrate the effectiveness and generalization of both GTR+ and LargeFine-Person. Code is available at: https://github.com/Flame-Chasers/GTR.
Read the full abstract ▾
All authors Mang Ye, Yucheng Ji, Yang Bai, Min Cao, Siyuan Chai, Bo Du, Min Zhang
Artificial Intelligence
EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics
Jiaxu Zhao, Bahar Radmehr, Fares Fawzi et al. (+2 more)
Large
language models (LLMs) are increasingly deployed as tutors, but it is unclear whether they support all students equally well. We introduce EduFair-Bench, a
benchmark for
auditing the pedagogical
fairness of LLM tutors—whether tutoring quality varies systematically with student demographics. EduFair-Bench pairs a multi-domain question bank (mathematics, physics, chemistry) with a controlled simulation in which a fixed LLM student interacts with each tutor across nine demographic levels spanning four dimensions: gender, immigration background, first
language, and socioeconomic status (SES). Tutoring quality is scored on five turn-level pedagogical metrics and four conversation-level dimensions, using an LLM judge validated against three-annotator consensus on 180 tutor turns. Bias is measured via paired Wilcoxon signed-rank tests and bootstrap effect-size confidence intervals. Two ablations (demographic cues conveyed through names; conflicting demographic information between tutor and student) disentangle tutor-driven from student-driven bias. Across five tutors, we find that model capability and demographic
fairness are largely orthogonal: the smallest model is the most consistent while the four more capable tutors all exhibit wide demographic gaps with no clear capability-to-
fairness ordering, pedagogy-specific RL training redistributes rather than removes bias, and
language- and immigration-related cues produce larger gaps than gender- and SES-related cues.
Read the full abstract ▾
All authors Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, Tanja Käser
Artificial Intelligence · Cross-listed from cs.RO
ARC: Autonomous Robotics Compliance A Three-Layer Governance Architecture for Deployed Autonomous Systems
Tord Eide and Einar Holt
Proposed governance framework for autonomous robotic systems, introducing a three-layer compliance architecture (ARC) instantiated through model safety validation, cognitive certification
benchmarks, and operational authorization standards.
Artificial Intelligence · Cross-listed from cs.CL
LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification
Hui Ye, Jing Zhang, Xiulong Yang et al. (+1 more)
Large-scale multi-label text classification assigns a small subset of relevant labels to each document from a vocabulary containing thousands or tens of thousands of candidate labels. Although pretrained
language models have improved semantic text representations, most representation-based approaches center their prediction pipelines on a primary encoder or combine auxiliary features within a single ranker. The complementarity between heterogeneous
language models therefore remains insufficiently explored. We propose DualMLC, a dual-branch framework that processes the same document through an autoregressive decoder-only
language model and a bidirectional encoder. Each branch maintains its own representation pathway and independently estimates relevance scores over the shared label space. DualMLC combines the two score vectors through late logit fusion, allowing shared evidence to reinforce relevant labels and branch-specific evidence to compensate for limitations in the other branch's representation. DualMLC achieves state-of-the-art results on three widely used large-scale multi-label text classification
benchmarks. Ablation results further confirm that integrating the heterogeneous predictors produces stronger rankings than either branch alone. The source code is publicly available at https://github.com/huiyegit/DualMLC.
Read the full abstract ▾
All authors Hui Ye, Jing Zhang, Xiulong Yang, Rajshekhar Sunderraman
Artificial Intelligence · Cross-listed from cs.CV
UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction
Xinqiang Yu, Zekun qi, Jiawei He et al. (+6 more)
Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized but object-aware, or part-aware but limited to closed-set taxonomies, which weakens zero-shot transfer. We study text-conditioned 3D part
segmentation, where a free-form phrase selects a functional part on point cloud. We introduce UniPart, a feed-forward cross-modal 3D
Transformer that conditions CLIP text
embedding. To scale supervision, we build LangPart-1M with 160K+ Objaverse assets and 8M text to part pairs using multi-view consistent part generation. We further manually label a high-quality subset, LangPart-4K, for fine-tuning and evaluation. UniPart achieves strong zero-shot results on open-vocabulary part
benchmarks and transfers to
language-conditioned part grasping in real world.
Read the full abstract ▾
All authors Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang
Artificial Intelligence
Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging
Pengyang Zhou, Xiaobin Tu, Zhengxi Liu et al. (+7 more)
Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and
multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and
language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the
multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their
multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in
multimodal model merging. It contains two modules, i.e.,
multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses
Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse
multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.
Read the full abstract ▾
All authors Pengyang Zhou, Xiaobin Tu, Zhengxi Liu, Rongkun Xue, Haochen Li, Miancan Liu, Ziyuan Chen, Yinggui Wang, Jinkui Ren, Xiantao Zhang
Artificial Intelligence · Cross-listed from cs.LG
Behavior Quotient Learning for Low-Rank Adaptation of LLM Agents
Pengyang Zhou, Xiaobin Tu, Zhengxi Liu et al. (+7 more)
LLM-based
agents rely on heterogeneous interaction capabilities to accomplish complex tasks. Existing approaches often distribute these capabilities across multiple LoRA adapters, which increases adapter storage requirements and introduces routing overhead during inference. A single LoRA avoids this overhead, but learning from diverse
agent trajectories under a fixed rank budget presents two challenges. First, trajectories with different interaction traces and parameter gradients can induce equivalent changes in decision distributions, causing repeated updates to overemphasize redundant behavioral changes. Second, an aggregated update may exceed the rank budget of the adapter, and approximating it in weight space can distort the decision changes that it is intended to produce. We propose BQ-LoRA, a low-rank adaptation framework that organizes trajectory updates through a local behavior quotient
manifold. It contains two modules, i.e., behavior quotient balancing (BQB) and decision preserving compression (DPC). BQB constructs the quotient
manifold from decision distributions and reweights trajectory update directions according to their local density in the quotient tangent space. DPC projects the balanced gradient onto the intrinsic fixed rank tangent space and refactorizes the resulting target by jointly controlling effective weight error and distortion of decision distributions. Experiments on AppWorld and BrowseComp-Plus compare BQ-LoRA with standard LoRA and recent low-rank adaptation methods, while separate ablations evaluate the complementary contributions of both components.
Read the full abstract ▾
All authors Pengyang Zhou, Xiaobin Tu, Zhengxi Liu, Rongkun Xue, Haochen Li, Miancan Liu, Ziyuan Chen, Yinggui Wang, Jinkui Ren, Xiantao Zhang
Artificial Intelligence · Cross-listed from cs.LG
Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting
Junhao Zhao, David Michael Simberg, Jacob Kang et al. (+2 more)
In autoregressive
forecasting, long prediction rollouts provide distant supervision, but backpropagation through time (BPTT) carries gradients from those losses through many autoregressive steps. Repeated Jacobian products can make distant gradients dominate the update while amplifying predictable signal and unpredictable innovation together; a large distant gradient therefore need not carry reliable learning signal. Motivated by this, we introduce Internal Dual-Wiener routing (Internal-DW), a backward-only intervention that preserves the full forward rollout and all step losses while reliability-weighting internal gradient routes. At each residual block, we derive bounded Wiener gains for the identity and nonlinear routes that balance preserving predictable learning signal against suppressing unpredictable variation, and estimate them from route-level gradient statistics and an explicit noise model. In a controlled system with known gradient signal-to-noise ratio (SNR), we show that distant gradients can grow even as their SNR falls, and that Internal-DW reduces error in recovering predictable gradient signals and improves
forecasting. On four history-dominated, weak-drive testbeds, Internal-DW reduces forecast error by 5.2%-13.8% relative to full BPTT, outperforms gradient clipping and Jacobian regularization on three testbeds, with similar performance on shear flow, and outperforms validation-selected truncated BPTT (TBPTT) on three. It also extends or preserves the fitted optimal training-horizon range across these four testbeds. Across
benchmarks, the current Internal-DW
estimator has a clear applicability boundary: its benefit diminishes or reverses when usable history is limited or when the selected sampler fails to represent dominant drive-dependent variation. The results show that retaining long-horizon supervision does not require trusting every backward contribution equally.
Read the full abstract ▾
All authors Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
Artificial Intelligence · Cross-listed from cs.RO
A Multi-Vehicle Dataset with Camera, LiDAR, and Radar Sensors and Scanned 3D Models for Custom Auto-Annotation using RTK-GNSS
Philipp Berthold, Bianca Forkel and Mirko Maehlisch
Datasets are a crucial element in the development of perception
algorithms. They relate sensor measurement data to annotated reference information and allow for the deduction of sensor and object characteristics. In autonomous driving, the reference data commonly consist of semantic image
segmentation, point-wise associations, or bounding box annotations. The
dataset proposed in this work, however, aims to dig deeper into the evaluation of measurement principles and provides scanned 3D models of all vehicles together with a pose and continuous kinematics reference obtained by RTK-GNSS. Combined, the state of the complete dynamic surrounding of the sensor vehicle is known for any point in time. Subsequent reference formats can be easily computed in user-defined granularity. This
dataset involves single-object and multi-object recordings with seven target vehicles. In particular, measurement effects such as occlusion, as well as reflections, can be evaluated, as the normals of the shape of the target vehicles are known. We describe the
dataset, discuss the technical background of its development, and briefly present exemplary evaluations.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
3D CT-to-PET Translation via Latent Brownian Bridge Diffusion
Sarita Mourya, Francesco Di Feola, Pierangelo Veltri et al. (+1 more)
Computed
tomography (CT) and positron emission
tomography (PET) provide complementary anatomical and functional information for cancer diagnosis and treatment planning. However, the widespread use of PET is limited by high radiation exposure, elevated costs, and restricted availability. To address these limitations, deep learning-based CT-to-PET translation has emerged as a promising approach for synthesizing PET-like information directly from CT images, although accurately modeling the large cross-modal gap remains challenging. In this work, we propose a 3D CT-to-PET translation framework based on latent Brownian Bridge
Diffusion (BBDM). The method consists of two stages. First, a Variational Autoencoder (VAE) is trained on paired CT-PET patches, integrating contrastive learning to improve latent
alignment between anatomical and metabolic representations. Second, a BBDM is trained in the latent space to translate CT latent representations into their corresponding PET counterparts. The translated PET latents are then decoded and stitched to reconstruct the final 3D PET volume. We evaluate the proposed approach on two publicly available
datasets. Quantitative results based on image fidelity and lesion-level PET-specific metrics demonstrate improved performance compared with competing methods. In particular, the proposed approach improves PET signal fidelity, better preserves clinically relevant uptake patterns, and shows improved performance in preserving small-lesion metabolic activation, paving the way for virtual imaging applications.
Read the full abstract ▾
All authors Sarita Mourya, Francesco Di Feola, Pierangelo Veltri, Paolo Soda
Artificial Intelligence
MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Youssef Mohamed, Ahmed Heakl, Qinrong Cui et al. (+13 more)
Medical
benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic
benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-
agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor
agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn
benchmarks miss.
Read the full abstract ▾
All authors Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang, Rafiq Ali, Bdour Babillie, Nazira Dunbayeva, Lang Gao, Omar Hussein, Ahmed Nada, Ahmed Mohamed Magdy Mohamed, Jinghui Liu, Salman Khan, Imran Razzak, Yuxia Wang, Xiuying Chen
Artificial Intelligence · Cross-listed from cs.CR
A Graph-Based Approach for Mapping Kernel-Level Telemetry to MITRE ATT&CK
Matteo Lupinacci, Luigi Arena, Francesco Blefari et al. (+1 more)
Mapping observed system behavior to standardized frameworks like MITRE ATT&CK is essential for threat-informed defense, but remains largely manual. Existing automated methods depend on Cyber Threat Intelligence reports, which offer only retrospective accounts of attacks. Low-level telemetry, i.e. kernel-level system calls, instead provides evidence of adversary behavior, yet its volume and complexity have limited its use for automated mapping. We present a methodology that collects kernel-level events via eBPF, correlates attacker commands into a provenance graph, and derives compact graph representations suitable for LLM-based
reasoning. These representations are mapped to the MITRE ATT&CK framework using both pure LLM prompting and
retrieval-augmented generation (RAG) grounded in the ATT&CK knowledge base, producing ranked technique candidates along with supporting rationales. We implement this methodology as an end-to-end pipeline, named Trace2ATT&CK and evaluate it on 347 Linux Atomic Red Team tests using locally deployed open-weights LLMs. RAG consistently improves ATT&CK mapping performance over pure prompting, while provenance graph substantially outperforms raw telemetry. These results show that local inference over graph-based behavioral descriptions can make automated ATT&CK mapping from kernel-level telemetry operationally viable, without compromising data confidentiality.
Read the full abstract ▾
All authors Matteo Lupinacci, Luigi Arena, Francesco Blefari, Angelo Furfaro
Artificial Intelligence · Cross-listed from cs.CR
Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Tasks
Sebastiano Nordio and Michele Lotto
The proliferation of highly capable open-weight Small
Language Models (SLMs) democratizes access to advanced cybersecurity capabilities, posing an escalating risk as these models can bypass proprietary API guardrails when deployed locally. However, SLMs deployed as autonomous
agents often struggle with long-horizon, exploratory tasks like cybersecurity Capture The Flag (CTF) challenges due to context bloat and cognitive degradation from accumulated tool-call outputs. To understand and mitigate this cybersecurity threat, we introduce context
segmentation, a two-level agentic framework that divides complex exploitation tasks into manageable, contextually isolated sub-problems. Evaluating on the picoCTF
dataset using
memory-constrained gemma-4 models, we demonstrate that for the E4B model, our strategy acts as an intelligent search, achieving competitive rewards with superior
token efficiency compared to brute-force retries, and successfully solving 18.52% of tasks that standard agentic execution fails to complete. Code is available at https://github.com/9xeb/context-
segmentation.
Read the full abstract ▾
Artificial Intelligence
Scaling Clinical Judgment to Evaluate Medical AI
Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur et al. (+15 more)
Blinded physician evaluation has been considered by many to be the gold standard for assessing clinical
reasoning in large
language models (LLMs). This is difficult to scale; thus, prior studies typically rely on small physician panels, often from a single institution or specialty, which both limits the scientific questions investigated and makes it unclear whether findings would be reproduced with a different set of evaluators. To more rigorously and scalably study clinical
reasoning in AI models, here we introduce PrecepTron, an LLM fine-tuned for physician-level evaluation of open-ended responses. PrecepTron was trained using low-rank adaptation (LoRA) of a 32-billion-parameter model on a small number of physician examples. We also release GRAND-ROUNDS, a new large-scale physician-annotated
benchmark of 9,217 scored responses from 160 clinicians across seven studies. We show that frontier LLMs in typical "LLM-as-a-judge" approaches often disagree with physicians and with each other, but fine-tuning PrecepTron on a small number of cases enables physician-level consistent scoring across tasks. We use PrecepTron to reproduce headline findings from five influential studies assessing LLMs for clinical care in JAMA, Science, and Nature Medicine without new human grading. Using PrecepTron, we then pose new questions about how LLMs reason in medicine that would have been infeasible with human grading alone, including measuring the diagnostic accuracy of frontier LLMs when clinical cases are provided piecemeal, even
token by
token. Together, PrecepTron and GRAND-ROUNDS provide a foundation for reproducible, large-scale study of how LLMs reason in medicine. All code, data, and labels are made freely available for researchers.
Read the full abstract ▾
All authors Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur, Byron Crowe, Anthony M. Pettinato, Aashna P. Shah, Adrian D. Haimovich, Liam G. McCoy, Daniel Restrepo, Jason A. Freed, Ethan Goh, Jonathan H. Chen, Laura Zwaan, Katherine E. Goodman, Daniel J. Morgan, Raja-Elie E. Abdulnour, Adam Rodman, Arjun K. Manrai
Artificial Intelligence · Cross-listed from cs.CV
Online Video Agent Harness for Long Video Understanding
Sen Yang, Boqiang Duan, Jing Yang et al. (+7 more)
Long
video understanding often behaves like a visual needle-in-a-haystack problem: query-relevant evidence is sparsely distributed across long temporal spans, while packing dense frames into a single VLM context incurs context rot and high cost. Existing
video agents often rely on query-agnostic offline preprocessing or ad hoc tool sets, which can miss query-specific details and waste computation. In this work, we present VideoXAgent, a purely online
video-
agent harness for long
video understanding that starts from the given
video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates
multimodal evidence to produce a final answer while resolving conflicts among observations. To support this on-demand invocation, we design a suite of heterogeneous expert tools guided by a data-driven taxonomy of atomic capabilities, spanning scripts, VLMs, and domain models (e.g., detection, OCR, ASR, face recognition). The harness further enforces objective evidence prompting and budget-aware control to curb
hallucination and non-termination. Across
Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, VideoXAgent is competitive with frontier LMMs and
video agents under a smaller context footprint—about 50k
tokens of
agent context per sample, even on hour-long
videos. In particular, on complex
video-
reasoning benchmarks such as MINERVA, it matches this level while using only about 15% of the context of a 1,024-frame dense-packing baseline. Notably, the harness remains effective with a visually weak or even text-only orchestrator, suggesting that strong long-
video understanding can emerge from progressive agentic evidence seeking rather than from packing the full
video into a single context. Project page: https://go-
agent-x.github.io/
video_
agent_harness/
Read the full abstract ▾
All authors Sen Yang, Boqiang Duan, Jing Yang, Weihao Bo, Jie Liu, Boyuan Tong, Ze Feng, Wenkang Zhang, Jingdong Wang, Hua Wu
Artificial Intelligence · Cross-listed from physics.ao-ph
4D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modeling
Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen et al. (+12 more)
We present BEAST, the first-ever
Bayesian Swin
Transformer for atmospheric
forecasting on 0.25$^\circ$ global resolution able to accurately quantify both aleatoric and epistemic uncertainty. To overcome the associated computational bottlenecks, we devise an orthogonal 4D-parallelization scheme that introduces a unique domain-tensor-parallelism strategy and a novel uncertainty parallel method, enabling us to fully leverage GPU capacity and efficiently scale model training. For a 2.4-billion-parameter model, we achieve a peak performance of 3.96 EFLOP/s on 20,480 NVIDIA GH200 GPUs on the JUPITER supercomputer. We train BEAST as a 700-million-parameter model with 96 random weight samples on 384 nodes on 40 years of data for nearly one million gradient updates. This model achieves predictive skill scores competitive with state-of-the-art probabilistic atmospheric AI models and numerical models, and can predict extreme events with exceptional skill, while generating large ensembles 3 to 4 times faster than the current-best AI model. Our contribution unlocks the potential of high-fidelity uncertainty quantification in atmospheric AI models, heralding a new era for AI-based models in
climate and Earth system sciences.
Read the full abstract ▾
All authors Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen, Mathis Bode, Iida Hakulinen, Andreas Herten, Chelsea Maria John, Thorsten Kurth, Anni Moisala, Asena Karolin Özdemir, Kaleb Phipps, Oskar Taubert, Arvid Weyrauch, Markus Götz, Charlotte Debus
Artificial Intelligence · Cross-listed from cs.LG
RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
Luca Herranz-Celotti and Vincent Guigue
Linear
attention and state-space models provide linear-time sequence modeling, but their recurrent
memory remains a second-order tensor (a
matrix), limiting the order of interactions that can be represented in the state. We introduce the RunningTensor, which generalizes this
memory to an order-$o$ tensor, updated by a rank-1 outer product and read by contracting against $o-1$ vector queries. Order $2$ recovers linear
attention; we study order $3$ as a proof of concept, retaining both recurrent and parallel forms while remaining linear in sequence length $T$ and improving working
memory capacity from $\mathcal{O}(W^2)$ to $\mathcal{O}(W^o)$. On synthetic multi-query associative recall, RunningTensor outperforms linear-
attention and SSM baselines. After pretraining, it also improves performance on
language-understanding and non-synthetic
retrieval tasks, suggesting that higher-order recurrent state can provide useful additional
memory capacity beyond
matrix-valued state.
Read the full abstract ▾
Artificial Intelligence
K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments
Guangsheng Yu, Yanna Jiang, Qin Wang et al. (+2 more)
Unlearning
benchmarks such as TOFU and MUSE certify forgetting by reading the model's final answer, where a model that refuses to answer already counts as having forgotten. We show that this model-level certificate does not transfer once the model is deployed as an
agent. We introduce K-Bench, a
benchmark that scores LLM unlearning under agentic deployment. K-Bench inspects all six channels a ReAct
agent exposes, including its chain-of-thought (CoT), tool calls and tool observations, and elicited summary. A query counts as leaked if the secret appears in any of them. Each experiment places the secret in exactly one of the
agent's three sources (the weights, the
prompt, or the
retrieval store). The K-Score is computed separately for each source and credits forgetting only when the
agent remains usable. Clearing the answer channel does not make the secret unrecoverable. On structured
retrieval, the secret stays verbatim in the tool-observation channel and the aggregate leak rate is unchanged. When the secret lives in the
prompt or the
retrieval store, TOFU and MUSE report no leakage, while the deployed
agent still leaks it on 22–86% of queries. When the secret is in the weights, none of the twenty evaluated published methods demonstrably removes it, and only an input-corruption intervention reaches selective forgetting under the evaluated observer. The top-ranked method changes across base models. A refusal-tuning method resists the evaluated extraction without verified knowledge removal.
Read the full abstract ▾
All authors Guangsheng Yu, Yanna Jiang, Qin Wang, Baihe Ma, Xu Wang
Artificial Intelligence
Interpreting the predictions of neural network classification based on a Taylor Coefficient Analysis (TCA)
Markus Klute, Artur Monsch, Lars Sowa et al. (+1 more)
We introduce a rigid and comprehensive taxonomy and paradigm for characterizing the influence of the input feature space $X$ on the predictions $\hat{y}$ of a neural network (NN) used for event classification, based on a Taylor expansion of $\hat{y}$ in $X$. The complete process of introspection we refer to as Taylor Coefficient Analysis (TCA). Based on two simplistic example tasks, which can be easily understood and bencmarked, we illustrate the power of the TCA when it comes to revealing, what properties of $X$ have led to what value of $\hat{y}$, of a given NN model, building up intuition for the method. A more complex application is meant to represent $X$ of a typical classification task at a CERN LHC experiment. Based on this application, we play through the different levels of introspection that the TCA offers and discuss a number of practical aspects for a TCA application typical for the analysis of CERN LHC data. We conclude with a study to support the assumption that those properties of $X$ most relevant for tasks of the complexity typical for a CERN LHC experiment, are usually caught by a TCA up to the second order.
Read the full abstract ▾
All authors Markus Klute, Artur Monsch, Lars Sowa, Roger Wolf
Artificial Intelligence · Cross-listed from cs.CL
Cognition on Graph: Navigating Massive Knowledge Space via Cognitive Cycles and Bidirectional Graph-Text Synergy
Gengxian Zhou, Jian Xu, Zichen Tang et al. (+3 more)
Retrieval-Augmented Generation (RAG) has empowered Large
Language Models (LLMs) to tackle knowledge-intensive tasks. However, navigating global, heterogeneous knowledge bases (large-scale knowledge graphs and text corpora) for complex
reasoning remains a challenge. Existing methods typically employ reactive, graph-driven exploration strategies, which blindly follow graph
topology without adapting to the question context or evolving exploration progress, and lack deep bidirectional synergy between graph and text. To address these limitations, we propose CoG (Cognition on Graph), a cognitive-inspired, training-free framework for adaptive knowledge exploration. Drawing inspiration from human problem-solving, CoG performs a continuous plan-explore-reflect cycle, where it proactively formulates investigation plans, performs dual-source
retrieval, and dynamically reflects on progress to adjust strategies. Crucially, it establishes deep bidirectional synergy between structured graph and unstructured text, where entities extracted from text dynamically guide graph exploration to bridge knowledge gaps. Extensive experiments on seven multi-hop QA
benchmarks demonstrate that CoG significantly outperforms state-of-the-art methods while achieving superior exploration efficiency. Our code and
datasets are available at https://github.com/zhougengxian/CoG.
Read the full abstract ▾
All authors Gengxian Zhou, Jian Xu, Zichen Tang, Shiming Xiang, Haihong E, Cheng-Lin Liu
Artificial Intelligence
MPT: Missing Prototype Tracking via Barycentric Reconstruction in Vehicular Federated Learning
Hanju Jang, Gyeongmin Han, Sungmin Lee et al. (+3 more)
Cross-vehicle
federated learning enables vehicles to collaboratively improve perception models while keeping locally collected driving data private. However, vehicle participation is transient, and a vehicle may depart before training converges while permanently taking its local data. When this departing vehicle holds most samples of a target class, the class becomes rare in the remaining FL network, and its recognition can silently degrade as the shared backbone continues to evolve. Recovering the class is difficult since the few remaining samples provide a noisy prototype estimate, while FL
privacy constraints prevent centralized access to raw data or per-sample features. This paper presents MPT, a cross-vehicle FL framework that maintains rare-class recognition by reconstructing its prototype at every round from
privacy-preserving class-level statistics. MPT combines a barycentric decomposition that tracks drift shared with remaining-class prototypes, a covariance-based residual prediction that estimates out-of-span drift, and an adaptive calibration that weighs the remaining rare-class samples according to their reliability. We evaluate MPT on three vehicle classification tasks and four backbones against representative calibration and drift-compensation baselines. MPT outperforms all baselines in rare class F1, reaching 0.516 on the nuImages
dataset with only 1% of rare-class samples remaining, without raw data, per-sample features, or retraining.
Read the full abstract ▾
All authors Hanju Jang, Gyeongmin Han, Sungmin Lee, Kichang Lee, Chunghan Lee, JeongGil Ko
Artificial Intelligence
Unified Agentic Video Editing Across Levels of Complexity and Creativity
Surabhi S. Nath, Kim Ferres, Milan Petrović et al. (+1 more)
Editing is a core component of
video production, requiring creative planning and decisions under multiple constraints. Here, we report methods for agentic tooling for automated
video editing across three tasks varying in editorial goal, complexity and creativity, namely scene previews,
video summaries and cinematic trailers. We evaluate the outputs and discuss implications for automation and agency.
Read the full abstract ▾
All authors Surabhi S. Nath, Kim Ferres, Milan Petrović, Lion Schulz
Artificial Intelligence · Cross-listed from cs.SE
GraphAHA: Graph-Based Adaptive Search with Heterogeneous Actions for Test-Time Code Generation
Xitao Li, Haijun Wang, Gege Yuan et al. (+4 more)
Test-time scaling improves code generation by spending additional inference budget (e.g., calls or
tokens) on direct sampling, feedback-conditioned repair, and
reasoning-guided implementation. Search-based methods can allocate this budget adaptively, but two challenges remain. First, tree-structured search treats each generation history as a separate state even when trajectories converge to the same program, duplicating evaluation and preventing statistics from being shared. Second, sampling, repair, and
reasoning have complementary and state-dependent payoffs, making online allocation among them difficult under a finite budget. To address these challenges, we propose an adaptive graph search method with heterogeneous actions (GraphAHA). GraphAHA organizes the test-time code generation in a typed directed acyclic graph. Equivalent programs are merged into a single code node, allowing their downstream search statistics to be reused across all discovery paths. Hierarchical Thompson sampling then selects whether to generate a new state or follow an existing successor and, for generation, chooses among the type-valid sampling,
reasoning, implementation, and repair operations. Evaluated on LiveCodeBench and CodeContests with Qwen2.5-Coder and DeepSeek-Coder, GraphAHA achieves the best score in 18 of 20 cases. For Pass@1 measured using visible tests, it outperforms the strongest baseline for both models on both
benchmarks by 4.1 percentage points on average, demonstrating more effective use of a fixed inference budget.
Read the full abstract ▾
All authors Xitao Li, Haijun Wang, Gege Yuan, Qiyuan Wu, Jiali Wei, Ming Fan, Xiaofei Xie
Artificial Intelligence
SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy
Xiefeng Wu, Shu Zhang, Zhaojie Chu et al. (+1 more)
Offline-to-online reinforcement learning reduces interaction cost for real-world
robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the
entropy formulation: the standard log-
entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari)
benchmarks under both single-demonstration and standard
dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual
benchmarks, and transfers to four real-
robot platforms including manipulation, wheeled, quadruped, and
humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: https://scq-rl.github.io.
Read the full abstract ▾
All authors Xiefeng Wu, Shu Zhang, Zhaojie Chu, Mingyu Hu
Artificial Intelligence
Assisted Spatial Cognition Through Vision-Language Models
H. Riaz, J. B. Fernandez, I. Mills et al. (+3 more)
Multimodal AI, powered by Large
Language Models (LLMs) and
Vision-
Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide who face persistent challenges in navigating indoor and outdoor environments due to limited spatial awareness and insufficient environmental cues. Existing
navigation aids often lack comprehensive 3D scene understanding, relying on constrained route-based strategies that hinder user autonomy. In this paper, we introduce a novel end-to-end framework that integrates LLMs, VLMs and digital twin technologies to deliver a spatially cognitive
navigation support for visually impaired and neuro-divergent users. Our system captures
video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds from
monocular RGB sequences in real-time. Our custom post-processing
algorithm ensures accurate point cloud
alignment across multiple viewpoints without requiring predefined reference points. This enhances the capabilities of SpatialLM to produce structured 3D representations, including architectural elements and oriented object bounding boxes. The enriched spatial data is then processed by a locally deployed LLM, which interprets 3D contexts to generate detailed scene descriptions and precise distance measurements between users and surrounding objects. We evaluated our approach across diverse
video scenarios featuring various perspectives, looped walking views and captured in multiple environments. The evaluation results demonstrate consistent accuracy in 3D scene interpretation and object localisation, underscoring the potential of our system as a transformative assistive
navigation solution that combines advanced visual perception with spatial
reasoning
Read the full abstract ▾
All authors H. Riaz, J. B. Fernandez, I. Mills, D. Hickey, F. Cleary, M. I. Ali
Artificial Intelligence
What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework
Ioannis Prokopiou, Athanasios Aidinis, Panagiotis-Christos Kyrmpatsos et al. (+1 more)
Agentic pipelines for structured-query generation are rapidly expanding, but it is unclear which part of the loop produces the gain. We use LAST-CQ – a five-
agent, training-free, execution-grounded Text-to-Cypher framework – as an instrumented testbed, running three counterfactuals over 2,471 live-database queries and six backbones spanning three vendor scale tiers. Removing correction is worth between 3.1% aggregate execution-BLEU against the single-pass system and 12.3% against a no-refinement counterfactual (up to 80.7% for the weakest backbone). Replacing schema-grounded, LLM-synthesised feedback with raw database error strings costs almost nothing (20.9% vs. 19.9% naive exact match; <0.2% end-to-end; equivalent within $\pm 0.075$ set-F1 by two one-sided tests). Spending the same call budget on parallel sampling degrades quality by 10-11%. What works is detecting failure and routing it to a retry, not the feedback sophistication or number of samples. LAST-CQ itself recovers 91.7% of queries that fail under single-pass generation, while a query that succeeds first time still costs exactly one LLM call. We also show that n-gram overlap on serialised results is not a bound in either direction: it over-scores against set equivalence on 65.9% of results while under-scoring against judged semantics. Finally, we calibrate our LLM judge against blind human labels and find it optimistic by 9 points.
Read the full abstract ▾
All authors Ioannis Prokopiou, Athanasios Aidinis, Panagiotis-Christos Kyrmpatsos, Pantelis Vikatos
Artificial Intelligence
Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
Mykhailo Kozyrev, Andrei Kozyrev and Anton Podkopaev
Coding
agents increasingly read repository knowledge from SKILLs — plain .md files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a
benchmark. A bare repository comes with no
benchmark, and the synthetic tasks prior work builds are small enough that a capable
agent saturates them with no document at all. We mine harder tasks — merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same
agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by $4.9$pp on average, and the ones SkillOpt finds leave it where it started, $0.1$pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the
dataset size a single repository supplies it cannot be separated from the
agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.
Read the full abstract ▾