Artificial Intelligence · Cross-listed from cs.RO
DexFLEX: Contact-Aware Foundation Controller for Command-Guided Dexterity
Weikang Wan, Zeyu Lai, Jianyu Hou et al. (+4 more)
Dexterous
robot hands can receive useful motion intent from teleoperation or learned policies, but successful execution also depends on contact decisions that these commands rarely specify. We introduce DexFLEX, a contact-aware foundation controller that turns upstream fingertip-motion drafts into contact-consistent joint commands. Instead of treating a draft as a trajectory to copy, DexFLEX treats it as evidence about intent: from the current tactile-proprioceptive state, it proposes feasible short-horizon motion chunks, predicts their contact consequences, and selects the candidate that best follows the command while preserving future contact stability. At inference time, a simple Draft-Dream-Select procedure combines pure-prior proposals for recovery with draft-seeded proposals for responsiveness, then decodes the selected motion into executable joint-space targets. Across degraded-command simulation, real-world shared control, and visuomotor policy learning, the same trained controller improves robustness without retraining, increasing real-world teleoperation success from 29.2% to 78.3% and reaching 46.7% policy-learning success, 3.5 times direct joint-action prediction, and 1.56 times prior-only correction.
Videos are available at dex-flex.github.io.
Read the full abstract ▾
All authors Weikang Wan, Zeyu Lai, Jianyu Hou, Yuchen Zhou, Jiawei Fu, Henrik I Christensen, Hao Su
Artificial Intelligence
When Pleasing the User Compromises Care: AI Sycophancy in Caregiver-Mediated Healthcare
Priyadarsi Mishra, Arjun Beniwal, Ziyu Yi et al. (+2 more)
Large
language model (LLM)
alignment often assumes that the user interacting with a model is also the primary beneficiary of its behavior. Yet in many consequential settings, one person directs the system while another bears the consequences. We study this problem in caregiver-mediated healthcare, where parents, pet owners, and caregivers make decisions for children, animals, and older adults. We introduce an evidence-grounded
benchmark spanning veterinary, pediatric, and older-adult care, and test whether AI advisors maintain appropriate care when users express clinically inferior preferences. Holding patient facts fixed, we vary stated care preferences and communication styles and evaluate both initial recommendations and multi-turn follow-up. Across frontier and open-weight models, inferior preferences substantially increase switching from appropriate to inappropriate care beyond neutral conversational drift and increase agreement with the requested action, while communication style alone has a smaller effect. These failures accumulate across turns, including on cases models initially answer correctly, and often carry moderate or severe potential harm. In matched eldercare cases, the effect of an inferior preference is similar whether it is expressed by the patient or by a caregiver. Together with models' limited tendency to elicit patient preferences in caregiver conversations, this suggests that current models are highly sensitive to expressed user preferences but less sensitive to whose preferences are represented and who bears the consequences. Our results expose a broader limitation of user-centric
alignment in settings with multiple stakeholders.
Read the full abstract ▾
All authors Priyadarsi Mishra, Arjun Beniwal, Ziyu Yi, Yu Ding, Sheng Liu
Artificial Intelligence
Actionable Meta-Requirements for AI Robustness Auditing
Joshua Owotogbe, Roya Nasiri, Indika Kumara et al. (+2 more)
AI-enabled systems must remain dependable when the conditions under which they operate change (e.g., due to input perturbations, distribution shifts, or changes in external dependencies). This makes robustness auditing of such systems important for judging whether the available evidence is sufficient to support a defined robustness conclusion. However, designing an AI robustness auditing framework is challenging. A key unresolved issue is determining the requirements such a framework should support; these requirements have not yet been consolidated. We address this gap through meta-requirements: higher-order requirements specifying what a class of auditing frameworks should support. In design science research, literature and domain experts are common sources of meta-requirements; accordingly, we derive the initial set from literature and validate and refine it with practitioners. Our systematic literature review retained 74 of 2,890 records and derived 43 meta-requirements across 16 categories. We evaluated them with 128 practitioners, obtaining 5,504 ratings and 586 substantive comments. Overall, 81.20% of ratings indicated agreement or strong agreement. In particular, 41 of the 43 literature-derived meta-requirements were classified as strongly supported. Practitioner feedback refined the requirements and identified an additional requirement concerning external components and supply chain dependencies. The final set comprises 44 meta-requirements across 16 categories, of which 43 received quantitative validation. These meta-requirements can help organizations implement AI robustness auditing frameworks, assess existing frameworks for completeness, and refine them.
Read the full abstract ▾
All authors Joshua Owotogbe, Roya Nasiri, Indika Kumara, Damian Andrew Tamburri, Willem-Jan van den Heuvel
Artificial Intelligence · Cross-listed from cs.CR
Natural Code Transformation Confuses Agents in Vulnerability Analysis
Jie Zhu, Ronik Bhaskar, Heather Zheng et al. (+2 more)
Having demonstrated a strong potential for automating vulnerability discovery, AI
agents can also be used to attack software systems at scale. Today's software systems need mechanisms to level the playing field by slowing down potential attacks, giving them time to patch vulnerabilities and strengthen software. Along this line, our study of
agent trajectories in vulnerability analysis shows a surprising finding:
agents produce dramatically different results across real-world code commits that carry the same functionality and the same vulnerability. This phenomenon appears consistently in multiple code domains and vulnerability categories. It motivates us to study natural and functionality-preserving code transformations derived from real-world software development, and their potential as a defense against attacker
agents. We build a corpus of natural transformations by mining recurring development patterns from real-world code commits, and apply them to evaluate impact on agentic vulnerability analysis. Our results show that even a single form of transformation can effectively "confuse" frontier
agents, e.g., Codex with GPT-6 Sol, reducing successes in vulnerability discovery. We also study different
agent failures, whether transformations can be composed to provide stronger protection, and adaptive attackers who seek to reverse them. Overall, our study of natural code transformation significantly expands the research space of using code transformations to defend against malicious
agents.
Read the full abstract ▾
All authors Jie Zhu, Ronik Bhaskar, Heather Zheng, Ben Y. Zhao, Kexin Pei
Artificial Intelligence
Which Metadata Matters for LLM Tool Selection in MCP? Evidence from Controlled Factor-Level Interventions
Xiaofan Li, Fengchun Qiao and Xing Gao
In Model Context Protocol (MCP) systems, LLMs select tools from metadata rather than from their implementations. Names and descriptions summarize what a tool does, while the parameter schema specifies the fields a tool accepts and how they are organized. Our analysis of metadata for 367,174 MCP tools reveals substantial schema variation among functionally similar tools. Many of these tools share names and similar descriptions but expose different parameter schemas. This raises a key question: can parameter-schema differences influence tool selection before any call is made? We study this question through controlled metadata interventions across five LLMs and five tasks. Each comparison changes exactly one metadata factor (server name, tool name, description, or parameter schema) while keeping the others fixed. We balance tool positions and use conditional logistic regression to estimate position-adjusted effects. We find that: (1) parameter-schema changes produce the largest and most consistent selection shifts among the tested interventions. Models tend to favor separate top-level parameters over grouped or composite forms. (2) Descriptions that explain parameters also tend to increase selection, while name effects are weaker and less consistent. (3) Targeted comparisons show that schema effects depend on the specific parameter names and on how the request represents the same information. When only one schema satisfies an explicit request requirement, all five models select the compatible tool despite position preferences. These results show that parameter schemas influence not only how tools are called, but also whether they are selected. For MCP developers and evaluators, schema design and tool ordering can therefore affect which implementation is invoked even when functionality is unchanged.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
TraceSampler: Making Sampling Errors Predictable and Correctable for Accelerated Generative Sampling
George Matthew, Guoqiang Zhang and Kenta Niwa
Accurate sampling from continuous-time generative models can require many model evaluations. Low-step sampling reduces cost, but coarse updates introduce endpoint errors that can accumulate and degrade generation quality. We introduce TraceSampler, a post-training method that corrects these errors without modifying the pretrained model. Rather than optimising low-step updates directly, TraceSampler models endpoint error as a sample-specific prediction problem. A higher-step reference integration over the same interval provides the supervised target, while a compact representation makes high-dimensional error predictable from the sampling trace. This representation combines shared residual structure, trace-derived directions, and a low-rank remainder, so shallow per-step MLP heads predict sample-dependent coefficients. These coefficients drive a two-stage learned residual correction: the first stage predicts the full endpoint residual, while an intermediate backbone evaluation provides new trace information to predict the remaining residual; this evaluation is reused by the following step. TraceSampler is evaluated with Rectified Flow on CIFAR-10, SiT-XL/2 on ImageNet-256, and SD3.5 Medium for text-to-image generation. At 10 steps, it reduces trajectory MSE relative to matched-step Euler by 90.5% on CIFAR-10 and 85.6% on ImageNet-256, while achieving FID scores of 2.692 and 2.189, close to 120-step Heun references of 2.601 and 2.056 with 24$\times$ fewer backbone evaluations. On SD3.5 Medium, TraceSampler improves image quality and
prompt following; at 10 steps, ImageReward rises from 0.752 to 0.906 while trajectory MSE falls by 70.7%. These gains preserve Euler's backbone-evaluation budget, add less than 0.4% inference FLOPs, and incur negligible wall-clock overhead, showing that coarse-step sampling error can be made predictable from the sampling trace and efficiently corrected.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference
Anxhelo Diko, Jieyu Lin, Stefano Ambrogio et al. (+5 more)
Foundation models like V-JEPA are increasingly deployed across a heterogeneous edge-cloud continuum spanning smart glasses, phones,
robots, and servers. Yet, they are typically trained as rigid, fixed-capacity networks bound to one device, supporting neither reconfigurable compute budgets nor collaborative execution. Existing approaches address only one axis at a time: slimmable and NAS-based methods provide width elasticity but assume single-device inference, whereas split-computing methods enable cross-device collaboration only for fixed partitions and static capacities. We introduce Elastic V-JEPA (EV-JEPA), a single model trained once and deployable everywhere along the continuum. Its width is reconfigurable, and it is natively partitionable across heterogeneous devices. At its core is Transition-Aware Triplet Training (T3). At each step we pick a narrow and a wide width, and train three paths: each width alone, plus a hybrid path that runs the early layers narrow and the rest wide. The hybrid switch point is where one device hands off to another, so the same weights learn both single-device and cross-device execution. Across various
benchmarks, EV-JEPA falls within 1 accuracy point of independently trained baselines while improving over previous elastic training approaches by up to 11 accuracy points. Narrow-to-wide cross-device hybrid execution recovers to within 1.1 points of the highest-capacity standalone model and supports many-to-one, any-to-any, and variable-split configurations. Finally, we show that the elastic principle extends beyond the backbone to an elastic
token compression mechanism that exposes controllable trade-offs among on-device computation, latency, and accuracy.
Read the full abstract ▾
All authors Anxhelo Diko, Jieyu Lin, Stefano Ambrogio, Ziyun Li, Mahmoud Assran, Théo Moutakanni, Barbara De Salvo, Chiao Liu
Artificial Intelligence · Cross-listed from cs.CV
MvFFN: Multi-view Floor plan Feed-forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction
Yuguang Li, Yichuan Deng, Zixuan Liu et al. (+5 more)
Reconstructing accurate camera poses and floor-plan layouts from sparse, unposed, wide-baseline RGB panoramas remains a challenging open problem. Direct multi-room floor-plan prediction is especially difficult in this setting: pose, geometry, semantics, and global layout structure are tightly coupled, yet a complete floor-plan is hard to learn as a single global output. Our key insight is to keep the model's predictions strictly local. Multi-view Ordered Wall Instance Segmentation (MOIS) is designed around this concept. Each panorama column predicts only the wall it observes plus the next two adjacent walls in clockwise order. The global wall partition, room layouts, and shared coordinates emerge from deterministic aggregation of overlapping local predictions. Building on MOIS, we propose Multi-view floor-plan Feed-Forward Network (MvFFN), which jointly predicts coarse camera poses, dense global geometry, and per-wall semantic outputs. A downstream Layout Chaining (LC) post processing pipeline then assembles these predictions into multi-room floor-plans with shared global coordinates, consistent wall identities, and connected room structure. The dense MOIS predictions recover cross-view wall identity and local within-room connectivity more accurately than composed modular baselines, and the assembled layouts substantially outperform prior methods on wall and junction level accuracy. We also introduce the ZInD-CrossView
dataset, which augments ZInD with globally unique wall instances and cross-view segmentation labels for this task.
Read the full abstract ▾
All authors Yuguang Li, Yichuan Deng, Zixuan Liu, Ivaylo Boyadzhiev, Xiuyuan Wu, Xinyu Zou, Linda Shapiro, Alex Colburn
Artificial Intelligence
ConicFlow: constrained Inference-time optimal control for flow matching models
Shervin Khalafi, Kasra Fallah, Sung Cho et al. (+3 more)
We propose ConicFlow, which frames constrained sampling from a flow matching model at inference time as an optimal control problem that balances deviation from the pretrained trajectory against reward, subject to hard terminal constraints. We discretize this problem and adapt receding-horizon control, optimizing only a short window of the trajectory at each step for computational tractability. We address the resulting sequence of receding-horizon subproblems with a receding-horizon primal-dual
algorithm. Its dual multipliers capture the difficulty of satisfying each constraint and are refined throughout sampling, steering generation toward feasible final samples. We further propose a terminal-estimate variant that avoids accumulating local truncation errors by evaluating the reward and constraints at the predicted terminal point rather than at the end of the current horizon. Under standard regularity assumptions, we establish finite-horizon guarantees on suboptimality. Under convex-affine structure, we derive a sharper bound that scales linearly with omitted horizon length, and we show that the terminal-estimate variant is provably tighter when the flow dynamics change slowly. Experiments in both low- and high-dimensional settings with a recent large-scale flow matching model show that ConicFlow achieves high reward and strong constraint satisfaction while deviating less from the pretrained flow than recent baselines.
Read the full abstract ▾
All authors Shervin Khalafi, Kasra Fallah, Sung Cho, Dongsheng Ding, James Anderson, Alejandro Ribeiro
Artificial Intelligence · Cross-listed from eess.AS
RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can
Yi-Jen Shih, Puyuan Peng, Haohe Liu et al. (+2 more)
Recent
Speech Full-Duplex (FDX) models are capable of engaging in naturalistic conversation by listening and speaking at the same time. Building on this foundation, recent architectures introduce internal thinking streams to enhance their intelligence. To guarantee these streams process successfully in real time, existing methods enforce a fixed decoding rate, which assumes a hardware profile capable of a specific latency. We argue this static approach is severely suboptimal. Because internal thinking is intrinsically asynchronous, an ideal FDX model should sustain real-time conversational flow while allowing each stream to decode at its own hardware-adaptive rate. Thinking streams must process as fast as the hardware allows to deliver accurate answers faster. To solve this, we propose RUBATO, a novel multistream positional encoding that assigns independent clocks to entirely decouple the streams. This structural isolation empowers the model to zero-shot interpolate across any inference decoding speed. It drastically simplifies
dataset construction by requiring training on only two extreme decoding speeds while smoothly adapting to any intermediate rate during deployment. RUBATO demonstrates consistent robustness across five spoken
reasoning tasks. On Spoken GSM8K, a baseline model collapses to 1.7% accuracy at interpolated speeds. Conversely, RUBATO-Full sustains 48.7% to 51.1% accuracy across all rates, leveraging faster hardware to achieve a 6.5x shorter time-to-answer with a constant time-to-first-response. This advantage holds on BIG-Bench Audio and amplifies significantly during complex full-duplex interruptions. Specifically, when a user interjects while the model is actively speaking and
reasoning, RUBATO-Full achieves 40.0% to 45.1% accuracy, whereas the baseline fails completely.
Read the full abstract ▾
All authors Yi-Jen Shih, Puyuan Peng, Haohe Liu, David Harwath, Abdelrahman Mohamed
Artificial Intelligence
Controllable Diverse Sampling through Pullback Directions
Sylvain Dehayem Kenfouo, Aaron Zweig, Mingxuan Zhang et al. (+1 more)
Conditional
diffusion models can generate high-quality images that align closely with a given
prompt. However, samples generated from different random seeds may remain visually similar, limiting creative exploration. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or
prompt alignment. We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-
embedding space to the image space to identify directions of maximal output sensitivity. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal. Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on
prompt-conditioned image generation and inpainting.
Read the full abstract ▾
All authors Sylvain Dehayem Kenfouo, Aaron Zweig, Mingxuan Zhang, Elham Azizi
Artificial Intelligence · Cross-listed from cs.CV
GeoFix: Geometry Fixing for 4D Consistent Generation
Yeobin Hong, Hyojun Go, Dohun Lee et al. (+7 more)
Video diffusion models increasingly serve as priors for 4D generation, but the generated
videos often lack
geometric consistency. Recent approaches address this through post-training by designing
geometric rewards, yet this can degrade texture and fine detail, requiring auxiliary appearance rewards and careful reward balancing. To address this challenge, we propose GeoFix, a unified framework that improves
geometric consistency and visual quality with a single
geometric reward, applicable to both training and inference-time reward
alignment. Our reward measures cross-view depth consistency in static scene regions, targeting structural inconsistencies without directly constraining appearance or penalizing valid object motion. Crucially, depth agreement increases with camera field-of-view overlap even in real
videos and 4D
datasets, allowing a smaller camera movement to be mistaken for better geometry. We account for this effect by calibrating the reward against the overlap-dependent trend. GeoFix achieves a 39% improvement in WorldScore 3D consistency and a 4% increase in VBench Quality Score.
Read the full abstract ▾
All authors Yeobin Hong, Hyojun Go, Dohun Lee, Jeongjae Lee, Sunghwan Hong, Federico Tombari, Zhaochong An, Jong Chul Ye, Konrad Schindler, Dominik Narnhofer
Artificial Intelligence · Cross-listed from cs.LG
The RNN Homunculus: Learning to Gate Long-Term Memory
Zeyu Fan, Alexandru-Dragos Manolache and Jan van Gemert
Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range
memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range
memory and show how training data can support this process by exposing the same task over shorter dependencies.
Read the full abstract ▾
Artificial Intelligence
Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference
Xingyu Xie, Xiaoyong Xie, Zhouchen Lin et al. (+1 more)
Tensor parallelism ends every layer with an all-reduce that sits between two dependent
matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth. Sending 4-bit instead of 16-bit values would cut this cost by nearly 4x, but the all-reduce must add the values it carries, and rounding the sum injects an error at every call. Model quality depends on how these errors accumulate over layers, yet existing methods treat each call on its own. We present Echo, a 4-bit all-reduce for training and inference that controls the accumulated error. Echo moves data in 4 bits and sums in high precision, so each value is quantized at most twice per call for any number of GPUs. The summation reveals each injected error, and Echo feeds it back into later calls, so old errors cancel instead of accumulating. We prove that the accumulated error stays bounded independently of depth. On A100 GPUs, Echo is 2.09x faster than the 16-bit NCCL all-reduce on 1 GB messages. It speeds up the time to first
token by up to 1.16x and the training step by up to 1.09x on an idle interconnect, and by up to 1.44x and 1.38x as background traffic grows. It keeps the average accuracy over seven
benchmarks within 0.4 points of 16-bit inference on Qwen3-8B. In 10B-
token pretraining, its final loss is less than 0.9% above that of 16-bit training. Echo needs no FP8 or FP4 hardware.
Read the full abstract ▾
All authors Xingyu Xie, Xiaoyong Xie, Zhouchen Lin, Kim-Chuan Toh
Artificial Intelligence · Cross-listed from cs.RO
Akule: Fast and Scalable Multi-Robot Motion Planning via Sparse Interaction Diffusion
Manav Vora, Gokul Puthumanaillam, Aditya Penumarti et al. (+2 more)
Diffusion models provide expressive trajectory priors for multi-
robot motion planning (MRMP), but scaling them to large teams requires coordinating individual motions without expensive joint computation. We introduce Akule, a sparse interaction
diffusion framework that augments a frozen single-
robot prior with three learned modules: a reusable pairwise correction model $R_{\phi}$, a contextual selector $U_{\psi}$ that determines which corrections to evaluate, and a composer $G_{\omega}$ that combines their effects through context-dependent signed weights. This decomposition separates which interactions merit computation from how their effects should be combined into coordinated denoising updates. We train $R_{\phi}$ using matched pairwise trajectory optimizations, then learn $G_{\omega}$ and $U_{\psi}$ from physically optimized joint trajectories without requiring a prescribed ground-truth interaction graph. The resulting joint proposals are passed to a downstream search-based planner for validation and repair. Across diverse MRMP
benchmarks, Akule achieves competitive planning success, complete-planner speedups of up to $25\times$, zero-shot population transfer, and successful planning with up to $256$
robots. Ablations further demonstrate that learned selection and composition reduce pre-repair conflicts and downstream repair effort relative to heuristic selection and uniform summation.
Read the full abstract ▾
All authors Manav Vora, Gokul Puthumanaillam, Aditya Penumarti, Jane Shin, Melkior Ornik
Artificial Intelligence · Cross-listed from cs.CL
Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study
Ruomeng Ding, Qianli Shen, Zhaoyang Han et al. (+3 more)
LLM data engineering requires iterative experimentation to determine how data selection, rewards, and curricula affect model learning, making it a natural setting for AI-for-AI (AI4AI) research. Can
agents use experimental feedback to progressively improve strategies, and do the resulting strategies improve target-model performance on held-out data? We study how
agents use experimental feedback to develop data-engineering strategies under fixed training and evaluation protocols. Our Agentic Data Engineering (ADE) framework lets
agents pursue, revise, or combine research directions using cumulative findings. Strategies are selected by in-loop scores and evaluated on held-out
benchmarks. Across four tasks-math and code data selection, math reward design, and math curriculum learning-selected strategies achieve absolute gains over generic baselines of up to 13.34% on AIME25 Pass@32 and 5.85% averaged across four held-out
benchmarks under fixed training conditions. In-loop scores improve over successive experiments, while held-out trajectories show that later in-loop gains do not always improve generalization. Comparisons of research organization and feedback access favor combining evidence analysis with cumulative findings. Research trajectories show how earlier findings inform strategy revisions and combinations across research directions.
Read the full abstract ▾
All authors Ruomeng Ding, Qianli Shen, Zhaoyang Han, Meixin Chen, Daoyuan Chen, Yaliang Li
Artificial Intelligence · Cross-listed from cs.CV
Latent World Action Model for GUI Agents
Seungjun Yu, Hojun Choi, Seojeong Park et al. (+2 more)
Pixel-space world models have recently emerged as a promising approach for GUI
agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large
multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.
Read the full abstract ▾
All authors Seungjun Yu, Hojun Choi, Seojeong Park, Jaeyo Shin, Hyunjung Shim
Artificial Intelligence · Cross-listed from cs.CV
CAPTOR: a vision-language foundation model for Chest-Abdomen-Pelvis Oncologic CT
Séléna Jean-Mactoux, Sophie Loizillon, Ghina Jardali et al. (+3 more)
Purpose : To develop and evaluate CAPTOR, a deep-learning
vision-
language foundation model for oncologic chest-abdomen-pelvis (CAP) CT interpretation. Materials & Methods : CAPTOR (Chest-Abdomen-Pelvis foundaTion model for Oncologic Radiology) is a contrastive
vision-
language model, trained using a pretraining strategy for oncologic CAP CT (in single- or multiple-series acquisitions), on a multicenter retrospective
dataset from North America, Europe and Asia, predominantly from tumor lesion examinations. CAPTOR was compared with state-of-the-art CT foundation models on three downstream tasks: disease detection, survival prediction using TumorImagingBench and image
retrieval. Classification and
retrieval were both evaluated on held-out internal and public test sets, stratified by anatomical coverage and indication. Bootstrap resampling was used to estimate confidence intervals, and pairwise comparisons were performed using permutation tests. Results : CAPTOR was trained and tested on 25,512 and 10,794 CT-report pairs respectively. TumorImagingBench
datasets comprised between 134 and 421 studies each. CAPTOR outperformed all baseline models, on the classification task, on the CAP subset (AUC 0.767, 95%CI: 0.734-0.793, vs. 0.695, 95%CI: 0.645-0.736 for the next-best, p<0.001) and oncologic subset (AUC 0.784, 95%CI: 0.759-0.805, vs. 0.734, 95%CI: 0.703-0.767 p<0.001), ranked first on three of the four oncologic prognostic
benchmarks, and achieved superior
retrieval performance on CAP and oncologic subsets. Conclusion : A foundation model trained on oncology-enriched CAP CT data outperformed relevant state-of-the-art models in its target clinical context, underscoring the importance of clinically aligned pretraining. We release our preprocessing, training and inference code : https://github.com/Guerbet-AI/CAPTOR and model weights : https://huggingface.co/Guerbet-AI/CAPTOR
Read the full abstract ▾
All authors Séléna Jean-Mactoux, Sophie Loizillon, Ghina Jardali, Amandine Crombé, Nathalie Lassau, Alexandre Bône
Artificial Intelligence
Harness^2: Recursive Agent Harnessing for an Open World
Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu et al. (+6 more)
LLM
agents' capabilities depend not only on their underlying models but also on their execution harnesses. Recursive self-improvement (RSI) offers a promising approach to automatically evolve these harnesses. However, deploying RSI in open-world settings still faces two critical challenges. First, open-world tasks are typically unseen during the harness design phase and novel at test time, leaving no ground-truth verifiers to reliably guide online harness edits. Second, the RSI process remains highly task-specific and fails to generalize to novel scenarios. We introduce Harness^2, a plug-and-play, recursive
agent harnessing framework that addresses these challenges through two connected levels of recursion. Task-level recursion refines the current task's harness using a contrastive Propose-Probe-Compose step that reliably guides edits without ground-truth verifiers, and can be scaled further via parallel or sequential recursion. Harnessing-level recursion then extracts experience from these edits to update a continual improvement harness, enabling the
agent to better adapt to future tasks. Across three professional
benchmarks and four model-harness configurations, a single step consistently improves GLM-5 and Gemini 3.5 Flash over their base harnesses, yielding gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB, with further gains as the task-level recursion budget increases.
Read the full abstract ▾
All authors Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister
Artificial Intelligence · Cross-listed from cs.LG
Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
Zongyue Qin, Haoran Deng, Jason Cong et al. (+1 more)
Softmax
attention in
Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear
attention alleviates this issue by replacing the exponential kernel with alternative functions that enable linear-time computation. Among existing linear
attention approaches, recent studies have shown that
polynomial kernels are particularly effective, as they exhibit sparse and spiky behavior similar to softmax
attention, which emphasizes large dot products while suppressing irrelevant interactions. However, the exact computation of high-degree
polynomial kernels is infeasible for high-dimensional representations. As a result, prior work relies on approximate
polynomial kernels. This introduces a non-negligible approximation error. In this paper, we show that, from the perspective of
polynomial kernel approximation, existing linear
attention methods are still suboptimal. We propose Learnable Low-Rank
Polynomial Sketch (LLoPS), a principled and flexible framework for approximating
polynomial kernels with linear
attention. Our method learns a low-rank
polynomial sketch that provably achieves a strictly smaller approximation error than existing approaches. Experiment results show that LLoPS achieves the highest performance across extensive
benchmarks, comparing with various linear
attention baselines.
Read the full abstract ▾
All authors Zongyue Qin, Haoran Deng, Jason Cong, Yizhou Sun
Artificial Intelligence · Cross-listed from cs.SE
Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
Samuele Giampieri
Coding
agents such as Claude Code also follow text files that developers write for them: a project file, skills the
agent may choose to open, and rules that load automatically when the
agent reads matching files. We call these files the
agent's harness. Developers add to it whenever they correct the
agent, but one success proves little: the same
agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the
agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the
agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the
agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the
agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.
Read the full abstract ▾
Artificial Intelligence
Should SLMs be in the Driver's Seat? A Study of Local Models as Interpreters in Cloud-Written Agent Programs
Boris Radovič, Marco Canini and Veljko Pejović
The most accurate
agents executing user-provided tasks are driven by large, cloud-hosted models, which solve them by reading the private data of the user.
Agents powered by models small enough to run on edge devices keep that data local, but yield far worse results. A third option is to have cloud models write a program that devices execute: the data stays local, but as the program is written before any data is seen, it fails whenever a step depends on interpreting content that appears only at run time. We explore an intermediate architecture, in which the cloud-written program defers bounded questions over user data to a small local model through an ordinary function call. On AppWorld this improves the success rate of a code-only
agent in all six combinations of backbone model and test partition we evaluate, by up to 14.9pp, recovering a substantial part of the gap to a cloud-hosted ReAct loop while keeping user data on the device.
Read the full abstract ▾
Artificial Intelligence
Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models
Laura Yao, Shixing Yu, Jie Peng et al. (+3 more)
Explaining a
vision–
language model (VLM) prediction requires tracing training-example influence to the
multimodal evidence that mediates it, yet existing attribution methods largely operate only at the sample level. Recent latent-mediated influence methods provide feature-level attribution in
language models, but lack the
multimodal and spatial structure needed to ground influence in VLMs. We introduce MOSAIC (
Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization). It learns a shared sparse image–text latent space whose concepts are 1) sparse within each modality, 2) supported on contiguous image regions, 3) aligned to the phrases that describe them, and 4) approximately decorrelated. It then applies EK-FAC to decompose training-example influence through these concepts and localize their contributions to specific text phrases and image regions. We evaluate MOSAIC on Qwen and Gemma across captioning, chest X-ray understanding, and visual question answering. The learned features are more localized and coherent without degrading task behavior, and ablating influence-ranked latents changes model outputs more than ablating latents ranked by activation or frequency. We find that visual evidence can substantially reorganize training-data attribution while preserving more of the shared latent structure. MOSAIC thus connects where model behavior comes from with what
multimodal evidence carries that influence.
Read the full abstract ▾
All authors Laura Yao, Shixing Yu, Jie Peng, Promit Ghosal, Kyra Gan, Tianlong Chen
Artificial Intelligence
Visualizing the World Model in an Agent's Value Function
Shawheen Naderi, Fatemeh Akherati and Yubei Chen
Could model-free reinforcement learning
algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization
agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning
agents that will prevent us from building truly advanced machine intelligence.
Read the full abstract ▾
Artificial Intelligence
Unconstrained CTI-REALM with Open-Weights
Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal et al. (+1 more)
Organisations that want
language-model
agents for security work face a choice between open-weight models, which they can run and audit themselves, and far more expensive closed frontier models. We study that choice on CTI-REALM (arXiv:2603.13517), a
benchmark in which an
agent turns a threat-intelligence report into a working detection by querying a live telemetry store. We evaluate twenty-six models, nineteen open-weight and seven closed, under a common harness, scorer and panel of LLM judges. Our main finding is that an open-weight model can match the strongest closed models at less than a tenth of their cost, provided it is given enough turns. The
benchmark's published limit on
agent messages is a confound: a task that exhausts the limit scores zero on every checkpoint, including those it had already passed, so results at that limit partly measure which models finish in time. At the published limit the leading open-weight and closed models cannot be distinguished. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. Among the leading models, price is a poor guide to capability: models with similar rewards differ in cost by more than an order of magnitude. Because the
benchmark's checkpoint scores largely record what an
agent did or mentioned, we also introduce stage-level metrics that score what the
agent submits, by executing its detection queries in the telemetry store and judging its outputs against the answer key. Rather than combining them into one more overall reward, we show that different models lead different stages, which suggests routing each stage of the workflow to the model that performs it best rather than choosing a single model by its overall score. We also report how many
tokens each model generates, and find that the share of output spent on thinking does not track the reward.
Read the full abstract ▾
All authors Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal, Shubham Vishwakarma
Artificial Intelligence · Cross-listed from cs.DL
Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis
Daniel J. Evans, J. P. Naiman, JooYoung Seo et al. (+1 more)
Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large
language and
multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/
datasets/ReadingTimeMachine/visual\_qa\_multipanel
Read the full abstract ▾
All authors Daniel J. Evans, J. P. Naiman, JooYoung Seo, J. Stephen Downie
Artificial Intelligence · Cross-listed from cs.CL
Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi et al. (+6 more)
Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload. However, DLMs degrade quickly with fewer sampling steps, process the full sequence at every step, and require many passes over the
dataset to reach good quality. We introduce Next-Concept Modeling (NCM), a two-stage, coarse-to-fine framework: a multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, and a
transformer generates one level at a time, predicting all codes within a level in parallel. At matched parameter scales, NCM achieves better generative perplexity than the strongest DLM baseline on LM1B while running $21\times$ faster. On OpenWebText, NCM is competitive with the strongest DLM while running $10\times$ faster; on XSum, it matches the strongest DLM's ROUGE-L while being $31\times$ faster. On the
reasoning task of Sudoku, NCM outperforms autoregressive and
diffusion baselines across all difficulty levels, using fewer function evaluations. On the
reasoning task of Sudoku, NCM outperforms autoregressive and
diffusion baselines across all difficulty levels, using fewer function evaluations.
Read the full abstract ▾
All authors Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, Christopher Ré
Artificial Intelligence · Cross-listed from eess.IV
Beyond Gate Weights: Decision-Level Interpretation of Radiomics-Image Feature Fusion for RCC Classification
Yuan Liang, Sourav Bhattacharjee and Abraham Campbell
Gate coefficients describe how a fusion layer weights intermediate features, but they do not necessarily reflect the magnitude of branch contributions at the classifier output. We investigate this distinction in a fixed 3D MedVAE–radiomics model for clear
cell versus non-clear
cell renal
cell carcinoma classification from contrast-enhanced CT. Because the model uses a linear classification head, the binary decision logit can be decomposed exactly into a gated image term, a direct radiomics term, and a bias. We define effective radiomics dominance as the direct radiomics term's share of the summed absolute non-bias branch contributions and compare it with structural radiomics weighting. Internally, mean structural radiomics weighting was 0.823, whereas mean effective radiomics dominance was 0.148, with a modest case-wise association (h̊o} = 0.290). The gated image term retained substantially stronger discriminative ranking information than the direct radiomics term. The same structural-to-effective discrepancy was observed in exploratory external evaluation. Branch-targeted permutation, integrated-gradient, and Shapley-style analyses further identified radiomics descriptors influencing the direct radiomics term. Because radiomics also generates the gate, the gated image term remains radiomics-dependent, so additive branch contribution should not be interpreted as total input influence. These findings show that structural gate weighting and decision-level branch contribution represent distinct aspects of gated feature fusion and should be interpreted separately.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from eess.IV
Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification
Yuan Liang, Morgan May, Abraham Campbell et al. (+3 more)
Reliable probability estimates are important for the clinical interpretation of machine-learning classifiers, yet radiomics–image fusion models for renal
cell carcinoma (RCC) are commonly evaluated primarily by discrimination. Using 396 patients from the KiTS23
dataset, we evaluated whether adding radiomics to four image-model families improved probabilistic performance in addition to discrimination. Seventeen individual models and four probability-average ensembles were assessed using discrimination, probability calibration, and proper scoring measures. Across 12 matched image-only versus fusion comparisons, AUC and AP improved in all 12, while Brier score and negative log likelihood (NLL) improved in 11 of 12 and expected calibration error (ECE) in eight of 12. The 3D MedVAE gated-fusion model achieved the strongest overall individual-model profile. Probability averaging across concatenation, cross-
attention, and gated fusion produced favourable point estimates across all five primary metrics relative to image-only models in all four families, although paired-bootstrap support was stronger for Brier score and NLL than for ECE. These findings show that radiomics–image fusion can improve probabilistic prediction quality beyond discrimination, while calibration behaviour varies across image architectures and fusion strategies. We show that our proposed probability-average ensemble models improve discrimination and calibration simultaneously across all architectures in the study.
Read the full abstract ▾
All authors Yuan Liang, Morgan May, Abraham Campbell, Sourav Bhattacharjee, Simon Caton, Pierpaolo Dondio
Artificial Intelligence
VETNET: Assembling Benchmark Datasets for AI-Driven Veterinary Innovation
Katherine J. Koebel, Renata Ivanek, Parminder Basran et al. (+33 more)
VETNET represents an ecosystem of live
benchmark datasets and evaluation frameworks designed to accelerate AI-driven innovation in animal science and veterinary medicine through standardized, high-quality
datasets and multi-institutional collaboration across academic and non-academic sectors. This framework addresses the gap between AI advances in human medicine and the fragmented state of veterinary AI development.
Read the full abstract ▾
All authors Katherine J. Koebel, Renata Ivanek, Parminder Basran, Jennifer Sun, Casey Cazer, Gayatri Anil, Tommaso Banzato, Liz Barton, Dottie Cimino Brown, Leszek Bukowski, Marta Castelhano, Simon Coghlan, Christopher De Sa, Nikki Dutta, Hala Ghali, Julio Giordano, Joshua Glass, David C. Hogg, Nicholas Hollingshead, Adam Kiehl, Ziang Liu, Monica Meduri, Param Mohapatra, Kim Nayyer, Amelia Nelson, Peter-John Mantyla Noble, Christopher Pinard, Beryl Rabindran, Scott Ross, Ron Seccia, Moira Sheehan, Fei Wang, Tracy L. Webb, Kilian Q. Weinberger, Heather M. Whitney, Chen Xin