Computation and Language
Indic-TunedLens: Interpreting Multilingual Models in Indian Languages
Mihir Panchal, Deeksha Varshney, Mamta et al. (+1 more)
Multilingual large
language models (LLMs) are increasingly deployed in linguistically diverse regions like India, yet most interpretability tools remain tailored to English. Prior work reveals that LLMs often operate in English centric representation spaces, making cross lingual interpretability a pressing concern. We introduce Indic-TunedLens, a novel interpretability framework specifically for Indian
languages that learns shared affine transformations. Unlike the standard Logit Lens, which directly decodes intermediate activations, Indic-TunedLens adjusts hidden states for each target
language, aligning them with the target output distributions to enable more faithful decoding of model representations. We evaluate our framework on 10 Indian
languages using the MMLU
benchmark and find that it significantly improves over SOTA interpretability methods, especially for morphologically rich, low resource
languages. Our results provide crucial insights into the layer-wise semantic encoding of
multilingual transformers. Our model is available at https://huggingface.co/spaces/MihirRajeshPanchal/IndicTunedLens. Our code is available at https://github.com/MihirRajeshPanchal/IndicTunedLens.
Read the full abstract ▾
All authors Mihir Panchal, Deeksha Varshney, Mamta, Asif Ekbal
Computation and Language · Cross-listed from cs.AI
NL2LOGIC: AST-Guided Translation of Natural Language into First-Order Logic with Large Language Models
Rizky Ramadhana Putra, Raihan Sultan Pasha Basuki, Yutong Cheng et al. (+1 more)
Automated
reasoning is critical in domains such as law and governance, where verifying claims against facts in documents requires both accuracy and interpretability. Recent work adopts structured
reasoning pipelines that translate natural
language into first-order logic and delegate inference to automated solvers. With the rise of large
language models, approaches such as GCD and CODE4LOGIC leverage their
reasoning and code generation capabilities to improve logic parsing. However, these methods suffer from fragile syntax control due to weak enforcement of global grammar constraints and low semantic faithfulness caused by insufficient clause-level semantic understanding. We propose NL2LOGIC, a first-order logic translation framework that introduces an abstract syntax tree as an intermediate representation. NL2LOGIC combines a recursive large
language model based semantic parser with an abstract syntax tree guided generator that deterministically produces solver-ready logic code. Experiments on the FOLIO, LogicNLI, and ProofWriter
benchmarks show that NL2LOGIC achieves 99 percent syntactic accuracy and improves semantic correctness by up to 30 percent over state-of-the-art baselines. Furthermore, integrating NL2LOGIC into Logic-LM yields near-perfect executability and improves downstream
reasoning accuracy by 31 percent compared to Logic-LM's original few-shot unconstrained translation module.
Read the full abstract ▾
All authors Rizky Ramadhana Putra, Raihan Sultan Pasha Basuki, Yutong Cheng, Peng Gao
Computation and Language · Cross-listed from cs.LG
LLM Compression by Block Removal with Constrained Binary Optimization
David Jansen, Roman Rausch, Ali Hashemi et al. (+2 more)
In this paper, we formulate the compression of large
language models (LLMs) by optimally deleting
transformer blocks (“block removal”) as a constrained binary optimization (CBO) problem that can be mapped to a physical system (Ising glass), whose energies are a strong proxy for downstream model performance. This formulation enables an efficient ranking of a large number of candidate block-removal configurations yielding many high-quality, non-trivial solutions beyond those only removing consecutive regions. Our method performs strongly in the deep compression regime, such as for 50% compression of Llama-3.3-70B-Instruct, where we achieve an almost 23 percentage point increase on the MMLU
benchmark compared to other state-of-the-art (SOTA) block-removal methods. For lighter compression, it performs on par with those methods across several
benchmarks for Llama-3.1-8B-Instruct, Qwen3-14B (both before and after retraining), as well as Llama-3.3-70B-Instruct. The approach is computationally efficient and requires only forward and backward passes on a calibration
dataset for a few active parameters. Additionally, we demonstrate that using good heuristic solvers for the CBO problem provides solutions that perform well on downstream tasks in negligible runtime when it is unfeasible to solve the problem exactly. The method can be readily applied to any architecture. We illustrate this generality on the recent NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model, which exhibits a highly inhomogeneous and challenging block structure, and where we outperform SOTA for AIME25 and GPQA when removing either 2
attention layers or 3 mixture-of-experts layers.
Read the full abstract ▾
All authors David Jansen, Roman Rausch, Ali Hashemi, David Montero, Román Orús
Computation and Language
Xinyun Wang, Min Zhang, Sen Cui et al. (+4 more)
Diffusion language models enable parallel
token generation through block-wise decoding, but their irreversible commitments can lead to stagnation, where the reverse
diffusion process fails to make further progress under a suboptimal context.We propose Reversible
Diffusion Decoding (RDD), a decoding framework that introduces reversibility into block-wise
diffusion generation. RDD detects stagnation as a state-dependent failure of the reverse process and enables efficient backtracking to earlier blocks without recomputation via cached model states. To avoid repeated failure trajectories, RDD applies confidence-guided re-masking to selectively reinitialize uncertain
tokens while preserving reliable context.This reversible formulation allows decoding to recover from early commitment errors while maintaining the parallel efficiency of
diffusion-based generation. Experiments show that RDD improves generation robustness and quality over baselines with minimal computational overhead.
Read the full abstract ▾
All authors Xinyun Wang, Min Zhang, Sen Cui, Zhikang Chen, Bo Jiang, Kun Kuang, Mingbao Lin
Computation and Language
Word-Centered Semantic Graphs for Interpretable Diachronic Sense Tracking
Imene Kolli, Kai-Robin Lange, Jonas Rieger et al. (+1 more)
We propose an interpretable, graph-based framework for analyzing semantic shift in diachronic corpora. For each target word and time slice, we induce a word-centered semantic network that integrates distributional similarity from diachronic Skip-gram
embeddings with lexical substitutability from time-specific masked
language models. We identify sense-related structure by clustering the peripheral graph, align clusters across time via node overlap, and track change through cluster composition and normalized cluster mass. In an application study on a corpus of New York Times Magazine articles (1980 - 2017), we show that graph connectivity reflects polysemy dynamics and that the induced communities capture contrasting trajectories: event-driven sense replacement (trump), semantic stability with cluster over-
segmentation effects (god), and gradual association shifts tied to digital communication (post). Overall, word-centered semantic graphs offer a compact and transparent representation for exploring sense
evolution without relying on predefined sense inventories.
Read the full abstract ▾
All authors Imene Kolli, Kai-Robin Lange, Jonas Rieger, Carsten Jentsch
Computation and Language
Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
Kanishk Awadhiya
Rotary Positional
Embeddings (RoPE) have become the standard for Large
Language Models (LLMs) due to their ability to encode relative positions through
geometric rotation. However, we identify a significant limitation we term ”Spectral
Rigidity”: standard RoPE utilizes a fixed
geometric decay ($\theta^{-i}$) optimized for local syntactic coherence, which fails to capture the long-range, periodic structures inherent in recursive logic and algorithmic
reasoning. This results in a ”Structure Gap”, where models trained on shallow
reasoning chains fail to extrapolate to deeper recursive steps. In this work, we introduce Bifocal
Attention, an architectural paradigm that decouples positional encoding into two distinct modalities:
Geometric Eyes (Standard RoPE) for precise
token-level manipulation, and Spectral Eyes (Learnable Harmonic Operators) for tracking long-range recursive depth. We propose a novel training protocol, Spectral
Evolution, which initializes positional frequencies as static
geometric parameters but allows them to evolve via gradient descent into a harmonic basis optimized for the specific algorithmic
topology of the task.
Read the full abstract ▾
Computation and Language
Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks
Candida M. Greco, Lucio La Cava and Andrea Tagarelli
Despite the growing utility of Large
Language Models (LLMs) for simulating human behavior, the extent to which these synthetic personas accurately reflect world and moral value systems across different cultural conditionings remains uncertain. This paper investigates the
alignment of synthetic, culturally-grounded personas with established frameworks, specifically the World Values Survey (WVS), the Inglehart-Welzel Cultural Map, and Moral Foundations Theory. We conceptualize and produce LLM-generated personas based on a set of interpretable WVS-derived variables, and we examine the generated personas through three complementary lenses: positioning on the Inglehart-Welzel map, which unveils their interpretation reflecting stable differences across cultural conditionings; demographic-level consistency with the World Values Survey, where response distributions broadly track human group patterns; and moral profiles derived from a Moral Foundations questionnaire, which we analyze through a culture-to-morality mapping to characterize how moral responses vary across different cultural configurations. Our approach of culturally-grounded persona generation and analysis enables evaluation of cross-cultural structure and moral variation.
Read the full abstract ▾
Computation and Language
Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading
Jamiu Adekunle Idowu and Ahmed Almasoud
Automated essay scoring (AES) systems increasingly rely on large
language models, yet little is known about how architectural choices shape their performance across different essay quality levels. This paper evaluates single-
agent and multi-
agent LLM architectures for essay grading using the ASAP 2.0 corpus. Our multi-
agent system decomposes grading into three specialist
agents (Content, Structure,
Language) coordinated by a Chairman
Agent that implements rubric-aligned logic including veto rules and score capping. We test both architectures in zero-shot and few-shot conditions using GPT-5.1. Results show that the multi-
agent system is significantly better at identifying weak essays while the single-
agent system performs better on mid-range essays. Both architectures struggle with high-quality essays. Critically, few-shot calibration emerges as the dominant factor in system performance – providing just two examples per score level improves QWK by approximately 26% for both architectures. These findings suggest architectural choice should align with specific deployment priorities, with multi-
agent AI particularly suited for diagnostic screening of at-risk students, while single-
agent models provide a cost-effective solution for general assessment.
Read the full abstract ▾
Computation and Language
SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
Chaoyue He, Xin Zhou, Di Wang et al. (+3 more)
Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher
language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.
Read the full abstract ▾
All authors Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, Chunyan Miao
Computation and Language
SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
Bailin Wang, Dan Friedman, Tao Lei et al. (+1 more)
Block-wise sparse
attention offers significant efficiency gains for long-context modeling, yet existing methods often suffer from low selection fidelity and cumulative contextual loss by completely discarding unselected blocks. To address these limitations, we introduce Sparse Plus Linear
Attention (SPLA), a framework that utilizes a selection metric derived from second-order Taylor expansions to accurately identify relevant blocks for exact
attention. Instead of discarding the remaining "long tail," SPLA compresses unselected blocks into a compact recurrent state via a residual linear
attention (RLA) module. Crucially, to avoid IO overhead, we derive an optimized subtraction-based formulation for RLA – calculating the residual as the difference between global and selected linear
attention – ensuring that unselected blocks are never explicitly accessed during inference. Our experiments demonstrate that SPLA closes the performance gap in continual pretraining, surpassing dense
attention models on long-context
benchmarks like RULER while maintaining competitive general knowledge and
reasoning capabilities.
Read the full abstract ▾
All authors Bailin Wang, Dan Friedman, Tao Lei, Chong Wang
Computation and Language
Stability-Aware Prompt Optimization for Clinical Data Abstraction
Arinbjörn Kolbeinsson, Daniel Timbie, Sajjan Narsinghani et al. (+1 more)
Large
language models used for clinical abstraction are sensitive to
prompt wording, yet most work treats
prompts as fixed and studies uncertainty in isolation. We argue these should be treated jointly. Across two clinical tasks (MedAlign applicability/correctness and MS subtype abstraction) and multiple open and proprietary models, we measure
prompt sensitivity via flip rates and relate it to calibration and selective prediction. We find that higher accuracy does not guarantee
prompt stability, and that models can appear well-calibrated yet remain fragile to paraphrases. We propose a dual-objective
prompt optimization loop that jointly targets accuracy and stability, showing that explicitly including a stability term reduces flip rates across tasks and models, sometimes at modest accuracy cost. Our results suggest
prompt sensitivity should be an explicit objective when validating clinical LLM systems.
Read the full abstract ▾
All authors Arinbjörn Kolbeinsson, Daniel Timbie, Sajjan Narsinghani, Sanjay Hariharan
Computation and Language
Context Structure Reshapes the Representational Geometry of Language Models
Eghbal A. Hosseini, Yuxuan Li, Yasaman Bahri et al. (+2 more)
Large
Language Models (LLMs) have been shown to organize the representations of input sequences into straighter neural trajectories in their deep layers, which has been hypothesized to facilitate next-
token prediction via linear extrapolation.
Language models can also adapt to diverse tasks and learn new structure in context, and recent work has shown that this in-context learning (ICL) can be reflected in representational changes. Here we bring these two lines of research together to explore whether representation straightening occurs within a context during ICL. We measure representational straightening in Gemma 2 models across a diverse set of in-context tasks, and uncover a dichotomy in how LLMs' representations change in context. In continual prediction settings (e.g., natural
language, grid world traversal tasks) we observe that increasing context increases the straightness of neural sequence trajectories, which is correlated with improvement in model prediction. Conversely, in structured prediction settings (e.g., few-shot tasks), straightening is inconsistent – it is only present in phases of the task with explicit structure (e.g., repeating a template), but vanishes elsewhere. These results suggest that ICL is not a monolithic process. Instead, we propose that LLMs function like a Swiss Army knife: depending on task structure, the LLM dynamically selects between strategies, only some of which yield representational straightening.
Read the full abstract ▾
All authors Eghbal A. Hosseini, Yuxuan Li, Yasaman Bahri, Declan Campbell, Andrew Kyle Lampinen
Computation and Language
MERMAID: Memory-Enhanced Retrieval and Reasoning with Multi-Agent Iterative Knowledge Grounding for Veracity Assessment
Yupeng Cao, Chengyang He, Yangyang Yu et al. (+2 more)
Assessing the veracity of online content has become increasingly critical. Large
language models (LLMs) have recently enabled substantial progress in automated veracity assessment, including automated fact-checking and claim verification systems. Typical veracity assessment pipelines break down complex claims into sub-claims, retrieve external evidence, and then apply LLM
reasoning to assess veracity. However, existing methods often treat evidence
retrieval as a static, isolated step and do not effectively manage or reuse retrieved evidence across claims. In this work, we propose MERMAID, a
memory-enhanced multi-
agent veracity assessment framework that tightly couples the
retrieval and
reasoning processes. MERMAID integrates
agent-driven search, structured knowledge representations, and a persistent
memory module within a Reason-Action style iterative process, enabling dynamic evidence acquisition and cross-claim evidence reuse. By retaining retrieved evidence in an evidence
memory, the framework reduces redundant searches and improves verification efficiency and consistency. We evaluate MERMAID on three fact-checking
benchmarks and two claim-verification
datasets using multiple LLMs, including GPT, LLaMA, and Qwen families. Experimental results show that MERMAID achieves state-of-the-art performance while improving the search efficiency, demonstrating the effectiveness of synergizing
retrieval,
reasoning, and
memory for reliable veracity assessment.
Read the full abstract ▾
All authors Yupeng Cao, Chengyang He, Yangyang Yu, Ping Wang, K.P. Subbalakshmi
Computation and Language · Cross-listed from cs.AI
Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents
Zehong Wang, Fang Wu, Hongru Wang et al. (+8 more)
Large
language model (LLM)-based
agents exhibit strong step-by-step
reasoning capabilities over short horizons, yet often fail to sustain coherent behavior over long planning horizons. We argue that this failure reflects a fundamental mismatch: step-wise
reasoning induces a form of step-wise greedy policy that is adequate for short horizons but fails in long-horizon planning, where early actions must account for delayed consequences. From this planning-centric perspective, we study LLM-based
agents in deterministic, fully structured environments with explicit state transitions and evaluation signals. Our analysis reveals a core failure mode of
reasoning-based policies: locally optimal choices induced by step-wise scoring lead to early myopic commitments that are systematically amplified over time and difficult to recover from. We introduce
FLARE (Future-aware Lookahead with Reward Estimation) as a minimal instantiation of future-aware planning to enforce explicit lookahead, value propagation, and limited commitment in a single model, allowing downstream outcomes to influence early decisions. Across multiple
benchmarks,
agent frameworks, and LLM backbones,
FLARE consistently improves task performance and planning-level behavior, frequently allowing LLaMA-8B with
FLARE to outperform GPT-4o with standard step-by-step
reasoning. These results establish a clear distinction between
reasoning and planning.
Read the full abstract ▾
All authors Zehong Wang, Fang Wu, Hongru Wang, Xiangru Tang, Bolian Li, Zhenfei Yin, Yijun Ma, Yiyang Li, Weixiang Sun, Xiusi Chen, Yanfang Ye
Computation and Language · Cross-listed from eess.AS
Sylber 2.0: A Universal Syllable Embedding
Cheol Jun Cho, Nicholas Lee, Alan W Black et al. (+1 more)
Scaling spoken
language modeling requires
speech tokens that are both efficient and universal. Recent work has proposed syllables as promising
speech tokens at low temporal resolution, but existing models are constrained to English and fail to capture sufficient acoustic detail. To address this gap, we present Sylber 2.0, a self-supervised framework for coding
speech at the syllable level that enables efficient temporal compression and high-fidelity reconstruction. Sylber 2.0 achieves a very low
token frequency around 5 Hz, while retaining both linguistic and acoustic detail across multiple
languages and expressive styles. Experiments show that it performs on par with previous models operating on high-frequency baselines. Furthermore, Sylber 2.0 enables efficient TTS modeling which can generate
speech with competitive intelligibility and quality with SOTA models using only 72M parameters. Moreover, the universality of Sylber 2.0 provides more effective features for low resource ASR than previous
speech coding frameworks. In sum, we establish an effective syllable-level abstraction for general spoken
language.
Read the full abstract ▾
All authors Cheol Jun Cho, Nicholas Lee, Alan W Black, Gopala K. Anumanchipalli
Computation and Language
Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
Chenxi Liu, Yanshuo Chen, Ruibo Chen et al. (+3 more)
The
reasoning abilities of large
language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative
reasoning through Multi-
Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning(SDRL), a training framework where models learn from self-debate, equipping a single LLM with both strong standalone problem-solving ability and the capability to process diverse
reasoning trajectories in MAD. Given a
prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse
reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and
reasoning benchmarks show that SDRL consistently improves MAD performance across diverse debate protocols and
agent configurations, while simultaneously strengthening single-model
reasoning.
Read the full abstract ▾
All authors Chenxi Liu, Yanshuo Chen, Ruibo Chen, Tianyi Xiong, Tong Zheng, Heng Huang
Computation and Language · Cross-listed from cs.AI
JAF: Judge Agent Forest
Sahil Garg, Brad Cheezum, Sridhar Dutta et al. (+1 more)
Judge
agents are fundamental to agentic AI frameworks: they provide automated evaluation, and enable iterative self-refinement of
reasoning processes. We introduce JAF: Judge
Agent Forest, a framework in which the judge
agent conducts joint inference across a cohort of query–response pairs generated by a primary
agent, rather than evaluating each in isolation. This paradigm elevates the judge from a local evaluator to a holistic learner: by simultaneously assessing related responses, the judge discerns cross-instance patterns and inconsistencies, whose aggregate feedback enables the primary
agent to improve by viewing its own outputs through the judge's collective perspective. Conceptually, JAF bridges belief propagation and ensemble-learning principles: overlapping in-context neighborhoods induce a knowledge-graph structure that facilitates propagation of critique, and repeated, randomized evaluations yield a robust ensemble of context-sensitive judgments. JAF can be instantiated entirely via ICL, with the judge prompted for each query using its associated primary-
agent response plus a small, possibly noisy set of peer exemplars. While kNN in
embedding space is a natural starting point for exemplars, this approach overlooks categorical structure, domain metadata, or nuanced distinctions accessible to modern LLMs. To overcome these limitations, we develop a flexible locality-sensitive hashing (LSH)
algorithm that learns informative binary codes by integrating semantic
embeddings, LLM-driven hash predicates, supervision from categorical labels, and relevant side information. These hash codes support efficient, interpretable, and relation-aware selection of diverse exemplars, and further optimize exploration of CoT
reasoning paths. We validate JAF with an empirical study on the demanding task of cloud misconfigs triage in large-scale cloud environments.
Read the full abstract ▾
All authors Sahil Garg, Brad Cheezum, Sridhar Dutta, Vishal Agarwal
Computation and Language · Cross-listed from cs.SE
Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models
Henri Aïdasso, Francis Bordeleau and Ali Tizghadam
In principle, Continuous Integration (CI) pipeline failures provide valuable feedback to developers on code-related errors. In practice, however, pipeline jobs often fail intermittently due to non-deterministic tests, network outages, infrastructure failures, resource exhaustion, and other reliability issues. These intermittent (flaky) job failures lead to substantial inefficiencies: wasted computational resources from repeated reruns and significant diagnosis time that distracts developers from core activities and often requires intervention from specialized teams. Prior work has proposed machine learning techniques to detect intermittent failures, but does not address the subsequent diagnosis challenge. To fill this gap, we introduce FlaXifyer, a few-shot learning approach for predicting intermittent job failure categories using pre-trained
language models. FlaXifyer requires only job execution logs and achieves 84.3% Macro F1 and 92.0% Top-2 accuracy with just 12 labeled examples per category. We also propose LogSift, an interpretability technique that identifies influential log statements in under one second, reducing review effort by 74.4% while surfacing relevant failure information in 87% of cases. Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
A Systematic Literature Review on LLM Defenses Against Prompt Injection and Jailbreaking: Expanding NIST Taxonomy
Pedro H. Barcha Correia, Ryan W. Achjian, Diego E. G. Caetano de Oliveira et al. (+5 more)
The rapid advancement and widespread adoption of generative artificial intelligence (GenAI) and large
language models (LLMs) has been accompanied by the emergence of new security vulnerabilities and challenges, such as jailbreaking and other
prompt injection attacks. These maliciously crafted inputs can exploit LLMs, causing data leaks, unauthorized actions, or compromised outputs, for instance. As both offensive and defensive
prompt injection techniques evolve quickly, a structured understanding of mitigation strategies becomes increasingly important. To address that, this work presents the first systematic literature review on
prompt injection mitigation strategies, comprehending 88 studies. Building upon NIST's report on
adversarial machine learning, this work contributes to the field through several avenues. First, it identifies studies beyond those documented in NIST's report and other academic reviews and surveys. Second, we propose an extension to NIST taxonomy by introducing additional categories of defenses. Third, by adopting NIST's established terminology and taxonomy as a foundation, we promote consistency and enable future researchers to build upon the standardized taxonomy proposed in this work. Finally, we provide a comprehensive catalog of the reviewed
prompt injection defenses, documenting their reported quantitative effectiveness across specific LLMs and attack
datasets, while also indicating which solutions are open-source and model-agnostic. This catalog, together with the guidelines presented herein, aims to serve as a practical resource for researchers advancing the field of
adversarial machine learning and for developers seeking to implement effective defenses in production systems.
Read the full abstract ▾
All authors Pedro H. Barcha Correia, Ryan W. Achjian, Diego E. G. Caetano de Oliveira, Ygor Acacio Maria, Victor Takashi Hayashi, Marcos Lopes, Charles Christian Miers, Marcos A. Simplicio Jr
Computation and Language · Cross-listed from cs.CV
Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation
Ken Deng, Yifu Qiu, Yoni Kasten et al. (+2 more)
We study whether
vision-
language models (VLMs) can solve relative camera pose estimation (RCPE) from image pairs, a direct test of multi-view spatial
reasoning. We cast RCPE as a discrete verbal classification task and introduce VRRPI-Bench, built from real RGB-D frames with object-centric camera motion, and VRRPI-Diag, which isolates individual motion degrees of freedom. Humans (0.91) and specialized
geometric pipelines such as LoFTR (0.99) solve the task reliably, yet the best VLM reaches only 0.66 and most others remain near random. Our analyses show that this gap is not basic spatial competence: strong VLMs are near ceiling on single-image
benchmarks, but most remain near random once
reasoning must span views. They are unstable under source-target reversal (best 59.7% consistency) and remain weak even in simplified single-DoF settings, especially on optical-axis motions such as roll and depth translation (GPT-5: 0.46 on roll). These failures are useful: they localize concrete missing capabilities, namely cross-view correspondence, view-consistent
reasoning, and projective camera-motion understanding, making RCPE a targeted diagnostic for improving multi-view spatial
reasoning in VLMs.
Read the full abstract ▾
All authors Ken Deng, Yifu Qiu, Yoni Kasten, Shay B. Cohen, Yftah Ziser
Computation and Language · Cross-listed from cs.CR
RedSage: A Cybersecurity Generalist LLM
Naufal Suryanto, Muzammal Naseer, Pengfei Li et al. (+5 more)
Cybersecurity operations demand assistant LLMs that support diverse workflows without exposing sensitive data. Existing solutions either rely on proprietary APIs with
privacy risks or on open models lacking domain adaptation. To bridge this gap, we curate 11.8B
tokens of cybersecurity-focused continual pretraining data via large-scale web filtering and manual collection of high-quality resources, spanning 28.6K documents across frameworks, offensive techniques, and security tools. Building on this, we design an agentic augmentation pipeline that simulates expert workflows to generate 266K multi-turn cybersecurity samples for supervised fine-tuning. Combined with general open-source LLM data, these resources enable the training of RedSage, an open-source, locally deployable cybersecurity assistant with domain-aware pretraining and post-training. To rigorously evaluate the models, we introduce RedSage-Bench, a
benchmark with 30K multiple-choice and 240 open-ended Q&A items covering cybersecurity knowledge, skills, and tool expertise. RedSage is further evaluated on established cybersecurity
benchmarks (e.g., CTI-Bench, CyberMetric, SECURE) and general LLM
benchmarks to assess broader generalization. At the 8B scale, RedSage achieves consistently better results, surpassing the baseline models by up to +5.59 points on cybersecurity
benchmarks and +5.05 points on Open LLM Leaderboard tasks. These findings demonstrate that domain-aware agentic augmentation and pre/post-training can not only enhance cybersecurity-specific expertise but also help to improve general
reasoning and instruction-following. All models,
datasets, and code are publicly available.
Read the full abstract ▾
All authors Naufal Suryanto, Muzammal Naseer, Pengfei Li, Syed Talal Wasim, Jinhui Yi, Juergen Gall, Paolo Ceravolo, Ernesto Damiani
Computation and Language · Cross-listed from cs.LG
Discovering Hidden Gems in Model Repositories
Jonathan Kahana, Eliahu Horwitz and Yedid Hoshen
Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient
market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search
algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.
Read the full abstract ▾
Computation and Language
Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
Yingfa Chen, Zhen Leng Thai, Zihan Zhou et al. (+6 more)
Hybrid
Transformer architectures, which combine softmax
attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax
attention blocks can be converted into RNN blocks through parameter transfer and knowledge
distillation. However, these transfer methods require substantial amounts of training data (more than 10B
tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over
Transformer-based models. In this paper, we present
HALO (Hybrid
Attention via Layer Optimization), a pipeline for distilling
Transformer models into RNN-
attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using
HALO, achieving performance comparable to the original
Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B
tokens, less than 0.01% of their pre-training data
Read the full abstract ▾
All authors Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, Zhiyuan Liu
Computation and Language · Cross-listed from cs.CV
Bo Li, Yida Yin, Wenhao Chai et al. (+2 more)
We introduce UEval, a
benchmark to evaluate unified models, i.e., models capable of generating both images and text. UEval comprises 1,000 expert-curated questions that require both images and text in the model output, sourced from 8 real-world tasks. Our curated questions cover a wide range of
reasoning types, from step-by-step guides to textbook explanations. Evaluating open-ended
multimodal generation is non-trivial, as simple LLM-as-a-judge methods can miss the subtleties. Different from previous works that rely on
multimodal Large
Language Models (MLLMs) to rate image quality or text accuracy, we design a rubric-based scoring system in UEval. For each question, reference images and text answers are provided to a MLLM to generate an initial rubric, consisting of multiple evaluation criteria, and human experts then refine and validate these rubrics. In total, UEval contains 10,417 validated rubric criteria, enabling scalable and fine-grained automatic scoring. UEval is challenging for current unified models: GPT-5-Thinking scores only 66.4 out of 100, while the best open-source model reaches merely 49.1. We observe that
reasoning models often outperform non-
reasoning ones, and transferring
reasoning traces from a
reasoning model to a non-
reasoning model significantly narrows the gap. This suggests that
reasoning may be important for tasks requiring complex
multimodal understanding and generation.
Read the full abstract ▾
All authors Bo Li, Yida Yin, Wenhao Chai, Xingyu Fu, Zhuang Liu