Cryptography and Security
A Comparative Study of Hybrid Post-Quantum Cryptographic X.509 Certificate Schemes
Abel C. H. Chen
As
quantum computing hardware continues to advance, the integration of such technology with
quantum algorithms is anticipated to enable the decryption of ciphertexts produced by RSA and
Elliptic Curve Cryptography (ECC) within
polynomial time. In response to this emerging threat, the U.S. National Institute of Standards and Technology (NIST) finalized a series of Post-
Quantum Cryptography (PQC) standards in August 2024 and outlined a roadmap for PQC migration. Consequently, the design of X.509 certificates that adhere to PQC standards has become a crucial focus in the development of certificate management systems. To further strengthen security and facilitate a smooth migration process, several hybrid certificate schemes have been proposed internationally based on the X.509 certificate format, including the composite scheme, the catalyst scheme, and the chameleon scheme. This study presents a comprehensive analysis and comparison of these hybrid certificate schemes from multiple perspectives (e.g., certificate size, computational efficiency, and migration feasibility) to assess their suitability for various applications and services.
Read the full abstract ▾
Cryptography and Security
Unvalidated Trust: Cross-Stage Vulnerabilities in Large Language Model Architectures
Dominik Schwarz
As Large
Language Models (LLMs) are increasingly integrated into automated, multi-stage pipelines, risk patterns that arise from unvalidated trust between processing stages become a practical concern. This paper presents a mechanism-centered taxonomy of 41 recurring risk patterns in commercial LLMs. The analysis shows that inputs are often interpreted non-neutrally and can trigger implementation-shaped responses or unintended state changes even without explicit commands. We argue that these behaviors constitute architectural failure modes and that string-level filtering alone is insufficient. To mitigate such cross-stage vulnerabilities, we recommend zero-trust architectural principles, including provenance enforcement, context sealing, and plan revalidation, and we introduce "Countermind" as a conceptual blueprint for implementing these defenses.
Read the full abstract ▾
Cryptography and Security
LLM-based Multi-class Attack Analysis and Mitigation Framework in IoT/IIoT Networks
Seif Ikbarieh, Maanak Gupta and Elmahedi Mahalal
The Internet of Things has expanded rapidly, transforming communication and operations across industries but also increasing the attack surface and security breaches. Artificial Intelligence plays a key role in securing IoT, enabling attack detection, attack behavior analysis, and mitigation suggestion. Despite advancements, evaluations remain purely qualitative, and the lack of a standardized, objective
benchmark for quantitatively measuring AI-based attack analysis and mitigation hinders consistent assessment of model effectiveness. In this work, we propose a hybrid framework combining Machine Learning (ML) for multi-class attack detection with Large
Language Models (LLMs) for attack behavior analysis and mitigation suggestion. After benchmarking several ML and Deep Learning (DL) classifiers on the Edge-IIoTset and CICIoT2023
datasets, we applied structured role-play
prompt engineering with
Retrieval-Augmented Generation (RAG) to guide ChatGPT-o3 and DeepSeek-R1 in producing detailed, context-aware responses. We introduce novel evaluation metrics for quantitative assessment to guide us and an ensemble of judge LLMs, namely ChatGPT-4o, DeepSeek-V3, Mixtral 8x7B Instruct, Gemini 2.5 Flash, Meta Llama 4, TII Falcon H1 34B Instruct, xAI Grok 3, and Claude 4 Sonnet, to independently evaluate the responses. Results show that Random Forest has the best detection model, and ChatGPT-o3 outperformed DeepSeek-R1 in attack analysis and mitigation.
Read the full abstract ▾
Cryptography and Security
Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
Shaked Zychlinski and Yuval Kainan
Large
Language Models (LLMs) are susceptible to jailbreak attacks where malicious
prompts are disguised using ciphers and character-level encodings to bypass safety guardrails. While these guardrails often fail to interpret the encoded content, the underlying models can still process the harmful instructions. We introduce CPT-Filtering, a novel, model-agnostic with negligible-costs and near-perfect accuracy guardrail technique that aims to mitigate these attacks by leveraging the intrinsic behavior of Byte-Pair Encoding (BPE) tokenizers. Our method is based on the principle that tokenizers, trained on natural
language, represent out-of-distribution text, such as ciphers, using a significantly higher number of shorter
tokens. Our technique uses a simple yet powerful artifact of using
language models: the average number of Characters Per
Token (CPT) in the text. This approach is motivated by the high compute cost of modern methods - relying on added modules such as dedicated LLMs or perplexity models. We validate our approach across a large
dataset of over 100,000
prompts, testing numerous encoding schemes with several popular tokenizers. Our experiments demonstrate that a simple CPT threshold robustly identifies encoded text with high accuracy, even for very short inputs. CPT-Filtering provides a practical defense layer that can be immediately deployed for real-time text filtering and offline data curation.
Read the full abstract ▾
Cryptography and Security · Cross-listed from cs.LG
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Tao Tao and Maissam Barkeshli
We study the ability of
Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of pseudo-random number generators (PRNGs). PCGs introduce substantial additional difficulty over linear congruential generators (LCGs) by applying a series of bit-wise shifts, XORs, rotations and truncations to the hidden state. We show that
Transformers can nevertheless successfully perform in-context prediction on unseen sequences from diverse PCG variants, in tasks that are beyond published classical attacks. In our experiments we scale
moduli up to $2^{22}$ using up to $50$ million model parameters and
datasets with up to $5$ billion
tokens. Surprisingly, we find even when the output is truncated to a single bit, it can be reliably predicted by the model. When multiple distinct PRNGs are presented together during training, the model can jointly learn them, identifying structures from different permutations. We demonstrate a scaling law with modulus $m$: the number of in-context sequence elements required for near-perfect prediction grows as $\sqrt{m}$. For larger
moduli, optimization enters extended stagnation phases; in our experiments, learning
moduli $m \geq 2^{20}$ requires incorporating training data from smaller
moduli, demonstrating a critical necessity for curriculum learning. Finally, we analyze
embedding layers and uncover a novel clustering phenomenon: the top principal components spontaneously group the integer inputs into bitwise rotationally-invariant clusters, revealing how representations can transfer from smaller to larger
moduli.
Read the full abstract ▾
Cryptography and Security
Toward Automated Security Risk Detection in Large Software Using Call Graph Analysis
Nicholas Pecka, Lotfi Ben Othmane and Renee Bryce
Threat modeling plays a critical role in the identification and mitigation of security risks; however, manual approaches are often labor intensive and prone to error. This paper investigates the automation of software threat modeling through the clustering of call graphs using density-based and community detection
algorithms, followed by an analysis of the threats associated with the identified clusters. The proposed method was evaluated through a case study of the Splunk Forwarder Operator (SFO), wherein selected clustering metrics were applied to the software's call graph to assess pertinent code-density security weaknesses. The results demonstrate the viability of the approach and underscore its potential to facilitate systematic threat assessment. This work contributes to the advancement of scalable, semi-automated threat modeling frameworks tailored for modern cloud-native environments.
Read the full abstract ▾
Cryptography and Security
A DRL-Empowered Multi-Level Jamming Approach for Secure Semantic Communication
Weixuan Chen and Qianqian Yang
Semantic communication (SemCom) aims to transmit only task-relevant information, thereby improving communication efficiency but also exposing semantic information to potential eavesdropping. In this paper, we propose a deep reinforcement learning (DRL)-empowered multi-level jamming approach to enhance the security of SemCom systems over
MIMO fading wiretap channels. This approach combines semantic layer jamming, achieved by encoding task-irrelevant text, and physical layer jamming, achieved by encoding random Gaussian noise. These two-level jamming signals are superposed with task-relevant semantic information to protect the transmitted semantics from eavesdropping. A deep deterministic policy gradient (DDPG)
algorithm is further introduced to dynamically design and optimize the precoding
matrices for both task-relevant semantic information and multi-level jamming signals, aiming to enhance the legitimate user's image reconstruction while degrading the eavesdropper's performance. To jointly train the SemCom model and the DDPG
agent, we propose an alternating optimization strategy where the two modules are updated iteratively. Experimental results demonstrate that, compared with both the encryption-based (ESCS) and encoded jammer-based (EJ)
benchmarks, our method achieves comparable security while improving the legitimate user's peak signal-to-noise ratio (PSNR) by up to approximately 0.6 dB.
Read the full abstract ▾
Cryptography and Security
A Comprehensive Evaluation and Practice of System Penetration Testing
Chunyi Zhang, Jin Zeng and Xiaoqi Li
With the rapid advancement of information technology, the complexity of applications continues to increase, and the cybersecurity challenges we face are also escalating. This paper aims to investigate the methods and practices of system security penetration testing, exploring how to enhance system security through systematic penetration testing processes and technical approaches. It also examines existing penetration tools, analyzing their strengths, weaknesses, and applicable domains to guide penetration testers in tool selection. Furthermore, based on the penetration testing process outlined in this paper, appropriate tools are selected to replicate attack processes using target ranges and target machines. Finally, through practical case analysis, lessons learned from successful attacks are summarized to inform future research.
Read the full abstract ▾
Cryptography and Security
Interdependent Privacy in Smart Homes: Hunting for Bystanders in Privacy Policies
Shuaishuai Liu, Gergely Acs and Gergely Biczók
Smart home devices such as
video doorbells and security cameras are becoming increasingly common in everyday life. While these devices offer convenience and safety, they also raise new
privacy concerns: how these devices affect others, like neighbors, visitors, or people passing by. This issue is generally known as interdependent
privacy, where one person's actions (or inaction) may impact the
privacy of others, and, specifically, bystander
privacy in the context of smart homes. Given lax data protection regulations in terms of shared physical spaces and amateur joint data controllers, we expect that the
privacy policies of smart home products reflect the missing regulatory incentives. This paper presents a focused
privacy policy analysis of 20
video doorbell and smart camera products, concentrating explicitly on the bystander aspect. We show that although some of the vendors acknowledge bystanders, they address it only to the extent of including disclaimers, shifting the ethical responsibility for collecting the data of non-users to the device owner. In addition, we identify and examine real-world cases related to bystander
privacy, demonstrating how current deployments can impact non-users. Based on our findings, we analyze vendor
privacy policies in light of existing legal frameworks and technical capabilities, and we provide practical recommendations for both policy
language and system design to enhance transparency and empower both bystanders and device owners.
Read the full abstract ▾
Cryptography and Security
CyberNER: A Harmonized STIX Corpus for Cybersecurity Named Entity Recognition
Yasir Ech-Chammakhy, Anas Motii, Anass Rabii et al. (+2 more)
Extracting structured intelligence via Named Entity Recognition (NER) is critical for cybersecurity, but the proliferation of
datasets with incompatible annotation schemas hinders the development of comprehensive models. While combining these resources is desirable, we empirically demonstrate that naively concatenating them results in a noisy label space that severely degrades model performance. To overcome this critical limitation, we introduce CyberNER, a large-scale, unified corpus created by systematically harmonizing four prominent
datasets (CyNER, DNRTI, APTNER, and Attacker) onto the STIX 2.1 standard. Our principled methodology resolves semantic ambiguities and consolidates over 50 disparate source tags into 21 coherent entity types. Our experiments show that models trained on CyberNER achieve a substantial performance gain, with a relative F1-score improvement of approximately 30% over the naive concatenation baseline. By publicly releasing the CyberNER corpus, we provide a crucial, standardized
benchmark that enables the creation and rigorous comparison of more robust and generalizable entity extraction models for the cybersecurity domain.
Read the full abstract ▾
All authors Yasir Ech-Chammakhy, Anas Motii, Anass Rabii, Oussama Azrara, Jaafar Chbili
Cryptography and Security
SSCL-BW: Sample-Specific Clean-Label Backdoor Watermarking for Dataset Ownership Verification
Yingjia Wang, Ting Qiao, Xing Liu et al. (+3 more)
The rapid advancement of deep neural networks (DNNs) heavily relies on large-scale, high-quality
datasets. However, unauthorized commercial use of these
datasets severely violates the intellectual property rights of
dataset owners. Existing
backdoor-based
dataset ownership verification methods suffer from inherent limitations: poison-label watermarks are easily detectable due to label inconsistencies, while clean-label watermarks face high technical complexity and failure on high-resolution images. Moreover, both approaches employ static watermark patterns that are vulnerable to detection and removal. To address these issues, this paper proposes a sample-specific clean-label
backdoor watermarking (i.e., SSCL-BW). By training a U-Net-based watermarked sample generator, this method generates unique watermarks for each sample, fundamentally overcoming the vulnerability of static watermark patterns. The core innovation lies in designing a composite loss function with three components: target sample loss ensures watermark effectiveness, non-target sample loss guarantees trigger reliability, and perceptual similarity loss maintains visual imperceptibility. During ownership verification, black-box testing is employed to check whether suspicious models exhibit predefined
backdoor behaviors. Extensive experiments on
benchmark datasets demonstrate the effectiveness of the proposed method and its robustness against potential watermark removal attacks.
Read the full abstract ▾
All authors Yingjia Wang, Ting Qiao, Xing Liu, Chongzuo Li, Sixing Wu, Jianbin Li
Cryptography and Security
A Survey of Heterogeneous Graph Neural Networks for Cybersecurity Anomaly Detection
Laura Jiang, Reza Ryan, Qian Li et al. (+1 more)
Anomaly detection is a critical task in cybersecurity, where identifying insider threats, access violations, and coordinated attacks is essential for ensuring system resilience. Graph-based approaches have become increasingly important for modeling entity interactions, yet most rely on homogeneous and static structures, which limits their ability to capture the heterogeneity and temporal
evolution of real-world environments. Heterogeneous Graph Neural Networks (HGNNs) have emerged as a promising paradigm for anomaly detection by incorporating type-aware transformations and relation-sensitive aggregation, enabling more expressive modeling of complex cyber data. However, current research on HGNN-based anomaly detection remains fragmented, with diverse modeling strategies, limited comparative evaluation, and an absence of standardized
benchmarks. To address this gap, we provide a comprehensive survey of HGNN-based anomaly detection methods in cybersecurity. We introduce a taxonomy that classifies approaches by anomaly type and graph dynamics, analyze representative models, and map them to key cybersecurity applications. We also review commonly used
benchmark datasets and evaluation metrics, highlighting their strengths and limitations. Finally, we identify key open challenges related to modeling, data, and deployment, and outline promising directions for future research. This survey aims to establish a structured foundation for advancing HGNN-based anomaly detection toward scalable, interpretable, and practically deployable solutions.
Read the full abstract ▾
All authors Laura Jiang, Reza Ryan, Qian Li, Nasim Ferdosian
Cryptography and Security
PVMark: Enabling Public Verifiability for LLM Watermarking Schemes
Haohua Duan, Liyao Xiang and Xin Zhang
Watermarking schemes for large
language models (LLMs) have been proposed to identify the source of the generated text, mitigating the potential threats emerged from model theft. However, current watermarking solutions hardly resolve the trust issue: the non-public watermark detection cannot prove itself faithfully conducting the detection. We observe that it is attributed to the secret key mostly used in the watermark detection – it cannot be public, or the adversary may launch removal attacks provided the key; nor can it be private, or the watermarking detection is opaque to the public. To resolve the dilemma, we propose PVMark, a plugin based on zero-knowledge proof (ZKP), enabling the watermark detection process to be publicly verifiable by third parties without disclosing any secret key. PVMark hinges upon the proof of `correct execution' of watermark detection on which a set of ZKP constraints are built, including mapping, random number generation, comparison, and summation. We implement multiple variants of PVMark in Python, Rust and Circom, covering combinations of three watermarking schemes, three hash functions, and four ZKP protocols, to show our approach effectively works under a variety of circumstances. By experimental results, PVMark efficiently enables public verifiability on the state-of-the-art LLM watermarking schemes yet without compromising the watermarking performance, promising to be deployed in practice.
Read the full abstract ▾
Cryptography and Security
Who Grants the Agent Power? Defending Against Instruction Injection via Task-Centric Access Control
Yifeng Cai, Ziming Wang, Zhaomeng Deng et al. (+6 more)
AI
agents capable of GUI understanding and Model Context Protocol are increasingly deployed to automate mobile tasks. However, their reliance on over-privileged, static permissions creates a critical vulnerability: instruction injection. Malicious instructions, embedded in otherwise benign content like emails, can hijack the
agent to perform unauthorized actions. We present AgentSentry, a lightweight runtime task-centric access control framework that enforces dynamic, task-scoped permissions. Instead of granting broad, persistent permissions, AgentSentry dynamically generates and enforces minimal, temporary policies aligned with the user's specific task (e.g., register for an app), revoking them upon completion. We demonstrate that AgentSentry successfully prevents an instruction injection attack, where an
agent is tricked into forwarding private emails, while allowing the legitimate task to complete. Our approach highlights the urgent need for intent-aligned security models to safely govern the next generation of autonomous
agents.
Read the full abstract ▾
All authors Yifeng Cai, Ziming Wang, Zhaomeng Deng, Mengyu Yao, Junlin Liu, Yutao Hu, Ziqi Zhang, Yao Guo, Ding Li
Cryptography and Security
Who Moved My Transaction? Uncovering Post-Transaction Auditability Vulnerabilities in Modern Super Apps
Junlin Liu, Zhaomeng Deng, Ziming Wang et al. (+6 more)
Super apps are the cornerstones of modern digital life,
embedding financial transactions into nearly every aspect of daily routine. The prevailing security paradigm for these platforms is overwhelmingly focused on pre-transaction authentication, preventing unauthorized payments before they occur. We argue that a critical vulnerability vector has been largely overlooked: the fragility of post-transaction audit trails. We investigate the ease with which a user can permanently erase their transaction history from an app's interface, thereby concealing unauthorized or sensitive activities from the account owner. To quantify this threat, we conducted an empirical study with 6 volunteers who performed a cross-evaluation on six super apps. Our findings are alarming: all six applications studied allow users to delete transaction records, yet a staggering five out of six (83+%) fail to protect these records with strong authentication. Only one app in our study required biometric verification for deletion. This study provides the first concrete evidence of this near-ubiquitous vulnerability, demonstrating a critical gap in the current mobile security landscape and underscoring the urgent need for a paradigm shift towards ensuring post-transaction audit integrity.
Read the full abstract ▾
All authors Junlin Liu, Zhaomeng Deng, Ziming Wang, Mengyu Yao, Yifeng Cai, Yutao Hu, Ziqi Zhang, Yao Guo, Ding Li
Cryptography and Security
Confidential FRIT via Homomorphic Encryption
Haruki Hoshino, Jungjin Park, Osamu Kaneko et al. (+1 more)
Edge computing alleviates the computation burden of data-driven control in cyber-physical systems (CPSs) by offloading complex processing to edge servers. However, the increasing sophistication of cyberattacks underscores the need for security measures that go beyond conventional IT protections and address the unique vulnerabilities of CPSs. This study proposes a confidential data-driven gain-tuning framework using homomorphic encryption, such as ElGamal and CKKS encryption schemes, to enhance cybersecurity in gain-tuning processes outsourced to external servers. The idea for realizing confidential FRIT is to replace the
matrix inversion operation with a vector summation form, allowing homomorphic operations to be applied. Numerical examples under 128-bit security confirm performance comparable to conventional methods while providing guidelines for selecting suitable encryption schemes for secure CPS.
Read the full abstract ▾
All authors Haruki Hoshino, Jungjin Park, Osamu Kaneko, Kiminao Kogiso
Cryptography and Security · Cross-listed from cs.CV
Security Risk of Misalignment between Text and Image in Multi-modal Model
Xiaosen Wang, Zhijin Ge and Shaokang Wang
Despite the notable advancements and versatility of multi-modal
diffusion models, such as text-to-image models, their susceptibility to
adversarial inputs remains underexplored. Contrary to expectations, our investigations reveal that the
alignment between textual and Image modalities in existing
diffusion models is inadequate. This misalignment presents significant risks, especially in the generation of inappropriate or Not-Safe-For-Work (NSFW) content. To this end, we propose a novel attack called
Prompt-Restricted Multi-modal Attack (PReMA) to manipulate the generated content by modifying the input image in conjunction with any specified
prompt, without altering the
prompt itself. PReMA is the first attack that manipulates model outputs by solely creating
adversarial images, distinguishing itself from prior methods that primarily generate
adversarial prompts to produce NSFW content. Consequently, PReMA poses a novel threat to the integrity of multi-modal
diffusion models, particularly in image-editing applications that operate with fixed
prompts. Comprehensive evaluations conducted on image inpainting and style transfer tasks across various models confirm the potent efficacy of PReMA.
Read the full abstract ▾
Cryptography and Security
Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories
Maximilian Schreiber and Pascal Tippe
This paper presents a comprehensive empirical analysis of security vulnerabilities in AI-generated code across public GitHub repositories. We collected and analyzed 7,703 files explicitly attributed to four major AI tools: ChatGPT (91.52%), GitHub Copilot (7.50%), Amazon CodeWhisperer (0.52%), and Tabnine (0.46%). Using CodeQL static analysis, we identified 4,241 Common Weakness Enumeration (CWE) instances across 77 distinct vulnerability types. Our findings reveal that while 87.9% of AI-generated code does not contain identifiable CWE-mapped vulnerabilities, significant patterns emerge regarding
language-specific vulnerabilities and tool performance. Python consistently exhibited higher vulnerability rates (16.18%-18.50%) compared to JavaScript (8.66%-8.99%) and TypeScript (2.50%-7.14%) across all tools. We observed notable differences in security performance, with GitHub Copilot achieving better security density for Python (1,739 LOC per CWE) and TypeScript, while ChatGPT performed better for JavaScript. Additionally, we discovered widespread use of AI tools for documentation generation (39% of collected files), an understudied application with implications for software maintainability. These findings extend previous work with a significantly larger
dataset and provide valuable insights for developing
language-specific and context-aware security practices for the responsible integration of AI-generated code into software development workflows.
Read the full abstract ▾
Cryptography and Security
PEEL: A Poisoning-Exposing Encoding Theoretical Framework for Local Differential Privacy
Lisha Shuai, Jiuling Dong, Nan Zhang et al. (+5 more)
Local Differential
Privacy (LDP) is a widely adopted
privacy-protection model in the Internet of Things (IoT) due to its lightweight, decentralized, and scalable nature. However, it is vulnerable to poisoning attacks, and existing defenses either incur prohibitive resource overheads or rely on domain-specific prior knowledge, limiting their practical deployment. To address these limitations, we propose PEEL, a Poisoning-Exposing Encoding theoretical framework for LDP, which departs from resource- or prior-dependent countermeasures and instead leverages the inherent structural consistency of LDP-perturbed data. As a non-intrusive post-processing module, PEEL amplifies stealthy poisoning effects by re-encoding LDP-perturbed data via sparsification, normalization, and low-rank projection, thereby revealing both output and rule poisoning attacks through structural inconsistencies in the reconstructed space. Theoretical analysis proves that PEEL, integrated with LDP, retains unbiasedness and statistical accuracy, while being robust to expose both output and rule poisoning attacks. Moreover, evaluation results show that LDP-integrated PEEL not only outperforms four state-of-the-art defenses in terms of poisoning exposure accuracy but also significantly reduces client-side computational costs, making it highly suitable for large-scale IoT deployments.
Read the full abstract ▾
All authors Lisha Shuai, Jiuling Dong, Nan Zhang, Shaofeng Tan, Haokun Zhang, Zilong Song, Gaoya Dong, Xiaolong Yang
Cryptography and Security · Cross-listed from cs.SD
ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models
Weifei Jin, Yuxin Cao, Junjie Su et al. (+5 more)
Recent advances in Audio-
Language Models (ALMs) have significantly improved
multimodal understanding capabilities. However, the introduction of the audio modality also brings new and unique vulnerability vectors. Previous studies have proposed jailbreak attacks that specifically target ALMs, revealing that defenses directly transferred from traditional audio
adversarial attacks or text-based Large
Language Model (LLM) jailbreaks are largely ineffective against these ALM-specific threats. To address this issue, we propose ALMGuard, the first defense framework tailored to ALMs. Based on the assumption that safety-aligned shortcuts naturally exist in ALMs, we design a method to identify universal Shortcut Activation Perturbations (SAPs) that serve as triggers that activate the safety shortcuts to safeguard ALMs at inference time. To better sift out effective triggers while preserving the model's utility on benign tasks, we further propose Mel-Gradient Sparse Mask (M-GSM), which restricts perturbations to Mel-frequency bins that are sensitive to jailbreaks but insensitive to
speech understanding. Both theoretical analyses and empirical results demonstrate the robustness of our method against both seen and unseen attacks. Overall, \MethodName reduces the average success rate of advanced ALM-specific jailbreak attacks to 4.6% across four models, while maintaining comparable utility on benign
benchmarks, establishing it as the new state of the art. Our code and data are available at https://github.com/WeifeiJin/ALMGuard.
Read the full abstract ▾
All authors Weifei Jin, Yuxin Cao, Junjie Su, Minhui Xue, Jie Hao, Ke Xu, Jin Song Dong, Derui Wang
Cryptography and Security
SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar et al. (+1 more)
The ability of LLM
agents to plan and invoke tools exposes them to new safety risks, making a comprehensive red-teaming system crucial for discovering vulnerabilities and ensuring their safe deployment. We present SIRAJ: a generic red-teaming framework for arbitrary black-box LLM
agents. We employ a dynamic two-step process that starts with an
agent definition and generates diverse seed test cases that cover various risk outcomes, tool-use trajectories, and risk sources. Then, it iteratively constructs and refines model-based
adversarial attacks based on the execution trajectories of former attempts. To optimize the red-teaming cost, we present a model
distillation approach that leverages structured forms of a teacher model's
reasoning to train smaller models that are equally effective. Across diverse evaluation
agent settings, our seed test case generation approach yields 2 – 2.5x boost to the coverage of risk outcomes and tool-calling trajectories. Our distilled 8B red-teamer model improves attack success rate by 100%, surpassing the 671B Deepseek-R1 model. Our ablations and analyses validate the effectiveness of the iterative framework, structured
reasoning, and the generalization of our red-teamer models.
Read the full abstract ▾
All authors Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar, Amin Saied