Cryptography and Security · Cross-listed from cs.HC
How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape
Patrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby et al. (+2 more)
Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of attacks, lowering the barrier to entry for creating harmful content, including sophisticated propaganda and deepfakes. Conversely, defenders envision leveraging GenAI to detect and mitigate harmful content at scale, conduct investigations, deploy persuasive counternarratives, improve moderator wellbeing, and offer user support. This work provides a strategic framework for understanding GenAI's impact on Trust & Safety and charts a path for its responsible use in creating safer online environments.
Read the full abstract ▾
All authors Patrick Gage Kelley, Steven Rousso-Schindler, Renee Shelby, Kurt Thomas, Allison Woodruff
Cryptography and Security
QOC DAO – Stepwise Development Towards an AI Driven Decentralized Autonomous Organization
Marc Jansen and Christophe Verdot
This paper introduces a structured approach to improving decision making in Decentralized Autonomous Organizations (DAO) through the integration of the Question-Option-Criteria (QOC) model and AI
agents. We outline a stepwise governance framework that evolves from human led evaluations to fully autonomous, AI-driven processes. By decomposing decisions into weighted, criterion based evaluations, the QOC model enhances transparency,
fairness, and explainability in DAO voting. We demonstrate how large
language models (LLMs) and stakeholder aligned AI
agents can support or automate evaluations, while statistical safeguards help detect manipulation. The proposed framework lays the foundation for scalable and trustworthy governance in the Web3 ecosystem.
Read the full abstract ▾
Cryptography and Security · Cross-listed from cs.CY
How Do Data Owners Say No? A Case Study of Data Consent Mechanisms in Web-Scraped Vision-Language AI Training Datasets
Chung Peng Lee, Rachel Hong, Harry H. Jiang et al. (+3 more)
The internet has become the main source of data to train modern text-to-image or
vision-
language models, yet it is increasingly unclear whether web-scale data collection practices for training AI systems adequately respect data owners' wishes. Ignoring the owner's indication of consent around data usage not only raises ethical concerns but also has recently been elevated into lawsuits around copyright infringement cases. In this work, we aim to reveal information about data owners' consent to AI scraping and training, and study how it's expressed in DataComp, a popular
dataset of 12.8 billion text-image pairs. We examine both the sample-level information, including the copyright notice, watermarking, and metadata, and the web-domain-level information, such as a site's Terms of Service (ToS) and
Robots Exclusion Protocol. We estimate at least 122M of samples exhibit some indication of copyright notice in CommonPool, and find that 60% of the samples in the top 50 domains come from websites with ToS that prohibit scraping. Furthermore, we estimate 9-13% with 95% confidence interval of samples from CommonPool to contain watermarks, where existing watermark detection methods fail to capture them in high fidelity. Our holistic methods and findings show that data owners rely on various channels to convey data consent, of which current AI data collection pipelines do not entirely respect. These findings highlight the limitations of the current
dataset curation/release practice and the need for a unified data consent framework taking AI purposes into consideration.
Read the full abstract ▾
All authors Chung Peng Lee, Rachel Hong, Harry H. Jiang, Aster Plotnik, William Agnew, Jamie Morgenstern
Cryptography and Security · Cross-listed from cs.SE
A Self-Improving Architecture for Dynamic Safety in Large Language Models
Tyler Slater
Context: Large
Language Models (LLMs) rely on static, pre-deployment safety mechanisms that cannot adapt to
adversarial threats discovered after release. Objective: To design a software architecture enabling LLM-based systems to autonomously detect safety failures and synthesize defense policies at runtime, without retraining or manual intervention. Method: We propose the Self-Improving Safety Framework (SISF), grounded in the MAPE-K reference model. The framework couples a target LLM with a feedback loop: an Adjudicator detects breaches, a Policy Synthesis Module generates dual-mechanism defense policies (heuristic and semantic), and a Warden enforces them. We conducted seven experiments (10,061 evaluations) across four model families. Results: Across five reproducibility trials, SISF achieved a mean Attack Success Rate (ASR) of 0.27% (+/-0.15%), autonomously generating 240 policies per trial. Cross-model evaluation confirmed deployment portability. A held-out test showed a 68.5% proactive interception rate on unseen attacks. Stacked behind Llama Guard 4, the combined defense reduced residual ASR from 7.88% to 0.00%. Ablation confirmed both heuristic and semantic policy types are architecturally required. Conclusion: Self-adaptive architecture is a viable approach to LLM safety. SISF achieves sub-1% ASR through synchronous output monitoring, progressively shifting enforcement to fast, local Warden policies via the MAPE-K loop, offering a new pattern for building resilient AI systems.
Read the full abstract ▾
Cryptography and Security · Cross-listed from cs.LG
Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Private
Ruihan Wu, Erchi Wang, Zhiyuan Zhang et al. (+1 more)
Retrieval-augmented generation (RAG) enhances large
language models (LLMs) by retrieving documents from an external corpus at inference time. When this corpus contains sensitive information, however, unprotected RAG systems are at risk of leaking private information. Prior work has introduced differential
privacy (DP) guarantees for RAG, but only in single-query settings, which fall short of realistic usage. In this paper, we study the more practical multi-query setting and propose two DP-RAG
algorithms. The first, MURAG, leverages an individual
privacy filter so that the accumulated
privacy loss only depends on how frequently each document is retrieved rather than the total number of queries. The second, MURAG-ADA, further improves utility by privately releasing query-specific thresholds, enabling more precise selection of relevant documents. Our experiments across multiple LLMs and
datasets demonstrate that the proposed methods scale to hundreds of queries within a practical DP budget ($\varepsilon\approx10$), while preserving meaningful utility.
Read the full abstract ▾
All authors Ruihan Wu, Erchi Wang, Zhiyuan Zhang, Yu-Xiang Wang
Cryptography and Security
A Decentralized Retrieval Augmented Generation System with Source Reliabilities Secured on Blockchain
Yining Lu, Wenyi Tang, Max Johnson et al. (+2 more)
Existing
retrieval-augmented generation (RAG) systems typically use a centralized architecture, causing a high cost of data collection, integration, and management, as well as
privacy concerns. There is a great need for a decentralized RAG system that enables foundation models to utilize information directly from data owners who maintain full control over their sources. However, decentralization brings a challenge: the numerous independent data sources vary significantly in reliability, which can diminish
retrieval accuracy and response quality. To address this, our decentralized RAG system has a novel reliability scoring mechanism that dynamically evaluates each source based on the quality of responses it contributes to generate and prioritizes high-quality sources during
retrieval. To ensure transparency and trust, the scoring process is securely managed through blockchain-based smart contracts, creating verifiable and tamper-proof reliability records without relying on a central authority. We evaluate our decentralized system with two Llama models (3B and 8B) in two simulated environments where six data sources have different levels of reliability. Our system achieves a +10.7% performance improvement over its centralized counterpart in the real world-like unreliable data environments. Notably, it approaches the upper-bound performance of centralized systems under ideally reliable data environments. The decentralized infrastructure enables secure and trustworthy scoring management, achieving approximately 56% marginal cost savings through batched update operations. Our code and system are open-sourced at github.com/yining610/Reliable-dRAG.
Read the full abstract ▾
All authors Yining Lu, Wenyi Tang, Max Johnson, Taeho Jung, Meng Jiang
Cryptography and Security
LSEG: A Lightweight and Secure Key Exchange Protocol for Smart Grid Communication
Amna Zafar, Muhammad Asfand Hafeez and Arslan Munir
The increasing deployment of the Internet of Things (IoT) edge devices in modern smart grid environments requires secure and efficient communication protocols specifically designed for resource-constrained environments. However, most existing authentication schemes either impose excessive computational overhead or lack robustness against advanced cyber threats, making them unsuitable for resource-limited smart grid deployments. To address these limitations, this paper proposes a lightweight authentication and secure key exchange protocol for smart grid (LSEG) environments. The proposed LSEG protocol utilizes a unified
elliptic curve key pair, enabled by birational mapping between Ed25519 and Curve25519, for signing and key exchange. Initial keys are derived using the hash based message authentication code (HMAC) based key derivation function (HKDF), while ephemeral key pairs, generated through the
Elliptic Curve Diffie Hellman Ephemeral (ECDHE), are used in each session to ensure forward secrecy. Session communication is protected using ASCON128a, a lightweight, NIST-standardized, authenticated encryption
algorithm. Formal security proofs in the random oracle model validate the security properties of LSEG, including mutual authentication, forward secrecy, and resistance to impersonation, replay, and man in the middle attacks. Experimental results on both Raspberry Pi and Intel Core i9-based systems demonstrate practical efficiency, achieving execution times under 5.5 milliseconds on embedded hardware and a communication cost of only 1024 bits for the protocol's message exchanges. The results demonstrate that LSEG effectively balances security, efficiency, and compliance, making it a scalable solution for secure communication in smart grid infrastructures.
Read the full abstract ▾
Cryptography and Security
FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
Pukang Ye, Junwei Luo, Xiaolei Dong et al. (+1 more)
Data duplication within large-scale corpora often impedes large
language models' (LLMs) performance and
privacy. In
privacy-concerned
federated learning scenarios, conventional deduplication methods typically rely on trusted third parties to perform uniform deletion, risking loss of informative samples while introducing
privacy vulnerabilities. To address these gaps, we propose
Federated ReWeighting (FedRW), the first
privacy-preserving framework, to the best of our knowledge, that performs soft deduplication via sample reweighting instead of deletion in
federated LLM training, without assuming a trusted third party. At its core, FedRW proposes a secure, frequency-aware reweighting protocol through secure multi-party computation, coupled with a parallel orchestration strategy to ensure efficiency and scalability. During training, FedRW utilizes an adaptive reweighting mechanism with global sample frequencies to adjust individual loss contributions, effectively improving generalization and robustness. Empirical results demonstrate that FedRW outperforms the state-of-the-art method by achieving up to 28.78x speedup in preprocessing and approximately 11.42% improvement in perplexity, while offering enhanced security guarantees. FedRW thus establishes a new paradigm for managing duplication in
federated LLM training.
Read the full abstract ▾
All authors Pukang Ye, Junwei Luo, Xiaolei Dong, Yunbo Yang
Cryptography and Security
Biologically-Informed Hybrid Membership Inference Attacks on Generative Genomic Models
Asia Belfiore, Jonathan Passerat-Palmbach and Dmitrii Usynin
The increased availability of genetic data has transformed genomics research, but raised many
privacy concerns regarding its handling due to its sensitive nature. This work explores the use of
language models (LMs) for the generation of synthetic genetic mutation profiles, leveraging differential
privacy (DP) for the protection of sensitive genetic data. We empirically evaluate the
privacy guarantees of our DP modes by introducing a novel Biologically-Informed Hybrid Membership Inference Attack (biHMIA), which combines traditional black box MIA with contextual genomics metrics for enhanced attack power. Our experiments show that both small and large
transformer GPT-like models are viable synthetic variant generators for small-scale genomics, and that our hybrid attack leads, on average, to higher
adversarial success compared to traditional metric-based MIAs.
Read the full abstract ▾
Cryptography and Security
JPRO: Automated Multimodal Jailbreaking via Multi-Agent Collaboration Framework
Yuxuan Zhou, Yang Bai, Kuofeng Gao et al. (+2 more)
The widespread application of large VLMs makes ensuring their secure deployment critical. While recent studies have demonstrated jailbreak attacks on VLMs, existing approaches are limited: they require either white-box access, restricting practicality, or rely on manually crafted patterns, leading to poor sample diversity and scalability. To address these gaps, we propose JPRO, a novel multi-
agent collaborative framework designed for automated VLM jailbreaking. It effectively overcomes the shortcomings of prior methods in attack diversity and scalability. Through the coordinated action of four specialized
agents and its two core modules: Tactic-Driven Seed Generation and Adaptive Optimization Loop, JPRO generates effective and diverse attack samples. Experimental results show that JPRO achieves over a 60% attack success rate on multiple advanced VLMs, including GPT-4o, significantly outperforming existing methods. As a black-box attack approach, JPRO not only uncovers critical security vulnerabilities in
multimodal models but also offers valuable insights for evaluating and enhancing VLM robustness.
Read the full abstract ▾
All authors Yuxuan Zhou, Yang Bai, Kuofeng Gao, Tao Dai, Shu-Tao Xia
Cryptography and Security
Privacy on the Fly: A Predictive Adversarial Transformation Network for Mobile Sensor Data
Tianle Song, Chenhao Lin, Yang Cao et al. (+5 more)
Mobile motion sensors such as accelerometers and gyroscopes are now ubiquitously accessible by third-party apps via standard APIs. While enabling rich functionalities like activity recognition and step counting, this openness has also enabled unregulated inference of sensitive user traits, such as gender, age, and even identity, without user consent. Existing
privacy-preserving techniques, such as GAN-based obfuscation or differential
privacy, typically require access to the full input sequence, introducing latency that is incompatible with real-time scenarios. Worse, they tend to distort temporal and semantic patterns, degrading the utility of the data for benign tasks like activity recognition. To address these limitations, we propose the Predictive
Adversarial Transformation Network (PATN), a real-time
privacy-preserving framework that leverages historical signals to generate
adversarial perturbations proactively. The perturbations are applied immediately upon data acquisition, enabling continuous protection without disrupting application functionality. Experiments on two
datasets demonstrate that PATN substantially degrades the performance of
privacy inference models, achieving Attack Success Rate (ASR) of 40.11% and 44.65% (reducing inference accuracy to near-random) and increasing the Equal Error Rate (EER) from 8.30% and 7.56% to 41.65% and 46.22%. On ASR, PATN outperforms baseline methods by 16.16% and 31.96%, respectively.
Read the full abstract ▾
All authors Tianle Song, Chenhao Lin, Yang Cao, Zhengyu Zhao, Jiahao Sun, Chong Zhang, Le Yang, Chao Shen
Cryptography and Security · Cross-listed from cs.CV
Breaking the Stealth-Potency Trade-off in Clean-Image Backdoors with Generative Trigger Optimization
Binyan Xu, Fan Yang, Di Tang et al. (+2 more)
Clean-image
backdoor attacks, which use only label manipulation in training
datasets to compromise deep neural networks, pose a significant threat to security-critical applications. A critical flaw in existing methods is that the poison rate required for a successful attack induces a proportional, and thus noticeable, drop in Clean Accuracy (CA), undermining their stealthiness. This paper presents a new paradigm for clean-image attacks that minimizes this accuracy degradation by optimizing the trigger itself. We introduce Generative Clean-Image
Backdoors (GCB), a framework that uses a conditional InfoGAN to identify naturally occurring image features that can serve as potent and stealthy triggers. By ensuring these triggers are easily separable from benign task-related features, GCB enables a victim model to learn the
backdoor from an extremely small set of poisoned examples, resulting in a CA drop of less than 1%. Our experiments demonstrate GCB's remarkable versatility, successfully adapting to six
datasets, five architectures, and four tasks, including the first demonstration of clean-image
backdoors in regression and
segmentation. GCB also exhibits resilience against most of the existing
backdoor defenses.
Read the full abstract ▾
All authors Binyan Xu, Fan Yang, Di Tang, Xilin Dai, Kehuan Zhang
Cryptography and Security · Cross-listed from cs.CV
LiteUpdate: A Lightweight Framework for Updating AI-Generated Image Detectors
Jiajie Lu, Zhenkan Fu, Na Zhao et al. (+4 more)
The rapid progress of generative AI has led to the emergence of new generative models, while existing detection methods struggle to keep pace, resulting in significant degradation in the detection performance. This highlights the urgent need for continuously updating AI-generated image detectors to adapt to new generators. To overcome low efficiency and catastrophic forgetting in detector updates, we propose LiteUpdate, a lightweight framework for updating AI-generated image detectors. LiteUpdate employs a representative sample selection module that leverages image confidence and gradient-based discriminative features to precisely select boundary samples. This approach improves learning and detection accuracy on new distributions with limited generated images, significantly enhancing detector update efficiency. Additionally, LiteUpdate incorporates a model merging module that fuses weights from multiple fine-tuning trajectories, including pre-trained, representative, and random updates. This balances the adaptability to new generators and mitigates the catastrophic forgetting of prior knowledge. Experiments demonstrate that LiteUpdate substantially boosts detection performance in various detectors. Specifically, on AIDE, the average detection accuracy on Midjourney improved from 87.63% to 93.03%, a 6.16% relative increase.
Read the full abstract ▾
All authors Jiajie Lu, Zhenkan Fu, Na Zhao, Long Xing, Kejiang Chen, Weiming Zhang, Nenghai Yu
Cryptography and Security · Cross-listed from cs.LG
On Stealing Graph Neural Network Models
Marcin Podhajski, Jan Dubiński, Franziska Boenisch et al. (+3 more)
Current graph neural network (GNN) model-stealing methods rely heavily on queries to the victim model, assuming no hard query limits. However, in reality, the number of allowed queries can be severely limited. In this paper, we demonstrate how an adversary can extract a GNN with very limited interactions with the model. Our approach first enables the adversary to obtain the model backbone without making direct queries to the victim model and then to strategically utilize a fixed query limit to extract the most informative data. The experiments on eight real-world
datasets demonstrate the effectiveness of the attack, even under a very restricted query limit and under defense against model extraction in place. Our findings underscore the need for robust defenses against GNN model extraction threats.
Read the full abstract ▾
All authors Marcin Podhajski, Jan Dubiński, Franziska Boenisch, Adam Dziedzic, Agnieszka Pręgowska, Tomasz P. Michalak
Cryptography and Security
Harnessing Sparsification in Federated Learning: A Secure, Efficient, and Differentially Private Realization
Shuangqing Xu, Yifeng Zheng and Zhongyun Hua
Federated learning (FL) enables multiple clients to jointly train a model by sharing only gradient updates for aggregation instead of raw data. Due to the transmission of very high-dimensional gradient updates from many clients, FL is known to suffer from a communication bottleneck. Meanwhile, the gradients shared by clients as well as the trained model may also be exploited for inferring private local
datasets, making
privacy still a critical concern in FL. We present Clover, a novel system framework for communication-efficient, secure, and differentially private FL. To tackle the communication bottleneck in FL, Clover follows a standard and commonly used approach-top-k gradient sparsification, where each client sparsifies its gradient update such that only k largest gradients (measured by magnitude) are preserved for aggregation. Clover provides a tailored mechanism built out of a trending distributed trust setting involving three servers, which allows to efficiently aggregate multiple sparse vectors (top-k sparsified gradient updates) into a dense vector while hiding the values and indices of non-zero elements in each sparse vector. This mechanism outperforms a baseline built on the general distributed ORAM technique by several orders of magnitude in server-side communication and runtime, with also smaller client communication cost. We further integrate this mechanism with a lightweight distributed noise generation mechanism to offer differential
privacy (DP) guarantees on the trained model. To harden Clover with security against a malicious server, we devise a series of lightweight mechanisms for integrity checks on the server-side computation. Extensive experiments show that Clover can achieve utility comparable to vanilla FL with central DP, with promising performance.
Read the full abstract ▾
Cryptography and Security · Cross-listed from cs.SD
E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
Zhisheng Zhang, Derui Wang, Yifan Mi et al. (+6 more)
Recent advancements in
speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world applications. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to address the production large
language model (LLM)-based
speech synthesis. While previous studies have considered the protection for fine-tuning synthesizers, they assume manually annotated transcripts. Given the labor intensity of manual annotation, end-to-end (E2E) systems leveraging automatic
speech recognition (ASR) to generate transcripts are becoming increasingly prevalent, e.g., voice cloning via commercial APIs. Therefore, this E2E
speech synthesis also requires new security mechanisms. To tackle these challenges, we propose E2E-VGuard, a proactive defense framework for two emerging threats: (1) production LLM-based
speech synthesis, and (2) the novel attack arising from ASR-driven E2E scenarios. Specifically, we employ the encoder ensemble with a feature extractor to protect timbre, while ASR-targeted
adversarial examples disrupt pronunciation. Moreover, we incorporate the psychoacoustic model to ensure perturbative imperceptibility. For a comprehensive evaluation, we test 16 open-source synthesizers and 3 commercial APIs across Chinese and English
datasets, confirming E2E-VGuard's effectiveness in timbre and pronunciation protection. Real-world deployment validation is also conducted. Our code and demo page are available at https://wxzyd123.github.io/e2e-vguard/.
Read the full abstract ▾
All authors Zhisheng Zhang, Derui Wang, Yifan Mi, Zhiyong Wu, Jie Gao, Yuxin Cao, Kai Ye, Minhui Xue, Jie Hao