Computation and Language
PersonaTree: Structured Lifecycle Memory for Person Understanding in LLM Agents
Yubo Hou, Jingwei Song, Hongbo Zhang et al. (+4 more)
Persistent LLM
agents require
memory representations that make the formation of person understanding explicit across long term interaction. Existing
agent memory methods emphasize information retention and
retrieval, yet give limited account of how accumulated interaction evidence is abstracted into person understanding. We view this process as schema formation, where situated evidence is abstracted into reusable patterns and stable person level claims. We introduce PersonaTree, a structured lifecycle
memory framework that realizes this view as a three level persona tree with explicit support paths from evidence to claims. PersonaTree maintains the tree through conservative writing, confidence guided consolidation, and query conditioned path
retrieval, returning only the evidence depth required by each query. Across six person understanding and persistent
memory benchmarks with three answer backbones, PersonaTree ranks first in 12 of 18 compact scores and reaches the top two in 16 settings. Ablations show that hierarchy improves abstract person understanding on KnowMe, while support path
retrieval improves RealPref
alignment under a comparable context budget.
Read the full abstract ▾
All authors Yubo Hou, Jingwei Song, Hongbo Zhang, Zhisheng Chen, Bang Xiao, Tao Wan, Zengchang Qin
Computation and Language · Cross-listed from cs.AI
Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
Kyungmin Park and Taesup Kim
Safety-aligned Large
Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where
alignment concentrates in the first few output
tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short
token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's
alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-
token generation. Our work argues that robust safety
alignment requires training on the generation process itself, not only its outputs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
Yong Cao, Chuqiao Li, Xianghui Xie et al. (+2 more)
Reliable evaluation of human motion understanding is fundamental to advancing
embodied AI, robotics, and animation. However, existing
benchmarks suffer from coarse semantic granularity, undifferentiated difficulty, limited annotation quality, and pervasive answer ambiguity, leaving them unable to diagnose where current models fail. To bridge this gap, we introduce NextMotionQA, a comprehensive
benchmark that leverages
vision-
language models (VLMs) for semi-automated, expert-verified
dataset. NextMotionQA features three complementary tasks: multiple-choice question answering,
video captioning, and fine-grained error correction. Each task is systematically structured across three core semantic axes and stratified into three task complexity levels. Our extensive evaluation of twelve representative VLMs uncovers critical capability gaps and weakness that remain invisible under conventional, single-task evaluations. In a complementary direction, recent work has begun using VLMs as judges for text-to-motion evaluation; we ask whether they show the same degradation under harder tasks. We find that VLMs align strongly with expert ratings on coarse criteria (Cohen's \kappa=0.70) but break down on fine-grained, part-level judgment (\kappa=0.10), validating the paradigm in its strong regime while clarifying its limits.
Read the full abstract ▾
All authors Yong Cao, Chuqiao Li, Xianghui Xie, Gerard Pons-Moll, Andreas Geiger
Computation and Language
TIDE: Proactive Multi-Problem Discovery via Template-Guided Iteration
Soyeong Jeong, Jinheon Baek, Minki Kang et al. (+1 more)
Agents are widely deployed as assistants over documents, tools, and code. However, they typically act only on explicit user requests, which surface only the problems the user has noticed, while many other important problems coexist, hidden in plain sight, within the broader user context, with their total number unknown in advance. We frame this as the task of discovering multiple hidden problems from context, in which coexisting problems should be uncovered, grounded in supporting evidence, and paired with concrete actions. To this end, we introduce TIDE, a template-guided iterative framework with two complementary mechanisms. Specifically, motivated by the observation that single-pass prediction anchors on the most salient cases and yields generic claims, we propose iterative discovery, which surfaces a small batch of candidates per round while conditioning on what has already been found, so subsequent rounds extend coverage; and thought templates, reusable schemas distilled from previously solved cases that specify what contextual signals to attend to and how to connect them, anchoring each prediction in a recognizable problem class. We validate TIDE on two realistic settings, personal workspaces and software repositories, across four model backbones, showing substantial gains over single-shot and parallel multi-
agent baselines on task coverage, identification, and resolution.
Read the full abstract ▾
All authors Soyeong Jeong, Jinheon Baek, Minki Kang, Sung Ju Hwang
Computation and Language
Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026
Enes Yavuz Ugan, Maike Züfle, Yuka Ko et al. (+5 more)
With the advent of Large
Language Models, single-task and
token-based multi-task models have evolved into instruction-based systems that infer task and target
language implicitly from natural
language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four
languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.
Read the full abstract ▾
All authors Enes Yavuz Ugan, Maike Züfle, Yuka Ko, Supriti Sinhamahapatra, Fabian Retkowski, Seymanur Akti, Jan Niehues, Alexander Waibel
Computation and Language
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM
SooHwan Eom, Jay Shim, Gwanhyeong Koo et al. (+4 more)
The
Transformer's quadratic complexity with input length imposes an unsustainable computational load on large
language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for
vision-
language modeling by compressing visual
tokens based on input through the cross-
attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various
vision-
language understanding
benchmarks show that the proposed cross-modal projector enhances Mamba-based
multimodal LLMs, boosting both performance and throughput.
Read the full abstract ▾
All authors SooHwan Eom, Jay Shim, Gwanhyeong Koo, Haebin Na, Mark A. Hasegawa-Johnson, Sungwoong Kim, Chang D. Yoo
Computation and Language
Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
Jingwen Chen, Wenkai Yang, Shengda Fan et al. (+7 more)
Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large
language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-
distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-
distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.
Read the full abstract ▾
All authors Jingwen Chen, Wenkai Yang, Shengda Fan, Wenbo Nie, Chenxing Sun, Shaodong Zheng, Yangen Hu, Lu Pan, Ke Zeng, Yankai Lin
Computation and Language · Cross-listed from cs.CV
Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms
Jiashu Yao, Heyan Huang, Daiqing Wu et al. (+5 more)
GUI
agents today assume a static screen, where the world is frozen between two actions. However, real interfaces such as short-
video applications violate this assumption, as their content keeps playing, and a competent user must decide what to watch and for how long. We formalize this task as Living-Screen-Native GUI
agents and introduce LivingScreen, the first
benchmark instantiating it on short-
video platforms, with a faithful browser-based environment, a three-tier task suite, and metrics that jointly score accuracy and information efficiency. Evaluating extensive frontier models, we find that none reaches the human cost-accuracy performance, and that their dominant failure mode is over- and under-observation, pointing to observation control as a missing capability axis for future GUI
agents. All data and code will be available at https://github.com/BITHLP/LivingScreen.
Read the full abstract ▾
All authors Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo
Computation and Language
DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer
Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui et al. (+3 more)
Small
language models (SLMs) are efficient and scalable, but their
multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA)
languages. We introduce DuDi, a dual-signal
multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy
token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in
multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive
distillation baselines. Ablations and analyses confirm that sequence-level optimization,
token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for
multilingual SLMs.
Read the full abstract ▾
All authors Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui, Sarana Nutanong, Alham Fikri Aji, Peerat Limkonchotiwat
Computation and Language
SMADE-IE: Sparse Multi-Agent Framework with Evidence-Driven Debate for Zero-Shot Information Extraction
Kenfeng Huang, Yi Cai, Xin Wu et al. (+2 more)
Zero-shot information extraction (IE) with large
language models (LLMs) enables adaptation to new schemas and domains without task-specific training. Existing methods mainly follow three paradigms. Monolithic prompting is efficient but prone to missed mentions, boundary errors, and type confusion. Each-type prompting improves type-level focus but may produce overlapping or conflicting predictions, while multi-
agent debate can resolve such conflicts at the cost of irrelevant context, redundant interactions, and high
token overhead. To address these issues, we propose SMADE-IE, a sparse and evidence-driven multi-
agent framework. An Adaptive Mode Selector routes simple inputs to a lightweight Global Extraction Mode and ambiguous inputs to a Type-Centric Extraction Mode based on sample complexity and relevant types. Cross-type conflicts are resolved by an Evidence-Driven Debate module that uses Toulmin-style arguments, external evidence scoring, Beta-based confidence updates, and early stopping. Experiments on nine
benchmarks covering NER, RE, and JERE show that SMADE-IE improves average Partial F1 over the strongest baselines by 11.37, 3.83, and 14.46 points, respectively. Compared with the multi-
agent baseline CrossAgentIE, SMADE-IE reduces
token consumption by 85.1% on DocRED and 80.1% on CrossRE, demonstrating substantially higher inference efficiency. Code is available at https://github.com/Cppys/SMADE-IE.
Read the full abstract ▾
All authors Kenfeng Huang, Yi Cai, Xin Wu, Zikun Deng, Li Yuan
Computation and Language · Cross-listed from eess.AS
Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
Zhihan Li, Hankun Wang, Yiwei Guo et al. (+3 more)
Automatic
speech recognition systems commonly rely on reference transcriptions for evaluation, while reference-free approaches often depend on internal confidence estimation or auxiliary
language models. We propose READ (Reference-free Hypothesis Evaluation with Acoustic Discrepancy), a novel metric that evaluates ASR hypotheses directly from the
speech signal. READ emphasizes the acoustic grounding of hypotheses. It uses a pretrained auto-regressive TTS model to compute the conditional likelihood of
speech tokens given a text hypothesis, to measure fine-grained acoustic discrepancy between
speech and text. Without additional training, READ can be applied for hypothesis refinement. Experiments show that READ correlates with specific recognition errors and improves ASR outputs, achieving up to 20% relative error rate reduction, with particularly strong gains under noisy conditions.
Read the full abstract ▾
All authors Zhihan Li, Hankun Wang, Yiwei Guo, Bohan Li, Xie Chen, Kai Yu
Computation and Language
CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts
Shanu Kumar, Shubhanshu Khandelwal, Akhila Yesantarao Venkata et al. (+3 more)
Prompts tuned for accuracy often grow long, raising inference cost on every model call. The best accuracy-cost trade-off depends on the task and the budget, so
prompt optimization is a search over the Pareto front of accuracy and
prompt-
token cost rather than for one
prompt. The usual shortcut, collapsing the objectives into a weighted sum, fixes the trade-off weight before search and often recovers only a narrow region of the front, a failure we call scalarization collapse. We present CRAFT (Cost-aware Refinement And Front-aware Tuning), a Pareto-front
prompt optimizer that treats target-LLM validation calls as the scarce resource and allocates them to candidates near the optimistic candidate front. Each round, complementary accuracy-oriented and cost-oriented generators propose edits, Pareto-gap acquisition spends the per-round validation budget, and NSGA-II retention keeps a spread-out population. Across six classification and
reasoning benchmarks, CRAFT's retained fronts reach both high-accuracy and low-cost regions, while accuracy-only, cost-only, and weighted-sum baselines each concentrate in narrower regions. The accuracy-cost trade-off becomes a post-search choice, not a pre-search weight.
Read the full abstract ▾
All authors Shanu Kumar, Shubhanshu Khandelwal, Akhila Yesantarao Venkata, Parag Agrawal, Yova Kementchedjhieva, Manish Gupta
Computation and Language
LifeSide: Benchmarking Agents as Lifelong Digital Companions
Yuqian Wu, Zhijie Deng, Wei Chen et al. (+8 more)
Lifelong digital companions must integrate cross-session cues, continually update their understanding of users, and adapt to shifting
privacy boundaries. Existing evaluations fail to capture this, testing
memory recall and short-term empathy in isolation. To bridge this gap, we introduce \benchmark, a
benchmark centered on multi-session
Memory-
Emotion-Environment loops. By modeling users as persistent worlds with layered profiles and event trajectories, \benchmark uses multi-
agent simulation to project environmental dynamics into dialogue, preserving the critical gap between latent thoughts and observable expressions. Evaluating 2,000 personas and 111K tasks across
memory tracking, user understanding,
privacy control, and emotional companionship, our experiment results reveal a stark reality: even models that saturate current
memory benchmarks fail to sustain accurate user understanding and true companionship over long horizons.
Read the full abstract ▾
All authors Yuqian Wu, Zhijie Deng, Wei Chen, Junwei Li, Yutian Jiang, Junle Chen, Zhengjun Huang, Qingxiang Liu, Jing Tang, Jiaheng Wei, Yuxuan Liang
Computation and Language
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
Mengao Zhang, Xiang Yang, Chang Liu et al. (+2 more)
Many real-world questions over business, legal, and scientific corpora are natural-
language versions of database-style queries over records latent in text. Existing
retrieval-augmented generation (RAG) systems are optimized primarily for semantic relevance, but retrieving plausible passages does not guarantee correct query execution. We introduce QO-Bench, a diagnostic
benchmark for query-operator question answering over typed event tuples. The
benchmark covers 22,984 news articles and 614 corporate events, with 18 query templates instantiating 785 questions. Each gold answer is deterministically computed from typed event tuples and scored by recall, with answers matched to the gold tuples by exact match rather than an LLM judge. This design enables operator-level diagnosis such as joins and intersection. We evaluate RAG, ReAct RAG, GraphRAG, and information-extraction-to-SQL under matched conditions, with a long-context oracle ceiling to isolate
retrieval failure. A two-axis framework – index-time preservation versus query-time execution – predicts where each paradigm fails, and the results bear it out: systems retrieve relevant text but discard the typed values operators need, and the deployable paradigm ranking inverts across operators, with similarity
retrieval leading on filter/project and extraction-to-SQL on intersection and counting. Even given the gold evidence, a long-context oracle stays far from saturated, so operator execution – not
retrieval alone – is a core bottleneck that a stronger answer model does not remove. QO-Bench reframes the goal from passage relevance to query-operator-preserving
retrieval. The
benchmark, predictions, and code are released at https://github.com/ZHANG-MENGAO/qo-bench.
Read the full abstract ▾
All authors Mengao Zhang, Xiang Yang, Chang Liu, Tianhui Tan, Ke-wei Huang
Computation and Language
CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment
Nikodem Tomczak
Language models acting as
agents over knowledge graphs generate Cypher queries that fail structurally (crashing at the database) or semantically (executing but returning wrong results). We place a pre-execution gate between query generation and a production Neo4j database. The gate validates structure through a four-backend chain culminating in execution against a mirror graph at 5.6 ms median latency. Structurally broken queries are routed to a corrector that iterates structured error feedback through a
language model. On seven CypherBench schemas (2348 questions, ACL 2025) the pipeline maintains generation accuracy on every model tested, confirming it operates as a safe defensive layer. The corrector achieves 81% to 95% success across five models (mean 89%). On a template-generated corpus across nine schemas the gate catches 100% of parse errors, 100% of constraint violations, and 100% of schema-reference errors in path queries with labelled endpoints, at zero false positives across 1135 queries. Property sibling-swaps where the substituted name is valid on the target label score 0%, marking the formal boundary where structural validation ends and semantic validation must begin. A planner-based cost gate flags catastrophic plan structures before execution.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
VentAgent: When LLMs Learn to Breathe – Multi-Objective Arbitration for ARDS Ventilation
Teqi Hao, Yuxuan Fu, Xiaoyu Tan et al. (+4 more)
Mechanical ventilation for Acute Respiratory Distress Syndrome (ARDS) requires balancing competing physiological goals, including oxygenation, lung protection, and acid-base homeostasis. However, current data-driven methods, especially those imitating retrospective Electronic Health Records (EHR), often suffer from imitation bias. They may capture superficial correlations from inconsistent clinical demonstrations, such as associating passive ventilator settings with survival because such settings are common in stable patients, and thus fail to generalize to volatile or out-of-distribution phenotypes. Standard Reinforcement Learning (RL) methods also struggle with the
adversarial trade-offs of critical care and often produce opaque policies with limited clinical interpretability. To address these limitations, we introduce VentAgent, a hierarchical framework in which Large
Language Models (LLMs) act as transparent arbitrators for mechanical ventilation. We reformulate ventilation control as a dynamic Multi-Objective Arbitration process rather than single-objective optimization. VentAgent decomposes decision-making into three interpretable stages: Perception, Planning, and Orchestration. By leveraging the semantic
reasoning capabilities of LLMs, it synthesizes strategies from heterogeneous experts and resolves conflicting clinical priorities through an explicit coordination mechanism. Evaluations on a high-fidelity physiological simulator show that VentAgent outperforms state-of-the-art RL and classical control baselines. Moreover, it converts control decisions into human-readable
reasoning chains, offering a safer, more interpretable, and adaptable paradigm for critical care automation.
Read the full abstract ▾
All authors Teqi Hao, Yuxuan Fu, Xiaoyu Tan, Shaojie Shi, Bohao Lv, Yinghui Xu, Xihe Qiu
Computation and Language
RAMPART: Registry-based Agentic Memory with Priority-Aware Runtime Transformation
Nikodem Tomczak
RAMPART is a compile-time
memory model and pure in-RAM block registry for LLM-based
agents. Context assembly is a programmable runtime operation where content is compiled from a structured registry under explicit policy for ordering, inclusion, and eviction. Five composable primitives (promote, gate, write, evict, rollback) act on named addressable blocks before compilation at zero
prompt-
token cost. Provenance tags and non-evictable authorship flags implement a permissioned
memory model with block-level ownership. Controlled probes with Qwen3-8B Q4 show that compile-time placement and the structural relationship between blocks and the task query affect task success, with the cliff falling at roughly the seventh block position when the task follows the registry and the twelfth when it precedes. Grouping the critical block with content-adjacent neighbours and promoting the group as a unit lifts task success by tens of percentage points at positions where single-block placement fails. Cross-model replication on Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, and Qwen3-14B shows the content-priming effect appears at the same absolute positions across families, with magnitude varying with model strength. Block grouping raises Mistral's mean pass rate roughly fivefold at the hardest registry size, and a smaller model with the intervention can outperform a larger model without it in the mid-registry zone. Relevance gating reduces
prompt cost by 67.8% while recovering 83% of the promoted-condition success rate. Schema eviction produces 0% invocations against 100% with the schema present, a property policy-based approaches cannot guarantee by construction. Shared-registry coordination reduces inter-
agent communication to a method call at zero coordination
token cost.
Read the full abstract ▾
Computation and Language
Hybrid Adversarial Defence for Natural Language Understanding Tasks
Manar Abouzaid, Yang Wang, Chenghua Lin et al. (+1 more)
Large
Language Models (LLMs) are vulnerable both to
hallucination and
adversarial manipulation. Although these problems are closely related, existing defences typically address them separately. We investigate a hybrid defence framework that combines
entropy-based models, designed to reduce
hallucinations, with uncertainty-based models and
geometric-based models, designed to reduce vulnerability. Under in-domain tests on Natural
Language Understanding
datasets (FEVER, HotpotQA, CSQA, SIQA) we find our hybrid model improves both clean-task performance (up to 43.34% increase in accuracy) and
adversarial robustness (up to 64.92% improvement in accuracy and 62.27% reduction in attack success rate). For out-of-distribution
datasets (AeroEngQA, CPIQA) we see similar
adversarial robustness from our hybrid model (up to 57.14% improvement in accuracy). For
prompt injection (SafeGuard) and jailbreak detection (AdvBench, DAN)
datasets our hybrid model is also very strong (up to 51% reduction in attack success rate compared to state of the art baseline models). Overall, our results show that combining
entropy, uncertainty and
geometric features provides a more effective defence strategy than using any single feature alone for both in-domain and out-of-distribution tasks.
Read the full abstract ▾
All authors Manar Abouzaid, Yang Wang, Chenghua Lin, Stuart E. Middleton
Computation and Language
A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs
Huangchen Xu, Yuan Wu and Yi Chang
Multimodal Large
Language Models (MLLMs) are increasingly used for
video understanding, yet their reliability under multi-
video inputs remains poorly understood. We study positional bias in multi-
video summarization, where the quality of a per-
video summary can change with the
video's input slot even when the underlying content is unchanged. We construct a
benchmark from ActivityNet and News
videos, covering Cooking, Domestic, Leisure, and News settings with two- and four-
video inputs. We evaluate nine open-source and proprietary MLLMs and measure position effects with three complementary metrics: Coverage, Directional Positional Bias (DPB), and Middle-Edge Gap (MEG). Our results show that positional effects are domain- and model-dependent: signed directional bias can be small even when middle positions underperform, and increasing visual or generation budget does not uniformly remove the imbalance. We further analyze
prompt-level mitigation methods. Together, the results show that multi-
video summarization remains sensitive to input protocol and position, motivating more robust order-invariant
multimodal systems.
Read the full abstract ▾
Computation and Language
Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues
Hanbo Bi, Zhiqiang Yuan, Chongyang Li et al. (+7 more)
With the widespread adoption of multi-modal communication platforms, long-form dialogues interleaving text and images have become increasingly common. Users often need to retrieve coherent dialogue fragments related to specific topics, rather than isolated utterances. We propose Fine-grained Fragment
Retrieval (FFR), which locates semantically relevant multi-utterance, multi-image fragments in multi-modal long-form dialogues. We explore two settings: (1) FFR within Single-Dialogue, retrieving fragments from a given dialogue; and (2) FFR within Dialogue Corpus, retrieving from a large-scale corpus for open-domain scenarios. For (1), we introduce F2RVLM, a generation-based
retrieval model trained with reinforcement learning, using multi-objective rewards and difficulty-aware curriculum sampling to enhance fragment coherence. For (2), we develop FFRS, a two-stage system combining offline fragment-level indexing with online
retrieval. Specifically, each dialogue is decomposed into minimal semantic fragments encoded by a Fragment
Embedding Model (FEM) into a vector database; at inference, FEM rapidly recalls Top-K candidates, and F2RVLM performs fine-grained
reasoning to identify the most relevant sub-content. To support FFR, we construct MLDR, the longest multi-modal dialogue
retrieval dataset to date, and a WeChat-based real-world test set. Experiments on both
benchmarks demonstrate that F2RVLM and FFRS consistently achieve superior performance across single-dialogue and corpus-level FFR.
Read the full abstract ▾
All authors Hanbo Bi, Zhiqiang Yuan, Chongyang Li, Qiwei Yan, Zexi Jia, Jiapei Zhang, Xiaoyue Duan, Yingchao Feng, Jinchao Zhang, Jie Zhou
Computation and Language
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
Huangchen Xu, Yuan Wu and Yi Chang
Multimodal large
language models have made rapid progress in
video understanding, yet existing
benchmarks largely rely on simple
prompts and provide limited evidence about whether models can satisfy explicit output constraints. We introduce VCIFBench, a
benchmark for evaluating complex instruction following in
video understanding. VCIFBench constructs constraint-rich instructions from both
benchmark-adapted and directly
video-grounded
prompts, covering content, format, style, and structure requirements, and evaluates model outputs with a hybrid verification pipeline. The
benchmark contains 306 satisfiable test instructions, 540 DPO training instances, and a 100-item diagnostic set for evaluating whether models can recognize instruction conflicts. Experiments on 10 MLLMs show that joint constraint satisfaction remains challenging. Preference optimization improves instruction following for two model families, while Conflict-100 reveals that models usually execute a satisfiable-looking subset instead of detecting global incompatibility.
Read the full abstract ▾
Computation and Language
Cartridges at Scale: Training Modular KV Caches over Large Document Collections
Momchil Hardalov, Gonzalo Iglesias and Adrià de Gispert
Large
Language Models can reason over long contexts, yet prefilling millions of
tokens is wasteful as much of the content remains static across queries. Cartridges address this by distilling document collections into reusable key-value (KV) caches that eliminate prefilling while preserving accuracy. A critical limitation of this approach is that cartridges are monolithic and non-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance. We introduce Cartridges at Scale (CAS), a training framework for scalable multi-cartridge learning with dynamic distractor mixing and a
memory-efficient budget manager that rotates hundreds of per-document cartridges between GPU and persistent storage. Our approach scales to collections exceeding a million
tokens, improving over a monolithic cartridge by 10-31 points at comparable
token budgets. Oracle cartridge accuracy falls within 2-6 points of full in-context learning even at high compression. When paired with
retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3-4x fewer
prompt tokens.
Read the full abstract ▾
Computation and Language
Temporal Order Matters for Agentic Memory: Segment Trees for Long-Horizon Agents
Yifan Simon Liu, Liam Gallagher, Faeze Moradi Kalarde et al. (+3 more)
Long-horizon conversational
agents need to interact with users through evolving events, tasks, and goals. Such histories are naturally temporal, yet many existing
memory systems organize information primarily by topical similarity and may ignore the order in which events occur. We introduce Segment Tree
Memory, or SegTreeMem, a
memory architecture that represents conversation history as a temporally ordered Segment Tree over utterances. SegTreeMem incrementally inserts new utterances through an online rightmost-frontier update rule, preserving chronological order while forming hierarchical
memory segments. For
retrieval, SegTreeMem propagates relevance scores through the tree to combine local semantic matching with hierarchical temporal context. Across three long-horizon
memory benchmarks and two LLM backbones, SegTreeMem improves answer quality over flat
retrieval, graph-structured
memory, and tree-structured
memory baselines. Additional temporal-order permutation analysis shows that the performance gain depends on preserving temporal order during
memory construction, supporting the claim that temporal order is a key structure for agentic
memory.
Read the full abstract ▾
All authors Yifan Simon Liu, Liam Gallagher, Faeze Moradi Kalarde, Jiazhou Liang, Armin Toroghi, Scott Sanner
Computation and Language
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
Daria Ledneva and Denis Kuznetsov
Genomic foundation models increasingly adopt large
language model architectures, yet almost universally rely on fixed tokenization schemes such as $k$-mers, BPE, or single nucleotides, which impose arbitrary sequence boundaries that may obscure biologically relevant structure. We present LDARNet, a 110M-parameter hierarchical genomic foundation model that adapts H-Net-style dynamic chunking from autoregressive generation to masked
language modeling, combining BiMamba-2 state-space layers with local
attention, bidirectional routing, and a ratio-based regularizer to induce adaptive
token boundaries without supervision. Fine-tuned on 27 tasks from the Nucleotide
Transformer and Genomic
Benchmarks suites, LDARNet achieves 15/18 wins among compact models ($<$300M parameters) and the best overall result on 9 of the 10 histone modification tasks, outperforming models up to 20$\times$ larger. A FLOPs-matched controlled experiment isolates learned routing as the source of these gains: learned boundaries beat fixed-grid boundaries by up to 14 percentage points on histone tasks at identical compute. Nucleotide-resolution analysis further shows that the learned boundaries align with canonical promoter motifs and splice junctions without supervision, providing a biological interpretation for adaptive tokenization in genomic foundation models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization
Heng Cao, Fan Zhang, Jian Yao et al. (+8 more)
Personalizing large
language models requires adapting model behavior to individual users while preserving robustness and deployment-scale efficiency. Existing approaches typically personalize LLMs either at the input level, by retrieving user histories or constructing profile
prompts, or at the parameter level, by maintaining user-specific parameter-efficient modules. The former makes personalization sensitive to
retrieval quality and
prompt design, whereas the latter incurs storage and maintenance costs that grow with the user population. To address these limitations, we propose TAP-PER (Temporal Attentive Prefix for PERsonalization), a prefix-based framework that encodes user preferences as learnable representations, avoiding the serialization of user histories into
prompts and replacing heavy per-user adapters with lightweight user-state prefix
embeddings. Inspired by personalized recommendation systems, TAP-PER decomposes user modeling into user-state and query-conditioned components, and incorporates temporal signals to capture the evolving nature of user interests. Experiments on six LaMP tasks show that TAP-PER consistently outperforms
prompt-based and model-based baselines across classification, rating, and generation settings. Moreover, TAP-PER uses 130x fewer per-user parameters than OPPU and roughly half the total parameter footprint of PER-PCS at the 1,000-user scale, demonstrating scalable personalization without
prompt-serialized histories or heavy per-user adapters.
Read the full abstract ▾
All authors Heng Cao, Fan Zhang, Jian Yao, Yujie Zheng, Changlin Zhao, Lu Hao, Yuxuan Wei, Wangze Ni, Huaiyu Fu, Yuqian Sun, Xuyan Mo
Computation and Language
Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models
Boyan Han, Yiwei Wang, Yi Song et al. (+2 more)
Diffusion large
language models (dLLMs) offer bidirectional
attention and parallel generation, enabling them to exploit global context and naturally support format-constrained tasks like parseable JSON or
reasoning templates. While straightforward fixed anchors can enforce such constraints, they often impose rigid spans, leading to truncated
reasoning or redundant content. To overcome this, we propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling. This flexible mechanism ensures structural correctness and semantic coherence, avoiding the inefficiencies of fixed-span methods. Experiments on
reasoning benchmarks demonstrate that DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH. These results establish DIA as a robust pathway toward reliable, structure-aware generation.
Read the full abstract ▾
All authors Boyan Han, Yiwei Wang, Yi Song, Yujun Cai, Chi Zhang
Computation and Language
GENEB: Why Genomic Models Are Hard to Compare
Daria Ledneva, Mikhail Nuridinov and Denis Kuznetsov
Progress in genomic foundation models is difficult to assess due to fragmented
benchmarks, incompatible evaluation protocols, and task-specific reporting. As a result, claims of superiority or generality across models are often not directly comparable. We introduce GENEB, a large-scale diagnostic
benchmark that evaluates frozen representations from 40 genomic foundation models across 100 tasks spanning 13 functional categories under a unified probing-based protocol, including few-shot regimes. GENEB enables controlled comparison across model scale, architecture, tokenization, and pretraining data while explicitly exposing task-level trade-offs. Our analysis shows that aggregate leaderboards are unstable: model rankings vary sharply across task categories, scale provides only modest and inconsistent gains, and architectural and pretraining
alignment frequently outweigh parameter count. These results highlight limitations of current evaluation practices and position GENEB as a reference framework for principled comparison and category-aware model selection in genomic machine learning.
Read the full abstract ▾
Computation and Language
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Yaosheng Fu, Guangxuan Xiao, Xin Dong et al. (+2 more)
Sparse
attention reduces compute and
memory bandwidth for long-context LLM inference. However, two key challenges remain: (1) KV cache capacity still grows with sequence length, and offloading to CPU
memory introduces a PCIe transfer bottleneck; (2) the sparse selection step itself retains $O(T^2)$ complexity and can dominate
attention cost at long contexts. We propose SparDA, a decoupled sparse
attention architecture that introduces a fourth per-layer projection, the Forecast, alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling lookahead selection that overlaps CPU-to-GPU prefetch with current-layer execution. Because Forecast is decoupled from the
attention query, our GQA implementation uses one Forecast head per GQA group, reducing selection overhead versus the original multi-head selector. SparDA adds $<$0.5% parameters and trains only the Forecast projections by matching the original selector's
attention distribution. On two sparse-pretrained 8B models, SparDA matches or slightly improves accuracy and delivers up to 1.25$\times$ prefill speedup and 1.7$\times$ decode speedup over the sparse-
attention offload baseline. By enabling larger feasible batch sizes on a single GPU, SparDA further reaches up to 5.3$\times$ higher decode throughput than the non-offload sparse baseline. Our source code is available at https://github.com/NVlabs/SparDA.
Read the full abstract ▾
All authors Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa
Computation and Language
Self-Evolving Deep Research via Joint Generation and Evaluation
Han Zhu, Chengkun Cai, Yuanfeng Song et al. (+3 more)
Large
Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability. Unlike traditional question-answering (QA) tasks, deep research report generation lacks definitive ground-truth, making reward design inherently unverifiable and limiting effective reinforcement learning. Existing approaches mitigate this challenge with LLM-as-a-judge and query-dependent evaluation rubrics, but they still rely on static evaluators that cannot adapt their standards as the solver improves, leading to insufficient and eventually saturated optimization pressure. We address this limitation with a self-evolving co-evolutionary training framework for deep research evaluation and generation (SCORE), which tightly couples an evaluator and a solver in a shared-parameter learning process. Rather than treating generation and evaluation as isolated modules, we leverage their intrinsic connection to enable joint improvement within a single shared-parameter model. To restrict this process, we introduce a meta-harness, which dynamically controls the evaluation environment based on solver performance, encouraging valid evaluation dimensions and sufficiently deep evaluator search. Extensive experiments on deep research
benchmarks demonstrate consistent improvement in report generation quality, showing that co-evolving evaluation and generation is a promising direction for training open-ended research
agents.
Read the full abstract ▾
All authors Han Zhu, Chengkun Cai, Yuanfeng Song, Xing Chen, Sirui Han, Yike Guo
Computation and Language
SANE Schema-aware Natural-language Evaluation of Biological Data
Rolf Gattung, Martin Krueger and Markus Reischl
High-throughput microscopy generates large, structured
datasets capturing cellular responses to pharmacological perturbations, but accessing these
datasets typically requires SQL expertise. Large
language models offer a natural-
language alternative, yet their tendency to hallucinate raises concerns about result reliability . We present SANE Schema-Aware Natural-
language Evaluation, a novel paradigm for domain-specific text-to-SQL evaluation: schema-grounded, automatically generated
benchmarks tied to real and specific experimental structure. SANE makes evaluation more scalable, systematic, and reproducible. Using SANE, we evaluate a few-shot large
language model and show that, under constrained schemas with structured prompting and guardrails, accurate query generation is achievable without any model training or fine-tuning. Most failures stem from ambiguous or underspecified inputs and manifest as overly cautious clarification requests or answers to queries that should first be disambiguated, rather than incorrect SQL generation. These results indicate that few-shot large
language models can provide reliable database access in well-defined domains when combined with schema-aware prompting.
Read the full abstract ▾