Information Retrieval
The Recall Ceiling of LLM Recommendation Reranking
Zhaohui Wang
Some LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the scored set, either by injecting it into the candidate list or by scoring it against sampled negatives. Across three primary Amazon
datasets, we show that this protocol overestimates realistic NDCG@10 by 92–95%. The cause is a recall ceiling: realistic
retrieval covers only 2–19% of relevant items at $K=100$ across eight
datasets in three domains, imposing a deterministic upper bound on any closed-candidate reranker's top-$k$ NDCG. Under leave-one-out evaluation, $\mathbb{E}[\mathrm{NDCG}@k] \leq \mathrm{Recall}@|W_\pi|$, where $W_\pi$ is the reranker's candidate window. Under realistic
retrieval, none of the tested optimisation strategies significantly improves over the collaborative-filtering baseline on our primary Amazon
datasets. These strategies include
prompt engineering, model scaling over a 168$\times$ parameter range, sequential models, supervised neural rerankers, LoRA fine-tuning, hybrid
retrieval, score-aware prompting, and LLM+CF fusion. Text-aware
retrieval increases recall on one
dataset but does not improve end-to-end NDCG, while providing upstream CF scores mainly makes the LLM reproduce the CF order. We therefore propose the Recall-Aware Evaluation Protocol (RAEP): first classify the
retrieval-recall regime, then evaluate reranking where the ceiling permits meaningful differentiation. In the low-recall regimes measured here, improving
retrieval is more consequential than increasing reranker sophistication; this ordering need not hold in production systems with higher recall, richer features, or online feedback.
Read the full abstract ▾
Information Retrieval · Cross-listed from cs.CL
SCoR: A Hierarchical Framework for Forecasting Relations Between Scientific Concepts
Jingze Wang, Fred Sun and Shangqi Guo
Anticipating emerging research directions is a critical goal of AI-assisted science. Existing methods mainly predict which concepts will co-occur in future papers, but co-occurrence captures shared
attention rather than the scientific meaning of a connection, such as whether one method uses, combines, replaces, or contradicts another. We formulate research-direction discovery as hierarchical scientific-relation forecasting over a shared candidate-pair space, comprising three temporally aligned tasks: first co-occurrence, first scientific-relation formation, and relation type at formation. We construct SCoR-Graph from 187,848 cs.CV papers published between 2017 and 2026, yielding 270,687 consolidated concepts, 7.45 million co-occurrence edges, and 615,036 typed, directed relation edges. From cutoff-specific graph snapshots, we derive SCoR-Bench, a leakage-audited
benchmark for these three capabilities, with expert-verified gold labels for the entire relation-type test set. We further introduce HiSCoR, a task-adapted model family that models relation emergence as a temporally evolving, hierarchically constrained process by encoding pre-cutoff event histories and conditioning relation formation on future co-occurrence. On the held-out 2025-2026 window, HiSCoR achieves an AUROC of 0.9515, a 2.4% relative improvement over the strongest temporal-graph baseline, and improves population-AUPRC by 14.0%; its relation-type variant achieves a Macro-AUROC of 0.7795. Ablations show that semantic, co-occurrence, and typed-relation views provide complementary predictive evidence. SCoR advances research-direction forecasting from predicting which concepts will co-occur to anticipating whether and how evidence-backed scientific relations will emerge.
Read the full abstract ▾
Information Retrieval · Cross-listed from cs.CV
Weaving Visual Narratives: Agentic Image Bundle Composition Beyond Atomic Visual Matching
Rong Shan, Tianyi Xu, Congmin Zheng et al. (+9 more)
Image
retrieval has traditionally been formulated as a point-wise matching problem, where each candidate image is scored in isolation. However, this atomic paradigm fails to capture the complexity of human search intent within personal photo collections, where users often seek compact visual stories bound by structural relations rather than isolated snapshots. To address this limitation, we introduce **Image Bundle Composition (IBC)**, a novel paradigm that shifts the objective from ranking individual images to dynamically composing cohesive image bundles from a massive, unstructured photo pool. Since target bundles are not predefined, IBC presents a severe combinatorial explosion challenge and demands modeling non-decomposable joint relevance. To establish this paradigm, we construct **IBCBench**, the first IBC
benchmark dataset containing 109,467 images and 667 verified queries, built via a semi-automated verification pipeline. Furthermore, we propose **BundleWeaver**, an agentic framework that reformulates IBC as query-conditioned incremental hyperedge discovery. By employing a Large
Language Model to adaptively search for missing relational roles and utilizing a
Vision-
Language Model for whole-bundle verification, BundleWeaver effectively navigates the combinatorial space. Extensive experiments demonstrate that while state-of-the-art
embedding models and static decompose-and-rerank paradigms suffer from relational blindness, BundleWeaver achieves substantial performance gains, highlighting the necessity of shifting from atomic scoring to dynamic relational composition. Our
dataset and code are available.
Read the full abstract ▾
All authors Rong Shan, Tianyi Xu, Congmin Zheng, Wenteng Chen, Jiachen Zhu, Junjie Wu, Teng Wang, Weiwen Liu, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin
Information Retrieval
NormasTCU — A Brazilian Portuguese IR Dataset and an Evaluation of LLM-as-a-Judge for Relevance Assessment
Leandro Carísio Fernandes, Marcus Vinícius Borela de Castro, Leandro dos Santos Ribeiro et al. (+2 more)
Portuguese Information
Retrieval (IR) lacks public
datasets, and relevance assessment for specialized collections remains costly. While Large
Language Models (LLMs) increasingly support relevance assessment, their reliability in non-English specialized domains remains unclear. We introduce NormasTCU (https://huggingface.co/
datasets/LeandroRibeiro/NormasTCU), a Brazilian Portuguese IR
dataset with 14,469 legal documents, 46 queries, and 3,048 human judgments over 812 query-document pairs. Using NormasTCU, we evaluated LLM-as-a-judge for relevance assessment by prompting three models with two
prompt techniques to grade these pairs. We then compared the rankings of 15 IR systems derived from LLM-generated and human reference qrels. LLMs consistently showed a positive scoring bias (mean absolute error: 0.46–0.66 on a 0-2 scale). Furthermore, pair-level agreement with human judgments achieved only fair to moderate levels, with Cohen's kappa ranging from 0.32 to 0.53. Despite this bias, LLM-generated judgments often yielded highly similar system rankings for nDCG@10 and MRR (observed Kendall's tau greater than or equal 0.90, although the bootstrap confidence intervals did not always remain above this threshold), but were less reliable for P@10 and R@10. Notably, LLM-based rankings were sometimes more strongly correlated with the reference ranking than individual human annotations were. As a practical implication, our results suggest that LLMs could effectively support scalable relevance assessment in specialized Portuguese corpora when evaluated using nDCG or MRR (rank-aware metrics), but they should be avoided when relying on precision or recall.
Read the full abstract ▾
All authors Leandro Carísio Fernandes, Marcus Vinícius Borela de Castro, Leandro dos Santos Ribeiro, Leonardo Augusto da Silva Pacheco, Edans Flávius de Oliveira Sandes
Information Retrieval
A Versioned Unified Graph Index for Dynamic Timestamp-Aware Nearest Neighbor Search
Jun Woo Chung and Weijie Zhao
We present TiGER (Time-Integrated Graph for Efficient
Retrieval), a novel approach for performing fast time-aware approximate nearest neighbor searches on dynamic vector
datasets with flexibility over any possible time range. Our proposed
algorithm builds and maintains a unified graph for all vectors by leveraging an index structure based on integrated versioned connectivity, allowing arbitrary time intervals to be queried directly on the unified graph without having to traverse invalid vectors. This forgoes the need for post-search filtering or merging, or separate graphs for each possible composite range. Empirical evaluations show that our method attains up to a 5x improvement in queries per second (QPS) without compromising accuracy over baselines based on filtering or per-time-segment sub-graphs. We believe that this method will enable efficient temporal analysis across evolving
datasets in real-time recommendation systems, log analysis, and any scenario requiring fast similarity search over dynamic, time-segmented data.
Read the full abstract ▾
Information Retrieval
Beyond the Vacuum: Combinatorial Strategy Selection for Competitor-Aware Generative Engine Optimization
Vaibhav Sourirajan, Yao Zhang, Himanshu Kumar et al. (+3 more)
Generative Engine Optimization (GEO) has emerged as a novel paradigm for transforming content to increase visibility in Large
Language Model (LLM) responses. Traditional GEO methods, however, select rewriting strategies in isolation, ignoring a critical externality: as adoption of content optimization grows, optimal strategies for rewriting content change. We formalize GEO as a competitor-aware strategy selection problem and propose a two-phase pipeline to solve it: (1) We use
Bayesian Optimization of Combinatorial Structures (BOCS) to efficiently search the space of rewriting strategies, (2) We generate preference pairs and grounded
reasoning traces from the BOCS black-box observations to fine-tune a
language model to analyze a document corpus and propose optimal rewriting strategy combinations. We achieve state-of-the-art performance across several impression metrics over existing agentic and single-heuristic methods on both geo-bench and our synthetically augmented competitive
dataset geo-bench_comp. Our method also transfers to multiple out-of-distribution
datasets, proving effective across domains, queries, and document types.
Read the full abstract ▾
All authors Vaibhav Sourirajan, Yao Zhang, Himanshu Kumar, Sahil Wadhwa, Mann Patel, Amirfarrokh Iranitalab
Information Retrieval
LitCurate: A Configuration-Driven AI-Assisted Framework for Scientific Database Construction with an Application to Lower-Mantle Equation-of-State Data
Abin Shakya, Wilson Samuels, Dominica Wilson et al. (+4 more)
The growing scientific literature contains decades of experimental and computational results that could support data-driven and physics-based modeling, yet much of this infor- mation remains locked in publications and is not readily usable for large-scale analysis or sci- entific software. Building structured databases from the literature is particularly challenging whenrelevantstudiesmustfirstbediscoveredamonglargecollectionsofpapersandreported quantities must be extracted with enough scientific context to remain usable. We present LitCurate, an open-source framework for building scientific databases from the literature using large
language models within an auditable, stage-wise curation workflow. LitCurate integratesliteraturediscovery, relevancescreening, full-textprocessing, andstructuredinfor- mation extraction while retaining intermediate results and provenance, allowing researchers to inspect and revise individual stages rather than treating automated curation as a black- box process. We apply LitCurate to construct an equation-of-state database of lower-mantle and lower-mantle-relevant high-pressure mineral phases from experimental and theoretical studies, comprising 1,334 entries from 205 papers. The resulting
dataset links reported equation-of-state parameters to mineral phases, compositions, equation formulations, meth- ods, and parameter constraints, and labels values as source-reported or citation-reported when provenance can be determined. The records are available through a searchable web application. By connecting scientific literature to traceable, machine-readable data, LitCu- rate provides a reusable approach for transforming accumulated literature into resources for scientific analysis and computational modeling.
Read the full abstract ▾
All authors Abin Shakya, Wilson Samuels, Dominica Wilson, Gioia A. Marchi, Israa Draz, Chenxing Luo, Renata M. Wentzcovitch
Information Retrieval
misi: a Metric Inverted Sample Index
Edgar Chavez
We present misi, an inverted index for approximate nearest-neighbor search over general metric spaces whose vocabulary is a random sample of the database, of size proportional to $n$. Each object is represented by its $k_b$ nearest sample points, found by a pluggable inner index over the sample; queries are answered by an idf-weighted shared-neighbor vote followed by exact verification of $C$ candidates. The construction generalizes the NAPP index from a constant number of pivots to a linear-size vocabulary, which keeps posting lists at constant expected length $\rho = k_b/\alpha$ as $n$ grows and turns the index into a combinator: any high-recall index on $\alpha n$ points yields an index on $n$ points, for any metric. A probabilistic model gives a recall guarantee – $k_b$ logarithmic in $n$ over the overlap gap suffices, with a verification budget the index itself estimates – and a matching limit: the vote cannot resolve overlap differences below order $1/\sqrt{k_b}$. The design's strengths are structural: construction is $n$ independent searches – embarrassingly parallel, deterministic, $5{,}250$ s for $10^8$ vectors on 64 cores, $3.7\times$ faster than a matched-recall graph build – it streams under an enforced 3 GiB cap, and the portable artifact serves $10^8$ vectors from NVMe within an enforced 8 GB budget, below the working floor of the SSD-graph baseline. Its cost is query-time work: saturated graph baselines answer $6$-$16\times$ faster in RAM, and the verification budget for 0.99 recall grows as $n^{0.30}$. All results carry seeds, saturation sweeps and full configurations, are generated from run manifests, and include measured negative results. The intended applications weight construction cost, determinism,
memory footprint, or black-box metrics over peak throughput: frequently rebuilt corpora, batch similarity workloads, constrained-
memory serving.
Read the full abstract ▾
Information Retrieval
Scaling Graph Neural Networks for Friend Recommendation: Multi-Hash User Embeddings and Temporal Neighbor Sampling
Maksim Utushkin, Andrei Ovsiannikov and Alexander D'yakonov
Friend recommendation is inherently graph-structured: the relevance of a potential connection depends on multi-hop social context rather than user attributes alone. However, deploying message-passing GNNs on a production-scale social graph with hundreds of millions of users and tens of billions of edges requires addressing numerous modeling and systems challenges. We present a scalable end-to-end GNN ranking system for production social graphs, focusing on two design choices that are critical in this setting: multi-hash ID
embeddings and temporal neighbor sampling. Multi-hash
embeddings are common for high-cardinality features, but industrial GNN systems typically either ignore trainable IDs or accept full
embedding tables, exceeding 200 GB for our graph. We integrate multi-hash as the primary node representation, reducing the ID-
embedding table size by more than 98 percent while preserving ranking quality. Temporal neighbor sampling is well understood in principle, but existing implementations scan full adjacency lists, which is a non-starter for users with tens of thousands of friends. We implement timestamp-sorted CSR storage with binary search, reducing the per-node temporal sampling cost from $O(deg(v) + k)$ to $O(\log(deg(v)) + k)$. Beyond these components, we show that this combination scales and yields measurable production impact. On a graph with 194M users and 28B edges, offline ablations isolate each design choice's contribution. In an online A/B test, our system increases friend additions from recommendations by 16 percent and unique friend adders by 11.5 percent over a strong production baseline. We release our framework for distributed training and inference on large temporal graphs.
Read the full abstract ▾
Information Retrieval · Cross-listed from cs.CL
RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
Maayan Sharon and Tom Hope
Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction—zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (
Retrieval Across Typed Ideation Operations), a large-scale
benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision—previously used only for classification—to corpus-scale
retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers and LLM reranking helps further, but much room for improvement remains. Researchers blind to condition found the fine-tuned retriever's candidates more useful as inspirations. RATIO provides a scalable training and evaluation framework for
retrieval components that support literature-grounded ideation, opening up new research avenues in scientific inspiration
retrieval. We release full data, code, models, and annotations.
Read the full abstract ▾
Information Retrieval · Cross-listed from cs.AI
CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
Sil Hamilton, Albert Yu Sun, Oscar J. Romero et al. (+4 more)
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic
datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A
benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication
reasoning, filling a crucial gap in the benchmarking ecosystem.
Read the full abstract ▾
All authors Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov
Information Retrieval
Stageboost: Recommending Signals Based on Counterfactual Estimation
Darpan Singhal, Matan Mandelbrod, Tal Franji et al. (+3 more)
Signals are short textual or visual snippets displayed on the eBay View-Item (VI) page, providing additional, contextual information for users about the viewed item. The aim of displaying these signals is to facilitate intelligent purchase and to incentivize engagement. In this paper, we present a 2 stage xgboost based model that optimally populates the VI page with signals. This approach has shown a 0.08% lift in overall GMB (Gross Merchandise Bought) and 0.58% increase in Parts and Accessories GMB, primarily due to increase in conversion of high average price items in online experimentation.
Read the full abstract ▾
All authors Darpan Singhal, Matan Mandelbrod, Tal Franji, Manasa Kolla, Vipul Gaba, Yuri Brovman
Information Retrieval
Astar: Learning to Propose Evolution Directions for Self-Evolving Industrial AI Systems
Jinxin Hu, Hao Deng, Haibo Xing et al. (+12 more)
Modern AI systems advance through continuous iteration: a loop of proposing
evolution directions, implementing code, training, and evaluation. While the latter three stages are increasingly automated, the starting point — proposing effective
evolution directions — remains a critical bottleneck that still relies heavily on senior experts. In this work, we explore whether AI can take over this role. We find that general-purpose LLMs, even the advanced GPT-5.5, offer only generic and misaligned suggestions: the required expertise is accumulated through experience rather than explicitly codified, and thus hard to inject directly. To this end, we propose Astar, a training-based approach that learns a specialized
evolution-guiding model from the abundant iteration histories of industrial systems. Realizing this idea, however, raises four challenges: sparse supervision, noisy data, a vast direction space, and prohibitively expensive verification. We address them along two fronts. On the data side, we design a pipeline that turns noisy historical commits into a large, clean evolutionary corpus via pairwise sample expansion and noise filtering. On the model side, we train the model through mid-training, SFT, and RL, guiding
evolution direction generation with hierarchical hints and using the reward model in RL as a fast surrogate evaluator. Astar has been deployed in Alibaba's Lazada advertising system for
evolution direction proposal. Astar-8B achieves a single-proposal success rate of 0.6786 in real-execution evaluation, far exceeding human experts (0.3229) and the strongest general-purpose LLM (0.3071). More importantly, Astar closes the loop and enables fully automatic iteration: it guided 20 consecutive iterations over two weeks, improving offline Hitrate@200 by 23.6%, while an online A/B test yielded relative lifts of 4.86% in GMV and 1.82% in advertising revenue.
Read the full abstract ▾
All authors Jinxin Hu, Hao Deng, Haibo Xing, Lingyu Mu, Muyu Zou, Weiqin Yang, Sirui Chen, Bohao Wang, Zhezheng Hao, Hao Zhang, Zulong Chen, Shizhun Wang, Yu Zhang, Xiaoyi Zeng, Jiawei Chen
Information Retrieval
ProRetrieval: Learning to Orchestrate Hybrid Search via Executable Program Synthesis
Chengsong You, Zhen Sun, Yunhai Hu et al. (+17 more)
Real-world
retrieval often composes structured constraints with semantic intents over text and images through arbitrary Boolean logic. Existing hybrid pipelines such as reciprocal rank fusion or self-querying retrievers admit only a fixed form of composition, while recent reinforcement-learning retrievers train the
language model as a query generator for a single backend, leaving the orchestration of heterogeneous
retrieval paths outside its action space. We propose ProRetrieval, which recasts the
language model as a
retrieval orchestrator: given a natural-
language query, it synthesizes an executable program in a hybrid DSL interleaving SQL operators over structured fields with vector-
retrieval primitives over text and images, with SQL itself providing the logical
algebra that fuses heterogeneous candidate sets. We train Qwen3-4B with GRPO and DAPO under a hierarchical four-term reward, and evaluate on two new
benchmarks built from Amazon products and Enron email. Our 4B model surpasses GPT-5.5 (Hit@1 0.81 vs. 0.69 on e-commerce; 0.91 vs. 0.86 on email) and Claude Opus 4.7 and a comprehensive suite of
retrieval, LLM-augmented, structured-query, and graph-based baselines. Code: https://anonymous.4open.science/r/ProRetrieval/; data: https://huggingface.co/
datasets/anonymous-7219/ProRetrieval.
Read the full abstract ▾
All authors Chengsong You, Zhen Sun, Yunhai Hu, Junwei Zhou, Xiaoyu Cao, Binyu Li, Ziyan Zhao, Weiyao Wang, Liren Lu, Zhijie Ye, Yumo Cao, Yitao Long, Yiwei Xu, Qiyi Jiang, Xuanyi Fu, Yufan Chen, Yilun Li, Rongkang Xiong, Yiran Zou, Nan Du
Information Retrieval
Conversational Recommendation over Live E-Commerce Catalogues with Self-Refreshing Retrieval
Ante Kapetanovic, Tomislav Duricic, Dionizije Fa et al. (+2 more)
Conversational recommender systems based on large
language models (LLMs) are usually evaluated on static, pre-indexed item collections, yet e-commerce catalogues change continuously as products are added or removed, repriced, and restocked. We present a merchant-agnostic, multi-turn conversational shopping assistant that operates over such live catalogues. Its central component is a self-refreshing retriever that ingests a merchant product feed, enriches the records, and synchronizes them into a vector index. On each run, per-item hashes identify which products are new, changed, deleted, or unchanged, so only the delta is processed rather than rebuilding the whole catalogue. A controller-based dialogue layer consumes this index, using an LLM only for intent classification and preference elicitation while
retrieval, reranking, and diversity selection run as dedicated functions. Our demonstration is a WhatsApp shopping assistant in which catalogue changes reach the recommendations after the next successful sync. A live chatbot, documentation, and a recorded walkthrough are available at https://github.com/infobip/infobip-agentic-crs.
Read the full abstract ▾
All authors Ante Kapetanovic, Tomislav Duricic, Dionizije Fa, Andro Mercep, Emanuel Lacic
Information Retrieval
Topology-Masked Unified Backbone for Joint Feature Interaction and Multi-Domain Sequence Modeling
Zhihao Zhu, Dezheng Han, Jikang Xia et al. (+1 more)
Large-scale post-click conversion rate (CVR) prediction requires jointly modeling heterogeneous feature interactions and dependencies over multi-domain user behavior sequences. Existing industrial ranking models usually handle these two aspects with separate modules. Recent unified architectures attempt to incorporate them into a single framework, but such unification often relies on coordination between modules and does not fully organize all information sources within the same interaction space. To address this problem, we propose MaskRec, a
topology-masked unified
token interaction architecture for feature interaction and multi-domain sequence modeling. MaskRec transforms heterogeneous features, multi-domain behavior sequences, and contextual signals into unified
token representations, and further introduces learnable global
memory tokens and domain-level
memory tokens as information aggregation nodes. Based on this unified
token space, MaskRec designs a structured
attention mask, TopoMask, which selectively enables or blocks
attention connections according to the structural differences and modeling requirements of different information sources. In this way, heterogeneous feature interaction and multi-domain sequence modeling are performed within the same
topology-constrained
attention process. In addition, MaskRec incorporates a dual-path interactive query generation module to inject candidate-conditioned user–item interaction signals before the unified backbone. Experiments on the Tencent Advertising
Algorithm Competition
dataset show that MaskRec achieves stable improvements over the official baseline, validating the effectiveness of the proposed unified framework for industrial CVR prediction.
Read the full abstract ▾
All authors Zhihao Zhu, Dezheng Han, Jikang Xia, Shuaishuai Guo
Information Retrieval
When Memory Takes Gradients: Collaborative Vector Memory for Agentic Recommender Systems
Hanchong Chen, Xing Tang, Lingjie Li et al. (+2 more)
Agentic recommender systems ground each decision of a large
language model (LLM) in a persistent
memory of the user, and in existing
agents that
memory is text: a narrative written and maintained by further LLM calls. Text limits this
memory in two ways. It is updated one rewrite at a time, so exploiting the full interaction history is prohibitively expensive; and collaborative evidence, graded similarity over an entire catalog, does not survive translation into sentences. We propose CoVeMem (Collaborative Vector
Memory), which vectorizes the collaborative core of the
agent's
memory. Frozen LightGCN user and item states form the
memory bank; at each decision, the candidate set itself retrieves the most relevant historical states, which enter the LLM's context as soft
tokens alongside a light textual profile. Contrastive
alignment to item-semantic anchors, followed by listwise co-training with masked candidates, teaches the model to read these states and to rank through them; a pointwise yes/no readout scores each candidate. Across four instruction-grounded recommendation
benchmarks, CoVeMem matches or exceeds the strongest collaborative text-
memory agent on 19 of 20 metric
cells while requiring zero additional LLM calls for
memory maintenance beyond the shared static profile, against per-interaction calls for text
memory. The
memory now takes gradients: the full interaction history, out of reach for text, becomes available as training data for what the
agent remembers and for how it reads what it remembers.
Read the full abstract ▾
All authors Hanchong Chen, Xing Tang, Lingjie Li, Xiongfeng Shan, Xiuqiang He
Information Retrieval · Cross-listed from cs.CL
Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers
Markus Frohmann, Mahdiyar Ali Akbar Alavi, Elizabeth Lingg et al. (+1 more)
In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one
prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that
prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. No
prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged
distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.
Read the full abstract ▾
All authors Markus Frohmann, Mahdiyar Ali Akbar Alavi, Elizabeth Lingg, Navid Rekabsaz
Information Retrieval
STREAM: An Objective-Driven and Uncertainty-Aware Framework for Industrial Energy Data Acquisition
Zhipeng Ma, Bo Nørregaard Jørgensen and Zheng Grace Ma
Industrial energy management requires
datasets that connect energy use with equipment states, production batches, material flows, and process conditions. However, conventional acquisition workflows commonly emphasize connectivity and storage without verifying whether accessible signals satisfy the requirements of a defined energy-performance assessment. This paper presents STREAM, an objective-driven and uncertainty-aware framework comprising Specification of Objectives, Technical Requirements, Resource Mapping, Extraction from Sources, Archival Metadata, and Migration to Database. STREAM is the central workflow: objective-to-data traceability is its end-to-end output, while measurement, temporal, contextual, and processing uncertainty are assessed across all six stages. Compared with the original conceptual STREAM sequence, this paper adds stage-level artifacts, minimum-evidence gates, source-suitability rules, a metadata template, an uncertainty rubric, and case-specific traceability
matrices. The framework is validated through two industrial batch-process cases: induction-furnace melting in a foundry and cheese-powder drying using SCADA and production-order data. The results demonstrate that data accessibility is not equivalent to analytical suitability and show how STREAM supports transparent decisions about immediate data use, analytical restrictions, and prioritized infrastructure improvements.
Read the full abstract ▾
Information Retrieval
Beyond a Single Story: Meta-Reviewing Sparse and Incomplete User-generated Contents for Recommendation
Hongren Wang, Tianjun Wei, Yingpeng Du et al. (+2 more)
Data sparsity remains a long-standing challenge in recommender systems, and it becomes more severe for methods relying on user-generated content (UGC) such as textual reviews, which capture fine-grained preferences but require more user efforts to produce. As a result, UGC exhibits (1) missing reviews, where interactions lack any review, and (2) incomplete reviews, where available reviews cover only a subset of relevant attributes. Existing approaches often overlook these UGC-specific issues, leading to degraded accuracy. Motivated by meta-review in academic peer review, we propose MOSAIC (Meta-review On Sparse And Incomplete user-generated Content), which constructs a meta-review for each target user by aggregating attribute-sentiment evidence from neighbor users' reviews. A multi-gate mixture-of-experts (MMoE) architecture jointly optimizes rating prediction and meta-review attribute-sentiment prediction, while an
attention module personalizes the aggregated meta-review signals to each target user, yielding both refined rating predictions and attribute-level explanations. Experiments on four real-world
datasets demonstrate that MOSAIC consistently outperforms state-of-the-art baselines in both recommendation accuracy and explanation quality, mitigating UGC sparsity and incompleteness while delivering consistent gains for users with limited interaction history.
Read the full abstract ▾
All authors Hongren Wang, Tianjun Wei, Yingpeng Du, Jie Zhang, Yin-Leng Theng
Information Retrieval
BLANC: Discovering Patent White Space via Changes in Normalized Pointwise Mutual Information Between Multi-View Clusters
Shuichi Miyazawa and Kensuke Fujii
Identifying white space — the unexplored but potentially valuable regions of a patent landscape — is essential for strategic R&D planning, yet existing methods rely on manual patent mapping or apply single-view clustering without quantitative gap detection. We propose BLANC (Blank Landscape Analysis through NPMI Conditioning), a three-phase pipeline combining (1) multi-view neural topic modeling along three semantic dimensions (application/use, novelty, inventive step); (2) Normalized Pointwise Mutual Information (NPMI) to quantify cross-dimensional cluster association; and (3) conditional detection that flags combinations whose NPMI drops when the corpus is filtered by a user-specified keyword. The drop is captured by a new metric, $\Delta$NPMI, which identifies combinations "established globally, unexplored locally." Because white space has no ground truth, we evaluate BLANC on two public USPTO corpora — machine learning/AI (5,417 patents, CPC G06N) and glass compositions (1,982 patents, CPC C03C) — by artificially depleting known technology combinations and testing recovery. When three-quarters of a target pair's documents are removed, BLANC recovers 34.1% (ML/AI) and 27.3% (glass) of the depleted combinations, whereas size-matched removals not aimed at them (random documents, or those of a different established combination) essentially never do: the target is never recovered in 191 decoy trials. Collapsing the three semantic views into one recovers nothing, while prior co-occurrence measures also flag the target under random removal, offering no specificity. In a proprietary case (302 float glass / glass-ceramics patents), the keyword "fluorine" reveals a fluorine surface treatment $\times$ warpage suppression candidate ($\Delta$NPMI up to 0.48) that experts had independently identified.
Read the full abstract ▾
Information Retrieval
When Does Supervised Fine-Tuning Reduce Instruction Sensitivity?
Jaekeol Choi
Large
language models can exhibit substantial performance variation across alternative formulations of the same task instruction, yet it remains unclear how conventional task-specific supervised fine-tuning (SFT) changes this instruction sensitivity. We study this question by evaluating fixed model checkpoints under multiple paraphrased instructions and defining instruction sensitivity as the standard deviation of task performance across them. We conduct a controlled scale analysis with Qwen3 models at 1.7B, 4B, and 8B on MS MARCO, together with targeted cross-family checks using Mistral-7B and Gemma-2-9B. Before SFT, instruction sensitivity decreases sharply with Qwen3 model scale. At 1.7B and 4B, SFT consistently reduces sensitivity across training instructions, with reductions of approximately 54–71%. At 8B, individual sensitivity changes are not statistically distinguishable from zero, but paired contrasts between training instructions are statistically reliable under query-level bootstrap analysis and have consistent directions across all three random seeds. Gemma-2-9B shows the same directional training-instruction contrast as Qwen3-8B, whereas Mistral-7B does not, suggesting that the strength of this effect also varies across models. Experiments on ESCI-English further show that free-generation and likelihood-based forced-choice evaluation can yield qualitatively different robustness conclusions even when valid-label generation is nearly perfect and average task performance is similar. Overall, SFT does not uniformly reduce instruction sensitivity: its robustness effect depends on the adaptation setting, while measured sensitivity can additionally depend on the prediction and scoring protocol.
Read the full abstract ▾
Information Retrieval · Cross-listed from cs.CV
PailitaoGR: Latent Think-with-Images for Generative Image Retrieval
Xiaomeng Fan, Yueran Liu, Shengyu Zhou et al. (+6 more)
Generative
retrieval has demonstrated strong performance by directly generating product semantic identifiers (SIDs). Extending this paradigm to image search, however, is nontrivial because real-world query images contain diverse information, including the search target, useful auxiliary evidence, and irrelevant visual content. This requires the model to identify and focus on the search target while selectively utilizing auxiliary evidence. In this paper, we propose PailitaoGR, a Latent Think-with-Images method for generative image
retrieval, which internalizes target-focused perception and selective auxiliary-evidence utilization into a the generative
retrieval model, enabling Zooming without Cropping and Reading without OCR. Specifically, we design a target-focused perception mechanism that identifies and enhances visual
tokens of the search target, consisting of a target Enhancer and a learning strategy based on on-policy
distillation and
attention guidance loss, enabling the model to focus on search-target regions. We also design a selective auxiliary-evidence utilization mechanism that identifies and enhances visual
tokens of auxiliary evidence, including an auxiliary enhancer and an in-capacity incremental contrastive
distillation strategy, enabling the model to exploit auxiliary evidence. We construct training and validation sets sampled from real-world online image-search logs. Experiments show that our method outperforms existing baselines by an average of 13.8%, validating its effectiveness.
Read the full abstract ▾
All authors Xiaomeng Fan, Yueran Liu, Shengyu Zhou, Chenghan Fu, Wanxian Guan, Feng Li, Chuan Yu, Jian Xu, Bo Zheng