Computer Vision
DreamEdit3D: Personalization of Multi-View Diffusion Models for 3D Editing
Jinxin Ai, Matthias Nießner and Ziya Erkoç
While 2D
diffusion models have achieved remarkable success in identity-preserving personalization, extending this capability to 3D assets remains a significant challenge due to the complexities of multi-view consistency and spatial control. Inspired by these 2D advancements, we present a novel personalization method for text-guided 3D editing that enables compositional, object-level control through natural
language. Given a 3D input, we render orthogonal views and extract object-level
segmentation masks to isolate semantic components. We then learn distinct
token embeddings for each component through a tailored two-phase optimization strategy: multi-view textual inversion with
attention alignment, followed by full fine-tuning of multi-view
diffusion model. During inference, these disentangled
tokens seamlessly compose with editing
prompts to generate multi-view consistent images, which are subsequently lifted into high-fidelity textured 3D meshes. Extensive evaluations across diverse editing scenarios demonstrate that our method successfully transfers the flexibility of 2D personalization to 3D, achieving state-of-the-art edit faithfulness and identity preservation compared to existing baselines.
Read the full abstract ▾
Computer Vision
Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction
Kejun Ren, Lei Jin, Tianxin Huang et al. (+2 more)
Streaming 3D reconstruction under a strict constant-
memory budget hinges on how the recurrent state is updated as the stream evolves. We profile TTT3R-style per-
token gates across five
benchmarks and discover a structural bottleneck: the gate is intrinsically bounded in magnitude (median $0.31$; never exceeding $0.6$) and nearly frame-invariant, yielding an effective
memory horizon of only $\sim$3 frames per state
token, which serves as the structural origin of long-sequence drift. We trace this to a missing axis: existing inference-time methods modulate updates only at the per-
token, intra-frame level, while the orthogonal frame-level question of how strongly each frame should contribute to the state has been treated as content-independent. We close this gap with a scalar frame-level gate $\alpha_t \in (0, 1]$ derived in closed form from frame-to-frame changes of internal features—a graded write weight, inspired by classical Simultaneous Localization and Mapping (SLAM) keyframe selection, that never discards a frame and requires no parameters, no training, and no extra forward pass. Across six
benchmarks spanning camera pose,
video depth, and 3D reconstruction at sequence lengths up to $4,661$ frames, our gate cuts ATE by $51\%$ on long TUM-RGBD pose sequences, reduces AbsRel by $13.0\%$ on Bonn
video depth, and on KITTI long-sequence pose estimation surpasses both LongStream and Keyframe-VO in average ATE, while retaining strictly constant
memory at zero training cost. Code is available at https://github.com/rkj12345/AFG.
Read the full abstract ▾
All authors Kejun Ren, Lei Jin, Tianxin Huang, Lianming Xu, Li Wang
Computer Vision
Statistical Hand Shape Modeling from Clinical CT Scans Using Deep Learning and Implicit Skinning
Gokce Guven, Hasan Fehmi Ates, Deniz Karasahin et al. (+1 more)
Accurate
segmentation and statistical shape modeling of hand anatomy have significant implications for medical diagnostics, ergonomics, and biomechanics. This study proposes an AI-assisted reconstruction pipeline for segmenting and analyzing hand anatomy from 1,271 elbow-to-hand (e2h-CT) computed
tomography scans. A Pix2Pix-based conditional generative
adversarial network is first employed to remove plaster cast and background artifacts from CT volumes. The cleaned scans are then processed in 3D Slicer to extract skin and bone masks, which are converted into closed-surface mesh models. Segmented bone meshes are used to construct skeletal representations, enabling implicit skinning to align all hand models into a standardized anatomical configuration. Subsequently, non-rigid registration is performed on the hand skin surfaces using the Geodesic Based Coherent Point Drift++ (GBCPD++)
algorithm to establish point-wise correspondence across subjects. Principal Component Analysis (PCA) is then applied to the registered models to quantify anatomical shape variability. The Pix2Pix preprocessing stage achieved a Dice coefficient of 0.9856 and an IoU of 0.9720 on the held-out test set. Statistical modeling was performed on a subset of 90 scans in which the fingers were fully visible and anatomically separated. The resulting statistical shape distributions demonstrate strong agreement with the U.S. Army Anthropometric Survey (ANSUR II), supporting the anatomical validity of the reconstructed models. The proposed methodology demonstrates significant potential for advancing biomechanical modeling, ergonomic optimization, prosthetic design, and precision medical diagnostics.
Read the full abstract ▾
All authors Gokce Guven, Hasan Fehmi Ates, Deniz Karasahin, Kaan Erdogan
Computer Vision
SHED: Style-Homogenized Embedding Alignment for Domain Generalization
Kai Gan and Tong Wei
Domain generalization aims to enhance model robustness against unseen domains with
embedding distribution shifts. While large-scale
vision-
language models like CLIP exhibit strong generalization, their direct image-text
embedding alignment suffers from inherent information asymmetry: images encode both class semantics and domain-specific styles, whereas text
prompts primarily convey basic class cues. This asymmetry hinders generalization to novel domains in realistic scenarios. To address this, we propose Style-Homogenized
Embedding alignment for Domain-generalization (SHED), a novel CLIP-based method that aligns style-homogenized
embeddings instead of raw representations from encoders in CLIP. During training, SHED removes domain-specific style centroids from both image
embeddings computed per source domains and text
embeddings which are averaged across diverse
prompt templates and stripped of a global centroid. For inference, considering the lack of target domain information, SHED projects diverse textual domain centroids into the visual space and aggregates predictions via membership weighting. Extensive experiments on five
benchmarks show SHED achieves state-of-the-art performance, outperforming prior methods significantly (e.g., +4.0% on DomainNet vs. standard fine-tuning).
Read the full abstract ▾
Computer Vision
Expandable, Compressible, Mineable: Open-World Thermal Image Restoration
Pu Li, Huafeng Li, Yafei Zhang et al. (+3 more)
In open-world settings, thermal infrared (TIR) image degradations continuously emerge and evolve, while most existing all-in-one restoration methods are built on a closed-set assumption and struggle to continually adapt to novel degradations. To address this, we propose ECMRNet, an Expandable, Compressible, and Mineable Restoration Network for open-world TIR restoration from a continual learning perspective. Conceptually, ECMRNet unifies continual degradation learning as an "expand-compress-mine" closed-loop process, enabling sustained adaptation to new degradations with controllable
evolution. Structurally, ECMRNet decomposes intermediate representations into group-isolated subspaces, and achieves strict parameter isolation and fast adaptation to new degradations by freezing historical groups and isomorphically expanding new ones. To curb model growth as tasks accumulate, we present Structural
Entropy Pruning, which identifies and removes redundant channel groups via two-dimensional structural
entropy minimization, achieving information contribution-driven adaptive compression. Moreover, we design a Sub-degradation Knowledge Mining Module that dynamically retrieves and recombines transferable components from historical representations to improve restoration under compound degradations. Experimental results demonstrate that ECMRNet achieves superior overall performance across diverse single and compound degradations while using fewer parameters and lower computational cost. The source code is available at https://github.com/Kust-lp/ECMRNet.
Read the full abstract ▾
All authors Pu Li, Huafeng Li, Yafei Zhang, Wen Wang, Neng Dong, Jie Wen
Computer Vision
OmniVL-Guard Pro: A Tool-Augmented Agent for Omnibus Vision-Language Forensics
Jinjie Shen, Zheng Huang, Yuchen Zhang et al. (+7 more)
Existing
vision-
language forgery detection and grounding methods operate under a closed-world paradigm, assuming verification can be completed by the model alone. However, self-contained MLLMs are constrained by finite parametric knowledge, static training corpora, and limited perceptual resolution, creating a practical ceiling in dynamic open-world forensics – particularly for real-time event verification requiring external clues and forgery
segmentation demanding fine-grained scrutiny of local manipulations. To address these limitations, we shift from scaling up the self-contained model toward reaching beyond it. We propose OmniVL-Guard Pro, a tool-augmented
agent that extends unified forensics from closed-world prediction to open-world clues-driven
reasoning. OmniVL-Guard Pro integrates a tool environment spanning real-time event search, local cropping and zooming, edge-anomaly screening, face detection,
video frame extraction, and SAM3-based
segmentation. To generate high-quality tool-
reasoning trajectories, we introduce Tree-Structured Self-Evolving Tool Trajectory Generation, which produces diverse trajectories through seed guidance, guider-free self-
evolution, and weakly-hinted hard sample synthesis, yielding the Full-Spectrum Tool
Reasoning (FSTR)
dataset for training. We further propose Checker-Guided Agentic Reinforcement Learning (CGARL), which provides process-level supervision to penalize cases where the answer is correct but the
reasoning is distorted. Extensive experiments demonstrate that OmniVL-Guard Pro achieves state-of-the-art performance across various tasks, and exhibits strong zero-shot generalization. The FSTR
dataset and code for OmniVL-Guard Pro will be publicly released at https://github.com/shen8424/OmniVL-Guard-Pro.
Read the full abstract ▾
All authors Jinjie Shen, Zheng Huang, Yuchen Zhang, Yujiao Wu, Yaxiong Wang, Lechao Cheng, Shengeng Tang, Tianrui Hui, Nan Pu, Zhun Zhong
Computer Vision
Latent Action Control for Reasoning-Guided Unified Image Generation
Fuxiang Zhai, Sixiang Chen, Yingjin Li et al. (+4 more)
Unified
multimodal models can encode visual understanding and image generation within a shared backbone, yet understanding does not automatically translate into control: models may infer objects, relations, or knowledge cues but fail to instantiate them in the generated image. We propose Latent Action Control (LAC), which makes
reasoning actionable by representing it as hidden continuous actions inside a unified generator. Given a
prompt, LAC rolls out a role-structured latent trajectory for planning, internal visual drafting, diagnosis, and refinement, and injects these actions into the hidden stream that conditions flow-based generation, without producing
reasoning tokens or intermediate images. Since such action trajectories are unobserved, LAC learns them through prior-guided variational latent action
alignment from training-only rendered semantic priors, draft image features, and supervised halting signals, followed by Latent-Flow GRPO to align the latent-to-image rollout with terminal visual feedback. This provides a control path from inferred relations, bindings, and knowledge cues to the generation process. Instantiated on BAGEL-7B-MoT, LAC consistently improves compositional and knowledge-grounded generation across GenEval, WISE, and T2I-CompBench, with the largest gains on spatial relations, attribute binding, and world-knowledge-sensitive
prompts. Ablations and latent interventions show that the learned action trajectory is consumed by the generator, suggesting that unified generation benefits when understanding is not only encoded, but made actionable during generation.
Read the full abstract ▾
All authors Fuxiang Zhai, Sixiang Chen, Yingjin Li, Shuaibo Li, Jianyu Lai, Tengjun Huang, Lei Zhu
Computer Vision
Edit-GRPO: A Locality-Preserving Policy Optimization Framework for Image Editing
Shaodong Xu, Zexian Li, Zhendong Wang et al. (+5 more)
A fundamental challenge in image editing lies in preserving spatial locality: edits should improve targeted content without inadvertently altering surrounding regions. However, most optimization-based editing approaches treat images as holistic entities, causing global policy updates that undermine locality and introduce undesired context changes. We observe that this issue stems from a mismatch between localized editing intent and globally applied optimization signals. Motivated by this insight, we propose Edit-GRPO, preserving Locality while optimizing image editing, a locality-preserving policy optimization framework that explicitly decouples editing and preservation objectives. By assigning region-specific optimization signals to edit and non-edit areas, Edit-GRPO aligns policy updates with the spatial structure of editing tasks, enabling localized improvements while maintaining global visual coherence. This design effectively suppresses common artifacts such as context distortion and boundary inconsistency. Extensive experiments across diverse image editing scenarios demonstrate that Edit-GRPO significantly improves locality preservation while maintaining strong editing performance compared to existing optimization-based methods, validating the generality and effectiveness of the proposed framework.
Read the full abstract ▾
All authors Shaodong Xu, Zexian Li, Zhendong Wang, Litong Gong, Tiezheng Ge, Wengang Zhou, Bo Zheng, Houqiang Li
Computer Vision
Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers
Shaodong Xu, Zhendong Wang, Litong Gong et al. (+4 more)
Recent advances in
Diffusion Transformers (DiTs) demonstrate that aligning noisy latent states with well-trained semantic features-as pioneered by Representation
Alignment (REPA)-can substantially accelerate training and improve generation fidelity. Subsequent analysis(e.g., iREPA) suggests that these gains arise primarily from transferring spatial structure contained in pre-trained
vision representations. However, mostly existing
alignment methods employ point-wise matching objectives or rely on implicit architectural tweaks, which fail to explicitly model the spatial relational geometry inherent in
vision foundation models. We argue that such element-wise supervision is insufficient to capture the rich spatial
topology of visual representations, and that effective
alignment for generation should instead be formulated as an explicit structural constraint. To this end, we propose sREPA, a structural REPresentation
Alignment framework to enforce consistency in the relational geometry of feature maps, rather than merely matching individual feature points. By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence, along with improved sample quality, compared to state-of-the-art
alignment strategies. Our code and models will be released.
Read the full abstract ▾
All authors Shaodong Xu, Zhendong Wang, Litong Gong, Zexian Li, Wengang Zhou, Tiezheng Ge, Houqiang Li
Computer Vision
DEVIS-GRPO: Unleashing GRPO on Dynamic Extreme View Synthesis
Yi Zuo, Huimin Wu, Lingling Li et al. (+3 more)
Trajectory-controlled
video generation has become essential for controllable
video generation. While current methods perform well under small-view camera motions, they degrade significantly with large-view motions. Existing solutions for extreme-view synthesis typically require dedicated
video pairs, demanding substantial annotation effort. To address these limitations, we propose Dynamic Extreme VIew Synthesis-GRPO (DEVIS-GRPO), a GRPO-based framework for trajectory-controlled
video generation, the first online policy gradient method for extreme view
video generation. Central to our approach is a novel sampling strategy: Accumulative Dynamic Extreme VIew Synthesis (ADEVIS), which achieves large-view camera motions by progressively accumulating small-view increments. This method delivers two key advantages: 1) enhanced training efficiency, as it eliminates the need to warm-start the policy model by collecting expensive paired large-view
videos, and 2) increased sampling diversity, achieved by flexibly varying trajectory configurations. Finally, we designed a multi-level consistency-quality reward function to select high-quality samples for model optimization. Experiments on the Kubric-4D, iPhone, and DL3DV
datasets demonstrate our method's superiority. On Kubric-4D, we achieve relative improvements of 21.57% in PSNR and 7.31% in SSIM over the second-best method in non-occlusion areas. On iPhone, LPIPS is reduced by 18.56%.
Read the full abstract ▾
All authors Yi Zuo, Huimin Wu, Lingling Li, Fang Liu, Licheng Jiao, Qing Li
Computer Vision
P2GS: Physical Prior-guided Gaussian Splatting for Photometrically Consistent Urban Reconstruction
Kota Shimomura, Hidehisa Arai, Tsubasa Takahashi et al. (+2 more)
3D Gaussian Splatting (3DGS) has recently emerged as a powerful explicit representation enabling fast, high-fidelity rendering, making it a promising foundation for closed-loop simulators and perception models in autonomous driving. However, conventional 3DGS implicitly assumes consistent exposure and tone mapping across views. Real driving data violates this assumption due to heterogeneous camera pipelines and dynamic outdoor illumination, baking exposure discrepancies and sensor noise into the radiance field and producing artifacts and inconsistent illumination especially in static backgrounds crucial for realistic simulation. These issues are amplified in autonomous driving, where sparse viewpoints, varying exposures, and outdoor lighting interact, while prior work mainly targets dynamic-object reconstruction and overlooks cross-view photometric consistency. To address this limitation, we introduce P2GS, a physically consistent Gaussian Splatting framework that jointly decomposes a view-invariant linear HDR radiance field, per-view exposure scales, and tone-mapping functions from only LDR images without HDR supervision. P2GS employs a unified optimization strategy grounded in the physical image-formation process, enforcing relative-exposure consistency and HDR-domain radiance regularization. This yields a radiance field robust to inter-camera illumination differences while preserving the real-time efficiency of standard 3DGS. Experiments across real and simulated driving environments show that P2GS matches or surpasses prior methods in LDR reconstruction while providing substantially improved photometric consistency, reliable exposure normalization, and physically coherent illumination across diverse scenes.
Read the full abstract ▾
All authors Kota Shimomura, Hidehisa Arai, Tsubasa Takahashi, Takayoshi Yamashita, Hironobu Fujiyoshi
Computer Vision
Neuroscience-inspired Staged Representation Learning with Disentangled Coarse- and Fine-Grained Semantics for EEG Visual Decoding
Xiang Gao, Hui Tian, Yanming Zhu et al. (+2 more)
Decoding visual information from electroencephalography (
EEG) signals remains a fundamental challenge in
brain-computer interfaces and medical rehabilitation. Existing
EEG visual decoding methods mainly focus on learning a single global
EEG embedding for cross-modal
alignment, but they largely overlook the staged and hierarchical characteristics of human visual processing. To address this limitation, we propose a neuroscience-inspired staged representation learning framework that reformulates
EEG visual decoding as a stage-specific representation decomposition problem. The proposed framework organizes
EEG representation learning into three complementary phases: low-level visual representation learning, high-level semantic representation learning, and integrative information fusion. To strengthen semantic modeling, we further introduce a
multimodal dual-level semantic learning mechanism that separates coarse label-level semantics from fine image-level visual-semantic information. In addition, semantic latent channels are introduced as computational representation channels generated from observed visual
EEG signals, expanding the channel-level semantic representation space for structured semantic abstraction and cross-modal
alignment. Extensive experiments on the THINGS-
EEG benchmark demonstrate that the proposed method achieves superior performance under subject-dependent zero-shot evaluation and improved exact
retrieval under subject-independent zero-shot evaluation. Additional analyses, including layer-wise
retrieval, temporal accumulation, expanded multi-image
retrieval, and ablation studies, further support the effectiveness of staged decomposition and structured semantic modeling. These results suggest that explicitly modeling staged perceptual, semantic, and integrative representations provides an effective neuroscience-inspired framework for
EEG-based visual decoding.
Read the full abstract ▾
All authors Xiang Gao, Hui Tian, Yanming Zhu, Xuefei Yin, Alan Wee-Chung Liew
Computer Vision
Motion Cues from Image-based Point Tracking for LiDAR Scene Flow Estimation
Youngdong Jang, Gyeongrok Oh, Jong Wook Kim et al. (+6 more)
Self-supervised
LiDAR scene flow approaches rely on pseudo-labels that distinguish static and dynamic points, providing targeted supervision. These labels are typically derived from
LiDAR geometry, making them unreliable in sparse or occluded regions. The resulting noise misguides motion supervision and degrades scene flow learning, yet existing methods remain largely dependent on the same
geometric evidence. To address this, we introduce TrackCue, a tracking-guided framework that transfers dense image trajectories into reliable static-dynamic supervision. TrackCue anchors tracking queries to projected
LiDAR points, yielding image-space trajectories that provide motion evidence beyond sparse
LiDAR geometry. Since image-space displacement also reflects ego-vehicle motion, we present a visually consistent motion compensation strategy that compares each tracked trajectory with the rigid trajectory induced by ego-motion alone. It admits motion evidence only where both trajectories are valid and the deviation persists across frames, isolating true object motion while suppressing transient tracking errors. Finally, we perform visual motion cue lifting, which associates image-space object motion cues with
LiDAR points for label refinement. As a result, TrackCue substantially improves the precision of dynamic pseudo-labels, providing more reliable supervision and leading to better self-supervised scene flow estimation.
Read the full abstract ▾
All authors Youngdong Jang, Gyeongrok Oh, Jong Wook Kim, Hyunju Ryu, Hyung-gun Chi, SeungHyeon Kim, Seungryong Kim, Jonghyun Choi, Sangpil Kim
Computer Vision
HighSync: High-Quality Lip Synchronization via Latent Diffusion Models
Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi et al. (+1 more)
We present HighSync, an end-to-end
diffusion-based framework for high-fidelity lip synchronization that generates photorealistic talking-face
videos aligned with arbitrary input audio. Existing approaches consistently struggle to reconcile image quality with synchronization accuracy, producing either visually degraded outputs or temporally inconsistent lip movements. HighSync addresses both challenges simultaneously and, to our knowledge, is the first lip sync model to operate natively at 512*512 resolution, positioning it as a viable solution for professional production environments such as the film and broadcast industries. Central to our approach is the identification and systematic elimination of a data leakage phenomenon that has silently undermined temporal modeling in prior work, preventing models from developing a genuine dependence on the audio signal. Comprehensive evaluations across both perceptual quality and synchronization accuracy metrics confirm that HighSync achieves state-of-the-art performance on both fronts. Source code, pre-trained models, and supplementary
video results are publicly available at: https://github.com/saeed5959/high_sync
Read the full abstract ▾
All authors Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi, Mahdi Bagheri
Computer Vision
VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction
Xun Chen, Tianchen Deng, Rui Wang et al. (+5 more)
3D semantic occupancy prediction requires accurate 2D-to-3D feature lifting, yet current methods restrict camera geometry to initial projections. Subsequent operations like offset learning,
attention weighting, and cross-camera aggregation remain geometry-agnostic, ignoring essential physical constraints. We propose VGGT-Occ, a framework that embeds
geometric tokens throughout the entire pipeline. We introduce Projection-Aware Deformable
Attention (PA-DA) to inject geometry into all
attention stages. PA-DA projects 3D offsets back to image planes and leverages the projection Jacobian as an additive bias to suppress unreliable observations. Features are then integrated through a view-quality semantic gate for cross-view consistency. To optimize both efficiency and performance, we employ a sequential coarse-to-fine decoder with gated fusion, where low-resolution features are refined into higher resolutions, allocating computation by information density while substantially reducing decoder cost. Extensive evaluations demonstrate the effectiveness and accuracy of our approach. On SurroundOcc-nuScenes, VGGT-Occ achieves 33.00% IoU and 21.08% mIoU ($T{=}1$), and 33.64% IoU and 21.43% mIoU with $T{=}2$ inference, outperforming existing methods, with only ${\sim}41$M trainable parameters in the occupancy head. Code will be released publicly.
Read the full abstract ▾
All authors Xun Chen, Tianchen Deng, Rui Wang, Fangjinhua Wang, Junyi Ma, Hongming Shen, Hesheng Wang, Danwei Wang
Computer Vision · Cross-listed from cs.ET
BIDO: A Biometric Identity Online Authentication Framework
Aditya Mithra, Sibi Chakkaravarthy S and Srinivas Kankanala
Security systems demand continuous, cryptograph- ically robust identity verification without requiring subjects to carry physical
tokens, smart cards, or dedicated hardware authenticators. This paper presents BIDO (Biometric Identity Online), a device-free authentication standard that achieves Au- thenticator Assurance Level 2 (AAL2) per NIST SP 800-63B with- out storing long-lived biometric templates, facial images, or any other form of Personally Identifiable Information (PII). BIDO derives
Elliptic Curve Digital Signature
Algorithm (ECDSA) key material deterministically from a live biometric measurement salted with a user-defined memorized secret at every authen- tication event, eliminating persistent private-key storage while enabling verification from any commodity sensor terminal. The generated credentials are non-discoverable (non-resident) Web Authentication (WebAuthn) credentials, fully compatible with all FIDO2-enabled websites and services without modification on the server side. A multi-stage pipeline, comprising capture of 200 valid biometric samples, feature extraction using the Dlib 68- point facial landmark predictor, affine face
alignment, frontality gating, Euclidean distance computation from the inter-eye mid- point, floor-division quantization with divisor q = 8, inter-session drift stabilization, and majority-voting SHA-256 hash binding, produces a Verification Seed (Vseed) from which the WebAuthn credential is transiently derived and immediately zeroized after signing. Evaluated against three prominent face
benchmarks (VGGFace2, LFW, and MegaFace), achieving 99.51% verification accuracy on LFW and 92.14% Rank-1 identification accuracy on MegaFace Challenge 1 at 10^6 distractors, with a cryptographic False Accept Rate (FAR) of 0.03%, a False Reject Rate (FRR) of 0.90%.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
AIM: Adversarial Information Masking for Faithfulness Evaluation of Saliency Maps
Chia-Ying Hsieh, Hsin-Yuan Fang and Chun-Shu Wei
Post-hoc saliency methods are widely used to interpret deep neural networks, but their faithfulness is difficult to evaluate reliably. Existing evaluations mask features according to saliency-induced feature ordering and measure performance degradation, but this degradation can be confounded by the masking operator: zero masking may create out-of-distribution artifacts, while interpolation-based masking may preserve residual predictive information. We propose
Adversarial Information Masking (AIM), a saliency-guided
adversarial feature replacement framework for evaluating both saliency-map faithfulness and masking-operator reliability. AIM replaces selected features with values from an
adversarial counterpart of the input and compares degradation under complementary masking orders. We assess reliability using random-attribution bias and stability of explanation-method faithfulness rankings. Experiments on image, audio, and
EEG tasks suggest that AIM reduces masking-induced bias compared with zero and interpolation-based masking, while revealing modality-dependent differences between signed and unsigned attributions.
Read the full abstract ▾
Computer Vision
WOW-Seg: A Word-free Open World Segmentation Model
Danyang Li, Tianhao Wu, Bin Li et al. (+5 more)
Open world image
segmentation aims to achieve precise
segmentation and semantic understanding of targets within images by addressing the infinitely open set of object categories encountered in the real world. However, traditional closed-set
segmentation approaches struggle to adapt to complex open world scenarios, while foundation
segmentation models such as SAM exhibit notable discrepancies between their strong
segmentation capabilities and relatively weaker semantic understanding. To bridge these discrepancies, we propose WOW-Seg, a Word-free Open World
Segmentation model for segmenting and recognizing objects from open-set categories. Specifically, WOW-Seg introduces a novel visual
prompt module, Mask2
Token, which transforms image masks into visual
tokens and ensures their
alignment with the VLLM feature space. Moreover, we introduce the Cascade
Attention Mask to decouple information across different instances. This approach mitigates inter-instance interference, leading to a significant improvement in model performance. We further construct an open world region recognition test
benchmark: the Region Recognition
Dataset (RR-7K). With 7,662 classes, it represents the most extensive category-rich region recognition
dataset to date. WOW-Seg attains strong results on the LVIS
dataset, achieving a semantic similarity of 89.7 and a semantic IoU of 82.4. This performance surpasses the previous SOTA while using only one-eighth the parameter count. These results underscore the strong open world generalization capabilities of WOW-Seg. The code and related resources are available at https://github.com/AAwcAA/WOW-Seg-Meta.
Read the full abstract ▾
All authors Danyang Li, Tianhao Wu, Bin Li, Zhenyuan Chen, Yang Zhang, Yuxuan Li, Ming-Ming Cheng, Xiang Li
Computer Vision
CAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
Houji Wen, Jiangyong Yu, Jun Li et al. (+1 more)
Segment Anything Models (SAMs) are extensively used in computer
vision for universal image
segmentation, but deploying them on resource-constrained devices is challenging due to their high computational and
memory demands. Post-Training Quantization (PTQ) is a widely used technique for model compression and acceleration. However, existing PTQ methods fail to consider the cross-
attention architecture in the SAM decoder. This degradation primarily stems from the unique challenges posed by SAMs: (1)
Attention dissipation, where the
attention information in the decoder, which is crucial for representing
segmentation masks, collapses into a diffuse and non-semantic form under low-bit quantization; and (2) Reconstruction oscillation, where bidirectional coupling within the two-way
transformer introduces cross-branch error interference and destabilizes convergence. To tackle these issues, we propose CAR-SAM, a unified quantization framework tailored for SAMs. Firstly, to mitigate
attention dissipation, we introduce MatMul-Aware Compensation (MAC) mechanism that transfers activation-induced quantization errors from MatMul to preceding linear weights. Secondly, to mitigate oscillation in decoder optimization, we develop a Joint Cross-
Attention Reconstruction (JCAR) strategy that jointly reconstructs coupled
attention branches, suppressing oscillatory behavior and promoting stable convergence. Extensive experiments show that CAR-SAM robustly quantizes SAM models down to 4-bit precision, surpassing existing methods by 14.6% and 6.6% mAP on SAM-B and SAM-L respectively.
Read the full abstract ▾
All authors Houji Wen, Jiangyong Yu, Jun Li, Dawei Yang
Computer Vision
LASAR: Towards Spatio-temporal Reasoning with Latent Cognitive Map
Jinzhou Tang, Sidi Liu, Waikit Xiu et al. (+2 more)
A fundamental challenge in
embodied AI is verifying if
agents build internal models of spatial structure or merely learn to mimic task-specific expert trajectories. This is critical as foundational approaches rooted in action-centric tasks (e.g., VLN) and
reasoning-centric tasks (e.g., EQA) often share a common limitation: they lack a learning signal that forces them to encode fine-grained spatial relationships (like
topology or distance) over long-range, fragmented experiences. To address this, we first propose LASAR, an architecture featuring a dual-
memory system designed to maintain both episodic experiences and a semantic cognitive map. We then introduce Spatio-temporal Contextual Representation Learning (ST-CRL), a contrastive objective designed to train this architecture. ST-CRL leverages spatio-temporal cues from cognitive queries generated through annotated spatio-temporal context in simulation to build sample pairs, thereby forming the internal cognitive map from the
agent's experiences. Experiments demonstrate that our method achieves 2%-3.5% gains in both zero-shot generalization on standard VLN-CE and VSI-Bench
benchmarks. We also demonstrate that our proposed cognitive map has high self-consistency.
Read the full abstract ▾
All authors Jinzhou Tang, Sidi Liu, Waikit Xiu, Weixing Chen, Keze Wang
Computer Vision
DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta et al. (+1 more)
Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent
Multimodal Large
Language Models (MLLMs) have shown promise on general
vision-
language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural
language descriptions. Specifically, our method first generates spatially grounded captions enriched with
multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit
language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA
benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe
Read the full abstract ▾
All authors Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta, C. V. Jawahar
Computer Vision
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
Chenglizhao Chen, Yuchen Cao, Xinyu Liu et al. (+3 more)
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference
alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference
alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference
alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype
retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.
Read the full abstract ▾
All authors Chenglizhao Chen, Yuchen Cao, Xinyu Liu, Mengke Song, Guisheng Zhang, Xiaomin Yu
Computer Vision
Mind the Gap: Learning Modality-Agnostic Representations with a Cross-Modality UNet
Xin Niu, Enyi Li, Jinchao Liu et al. (+3 more)
Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an
adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap.
Read the full abstract ▾
All authors Xin Niu, Enyi Li, Jinchao Liu, Yan Wang, Margarita Osadchy, Yongchun Fang
Computer Vision
Towards Generalized Image Manipulation Localization via Score-based Model
Yunfei Wang, Bo Du, Zhe Yang et al. (+4 more)
With the rapid
evolution of synthetic media, Image Manipulation Localization (IML) has emerged as a critical component in multimedia forensics for ensuring the integrity of digital content. However, generalization remains a core challenge, as existing discriminative methods typically learn a fixed decision boundary that tends to overfit to specific training artifacts and fails to adapt to unseen manipulation types. To address this, we propose DiffIML, a novel framework that introduces score-based generative modeling to IML. Diverging from the direct estimation of hard boundaries, DiffIML approximates the score function, the gradient of the log-likelihood, to capture the intrinsic
geometric topology of mask distributions. This paradigm leverages structural priors to iteratively recover coherent masks from noise, thereby circumventing the brittleness associated with discriminative models. Under this formulation,
diffusion models serve as an effective numerical solver for the learned score function.To ensure practicality, we respectively resolve the efficiency and stability bottlenecks of standard
diffusion by: (1) utilizing a Lightweight Mask-Specific VAE for fast latent-space process and a decoupled architecture with a lightweight denoising UNet, (2) edge supervision and error prior to mitigate error accumulation during sampling. Extensive experiments of two distinct protocols on eight non-generative and three generative
benchmarks demonstrate that DiffIML consistently outperforms state-of-the-art methods, yielding remarkable generalization improvements on diverse unseen
datasets. The code is publicly available at https://github.com/scu-zjz/DiffIML.
Read the full abstract ▾
All authors Yunfei Wang, Bo Du, Zhe Yang, Xin Liu, Zhiyu Lin, Tianxin Xu, Ji-Zhe Zhou
Computer Vision
Zero-Shot Faithful Textual Explanations via Directional-Derivative Influence on Predictions
Toshinori Yamauchi, Hiroshi Kera and Kazuhiko Kawamoto
Zero-shot textual explanations aim to make image classifiers more transparent by probing their internal representations, without relying on task-specific supervision or LVLMs. However, existing methods often miss the features that truly drive the prediction, resulting in limited faithfulness to the evidence underlying the model's decision. To address this, we propose FaithTrace. Motivated by the idea that faithful explanations should describe concepts that strongly influence the prediction, FaithTrace directly measures how much the representation induced by the explanation changes the class logit. We introduce an influence score, computed as the directional derivative of the class logit along the text-induced direction in the classifier's feature space, and use it as a proxy for faithfulness. Moreover, we extend this influence score into quantitative evaluation metrics, helping fill the gap in faithfulness evaluation for textual explanations. Experiments show that FaithTrace yields more faithful explanations than baselines, facilitating a more accurate understanding of the model. The code will be publicly released.
Read the full abstract ▾
Computer Vision
Xi Liu, Weiwei Sun, Zhou Ren et al. (+3 more)
Diffusion priors have recently demonstrated strong capability in enhancing the quality of sparse-view 3D reconstruction by augmenting training views at novel viewpoints, but they inevitably introduce hallucinated content – artifacts inconsistent with the input views – into the final 3D model. To address this challenge, we propose
Hallucination-Aware
Diffusion prior (HAD), which estimates pixel-wise
hallucination score maps for augmented images by leveraging multi-view
reasoning capabilities from a feedforward novel view synthesis (NVS) network pre-trained on large-scale 3D data. These
hallucination scores enable selective masking of unreliable pixels during the progressive 3D reconstruction procedure, preventing the introduction of non-existent artifacts into the 3D model. To further enhance performance, we create multiple versions of augmented images at each novel view by conditioning the
diffusion prior on different input views, which are then fused into a final image that leverages the broader context across all input views. We show that our method substantially reduces
hallucination artifacts in
diffusion-assisted 3D reconstruction, thereby achieving state-of-the-art performance across multiple
benchmarks on novel view synthesis. Our project are publicly available at \href{https://xiliu8006.github.io/HAD-Project-website/}{project website}.
Read the full abstract ▾
All authors Xi Liu, Weiwei Sun, Zhou Ren, Chris Broaddus, Siyu Huang, Laurent Guigues
Computer Vision
Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks
Yachan Guo, JoseLuis Gomez Zurita, Danna Xue et al. (+2 more)
Although large-scale visual foundation models (VFMs) achieve remarkable performance in semantic understanding, they still underperform in instance-aware dense prediction tasks. They exhibit different biases in representation: for instance, promptable
segmentation models (e.g., SAM2) focus on fine-grained region boundaries, while self-supervised models (e.g., DINOv3) emphasize object-level structure. This observation highlights the potential of combining complementary features from different VFMs to enhance downstream dense prediction tasks. However, naive multi-VFM fusion seldom leads to reliable gains, and interpretable principles for leveraging their complementary features are still underexplored. In this work, we propose a metric-guided approach that effectively selects and aggregates complementary features from different VFMs based on explicit assessment scores. Specifically, we design a suite of label-free metrics in feature space across two aspects, Structural Coherence and Edge Fidelity, to assess features of VFM encoders. Guided by these scores, we identify complementary edge-strong and structure-strong encoder pairs, and integrate them via a master-auxiliary fusion scheme. This feature fusion requires no complex architectural changes and is trained only in a single stage. Our model shows consistent performance gains across multiple dense prediction tasks compared with the baselines, with better object-level semantics and more accurately localized boundaries. The code is available at {https://github.com/gyc-code/metric-guided-fusion}.
Read the full abstract ▾
All authors Yachan Guo, JoseLuis Gomez Zurita, Danna Xue, Yi Xiao, AntonioManuel Lopez Pena
Computer Vision
Prefix-Adaptive Block Diffusion for Efficient Document Recognition
Mingxu Chai, Ziyu Shen, Chenyu Liu et al. (+3 more)
Block
Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing. However, existing BDMs bind denoising and cache commitment to fixed block boundaries: parallelism shrinks during intra-block denoising, while generated
tokens cannot be cached until the whole block is completed. Moreover, intra-block bidirectional denoising conflicts with inter-block autoregression, creating inconsistent information flow that can challenge structure-sensitive recognition. We propose the Prefix-Adaptive Block
Diffusion Model (PA-BDM), which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix and treats the block size as a maximum candidate range rather than a fixed commitment unit. PA-BDM uses Confidence-gated Structural Loss (CSL) to build low-
entropy prefixes before extending training to longer continuations. During inference, Progressive Prefix Commitment (PPC) then dynamically commits the longest reliable prefix into the KV cache and resets the next candidate range from the updated prefix, restoring a large parallel decoding space at each step. Experiments show that the 3B PA-BDM achieves higher recognition scores on several
benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-
Diffusion.
Read the full abstract ▾
All authors Mingxu Chai, Ziyu Shen, Chenyu Liu, Jihua Kang, Tao Gui, Qi Zhang
Computer Vision
VGGT-CD: Training-Free Robust Registration for 3D Change Detection
Wei Zhang, Songhua Li, Yihang Wu et al. (+2 more)
3D change detection from multi-view images is essential for urban monitoring, disaster assessment, and autonomous driving. However, existing methods predominantly operate in the 2D domain, where viewpoint variations are mistaken for physical changes and depth is unavailable. While visual geometry foundation models like VGGT rapidly produce dense point clouds from unposed images, independent per-epoch reconstruction encounters fundamental obstacles: unpredictable inter-epoch scale ambiguity, registration-change paradox where scene changes corrupt
alignment, and pervasive edge-flying noise. To address these challenges, we present VGGT-CD, a training-free pipeline decoupling cross-temporal registration from dynamic-change interference. In the Coarse Stage, sparse keyframe joint inference establishes a unified metric space and yields an initial Sim(3) prior. In the Fine Stage, dense reconstructions are purified by isolating static-background correspondences. A closed-form centroid
alignment refines the translation while locking scale and rotation, using a residual self-check to mathematically guarantee non-degradation. Evaluated on an 11-scene
benchmark from the World Across Time
dataset, VGGT-CD reduces Absolute Trajectory Error by 44% outdoors and 59% indoors. It completes registration over 6 times faster, producing high-purity 3D change maps without task-specific training.
Read the full abstract ▾
All authors Wei Zhang, Songhua Li, Yihang Wu, Qiang Li, Qi Wang
Computer Vision
Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction
Yichang Jian, Boyuan Xiao, Zhenyuan Huang et al. (+2 more)
Planning from raw visual input remains a significant challenge for current
Vision-
Language Models (VLMs), when the complexity of input is beyond their one-step perception capability. Motivated by recent advances in Thinking with Images (TWI), a reasonable solution is to decompose the perception process into simpler steps by iteratively acquiring and incorporating local visual evidence. However, even though current VLMs are well-trained in general TWI ability, their perceptual bottleneck in the planning domain remains. To tackle this challenge, we formulate TWI as a tool to gradually build and reflect an accurate internal world model. We find that the resulting training-free planning strategy enables VLMs to solve tasks that are far beyond their initial capabilities, at the cost that too many TWI operations would significantly increase the computational overhead. To further improve efficiency, we propose Pattern Inference, a novel TWI strategy enabling VLMs to actively recognize known visual patterns in the new tasks and directly infer local world model structures. To obtain these patterns, we propose Pattern Induction, an online inductive learning strategy treating visual patterns as composite and reusable experts, which are autonomously discovered and optimized from experience. Experimental evaluations in FrozenLake, Crafter and CubeBench domains show that our approaches achieve a desirable balance between accuracy and efficiency.
Read the full abstract ▾
All authors Yichang Jian, Boyuan Xiao, Zhenyuan Huang, Yifei Peng, Yao-Xiang Ding