Computer Vision
FedMAD: Modulation-Aware Directional Aggregation for Federated Learning in Remote Sensing Image Classification
Bar{\i}ş Büyüktaş and Begüm Demir
Federated learning (FL) has recently attracted increasing
attention in remote sensing (RS) since it enables collaborative model training across decentralized RS image archives without requiring direct access to local data. However, FL performance significantly degrades when the data distributions between clients are heterogeneous, which often occurs due to geographical differences, seasonal changes, and varying image acquisition and atmospheric conditions. To address this challenge, in this letter, we propose a novel personalized FL framework (denoted as FedMAD) for RS image classification problems. The proposed framework separates globally shared representation parameters from client-specific adaptation parameters to preserve client-specific features while maintaining globally transferable representations. This is achieved by integrating lightweight modulation modules and local batch normalization layers into the backbone network. Although globally shared parameters are collaboratively optimized between clients, client-specific parameters remain local to preserve domain-specific feature characteristics. In addition, FedMAD introduces a modulation-aware directional aggregation strategy that dynamically adjusts the importance of aggregation for each client according to the
alignment of local modulation updates. This allows the global optimization process to suppress conflicting client updates originating from heterogeneous data distributions while enhancing the contribution of clients with consistent adaptation behaviors. The experimental results obtained on the BigEarthNet-S2 and EuroSAT
datasets demonstrate the effectiveness of FedMAD compared to state-of-the-art FL
algorithms under heterogeneous RS data distributions. The code of the proposed framework will be publicly available at https://git.tu-berlin.de/rsim/fedmad.
Read the full abstract ▾
Computer Vision
Soundwich: Video Generation with Layered and Controllable Audio
Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek et al. (+2 more)
Recent joint audio-
video generative models can synthesize realistic
videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where
speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-
video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared
video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the
video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.
Read the full abstract ▾
All authors Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek, Daniel Cohen-Or, Ali Mahdavi-Amiri
Computer Vision
SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
Mikhail Dereviannykh, Vikram Voleti, Simon Donne et al. (+3 more)
Recent
video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of
diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse
tokens carry the clip's global semantics while later
tokens further specify details. Existing flexible tokenizers only apply a representation-
alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible
video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained
token prefix alone. SemanTok achieves high semantic
alignment and
video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic
alignment on out-of-distribution classes and gives the decoder higher semantic
alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short
token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later
tokens.
Read the full abstract ▾
All authors Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss
Computer Vision · Cross-listed from cs.LG
Curvature Under Attack in hZACH-ViT: Gauge Symmetry, Boundary Saturation, and Adversarial Failure
Athanasios Angelakis and Marta Gomez-Barrero
Curvature is often treated as an intrinsic property of a representation, although its empirical effect also depends on coordinate scale, learned logit temperature, and numerical safeguards. We study this interaction in hZACH-ViT, a compact
Vision Transformer with Euclidean, Poincare, and spherical prototype heads. The backbone architecture, seed-specific initialization, 50-per-class training subset, and optimization protocol are matched across three MedMNIST
datasets and five seeds. At the fixed comparison
curvature $c=1$, Poincare has the lowest class-macro PGD attack-success rate in all 12
dataset-budget
cells and under a stronger CE+DLR multi-restart attack on all three
datasets, but it also has the lowest clean MacroF1. An end-to-end
curvature intervention changes the interpretation. Reducing Poincare
curvature to $c=0.1$ improves clean MacroF1 in every one of the 15 paired seed-
dataset comparisons and removes hard boundary clipping, yet on OrganAMNIST it increases strong attack success from $89.7\%$ to $99.3\%$ (paired difference $+9.57$ points; 95% hierarchical bootstrap CI $[+5.52,+14.03]$). At $c=1$, $40$-$47\%$ of clean Poincare features are hard-clipped, the radial Jacobian of the inherited map is nearly zero, and dimensionless attack trajectories are unusually long and inefficient. The spherical head provides a control: its
curvature change is an exact scale gauge to floating-point precision and produces much smaller attack differences. These results do not establish intrinsic
hyperbolic robustness. They identify an implementation-sensitive regime in which
curvature, scale, and proximity to the Poincare boundary jointly organize clean recognition and
adversarial representation motion.
Read the full abstract ▾
Computer Vision
Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality
Elias Rotondo, Lin Duan, Yanming Xiu et al. (+3 more)
Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare delivery, and risk-mitigation protocols. However, optimizing for end-user immersion and comfort remains challenging, as AR head-mounted displays contend with constrained scene geometry, spatial jitter, and temporal instability. User studies are the standard AR evaluation method for visual quality, but their cost, diminishing scalability, and inflexibility pose bottlenecks during iterative application design. To address this problem, we present an automated framework for AR content evaluation and refinement, built on
vision-
language models (VLMs), to evaluate and predict the visual fidelity of AR scenes as perceived by users. First, we introduce RateAR, a
benchmark of AR images and
videos collected across diverse scenes and environmental conditions, with good-to-excellent reliability (ICC(2,5) >= .90) across perceptual factors, including object placement, scale, and shadow consistency. Subsequently, we evaluate eleven commercial VLMs on the crafted
benchmark. Results support that VLM-based quality predictions strongly correlate with human subjective judgments, achieving Spearman's rank-order correlations of up to 0.8695. An ablation study further suggests that, compared to other prompting strategies, our contextual prompting yields better
alignment with human ratings while balancing introduced complexity cues. Building on these findings, we construct an automated AR content adjustment system and conduct a 21-participant user study. More than 90% of participants found that the system improved placement and size coherence of virtual content.
Read the full abstract ▾
All authors Elias Rotondo, Lin Duan, Yanming Xiu, Sangjun Eom, Conrad Li, Maria Gorlatova
Computer Vision
VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision
Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao et al. (+6 more)
Qualitative comparison figures are central evidence in computer
vision papers, and
vision-
language models (VLMs) are increasingly used to judge them. Yet existing
benchmarks score only scalar quality or overall preference, so a judge can be rewarded for picking the preferred image for the wrong visual reason. We introduce VisionQ, the first
benchmark built from peer-reviewed CV comparison figures that grounds every judgment in a named visual criterion: each question states the criterion, and a judge is credited only when it selects the output the authors identify as best on that criterion. We call this task criterion-conditioned visual discrimination. VisionQ comprises (1) a corpus of 1,409 CVPR and ICCV papers with 1,800+ validated comparison figures and 3,911 hand-annotated data points linking method crops to author-stated visual claims; (2) a six-axis, 51-leaf taxonomy of the visual criteria behind qualitative judgment; (3) a criterion-conditioned evaluation protocol that hides method names, captions, and paper identity and reports accuracy per criterion; and (4) VisionQ-Judge, a DPO-tuned Gemma-4-E4B judge trained on symmetric evidence pairs, which reduces last-option predictions by 7.0pp and improves accuracy by 2.5pp on a held-out test set. Evaluating 20 open- and closed-source VLM judges, we find that the strongest reach only 63.1% accuracy (chance 32.2%) and that reliability varies sharply across criteria. Code: https://github.com/ReML-AI/visionq. Data: https://huggingface.co/
datasets/visionq-anon-2026/VisionQ-1k.
Read the full abstract ▾
All authors Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
Computer Vision
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Wenhan Yang, Anirudh Rao and Ashwin Chandra
Speculative decoding has achieved substantial lossless speedups for LLMs, but remains less effective for large
vision-
language models (LVLMs), where lightweight drafters struggle to use rich
multimodal information. A second limitation is that standard
distillation supervises the drafter only along the original training trajectory, without modeling how target predictions shift after the drafter's own proposals. As drafting moves away from this trajectory, the drafter can increasingly disagree with the target, reducing acceptance in later steps. We propose HAWK to address both limitations. HAWK uses representation similarity to select informative target layers and learns how to combine their hidden states. For visual information, it directly provides the drafter with compressed visual hidden states from the target model instead of raw visual
tokens, making the visual information easier for a shallow drafter to use. HAWK also trains the drafter to capture how target predictions change after its own proposals, improving its agreement with the target during multi-step drafting. On SmolVLM-256M across ten
multimodal benchmarks, HAWK raises average acceptance length from 3.32 to 4.08 and speedup from 2.19x to 2.60x over EAGLE-3 under greedy decoding, and from 2.89 to 3.41 and 1.92x to 2.19x under sampling.
Read the full abstract ▾
Computer Vision
Just Align $\bm{x}$: Aligning Predictions, Not Representations
Yuyao Zhang, Yuwei Hu, Ziyang Mai et al. (+1 more)
Representation
alignment has become an effective way to accelerate
diffusion training, but its benefits do not transfer reliably to pixel-space clean-image prediction. In JiT, we find that auxiliary feature
alignment can improve access to semantic features while reducing access to image variation needed for clean-image prediction, creating a mismatch between the auxiliary objective and the denoising task. This suggests a different principle: auxiliary supervision should improve the prediction target itself rather than impose a separate representation target. We introduce JAx (Just Align x), a prediction-supervision method that aligns clean-image predictions across noise levels. JAx couples a noisier student observation with a cleaner observation through a Markov degradation that preserves the original JiT input distribution. Under this coupling, the oracle prediction from the cleaner state has the same conditional mean as the optimal JiT target, while its conditional target covariance is no greater. Thus, oracle prediction
alignment preserves the population JiT objective up to a constant while providing a lower-variance training target. To make this construction practical with an imperfect EMA teacher, JAx combines ground-truth supervision with a reliability-gated coupling band that selects nearby teacher states based on prediction risk. On ImageNet 256x256, JAx consistently improves FID and accelerates convergence across JiT-B/16, L/16, and H/16, without an external encoder or changes to the architecture or sampling procedure. Gradient diagnostics further show reduced minibatch gradient variance, while ablations demonstrate that the gains cannot be explained by time reweighting alone. These results show that prediction-space supervision provides a simple and principled alternative to representation
alignment for pixel-space generative models.
Read the full abstract ▾
All authors Yuyao Zhang, Yuwei Hu, Ziyang Mai, Yu-Wing Tai
Computer Vision · Cross-listed from cs.LG
Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference
Opegbemi Matthias Busoye, Tolulope Matthew Busoye and Eghonghon-aye Eigbe
Activation
memory, not compute, limits CNN inference on constrained hardware such as microcontrollers. Direct in-place convolution removes the dual-buffer cost, but the
memory-optimal formulation of Gural and Murmann assumes valid padding, unit stride, unit dilation, and odd square kernels, and needs a non-sequential traversal costing $2\times$ inference time in transposes. We identify two regimes in which their published closed form does not hold: (1) an under-allocation of exactly $(k-1)C_{in} \bmod (C_{out}-C_{in})$ scalars, active on every convolutional layer of their own deployed network and manifesting as a silent corruption of still-live input; (2) an unbounded overestimate, up to $2{,}432\times$, once the critical leg leaves the output grid. We correct both and generalize to arbitrary stride, dilation, padding, and rectangular kernels. We then propose Right In-Place (RiP) convolution, a bit-identical operation in which every layer reads its input right-aligned in a shared workspace and writes its output left-aligned from index zero. The debt is piecewise affine in the output pixel index, so evaluating its breakpoints in $O(1)$ yields the minimum safe gap without enumerating the output grid, with row-major access preserved. Across $10{,}000$ random layers RiP produced no corruption, and across 84 convolutional layers from 25 architectures it matches the herringbone workspace exactly on 58 and within 5% on 81, using 24.8% less
memory than dual buffering on average. Written into TinyEngine's kernels and deployed to a Raspberry Pi Pico 1 and Pico 2, it cuts peak activation
memory across eleven MCUNet models by 12.5 to 33.3% at unchanged cycle counts and bit-identical outputs, raising the number of models that fit the Pico 1's 256 KB SRAM from six to nine.
Read the full abstract ▾
Computer Vision
Discrete Annotation, Continuous Preference: Rethinking Supervision for Accurate and Generalizable Aesthetic Image Cropping
Ziqing Zhang, Xiao Liu, Kai Liu et al. (+4 more)
Aesthetic image cropping aims to identify the optimal crop of an image in terms of aesthetics and composition. While supervision based on annotated data is fundamental, the field has been hindered by a long-standing problem: existing
datasets suffer from (1) human subjectivity and (2) rigid discreteness confined to fixed sampling grids. These flawed annotations not only limit the accuracy and generalization of trained models but also severely distort fair evaluation. To overcome this, we propose to model human cropping preference as a multi-peaked, continuous, and sharp field over the crop space. We introduce the Continuous Preference Field (CPF), which recovers a dense preference landscape from discrete annotations through (1) peak clustering, (2) off-
lattice refinement, (3) negative shaping, and (4) field assembly. Based on this, we train CPIC, a VLM-based cropping model optimized via GRPO with the CPF reward, which overcomes template collapse, achieving state-of-the-art performance and exceptional out-of-domain generalization. Finally, to resolve the long-standing
benchmark evaluation crisis, we introduce CPICD, a comprehensive recalibration of existing ground-truth boxes. By leveraging the CPF to correct grid-bound artifacts across mainstream
benchmarks, CPICD establishes a rigorous and reliable foundation for future cropping research. Extensive experiments and user studies demonstrate the superiority of our CPF, CPIC, and CPICD. Code, model, and data are available at https://github.com/zzqingz/CPIC.
Read the full abstract ▾
All authors Ziqing Zhang, Xiao Liu, Kai Liu, Jianze Li, Weihang Zhang, Linghe Kong, Yulun Zhang
Computer Vision
Gestalt: Large Multimodal Interplay Model
Zequn Yang, Yu Miao, Haotian Ni et al. (+8 more)
In this paper, we propose Gestalt, a new paradigm of large
multimodal model built around
multimodal interplay. Despite rapid advances, large
multimodal models are reaching a bottleneck: existing approaches focus primarily on accommodating additional modalities while overlooking the distinct characteristics of each modality and the relations among them. Motivated by the multistage property of human multisensory perception, we propose a
multimodal interplay pyramid that organizes
multimodal modeling as a progression from modality-specific processing, through cross-modal
alignment, to deeper
multimodal integration. Guided by this pyramid, Gestalt adopts a unified discrete
diffusion framework and an interplay-partitioned architecture, with learnable interplay
tokens mediating cross-modal exchange and integration. The pyramid also structures its data organization and training strategy. Strong performance across image generation,
multimodal understanding, and text-only evaluation shows that Gestalt significantly improves cross-modal integration while preserving modality-specific information, effectively harnessing the strengths of
diffusion-based
multimodal models and offering a promising path toward unified
multimodal intelligence. Project page: https://GeWu-Lab.github.io/Gestalt.
Read the full abstract ▾
All authors Zequn Yang, Yu Miao, Haotian Ni, Ziheng Chen, Chengxiang Huang, Dongzhan Zhou, Kai Chen, Qi Zhang, Ji-Rong Wen, Yake Wei, Di Hu
Computer Vision
FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering
Haifeng Huang, Biyin Xu, Chunsheng Xin et al. (+1 more)
Query-aware keyframe selection enables
multimodal large
language models (MLLMs) to process long
videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the
video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact
algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-
video question-answering
benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.
Read the full abstract ▾
All authors Haifeng Huang, Biyin Xu, Chunsheng Xin, Yang Li
Computer Vision
PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop
Xinge Peng, Yiting Lu, Tianwu Zhi et al. (+4 more)
Vision-
Language Models (VLMs) have shown strong
multimodal reasoning capabilities, yet whether they truly capture the physical consistency underlying real-world dynamics remains unclear. Existing
benchmark paradigms often suffer from fragmented evaluation, focusing on isolated cognitive stages while overlooking the inherent synergy between perception,
reasoning, and physical judgment. The lack of a holistic perspective limits the ability to diagnose whether VLMs can reliably serve as evaluators of physical plausibility in generative content. To address these issues, we introduce PhysVista, a
benchmark designed to evaluate physical intelligence in VLMs through a closed cognitive loop framework inspired by the human seeing-
reasoning-assessment process. PhysVista restores this loop by jointly evaluating physical state perception, physical dynamics
reasoning, and physical plausibility assessment. It further distinguishes event-level
reasoning and scale-level
reasoning to enable fine-grained analysis of physical understanding. In addition, PhysVista incorporates both real-world and AI-generated
videos, allowing evaluation across diverse domains and emerging generative scenarios. Extensive experiments across a diverse set of VLMs reveal substantial limitations in physical
reasoning and plausibility assessment, highlighting a persistent gap between visual recognition and genuine physical understanding, and pointing toward more principled designs for physically grounded
multimodal intelligence.
Read the full abstract ▾
All authors Xinge Peng, Yiting Lu, Tianwu Zhi, Wen Wen, Jianzhao Liu, Xin Li, Zhibo Chen
Computer Vision
Memorizon: Training World Models Beyond Their Context Window
Tingting Liao, Xuezhi Liang, Hao Li et al. (+1 more)
Streaming world models should render a place consistently across repeated visits. Directly supervising such revisits requires training samples that capture both visits, often spanning minutes. Yet dense
attention over the full span incurs quadratic costs, making long-span supervision expensive. Memorizon breaks this coupling: long spans are needed for supervision, but not for
attention, since the two visits can share a forward pass without including every intervening frame. A training sample covers a span of any length but is scored only on its last $k$ chunks. Instead of tokenizing the history before them, each scored chunk retrieves its own top-$K$ latents by camera co-visibility, and the union of these requests forms a shared bank. The bank is bounded by $kK$, so the sequence stays bounded however long the span; at the shortest span the recipe is exactly conventional training. Adding the bank raises the cost of a step once; beyond that, a longer span costs little, and going from 100 to 400 s adds 12% to the step time. Against a sliding-window baseline,
retrieval raises revisit consistency on every split, and a span long enough to reach the first visit of each return adds a further 24% to 30%, at some cost in image quality; beyond that span, more length no longer helps. Filling the bank from another episode lowers revisit correlation by 83%, so the model uses what it retrieves. Project page: https://tingtingliao.github.io/memorizon
Read the full abstract ▾
All authors Tingting Liao, Xuezhi Liang, Hao Li, Guangyi Liu
Computer Vision
PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion
Lehan Yang, Daiqing Qi, Wenhao Zhang et al. (+9 more)
Representation
alignment (REPA) accelerates
diffusion transformer training, but its
alignment targets are almost exclusively semantic encoders such as DINOv2 and CLIP. Recent analysis points to spatial structure, not global semantics, as the carrier of the
alignment effect, yet dense-prediction foundation models trained to predict that structure remain overlooked as REPA targets. In pixel-space
diffusion, SAM2, Depth Anything v2, and Metric3D v2 each outperform the DINOv2-only GenEval baseline, with the two
geometric teachers leading the segmentation teacher. A flat sum of all four teachers, however, lands below the best single
geometric teacher, as semantic and
geometric gradients compete for one denoiser projection. We introduce PixelDense, which routes DINOv2 and SAM2 through a semantic projection stream, routes Depth Anything v2 and Metric3D v2 through a
geometric projection stream, and adds a weight-space orthogonality penalty that keeps the two streams in disjoint subspaces. All four teachers are frozen during training and dropped at inference. Applied to PixelGen and DeCo with a single recipe, PixelDense improves GenEval, DPG-Bench, and HPS v2.1, raises PixelGen-XXL's GenEval Overall from 0.7927 to 0.8093, and beats every single-teacher and unfactored multi-teacher variant. In partial-noise reconstruction, independent panoptic, depth, and surface-normal probes show up to 53.1% PQ gain and 36.0% depth AbsRel reduction at $\tau=0.5$ across COCO and Flickr30K. From random initialization, PixelDense also reaches the baseline's peak GenEval 1.23x faster. In SDEdit editing on PIE-Bench, PixelDense keeps more of the source background and layout at every edit strength, raising background PSNR by up to 2.2 dB.
Read the full abstract ▾
All authors Lehan Yang, Daiqing Qi, Wenhao Zhang, Avery Li, Yiqing Yang, Yifan Li, Yu Kong, Haitian Zheng, Zhifei Zhang, Zhe Lin, Varun Jampani, Sheng Li
Computer Vision
PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video
Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin et al. (+5 more)
Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint
reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular
video. Our approach augments a human reconstruction foundation model with learnable contact-force
tokens and a temporal
transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world
videos. We also introduce a real-world climbing
benchmark ForceWall with climbing
videos and corresponding ground-truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from
video.
Read the full abstract ▾
All authors Rikhat Akizhanov, Yangsong Zhang, Nikolai Kaliazin, Peter Wolf, Yoshihiko Nakamura, Pascal Fua, Fabio Pizzati, Ivan Laptev
Computer Vision · Cross-listed from cs.AI
Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation
Anson Y. Lam, Shuqing Li and Michael R. Lyu
Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings and caption wording become part of the measurement protocol. Across 300 frozen scenes from six generators, we vary eight render and caption factors for 19
alignment evaluators plus one perceptual-quality control, then test four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17/19
alignment evaluators, with
prompt-bootstrap lower bounds above 1 for 11/19. Rankings are more stable than scores, yet 18/19 evaluators change their point-estimate winner under some configuration. Pairwise protocol margin envelopes show which comparisons keep their direction across the tested settings. Selected pairs have opposite pointwise intervals, but no reversal survives simultaneous inference over the full search. Thus the observed winner changes are descriptive, not confirmed changes in generator superiority. Sensitivity remains separate: no evaluator, even the
prompt-free control, exceeds 67% tie-adjusted directional discrimination on layout scrambling, which is diagnostic rather than human-validated ground truth. The audit separates score stability, decision uncertainty, and targeted sensitivity, and recommends reporting (generator, score, card ID) with protocol-dependent comparisons and selection-aware uncertainty.
Read the full abstract ▾
Computer Vision
Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification
Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi et al. (+1 more)
Automated classification of
brain tumors from MRI is a heavily published application of deep learning in medical imaging, with reported accuracies on public
benchmarks routinely exceeding 98%. However, accuracy does not capture a critical dimension of
benchmark quality:
dataset integrity, defined as the independence of test from training data at the image, patient, and acquisition-source levels. We introduce a three-layer contamination framework comprising duplicate, patient, and source-label leakage to assess the public corpora on which this literature rests. We audit the three most widely used corpora against a chest-radiograph negative control and quantify each layer's effect on measured performance across nine architectures and three evaluation conditions. Contamination is severe at every layer: 28.8% of the dominant corpus's official test split has a near-twin in its own training split, a second corpus leaks 22.3% of its test images byte-identically, 95.5% of traceable test images share a patient with training, and file-header features containing no anatomy separate tumor from no-tumor at 0.959 balanced accuracy, at parity with fine-tuned ResNet backbones. The unexpected result is that removing every identified leaked test image leaves balanced accuracy essentially unchanged: stable performance after deduplication does not establish
benchmark integrity. Our findings establish
dataset integrity as a distinct, measurable axis of
benchmark quality that a stable leaderboard cannot certify. For biomedical research, reported accuracy on these corpora alone does not establish that a model has learned to recognize tumors rather than exploit
dataset-specific cues. We release the contaminated-file lists, recovered patient identifiers, and deduplicated splits.
Read the full abstract ▾
All authors Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala
Computer Vision
From Image Latent Space to Fuzzy Rules: Interpretable Analysis of Gastrointestinal Foundation Model
Michael D. Vasilakakis and Dimitris K. Iakovidis
Foundation models pretrained on large-scale
datasets demonstrate strong transferability to medical imaging tasks. However, understanding how their latent representations encode clinically relevant information remains an open challenge in safety-critical domains. This study proposes a prototype-based fuzzy-rule framework that interprets the patch-level features produced by the inner layers of pretrained foundation models, without any fine-tuning. Class-specific prototypes are learned by clustering in the feature space, yielding compact visual patterns. Patch features are then expressed as prototype similarities and classified by fuzzy rules with linguistic IF-THEN conditions that are human readable. The framework is applied across the final two blocks of ViT-S/16 backbones pretrained on ImageNet-1K and GastroNet-5M, and benchmarked against k-nearest neighbours, kernel SVM, and linear probing under identical frozen features, on wireless capsule endoscopy classification, gastrointestinal endoscopy classification, and colonic polyp segmentation. The experimental analysis shows that the proposed method, without backbone fine-tuning, reaches accuracy comparable to these black-box classifiers, and that domain-specific pretraining yields features that are both discriminative and symbolically compressible. Because the resulting rules are extracted from real data and expressed in interpretable terms, they are further used as an instrument to investigate synthetic medical images, providing a human-readable account of which real prototypes and rules a generator reproduces or fails to reproduce, localising where a synthetic image departs from real tissue rather than summarising it with a single score. The framework thus offers a transparent, depth-resolved view of how foundation models organise clinically relevant structure, together with a practical downstream use of the extracted rules.
Read the full abstract ▾
Computer Vision · Cross-listed from eess.IV
RIQE: a NIQE-style reference model for Computed Tomography
Fabio Mattiussi
The Natural Image Quality Evaluator (NIQE) scores an image by its statistical distance from a model fitted on pristine images, and its distributed model is fitted on photographs. We release the Radiology Image Quality Evaluator (RIQE), a NIQE-style model fitted on 3,792 full-dose slices from 158 patients of the public LDCT-and-Projection-data collection, with a declared intensity mapping, a manifest of every slice and a script that reproduces the fit. On 40 held-out patients, RIQE ranks reduced-dose reconstructions, simulated by projection-domain noise insertion, worse than the full-dose reconstruction of the same slice in 240 of 240 chest and 230 of 240 abdominal pairs, and ranks images with 20% more noise worse than their source in 97.5-100% of cases. Its preferences among filtered images, however, do not follow lesion signal. With a 4 mm, +10 HU lesion inserted in noisy abdominal slices, RIQE prefers bilateral filtering to the unfiltered image in every image up to a 32 HU residual, at which 29% of the lesion's matched-filter signal remains and its detectability index falls from 0.51 to 0.33; it never prefers Gaussian smoothing, which at the same 32 HU residual leaves 70% of the signal and a detectability index of 0.48. Fitted on photographs with the parameters published for NIQE, the same code ranks every simulated reduced-dose abdominal image better than its full-dose counterpart. RIQE is suited to ranking a degraded image against its source; under the conditions tested it should not be the sole criterion for selecting, comparing or tuning denoisers.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.LG
Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment
Jinho Chang and Jong Chul Ye
Recent advances in
distillation and flow-map models have enabled deterministic one- or few-step generation for high-quality data, facilitating a new branch of reward
alignment approaches that directly optimize the initial noise from a Gaussian distribution. However, most existing initial-noise optimization methods rely on first-order gradient information, which is either inapplicable or suffers from instability and inefficiency in black-box reward scenarios. Here, we introduce ZeNOVA, a stable and efficient initial noise
alignment method in a gradient-free manner. Specifically, we address existing
algorithms' major challenge in black-box scenarios through annealed soft-value guidance,
manifold-constrained hyperspherical Langevin dynamics, and Metropolis-Hastings jumping. Extensive experiments on image and
video generative models show that ZeNOVA outperforms all evaluated zeroth-order baselines by optimizing the initial noise toward higher rewards substantially more stably while exploiting the geometry of the Gaussian prior, demonstrating its practical applicability to various black-box reward
alignment.
Read the full abstract ▾
Computer Vision · Cross-listed from cs.RO
DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation
Haoyu Wang, Siyuan Qian, Yanjun Li et al. (+4 more)
Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human
video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
Read the full abstract ▾
All authors Haoyu Wang, Siyuan Qian, Yanjun Li, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao Tang
Computer Vision · Cross-listed from cs.GR
Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength
Candi Zheng and Yuan Lan
Diffusion models with
prompt and reference image-guided editing have seen rapid progress, yet they remain too coarse for pixel-level control. One promising direction is to incorporate a soft mask that specifies spatially varying edit strengths but training such fine-grained control demands expensive pixel-wise annotations, while existing zero-shot methods often yield unsatisfactory results. We introduce SoftPaint, a new zero-shot sampling method that leverages soft masks to enable a continuous spectrum of edits, from fully preserving the original content to completely re-synthesizing the masked region. Going beyond zero-shot inpainting methods, we design a Langevin-iteration-based sampler that respects per-pixel soft mask strengths, which applies universally to image and
video diffusion models, enabling tasks such as
video editing. The method is gradient-free,
memory-efficient, and achieves smooth, pixel-level edits across multiple image and
video backbones.
Read the full abstract ▾
Computer Vision
Hongyuan Tao, Xinggang Wang, Lianghui Zhu et al. (+7 more)
We present
Multimodal Flow, a fully continuous generative model of
language and
vision. Most unified
multimodal models either model both
language and quantized images as discrete
tokens or combine discrete
language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for
multimodal pretraining.
Multimodal Flow introduces a unified continuous architecture that integrates
multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual
token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint
attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on
multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves
multimodal modeling. With only 150B pretraining
tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets,
Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based
embedding flow modeling as a new fully continuous paradigm for unified
multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/
Multimodal-Flow.
Read the full abstract ▾
All authors Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li, Yunchao Wei, Bin Feng, Shaoyu Chen, Qian Zhang, Chang Huang, Kai Yu
Computer Vision · Cross-listed from cs.LG
Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis
Tian Xia, Minghao Liu, Yiqing Liang et al. (+2 more)
Multimodal large
language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on
prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores
matrix with one row per evaluation instance and one column per candidate
prompt;
cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto
prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the
cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the
prompt evolution search reads from - the scores
matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based
prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via
vision-encoder-tuned SFT or medical pretraining - is a prerequisite that
prompt search cannot replace - our recipe extends reflective
prompt evolution from text-only data to
multimodal clinical decision-making.
Read the full abstract ▾
All authors Tian Xia, Minghao Liu, Yiqing Liang, Laixi Shi, Jiayun Wang
Computer Vision
Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model
Liming Lu, Xianzheng Ma, Wenkun He et al. (+14 more)
Video world models are expected to predict how the physical world evolves, yet they often produce visually plausible
videos that violate basic physical principles. Existing approaches commonly assume that natural
language is insufficient to represent the physical knowledge required for reliable generation, and therefore introduce additional visual, latent, numerical, or planning-based signals. We revisit this assumption and introduce Physis-Lang, a self-evolving framework that treats physical
language as a shared and optimizable representation across data curation, model training, and
video generation. Physis-Lang represents physical processes through
language that describes their relevant entities, causes, interactions, governing principles, temporal
evolution, and effects. To improve this representation, we construct PhysCapBench, which decomposes physical processes into atomic assertions and evaluates captions using recall and precision. An agentic loop iteratively analyzes assertion-level errors and refines the instruction used to produce physical captions. Physis-Lang further converts model deficiencies into textual descriptions and uses
language-guided
retrieval to identify visually diverse
videos that cover missing physical processes. Experiments on four widely used physical
video benchmarks with Wan and Cosmos backbones demonstrate consistent improvements in physical plausibility. Notably, starting from open-source Cosmos3-Nano backbones, our Physis-Lang-enhanced models surpass the leading proprietary Veo 3.1 model.
Read the full abstract ▾
All authors Liming Lu, Xianzheng Ma, Wenkun He, Guanqi Zhan, Yilin Zhao, Junyu Chen, Mengyao Xu, Jiaojiao Fan, Wenhang Ge, Yuchao Gu, Yunze Liu, Boyi Li, Zhen Dong, Victor Prisacariu, Ming-Yu Liu, Song Han, Han Cai
Computer Vision
ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing
Xinghao Chen, Xiangbo Gao, Jiongze Yu et al. (+2 more)
Recent
video generation is increasingly realistic and controllable, yet
video editing remains less developed, particularly for precise local edits that must preserve the original scene dynamics.
Video scene text editing replaces text on scene surfaces, such as storefront signs, whiteboards, and product labels, while preserving the surrounding content, motion, and camera dynamics. Although scene text editing is well studied for images,
video scene text editing that achieves high visual quality, temporal consistency, and edit locality remains underexplored. Existing resources offer limited paired real-
video data, and general
video-editing metrics do not directly measure whether the requested text remains correct over time. We introduce ViTeX-Bench, a
benchmark suite comprising ViTeX-
Dataset and a three-axis evaluation protocol. The
dataset contains 387 real-world 720p
videos with text-region masks and editing instructions: 230 provide reviewed, pipeline-generated paired edits for training, and 157 form a frozen evaluation split. The protocol evaluates text correctness, visual and temporal quality, and edit locality through 13 metrics, with one primary metric per axis and a Pareto comparison of their trade-offs. OCR calibration, human evaluation, and annotation-sensitivity analyses support the interpretation of these scores. Across eight baselines from four editing families, accurate text, temporal stability, and scene preservation remain difficult to achieve together. We also release ViTeX-Edit-14B, an open-source reference editor fine-tuned on the paired training split with motion-aligned glyph-
video conditioning. It achieves CharAcc 0.688, the highest mean among the evaluated
video-native editors, and the lowest comparable text-crop Warp among raw editor outputs. ViTeX-Bench provides a reproducible foundation for studying these trade-offs in
video scene text editing.
Read the full abstract ▾
All authors Xinghao Chen, Xiangbo Gao, Jiongze Yu, Yuheng Wu, Zhengzhong Tu
Computer Vision
AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
Jiahao Zhang, Yeying Fan, Moitreya Chatterjee et al. (+5 more)
The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose
agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which
agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available.
Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight
agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how
agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly
agents and characterizing the gap between approximate structure recovery and precise reconstruction.
Read the full abstract ▾
All authors Jiahao Zhang, Yeying Fan, Moitreya Chatterjee, Suhas Lohit, Bernhard Egger, Tim K. Marks, Anoop Cherian, Stephen Gould
Computer Vision
Image Classifiers are Efficient Self-Supervised Video Representation Learners
Owais Iqbal, Sudipta Sarkar, Shyam Marjit et al. (+3 more)
We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in
videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image
Vision Transformers by representing
videos as super images which are grids composed of frames sampled from
videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared
Vision Transformer (ViT) encoder aligns their
embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient
video representation learning. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51 while requiring up to $32\times$ fewer and $160\times$ fewer
video pretraining epochs compared to prior
video self-supervised learning methods. Our proposed approach also shows strong performance in low-shot classification, confirming the transferability of the learned representations in a label-scarce scenario. Project Page: https://cvir.github.io/projects/videomsn.
Read the full abstract ▾
All authors Owais Iqbal, Sudipta Sarkar, Shyam Marjit, Omprakash Chakraborty, Anirban Chakraborty, Abir Das
Computer Vision · Cross-listed from cs.RO
Ego4WAM: What Matters When Scaling Egocentric Human Data for Robot Learning?
Zhihao Sun, Liu Liu, Xinjiang Wang et al. (+6 more)
Egocentric human data provides a scalable source of experience for
robot learning, but varies substantially in human-
robot alignment, behavioral coverage, and available supervision. Existing work shows favorable scaling with increasing human data, but it remains unclear which data properties drive downstream
robot gains and how to use such data throughout the training pipeline. We present a systematic study of egocentric human data with different
alignment and supervision under a unified world-action model framework. With the model backbone fixed, we disentangle the effects of human-
robot alignment, data duration and task diversity, action supervision, and data usage strategies. We find that aligned human demonstrations substantially improve out-of-distribution generalization and reduce target-task
robot data requirements; data duration and task diversity affect downstream capabilities differently; and
video-only supervision remains effective without action labels, providing a strong foundation for subsequent
video-action training. We validate these findings through closed-loop policy evaluation on both real
robots and RoboDojo. Rather than treating data duration as the sole scaling axis, Ego4WAM shows how
alignment, task diversity, available supervision, and usage strategy jointly shape the value of egocentric human data for
robot learning.
Read the full abstract ▾
All authors Zhihao Sun, Liu Liu, Xinjiang Wang, Haoyi Jiang, Wei Feng, Huiqiang Zhang, Xiaosong Jia, Zhizhong Su, Zuxuan Wu