Multimodal AI news — Page 2

AI systems that understand or generate combinations of text, images, audio, video, and other types of data.

ResearchOfficialarXiv Multiagent Systems

NetForge RL: Multi-Agent Cyber Defense Simulation with Durative Actions

NetForge RL is a multi-agent simulation environment designed for cyber defense research, featuring procedurally generated enterprise and operational technology (OT) networks. It supports red-blue team self-play under partial observability, with actions mapped to the MITRE ATT&CK framework and a JAX backend capable of 250,000 steps per second. The environment includes reference baselines, diagnostic probes for defensive skills, and an evaluation runner for reproducible benchmarking.

Why it matters: NetForge RL offers a realistic and reproducible testbed for developing and evaluating multi-agent reinforcement learning systems in cyber defense scenarios with adaptive adversaries and noisy, partial observations.

Jul 15, 2026

ResearchOfficialarXiv Machine Learning

SinAE: A Single-Architecture Flow-Matching Autoencoder for Cross-Domain Atomic Systems

SinAE is a flow-matching autoencoder that uses a single vanilla Transformer architecture to handle molecules, crystals, and proteins, without relying on domain-specific operators. It achieves near-lossless reconstruction across these domains and demonstrates strong generative performance, with joint training on molecules and crystals leading to improvements in both domains.

Why it matters: This work provides a unified approach for generative modeling across diverse atomic systems, enabling cross-domain transfer and potentially alleviating data scarcity issues.

Jul 15, 2026

ResearchOfficialarXiv Information Retrieval

Vision-Free Composed Image Retrieval Sets New Zero-Shot Benchmark with Attribute-Augmented Scoring and LLM Reranking

A new framework for Composed Image Retrieval (CIR) achieves state-of-the-art zero-shot performance on the CIRR dataset (44.04% R@1, an improvement of 8.79%) without using visual features. The approach combines attribute-augmented hybrid scoring to address the loss of visual detail and LLM-based reranking to ensure semantic consistency. Ablation studies confirm that both components contribute to the performance gains.

Why it matters: This work shows that vision-free methods can outperform traditional approaches on complex multimodal retrieval tasks, suggesting new directions for efficient and simplified retrieval systems.

Jul 15, 2026

ResearchOfficialarXiv Machine Learning

PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated Multimodal LLMs

A new framework called PFAdapter introduces hierarchical LoRA decomposition to separate global-shared and local-private parameters for federated fine-tuning of multimodal large language models (MLLMs). By synchronizing only the global-shared components and keeping local adaptations private, PFAdapter reduces communication costs by nearly 50% and achieves accuracy improvements of 2.4% to 4.8% on several medical and multimodal datasets. The approach also uses orthogonality regularization to enforce strict separation between parameter types, preventing redundant feature learning.

Why it matters: This work offers a practical advance for deploying personalized, communication-efficient AI at network edges, addressing key challenges in federated learning for multimodal models.

Jul 15, 2026

ResearchOfficialarXiv Information Retrieval

SlimPer: A Slim and Smart Personalization Model for Recommendation Systems

SlimPer introduces a new approach to personalized ranking in recommendation systems by iteratively refining a compact knowledge base, rather than relying on large intermediate representations that scale with user history length. This design achieves O(N) per-layer computational cost and fixed-size intermediate representations, allowing the model to handle over 10,000 user history events efficiently. Deployed on Instagram Reels and Feed, SlimPer has demonstrated measurable improvements in user engagement while unifying sparse, dense, and sequence features within a single model backbone.

Why it matters: SlimPer enables deeper and more efficient personalization in large-scale recommendation systems by decoupling model depth from user history length, reducing compute and memory requirements.

Jul 15, 2026

ResearchOfficialarXiv Computer Vision

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

A new preprint audits data leakage in whole-slide image (WSI) multimodal benchmarks and finds that patient-level and institutional-level overlaps significantly compromise reported zero-shot performance of vision-language models. The authors document case-level train-test overlaps of 92.3–100% on TCGA-derived benchmarks and show that this leakage is linearly decodable from foundation-model features. They recommend concrete steps for contamination-free evaluation practices.

Why it matters: This work reveals that widely-used WSI VQA benchmarks may not accurately measure genuine multimodal reasoning, calling into question reported advances in computational pathology.

Jul 15, 2026

ResearchOfficialarXiv Computer Vision

DM-KG: A Structured Knowledge Graph Framework Improves Spatial Reasoning in Vision-Language Models for Street View Imagery

A new framework called DM-KG introduces a structured knowledge graph to enhance spatial reasoning in vision-language models (VLMs) applied to street view imagery. By extracting and encoding directional and metric relationships between entities from 2D images, DM-KG significantly reduces distance estimation and direction judgment errors on spatial question-answering benchmarks.

Why it matters: This work offers a notable advance in improving the spatial cognition of VLMs, addressing a key limitation for their deployment in geospatial and geographic visual question answering tasks.

Jul 15, 2026

ResearchOfficialarXiv Computer Vision

VLM-Assisted Framework Improves EEG-to-Image Reconstruction Evaluation

A new evaluation framework uses four vision-language models (VLMs) to assess EEG-to-image reconstructions, introducing Tolerant Perceptual Alignment Scores (T-PAS) and Tolerant Semantic Alignment Scores (T-SAS). The distilled BCI-Coherence Score (BCS) demonstrates lower mean absolute error and higher correlation with human judgments than traditional pixel or representation-based metrics, addressing the challenge of distinguishing visual fidelity from semantic recoverability.

Why it matters: This framework offers a more meaningful way to evaluate EEG-to-image reconstructions, potentially advancing brain-computer interface research by better capturing both perceptual and semantic accuracy.

Jul 15, 2026

ResearchOfficialarXiv Computer Vision

DeGuNet: Depth-Guided Ultra-Compact Backbones for Efficient LiDAR-Camera 3D Detection

A new preprint introduces DeGuNet, an ultra-compact image backbone designed for LiDAR-camera 3D detection in autonomous driving. DeGuNet uses depth-guided representation learning to address parameter redundancy in current multi-modal frameworks, reducing GPU memory usage by up to 66.5% and achieving a 1.16x speedup. The method also improves mean average precision (mAP) by up to 6.20 points on the nuScenes dataset, demonstrating both efficiency and accuracy gains.

Why it matters: DeGuNet could enable more efficient and accurate 3D perception for autonomous vehicles by significantly reducing computational demands.

Jul 15, 2026

ResearchOfficialarXiv Computer Vision

Continual Learning for Heterogeneous Medical VQA: An Empirical Analysis

A new preprint systematically evaluates continual learning (CL) methods for medical visual question answering (MedVQA) across a range of clinical tasks, including classification, detection, cell counting, and report generation. The study finds that current CL methods have difficulty maintaining a balance between retaining old knowledge and learning new tasks when faced with diverse objectives and supervision formats. The authors also analyze the impact of task ordering and the evolution of model parameters during continual learning. Code and experimental setup will be made publicly available.

Why it matters: This work exposes key limitations in current continual learning approaches for medical VQA, highlighting challenges that must be addressed for robust real-world deployment.

Jul 15, 2026

ResearchOfficialarXiv Cryptography and Security

Audit Reveals Methodological Flaws in Inference-Time Defense Evaluations for Multimodal LLMs

A recent audit of inference-time defenses for multimodal large language models (MLLMs) found that three benchmark branches failed provenance checks, restricting reliable comparative results to only two datasets. The study also discovered that a legacy keyword-based safety protocol incorrectly counted empty strings as safe, with raw model responses unavailable to reassess this effect. Contrary to earlier claims, the archive does not support widespread model refusal, with a pooled refusal rate of just 0.52%.

Why it matters: This work exposes critical methodological issues in evaluating MLLM safety defenses, emphasizing the importance of traceable audits and provenance standards for trustworthy comparisons.

Jul 15, 2026

ResearchOfficialarXiv Computation and Language

KnowAct-GUIClaw: Personal GUI Assistant with Self-Evolving Memory and Skill

KnowAct-GUIClaw is a new framework for GUI automation that addresses limitations in cross-platform support and self-evolution found in previous systems like OpenClaw. The framework introduces a Know-Route-Act-Reflect paradigm, enabling the agent to accumulate user interaction experience and improve execution accuracy over time. In experiments, KnowAct-GUIClaw using the open-source Kimi-2.6 model achieved 64.1% on the MobileWorld benchmark, outperforming both open and closed-source alternatives such as Seed-2.0-Pro and GPT-5.5.

Why it matters: This work represents a notable advance in personal GUI assistants by enabling self-evolving memory and skill transferability across platforms and models, potentially improving automation efficiency and adaptability.

Jul 15, 2026

ResearchOfficialarXiv Computation and Language

CityBehavEx: Scalable LLM-Assisted Urban Simulation Platform Validated Against Real-World Mobility

CityBehavEx is an interactive urban simulation platform that integrates large language models (LLMs) with traditional human mobility models to efficiently simulate city-scale populations. The system can run simulations of 100,000 agents over 75 days in under one hour on a single consumer GPU, and supports empirical validation by comparing generated mobility patterns to real-world spatial, temporal, and semantic distributions. CityBehavEx also provides tools for inspecting agent behavior, debugging, and validating routines against real-world data, addressing scalability and validation challenges in prior LLM-based simulators.

Why it matters: This work offers a scalable and empirically validated approach to LLM-driven urban simulation, potentially advancing applications in urban planning and policy analysis.

Jul 15, 2026

ResearchOfficialarXiv AI/ML

MaxSAT-Based Feedback Improves Logical Consistency in Vision-Language Models for Sudoku

A new neuro-symbolic approach integrates a Maximum Satisfiability (MaxSAT) oracle with Vision-Language Models (VLMs) to improve their performance on Sudoku puzzles. The MaxSAT solver identifies consistent subsets of VLM-generated assignments and provides structured feedback, guiding the models to refine their solutions. Experiments across multiple VLMs show that this method enhances logical consistency and increases the number of correctly solved Sudoku instances.

Why it matters: This work shows that symbolic optimization can significantly improve the reliability of vision-language models on structured reasoning tasks.

Jul 15, 2026

ResearchOfficialarXiv AI/ML

Light-MER: Sub-1B Multimodal Emotion Model Matches or Exceeds Larger Models

Researchers introduce Light-MER, a multimodal emotion recognition model with fewer than 1 billion parameters, designed to achieve high performance through knowledge distillation from larger models. The framework incorporates optimal transport loss and a multi-reward optimization strategy to balance accuracy and efficiency. Experiments on nine benchmark datasets show that Light-MER achieves state-of-the-art results while enabling faster inference, making it suitable for deployment on resource-constrained devices.

Why it matters: This work demonstrates that high-quality multimodal emotion recognition is possible with much smaller models, enabling practical deployment on devices with limited computational resources.

Jul 15, 2026

ResearchOfficialarXiv AI/ML

Visual Access Boundaries in Vision-Language Model Reasoning

A new preprint introduces Visual Access Sweep, a causal intervention technique to study how Vision-Language Models (VLMs) use image information during Chain-of-Thought (CoT) reasoning. The study finds that VLMs do not require ongoing access to image tokens throughout CoT reasoning; instead, CoT mainly extends language-side computation over previously extracted image-derived hidden states. The research also shows that the effectiveness of CoT is limited by the model's ability to reliably read out perceptual attributes, rather than by its counting ability.

Why it matters: This work challenges common assumptions about CoT in VLMs and suggests that improving perceptual readout, rather than extending visual access, may be key to further performance gains.

Jul 15, 2026

ResearchOfficialarXiv AI/ML

FormalAnalyticGeo: A Neural-Symbolic Framework for Multimodal Analytic Geometry Problem Generation

Researchers introduce FormalAnalyticGeo, a scalable framework that leverages formal languages and four specialized LLM components to automatically generate multimodal analytic geometry problems with precise diagrams. The system produces the AnalyticGeo7K dataset, comprising over 7,000 verified problems, and achieves a median ground-truth relative error of 0.70%.

Why it matters: By enabling fully automatic generation of high-quality multimodal analytic geometry problems, this framework addresses the scarcity of annotated data and could accelerate research in math reasoning for multimodal large language models.

Jul 15, 2026

ResearchOfficialarXiv Software Engineering

LMVQA: LLM-Based Multimodal QA System for Diagram-Rich Technical Meeting Videos

Researchers introduced LMVQA, a large language model-based multimodal question-answering system designed for technical meeting videos containing diagram-rich content such as UML diagrams. LMVQA significantly improved answer accuracy over a state-of-the-art baseline, from 31% to 94% on an industrial dataset and from 21% to 88% on a public dataset. The system also reduced average response time and LLM API costs after a one-time video indexing process. Interviews with domain experts highlighted LMVQA's value for locating relevant information and tracing rationale in software engineering meetings.

Why it matters: LMVQA addresses the challenge of efficiently retrieving knowledge from complex, diagram-rich meeting recordings, which is crucial for software engineering teams relying on asynchronous communication.

Jul 14, 2026

ResearchOfficialarXiv Multiagent Systems

Self-Creating Random Walks for Decentralized Learning under Pac-Man Attacks

Researchers introduce the CREATE-IF-LATE (CIL) algorithm to address 'Pac-Man' attacks in decentralized learning, where malicious nodes can terminate random walks and disrupt learning. The CIL algorithm enables self-creating random walks, ensuring that the random walk population does not go extinct, remains bounded, and that stochastic gradient descent converges with a quantifiable deviation even under attack. Empirical results on synthetic and benchmark datasets support the theoretical guarantees.

Why it matters: This work provides a novel, decentralized defense against a stealthy adversarial threat in distributed learning systems, helping to maintain robust learning progress without centralized control.

Jul 14, 2026

ResearchOfficialarXiv Multiagent Systems

Recursive Multi-Agent Trading System Achieves Lower Drawdown Under Geopolitical Uncertainty

A new preprint introduces RMATS, a recursive multi-agent trading system composed of four specialized agents coordinated by a manager. In experiments spanning 561 trading days and multiple asset classes, RMATS achieved a maximum drawdown of 9.62%, outperforming traditional methods in downside protection during several geopolitical stress scenarios. Ablation studies indicate that each agent contributes to risk control, though RMATS underperforms return-maximizing baselines in bull markets.

Why it matters: This work presents a novel multi-agent architecture that advances risk-controlled portfolio management, particularly relevant for capital preservation during periods of geopolitical uncertainty.

Jul 14, 2026