Our comprehensive benchmarking of 17 state-of-the-art Multimodal Large Language Models (MLLMs) on the LIVEVQA dataset revealed significant difficulties in handling visual information beyond their knowledge cutoff dates. For instance, even top-performing models showed low accuracy on recent visual content when operating without external tools.

However, the integration of multimodal search capabilities leads to dramatic improvements.

1. Models augmented with multimodal search tools (e.g., GPT-4.1 with MM-Search) demonstrated an average accuracy increase of 327% in seeking live visual knowledge. Specifically, GPT-4.1's average accuracy more than doubled from 16.5% to 33.4% when using MM-Search, with particularly striking gains on challenging Level 2 questions (e.g., accuracy on News subset Level 2 rose from 5.2% to 27.2%).
2. Native image search capabilities, as seen in models like GPT-03, also provided substantial gains (e.g., from 3.0% to 15.8% on Level 2 questions). In contrast, simple text-based online searching did not yield significant improvements, underscoring the necessity of multimodal retrieval for dynamic visual information.

The research explored updating MLLMs with new visual knowledge using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and DoRA.

1. Rapid Adaptation: Visual information can be efficiently updated through fine-tuning within only one epoch. Models using direct multiple-choice questions with concise answers (MCQA format) yielded faster and more effective learning during the visual knowledge acquisition phase compared to other formats like QA (Question + Ground Truth) or QAR (Question + Ground Truth + Reasoning).
2. LoRA Rank Impact: Higher rank LoRA configurations consistently enhanced visual knowledge capabilities, particularly in assimilating recent visual entities. Models with higher ranks outperformed lower-rank counterparts by an average of 5.4% on the validation subset.
3. Benefit to General Reasoning: Training on the visually knowledge-intensive LIVEVQA dataset—particularly with straightforward answers and multiple-choice questions—led to a notable 4.2% improvement on the general multimodal reasoning benchmark MMMU.

While PEFT methods allow for efficient incorporation of new visual facts, this process is not without its challenges and trade-offs.

A consistent observation was the degradation in the model's foundational visual perception capabilities (as measured by the MMStar benchmark) after undergoing intensive visual knowledge updates, regardless of rank, training steps, or data formats. For example, models trained using the simple QA format exhibited a performance drop on MMStar from 65.80% to 58.16%. This suggests an inherent conflict between enhancing specific visual knowledge through intensive updates and preserving the model's broader visual perception abilities.

The benchmark results highlighted several aspects regarding model characteristics:

1. Larger Models Tend to Perform Better: For models sharing the same knowledge cutoff (e.g., the GPT-4.1 family), increased model size generally correlated with improved accuracy on LIVEVQA tasks across all difficulty levels. Proprietary models also typically maintained an advantage over open-source counterparts.
2. Overconfidence and Calibration Issues: A crucial finding was the positive correlation between stated confidence and accuracy across models, but with significant calibration issues. All evaluated MLLMs demonstrated a consistent pattern of overconfidence in their visual factuality assessments, with their performance falling significantly below the ideal calibration line. While larger models like GPT-4.1 showed comparatively better calibration than their smaller variants, substantial opportunities remain for improving MLLM calibration when encountering unknown visual knowledge.
3. Level 2 Questions Prove More Difficult: As anticipated, Level 2 questions, which require deeper cross-modal reasoning, generally resulted in significantly lower performance for models compared to Level 1 (visual entity recognition) questions across most data subsets (News, Video).