AI Learns to Deliver Enhanced Video Critiques

The Challenge of Evaluating Video Content in AI Research

When diving into the world of computer vision literature, Large Vision-Language Models (LVLMs) can be invaluable for interpreting complex submissions. However, they hit a significant roadblock when it comes to assessing the quality and merits of video examples that accompany scientific papers. This is a crucial aspect because compelling visuals are just as important as the text in generating excitement and validating the claims made in research projects.

Video synthesis projects, in particular, rely heavily on demonstrating actual video output to avoid being dismissed. It's in these demonstrations where the real-world performance of a project can be truly evaluated, often revealing the gap between the project's bold claims and its actual capabilities.

I Read the Book, Didn’t See the Movie

Currently, popular API-based Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) are not equipped to analyze video content directly. Their capabilities are limited to analyzing transcripts and other text-based materials related to the video. This limitation is evident when these models are asked to directly analyze video content.

*The diverse objections of GPT-4o, Google Gemini and Perplexity, when asked to directly analyze video, without recourse to transcripts or other text-based sources.*

Some models, like ChatGPT-4o, might even attempt to provide a subjective evaluation of a video but will eventually admit their inability to directly view videos when pressed.

*Having been asked to provide a subjective evaluation of a new research paper's associated videos, and having faked a real opinion, ChatGPT-4o eventually confesses that it cannot really view video directly.*

Although these models are multimodal and can analyze individual photos, such as a frame extracted from a video, their ability to provide qualitative opinions is questionable. LLMs often tend to give 'people-pleasing' responses rather than sincere critiques. Moreover, many issues in a video are temporal, meaning that analyzing a single frame misses the point entirely.

The only way an LLM can offer a 'value judgment' on a video is by leveraging text-based knowledge, such as understanding deepfake imagery or art history, to correlate visual qualities with learned embeddings based on human insights.

*The FakeVLM project offers targeted deepfake detection via a specialized multi-modal vision-language model.* Source: https://arxiv.org/pdf/2503.14905

While an LLM can identify objects in a video with the help of adjunct AI systems like YOLO, subjective evaluation remains elusive without a loss function-based metric that reflects human opinion.

Conditional Vision

Loss functions are essential in training models, measuring how far predictions are from correct answers, and guiding the model to reduce errors. They are also used to assess AI-generated content, such as photorealistic videos.

One popular metric is the Fréchet Inception Distance (FID), which measures the similarity between the distribution of generated images and real images. FID uses the Inception v3 network to calculate statistical differences, and a lower score indicates higher visual quality and diversity.

However, FID is self-referential and comparative. The Conditional Fréchet Distance (CFD) introduced in 2021 addresses this by also considering how well generated images match additional conditions, such as class labels or input images.

*Examples from the 2021 CFD outing.* Source: https://github.com/Michael-Soloveitchik/CFID/

CFD aims to integrate qualitative human interpretation into metrics, but this approach introduces challenges like potential bias, the need for frequent updates, and budget constraints that can affect the consistency and reliability of evaluations over time.

cFreD

A recent paper from the US introduces Conditional Fréchet Distance (cFreD), a new metric designed to better reflect human preferences by evaluating both visual quality and text-image alignment.

*Partial results from the new paper: image rankings (1–9) by different metrics for the prompt "A living room with a couch and a laptop computer resting on the couch." Green highlights the top human-rated model (FLUX.1-dev), purple the lowest (SDv1.5). Only cFreD matches human rankings. Please refer to the source paper for complete results, which we do not have room to reproduce here.* Source: https://arxiv.org/pdf/2503.21721

The authors argue that traditional metrics like Inception Score (IS) and FID fall short because they focus solely on image quality without considering how well images match their prompts. They propose that cFreD captures both image quality and conditioning on input text, leading to a higher correlation with human preferences.

*The paper's tests indicate that the authors' proposed metric, cFreD, consistently achieves higher correlation with human preferences than FID, FDDINOv2, CLIPScore, and CMMD on three benchmark datasets (PartiPrompts, HPDv2, and COCO).*

Concept and Method

The gold standard for evaluating text-to-image models is human preference data gathered through crowd-sourced comparisons, similar to the methods used for large language models. However, these methods are costly and slow, leading some platforms to stop updates.

*The Artificial Analysis Image Arena Leaderboard, which ranks the currently-estimated leaders in generative visual AI.* Source: https://artificialanalysis.ai/text-to-image/arena?tab=Leaderboard

Automated metrics like FID, CLIPScore, and cFreD are crucial for evaluating future models, especially as human preferences evolve. cFreD assumes that both real and generated images follow Gaussian distributions and measures the expected Fréchet distance across prompts, assessing both realism and text consistency.

Data and Tests

To evaluate cFreD's correlation with human preferences, the authors used image rankings from multiple models with the same text prompts. They drew on the Human Preference Score v2 (HPDv2) test set and the PartiPrompts Arena, consolidating data into a single dataset.

For newer models, they used 1,000 prompts from COCO's train and validation sets, ensuring no overlap with HPDv2, and generated images using nine models from the Arena Leaderboard. cFreD was evaluated against several statistical and learned metrics, showing strong alignment with human judgments.

*Model rankings and scores on the HPDv2 test set using statistical metrics (FID, FDDINOv2, CLIPScore, CMMD, and cFreD) and human preference-trained metrics (Aesthetic Score, ImageReward, HPSv2, and MPS). Best results are shown in bold, second best are underlined.*

cFreD achieved the highest alignment with human preferences, reaching a correlation of 0.97 and a rank accuracy of 91.1%. It outperformed other metrics, including those trained on human preference data, demonstrating its reliability across diverse models.

*Model rankings and scores on PartiPrompt using statistical metrics (FID, FDDINOv2, CLIPScore, CMMD, and cFreD) and human preference-trained metrics (Aesthetic Score, ImageReward, and MPS). Best results are in bold, second best are underlined.*

In the PartiPrompts Arena, cFreD showed the highest correlation with human evaluations at 0.73, closely followed by FID and FDDINOv2. However, HPSv2, trained on human preferences, had the strongest alignment at 0.83.

*Model rankings on randomly sampled COCO prompts using automatic metrics (FID, FDDINOv2, CLIPScore, CMMD, and cFreD) and human preference-trained metrics (Aesthetic Score, ImageReward, HPSv2, and MPS). A rank accuracy below 0.5 indicates more discordant than concordant pairs, and best results are in bold, second best are underlined.*

In the COCO dataset evaluation, cFreD achieved a correlation of 0.33 and a rank accuracy of 66.67%, ranking third in alignment with human preferences, behind only metrics trained on human data.

*Win rates showing how often each image backbone's rankings matched the true human-derived rankings on the COCO dataset.*

The authors also tested Inception V3 and found it to be outmatched by transformer-based backbones like DINOv2-L/14 and ViT-L/16, which consistently aligned better with human rankings.

Conclusion

While human-in-the-loop solutions remain the optimal approach for developing metric and loss functions, the scale and frequency of updates make them impractical. cFreD's credibility hinges on its alignment with human judgment, albeit indirectly. The metric's legitimacy relies on human preference data, as without such benchmarks, claims of human-like evaluation would be unprovable.

Enshrining current criteria for 'realism' in generative output into a metric function could be a long-term mistake, given the evolving nature of our understanding of realism, driven by the new wave of generative AI systems.

*At this point, I would normally include an exemplary illustrative video example, perhaps from a recent academic submission; but that would be mean-spirited – anyone who has spent more than 10-15 minutes trawling Arxiv's generative AI output will have already come across supplementary videos whose subjectively poor quality indicates that the related submission will not be hailed as a landmark paper.*

*A total of 46 image backbone models were used in the experiments, not all of which are considered in the graphed results. Please refer to the paper's appendix for a full list; those featured in the tables and figures have been listed.*

First published Tuesday, April 1, 2025