# The Semantic Gap in Art: AI Visual-Textual Alignment Study
> A research analysis of how multimodal models like CLIP and ResNet struggle with narrative art vs. simple textures, quantifying the semantic gap.

Tags: ai-research, computer-vision, multimodal-learning, art-history, clip-model, semantic-gap, neural-networks
## The Semantic Gap in Art
- Investigates the limits of visual-textual alignment in modern AI architectures.
- Research team: Talih, Sluimer, Speh, van Goethem (UvA).

## Core Research Question
- Do multimodal models truly 'understand' artistic narrative and cultural symbols, or just surface textures?

## The Semantic Binary
- **Additive (Simple):** Meaning from visible objects (Still-life, Landscape).
- **Narrative (Complex):** Meaning from symbols and relationships (Mythology, History).

## Hypotheses
- H1: Embeddings organize by Art Type more than by School or Timeframe.
- H2: Retrieval performance is significantly lower for complex 'Narrative' art.
- H3: ResNet suffers from texture bias; CLIP shows improved but imperfect semantic alignment.

## Quantitative Results
- **Distance Ratio:** Art Form (0.78) and Author (0.72) show much tighter clustering than School (0.98) or Timeframe (0.96).
- **The Semantic Gap:** Narrative art is 1.4x to 3.8x harder to retrieve than additive art.
- **Model Performance:** CLIP achieved a median rank of 5 for simple art vs 19 for complex art; ResNet+BERT performed worst with a median rank of 65 for complex scenes.

## Texture Bias Problem
- Standard CNNs prioritize local patterns (brush texture) over global relationships.
- Qualitative failures show ResNet matching color and texture for Mythology queries but failing to capture the story content.

## Conclusions
- Explicit (Additive) art is largely solved by AI.
- Symbolic (Narrative) art remains a significant failure mode for current architectures.
- Massive pre-training (CLIP) reduces the gap but does not close it entirely.
---
This presentation was created with [Bobr AI](https://bobr.ai) — an AI presentation generator.