Introduction

In the era of AI-powered search, finding relevant information across different data types—text and images—has become crucial. This blog post explores how to build a powerful multimodal Retrieval-Augmented Generation (RAG) system using CLIP (Contrastive Language-Image Pre-training) and Pinecone vector database.

What is Multimodal RAG with CLIP?

Multimodal RAG with CLIP uses a shared vector space approach where text and images are embedded into the same mathematical space. This makes it possible to:

  • Search for images using text descriptions
  • Find relevant text using image queries
  • Combine both modalities in a single search system

The Power of CLIP

CLIP, developed by OpenAI, was trained on 400 million image-text pairs. Its paired encoders make it a useful baseline for multimodal retrieval, although domain-specific data may require evaluation or fine-tuning. See the original CLIP paper for the training method and limitations.

How Does CLIP + Pinecone Work?

The system operates through a sophisticated pipeline:

1. Dual-Index Architecture

Text Index (1536 dimensions) → OpenAI Embeddings
Image Index (512 dimensions) → CLIP Embeddings

2. Processing Pipeline

graph LR
    A[PDF/Images Input] --> B[Text Extraction]
    A --> C[Image Extraction]
    B --> D[OpenAI Text Embeddings]
    C --> E[CLIP Image Embeddings]
    D --> F[Pinecone Text Index]
    E --> G[Pinecone Image Index]
    H[User Query] --> I[Cross-Modal Search]
    F --> I
    G --> I
    I --> J[Combined Results]

3. Cross-Modal Search Magic

When you search with text, the system:

  1. Creates text embeddings using OpenAI's model
  2. Generates CLIP embeddings for image search
  3. Queries both indexes simultaneously
  4. Returns relevant text passages AND images

Implementation Overview

My implementation (multimodal_clip_pinecone.py) includes several key components:

Core Features:

# Key Functions:
- encode_text(): Generate OpenAI text embeddings
- encode_image(): Create CLIP embeddings for images
- index_figures_folder(): Process image directories
- index_pdf(): Extract and index PDF content
- search(): Unified search across modalities

Architecture Highlights:

  1. Separate Namespaces:

  2. TEXT_NAMESPACE = "medical_text"

  3. IMAGE_NAMESPACE = "medical_images"

  4. Smart PDF Processing:

  5. Extracts text in chunks (1000 characters)

  6. Extracts embedded images automatically

  7. Maintains page-level metadata

  8. Intelligent Figure Recognition:

  9. Automatically extracts figure numbers from filenames
  10. Links figures to their descriptions

Input and Output Examples

What You Can Input:

  1. PDF Documents: Medical papers, research documents, technical manuals
  2. Image Folders: Figures, diagrams, charts (JPG, PNG)
  3. Text Queries: Natural language questions

Example Inputs and Outputs:

Input Query: "Show me diagrams about somatosensory pathways"

Output:

{
  "text_results": [
    {
      "content": "The somatosensory pathway consists of three neurons...",
      "page": 5,
      "score": 0.89
    }
  ],
  "image_results": [
    {
      "figure_number": "3",
      "filepath": "/figures/figure-3-1.jpg",
      "score": 0.92
    }
  ]
}

Real-World Search Examples:

  • Medical Research: "Find all images showing neural pathways with their explanations"
  • Technical Documentation: "Show circuit diagrams for power supply units"
  • Educational Content: "Get all figures related to cell division with descriptions"

Key Features and Benefits

1. Unified Search Experience

  • Single query searches both text and images
  • No need to specify search type
  • Automatic cross-modal understanding

2. Intelligent Indexing

  • Automatic figure number extraction
  • PDF image extraction with page tracking
  • Metadata preservation for context

3. Scalable Architecture

  • Separate indexes for optimal performance
  • Serverless Pinecone deployment
  • Batch processing support

4. Interactive Results

  • Automatic image display on relevant OS
  • Text snippets with context
  • Relevance scoring for ranking

Use Cases

1. Medical Research

  • Search medical literature with symptom descriptions
  • Find anatomical diagrams using text queries
  • Cross-reference images with research papers

2. Technical Documentation

  • Locate circuit diagrams from descriptions
  • Find code architecture diagrams
  • Search troubleshooting images

3. Educational Platforms

  • Students search for visual explanations
  • Teachers find relevant diagrams for lessons
  • Create visual learning experiences

4. Digital Libraries

  • Archive historical documents with images
  • Search museum collections
  • Academic paper repositories

Getting Started

Prerequisites:

pip install torch transformers pinecone-client openai pillow PyMuPDF

Environment Variables:

OPENAI_API_KEY=your_openai_key
PINECONE_API_KEY=your_pinecone_key

Basic Usage:

# Initialize and index documents
python multimodal_clip_pinecone.py

# Search for content
Enter your search query: "neural pathways in the brain"

Advanced Features in My Implementation

1. Automatic OS Detection

The system automatically detects your operating system and opens images using the appropriate viewer:

  • Windows: start
  • macOS: open
  • Linux: xdg-open

2. Smart Caching

  • Checks for existing indexes before re-indexing
  • Provides options to re-index data
  • Maintains index statistics

3. Flexible Configuration

  • Customizable chunk sizes
  • Adjustable search result counts
  • Configurable embedding models

Performance Optimization Tips

  1. Batch Processing: Index multiple documents simultaneously
  2. Namespace Strategy: Use separate namespaces for different document types
  3. Metadata Filtering: Add custom metadata for refined searches
  4. Index Monitoring: Track vector counts and search performance

Conclusion

This multimodal RAG design combines unified embeddings with a dual-index architecture so text and images can be retrieved through the same search experience.

Whether you're building a medical research platform, technical documentation system, or educational tool, this approach provides the foundation for next-generation search experiences.

Key Takeaways:

  • CLIP enables true multimodal understanding
  • Dual-index architecture optimizes performance
  • Cross-modal search enhances user experience
  • Practical applications span multiple industries

Ready to build your own multimodal RAG system? The complete code is available in the repository, and with just a few environment variables, you can start searching across text and images seamlessly.