multimodal-clip

Topics in the source

• CLIP - Contrastive Language-Image Pre-Training
• When to use CLIP
• Quick start
• Installation
• Zero-shot classification
• Available models
• Image-text similarity
• Semantic image search

OpenAI's model that understands images from natural language. Use when: Zero-shot image classification (no training data needed) Image-text similarity/matching Semantic image search Content moderation (detect NSFW, violence) Visual question answering Cross-modal retrieval (image→text, text→image) Metrics: 25,300+ GitHub stars Trained on 400M image-text pairs Matches ResNet-50 on ImageNet (zero-shot) MIT License Use alternatives instead: BLIP-2: Better captioning LLaVA: Vision-language chat Segment Anything: Image segmentation | Model | Parameters | Speed | Quality | |-------|------------|-------|---------| | RN50 | 102M | Fast | Good | | ViT-B/32 | 151M | Medium | Better | | ViT-L/14 | 428M | Slow | Best | 1. Use ViT-B/32 for most cases - Good balance 2. Normalize embeddings - Required for cosine similarity 3. Batch processing - More efficient 4. Cache embeddings - Expensive to recompute 5. Use descriptive labels - Better zero-shot performance 6. GPU recommended - 10-50× faster 7. Preprocess images - Use provided preprocess function | Operation | CPU | GPU (V100) | |-----------|-----|------------| | Image encoding | 200ms | 20ms | | Text encoding | 50ms | 5ms | | Similarity compute

This is an excerpt. Refer to the complete source and its current requirements.