OWL-ViT Base Patch32

What is owlvit-base-patch32?

OWL-ViT (Vision Transformer for Open-World Localization) is an advanced zero-shot text-conditioned object detection model that leverages the power of CLIP as its multi-modal backbone. The model combines a ViT-like Transformer for visual processing with a causal language model for text understanding, enabling users to query images with multiple text prompts simultaneously.

Implementation Details

The model architecture consists of a CLIP backbone with a ViT-B/32 Transformer for image encoding and a masked self-attention Transformer for text encoding. The model removes CLIP's final token pooling layer and adds lightweight classification and box heads to each transformer output token, enabling precise object detection capabilities.

• Multi-modal architecture combining vision and text processing
• Trained on large-scale image-caption datasets including YFCC100M
• Fine-tuned on COCO and OpenImages datasets
• Supports zero-shot text-conditioned object detection

Core Capabilities

• Zero-shot object detection without prior training on specific objects
• Multiple text query processing per image
• Bipartite matching loss for accurate object detection
• Open-vocabulary classification using text embeddings