[Paper] [Citations] [Clip Colab] [Coca Colab]

⚠️ Main branch training stack notice

main now uses the post-refactor training stack by default. Training is organized around TrainingTask wrappers, dict-based batches, FSDP2 support, NaFlex image/audio pipelines, and multiple torch.compile strategies. The scope has grown well beyond the original refactor and now includes several new model families. For the older release-stable training API, pin to the v3 branch or the latest 3.x release on PyPI. Inference usage of pretrained image/text models is still intended to be compatible, but training scripts and downstream integrations should review the changes below before upgrading.

New / experimental model families on main:

• NaFlex CLIP — variable-resolution/aspect image towers (timm naflexvit) with token-budget batching (--use-naflex, naflex_* configs)
• NaFlex CLAP — audio-text contrastive training with variable-duration audio (naflexclap_* configs)
• NaFlex GenLIP / GenLAP — generative image/audio captioning with prefix-LM attention and packed [media ; text] rows (naflexgenlip_*, naflexgenlap_* configs, tiktoken text)
• Modern text tower — text_cfg.text_arch="modern": RoPE, SwiGLU/ReLU², RMSNorm, masked pooling (eos/mean/map), with optional qk-norm, gated attention, register tokens, sandwich norm, value residuals, and zero-init residual (moderntext-* configs)
• Variable-length text — text_cfg.variable_text=true pads captions to the per-batch max instead of a fixed context length (works with the modern tower and HF towers)
• Hugging Face ModernBERT text towers — e.g. gte-modernbert-base-ViT-B-32-256
• MaMMUT — single text decoder used in two passes per the paper: bi-directional without cross-attention for contrastive learning, causal with cross-attention for captioning (trains via the CoCa task/loss). Two config families, mirroring coca/coca2: mammut_* reproduces the original LAION-fork numerics exactly via legacy flags (pool_type="avg_all", use_pad_mask=false) and loads the released