AI in Vision and Image Generation

Vision Fundamentals

ViT — Vision Transformer

• arXiv: 2010.11929 (Dosovitskiy et al., Google, 2020)
• Mechanism: Splits image into patches → treats each patch as a token → standard Transformer
• Result: Matches CNNs on ImageNet given enough data
• Impact: Unified vision and language; foundation of nearly all modern VLMs

CLIP (OpenAI, 2021)

• arXiv: 2103.00020
• Mechanism: Trains an image and text encoder to align representations (contrastive learning)
• Data: 400M image-text pairs from the internet
• Capability: Zero-shot image classification; text-based search in images
• Impact: Foundation of Stable Diffusion, DALL-E 2, Midjourney, and hundreds of others

SigLIP (Google, 2023)

• arXiv: 2303.15343
• Improvement over CLIP: Sigmoid loss (not softmax); more efficient; better quality
• Adoption: Gemma 3, PaliGemma, many modern VLMs

DINOv2 (Meta, 2023)

• arXiv: 2304.07193
• Training: Self-supervised (no labels); knowledge distillation from itself
• Features: Dense representations; excellent for segmentation, depth estimation
• Use: Foundation for segmentation and depth models

---

Segmentation

SAM — Segment Anything Model (Meta, 2023)

• arXiv: 2304.02643
• Data: SA-1B: 1B masks across 11M images (largest segmentation dataset)
• Capability: Segments any object given a point, box, or text as a prompt
• Zero-shot: Works without additional training

SAM 2 (Meta, 2024)

• arXiv: 2408.00714
• Extension: Segmentation in video — tracks objects across frames
• Speed: 44 FPS on video
• Uses: Video editing, medical tracking, AR

---

Image Generation with Diffusion

Stable Diffusion 1.x / 2.x (Stability AI, 2022)

• arXiv: 2112.10752 (LDM — Latent Diffusion Models)
• Mechanism: Diffusion in the latent space (4× compressed) → more efficient
• CLIP: Text conditioned by the CLIP text encoder
• Open-source: Open weights; foundation of the open image generation ecosystem

Stable Diffusion 3 / 3.5 (Stability AI, 2024)

• arXiv: 2403.03206 (SD3)
• Architecture: Multimodal Diffusion Transformer (DiT with integrated text tokens and image tokens)
• Improvement: Typography (text in images), composition of multiple objects
• SD 3.5 Large: 8B parameters; open-source state of the art

FLUX.1 (Black Forest Labs, 2024)

• Origin: Original Stable Diffusion team (Robin Rombach et al.)
• Variants: flux-dev (open), flux-schnell (open), flux-pro (API)
• Architecture: Flow matching + Transformer; no UNet
• Quality: Better typography and realism than SD3 in many cases

AuraFlow (Fal, 2024)

• Open-source: Yes; Flow Matching architecture
• Alternative: Lower-compute-cost alternative to FLUX.1

---

Generation with Proprietary Models

DALL-E 3 (OpenAI, 2023)

• Mechanism: Trained with synthetic captions generated by GPT-4 (vs original captions)
• Result: Better text-image fidelity; text in images
• Integration: ChatGPT; OpenAI API

GPT-4o Native Image Generation (2025)

• New: GPT-4o generates images natively (without a separate DALL-E)
• Capability: Image editing with context; precision in text

Midjourney v6 / v7 (2024–2025)

• Company: Independent
• Highlight: Photographic realism; aesthetics; widely used by artists
• Access: Discord + web; no public API

Imagen 3 (Google, 2024)

• Cascaded diffusion with T5-XXL text encoder
• Quality: Competitive with DALL-E 3; integrated into Google Workspace

---

Control and Personalization

ControlNet (2023)

• arXiv: 2302.05543
• Mechanism: Additional conditioning (pose, depth, edge, segmentation) via parallel networks
• Impact: Enables precise composition control without retraining SD

IP-Adapter (2023)

• arXiv: 2308.06721
• Mechanism: Adapter that conditions SD on a reference image (style/content)
• Use: "Generate an image in this style" without fine-tuning

LoRA for Images

• DreamBooth: Fine-tunes SD on 3–20 images of a concept
• LoRA: Lightweight adapter; can represent a character, style, object
• Civitai: Community of image LoRAs

---

Super-Resolution and Restoration

Real-ESRGAN

• Upscaling: 4× with real artifacts
• Use: Restoration of old photos, video upscaling

BSRGAN / SwinIR

• Restoration of degraded images (blur, noise, JPEG compression)

---

Vision-Language Models (VLMs)

---

Vision Benchmarks