ZoomLDM (CVPR 2025), Hummingbird (ICLR 2025), Gen-SIS
Srikar Yellapragada, Varun Belagali, and Ming-Quan Le
The next Biomath seminar will take place on Tuesday (Tomorrow), March 25th, 2025, at 10:30 in Room e.212, Bouygues Building. We will have the pleasure of hosting Srikar Yellapragada, Varun Belagali, and Ming-Quan Le, visiting students from Stony Brook University working under the supervision of Professor Dimitris Smaras. They will present their recent work on generative models for image synthesis and self-supervised learning:
Title 1: ZoomLDM: Latent Diffusion Model for Multi-Scale Image Generation
Speaker: Srikar Yellapragada
Abstract: Diffusion models have significantly advanced image generation capabilities, yet their extension to domains involving extremely large images — such as digital pathology and satellite imagery — remains challenging. These domains typically involve gigapixel-scale images, making direct modeling impractical. Traditional diffusion-based methods address this by generating fixed-size image patches, but this approach misses critical global context and structural coherence necessary for high-quality synthesis.
In this talk, I will introduce our recent work, ZoomLDM (accepted to CVPR 2025), a multi-scale diffusion model designed to overcome these limitations. ZoomLDM employs a magnification-aware conditioning mechanism using self-supervised learning (SSL) embeddings, allowing the synthesis of fixed-size image patches at multiple zoom levels. This approach achieves SoTA generation quality across scales and excels in generating accurate image thumbnails even from limited data. The multi-scale capabilities of ZoomLDM further enable coherent large-image synthesis at resolutions up to 4096×4096 pixels, as well as effective 4× super-resolution.
Title 2: Hummingbird: High-Fidelity Image Generation via Multimodal Context Alignment (ICLR 2025)
Speaker: Ming-Quan Le
Abstract: While diffusion models are powerful in generating high-quality, diverse synthetic data for object-centric tasks, existing methods struggle with scene-aware tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning, where it is critical to preserve scene attributes in generated images consistent with a multimodal context, i.e. a reference image with accompanying text guidance query. To address this, we introduce Hummingbird, the first diffusion-based image generator which, given a multimodal context, generates highly diverse images w.r.t. the reference image while ensuring high fidelity by accurately preserving scene attributes, such as object interactions and spatial relationships from the text guidance. Hummingbird employs a novel Multimodal Context Evaluator that simultaneously optimizes our formulated Global Semantic and Fine-grained Consistency Rewards to ensure generated images preserve the scene attributes of reference images in relation to the text guidance while maintaining diversity. As the first model to address the task of maintaining both diversity and fidelity given a multimodal context, we introduce a new benchmark formulation incorporating MME Perception and Bongard HOI datasets. Benchmark experiments show Hummingbird outperforms all existing methods by achieving superior fidelity while maintaining diversity, validating Hummingbird’s potential as a robust multimodal context-aligned image generator in complex visual tasks. Project page and paper.
Title 3: Gen-SIS: Generative Self-Augmentation Improves Self-Supervised Learning
Speaker: Varun Belagali
Abstract: Self-supervised learning (SSL) methods have emerged as strong visual representation learners by training an image encoder to maximize similarity between features of different views of the same image. To perform this view-invariance task, current SSL algorithms rely on hand-crafted augmentations, such as random cropping and color jittering, to create multiple views of an image. Recently, generative diffusion models have been shown to improve SSL by providing a wider range of data augmentations. However, these diffusion models usually require pre-training on large-scale image-text datasets, which might not be available for many specialized domains like histopathology. In this presentation, we introduce Gen-SIS, a diffusion-based augmentation technique trained exclusively on unlabeled image data, eliminating any reliance on external sources of supervision such as text captions. Leveraging the ability to interpolate in the encoder latent space, we introduce a novel pretext task for SSL: disentangling the two source images from an interpolated synthetic image. Gen-SIS improves self-supervised learning across various methods and encoder variants, highlighting its broad applicability.