PyTorch Expert Exchange: Efficient Generative Models: From Sparse to Distributed Inference
In this talk, we explore the advancements in making generative models more efficient, focusing on two key techniques: sparse inference and distributed parallel inference. We begin by discussing Spatially Sparse Inference during image editing, a general-purpose method that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models. Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to measured latency reduction on off-the-shelf hardware. Despite these improvements, generating high-resolution images with diffusion models remains computationally demanding, particularly for interactive applications. We propose DistriFusion to tackle this problem by leveraging parallelism across multiple GPUs. DistriFusion uses Displaced Patch Parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step. Therefore, its communication cost can be hidden into to the computation through asynchronous communication. Together, these innovations represent a substantial step forward in making generative models more practical and scalable for real-world applications.
Resources:
GAN Compression: Efficient Architectures for Interactive Conditional GANs (arXiv: https://urldefense.com/v3/__https://a..., code: (https://github.com/mit-han-lab/gan-co...)
Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models (arXiv: https://urldefense.com/v3/__https://a..., code: https://github.com/lmxyy/sige)
DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models (arXiv: https://urldefense.com/v3/__https://a..., code: https://github.com/mit-han-lab/distri...
The slides are at: https://www.dropbox.com/scl/fi/g6rt6n...
