Lipschitz Singularities in Diffusion Models
Diffusion models, which employ stochastic differential
equations to sample images through integrals, have
emerged as a dominant class of generative models.
However, the rationality of the diffusion process itself
receives limited attention, leaving the question of
whether the problem is well-posed and well-conditioned.
In this paper, we explore a perplexing tendency of
diffusion models: they often display the infinite
Lipschitz property of the network with respect to time
variable near the zero point. We provide theoretical
proofs to illustrate the presence of infinite Lipschitz
constants and empirical results to confirm it. The
Lipschitz singularities pose a threat to the stability
and accuracy during both the training and inference
processes of diffusion models. Therefore, the mitigation
of Lipschitz singularities holds great potential for
enhancing the performance of diffusion models. To
address this challenge, we propose a novel approach,
dubbed E-TSDM, which alleviates the Lipschitz
singularities of the diffusion model near the zero point
of timesteps. Remarkably, our technique yields a
substantial improvement in performance. Moreover, as a
byproduct of our method, we achieve a dramatic reduction
in the Fréchet Inception Distance of acceleration
methods relying on network Lipschitz, including DDIM and
DPM-Solver, by over 33%. Extensive experiments on
diverse datasets validate our theory and method. Our
work may advance the understanding of the general
diffusion process, and also provide insights for the
design of diffusion models.
