SONIC: Spectral Optimization of Noise
for Inpainting with Consistency

Seungyeon Baek1, Erqun Dong1, Shadan Namazifard1, Mark J. Matthews2 *, Kwang Moo Yi1
1The University of British Columbia, 2Google DeepMind
*Participated in an advisory capacity only.
Paper arXiv Code
Descriptive Text

We propose a novel training-free method of inpainting that focuses exclusively on the initial seed noise. (Top row) We show the denoising result of an initial seed noise, as we optimize the seed noise using our method. We optimize the seed noise to faithfully regenerate the non-masked regions of the input image, so as to obtain more consistent inpainting results. (Bottom row) Inpainting results of competing methods, with our final result on the right.

Abstract

We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting, in practice, their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue that the missing ingredient for training-free inpainting is the optimization (guidance) of the initial seed noise. We propose to optimize the initial seed noise to approximately match the unmasked parts of the data—with as few as a few tens of optimization steps. We then apply conventional training-free inpainting methods on top of our optimized initial seed noise. Critically, we propose two core ideas to effectively implement this idea: (i) to avoid the costly unrolling required to relate the initial noise and the generated outcome, we perform linear approximation; and (ii) to stabilize the optimization, we optimize the initial seed noise in the spectral domain. We demonstrate the effectiveness of our method on various inpainting tasks, outperforming the state of the art.

Overview

Descriptive Text

Method overview. We optimize the initial seed noise in the spectral domain XT, starting from a random noise xT, such that our denoised latent matches the masked observation y in the latent space. To allow partial observations to be encoded, we use nearest-pixel filling before passing it into the encoder. We then compute the masked mean square error in the latent space, comparing it with a fully denoised latent, and update XT accordingly. Importantly, we linearize the entire T-step denoising process, essentially disconnecting the gradient flow passing through it. This allows us to optimize the initial seed noise XT without back-propagating through the denoiser.

Optimizing the Initial Seed Noise

Loading...
Initialization Optimization

This interactive slider provides a visualization of optimizing the initial seed noise in the spectral domain. The image progressively changes to match the observed regions during the optimization process.

Spatial vs. Spectral Optimization

We compare optimizing the initial noise in the spatial domain versus our proposed spectral domain optimization. Spatial optimization is less reliable and struggles to converge effectively, while our spectral approach (optimizing in the FFT domain) achieves stable and superior convergence. Below we show the loss curves and corresponding x₀ reconstructions demonstrating this difference.

Spatial Loss 0.05
Spatial Reconstruction 0.05
Spectral Loss 3.0
Spectral Reconstruction 3.0

Qualitative Comparisons

For additional qualitative comparisons, please refer to the Supplementary Materials.

FFHQ
DIV2K
BrushBench

Additional Applications

Our initial noise optimization method can also be applied to text-to-video models for video inpainting tasks. Below, we compare the output from our method prepended to Wan2.1[1] against ProPainter[2], a video inpainting baseline. To compare the results, please move your cursor left and right.


Masked Ground Truth
Loading video
Inpainted Output

Prompts

To facilitate reproduction, we provide prompts used for all experiments. We provide the same prompts to all methods.


FFHQ Prompts
DIV2K Prompts