As part of my ongoing effort to learn more about deep learning, I have been building diffusion and flow-matching models from scratch. My initial goal was to build a model for audio using mel-spectrogram inputs. However, evaluating generated mel-spectrograms by eye was difficult, and converting them back to audio added another layer of complexity. I decided to pivot to images first so I could focus on learning the core mechanics of generative modeling. Specifically, I chose a classic celebrity image dataset.
tl;dr In this post, I build a flow-matching image generator from scratch by following Back to Basics: Let Denoising Generative Models Denoise. A Python notebook with the full implementation is available here.
Flow matching and diffusion models generate images by progressively transforming noise into clean data. Suppose x0 represents pure noise and x1 represents a clean image. The model learns to transform x0 into x1 through updates similar to:
In many diffusion formulations, models are trained to predict noise ϵ. In this framework, the model estimates the noise residual that was added to the data to reach the current state x(t) at time step t. By calculating this predicted noise, we can iteratively move toward a cleaner image by “subtracting” the predicted noise at each step.
A recent paper I came across, Back to Basics, suggests that denoising models learn more effectively when they focus on the data distribution rather than the noise distribution. Specifically, it proposes predicting the clean image x from t and a noisy input z, then computing mean squared error (MSE) in velocity space.
The simplicity of this formulation was eye-opening, and it became the basis for my implementation. The algorithm below comes directly from the paper; JiT stands for Just Image Transformer.
This algorithm will serve as the blueprint for the implementation in this post.
You can find the full notebook with data loading, the model, the training loop, and sampling here.
Instead of copying the notebook verbatim, I will give a high-level overview and highlight a few areas I found interesting while building the project.
I used a standard Transformer architecture based on self-attention, with modern components such as QK norm, SwiGLU activation, and RoPE embeddings. As an experiment, I also tried using Derf instead of RMSNorm. The model generates 256x256 images.
To process an image represented as a tensor of shape (C, H, W), the Transformer needs to encode it as a sequence of tokens. Following standard practice, we patchify the image by dividing it into small squares of size p. This produces a tensor of shape (T, C'), where T = H * W // (p * p) and C' = C * p * p. This assumes a stride of p, which produces non-overlapping patches.
The patchify operation is available through torch.nn.functional:
F.unfold(
img_tensor,
kernel_size=patch_size,
stride=patch_size,
)
The inverse operation is:
folded_sum = F.fold(
unfolded,
output_size=(H, W),
kernel_size=patch_size,
stride=stride,
)
The following visualization shows patchification when stride equals patch_size.
Setting stride to 50% or 25% of patch_size creates overlapping patches, which can reduce boundary artifacts. When folding those patches back into an image, we need a normalization mask to account for the overlap.
# Create a normalization mask by folding a tensor of ones
ones = torch.ones_like(unfolded)
norm_mask = F.fold(
ones,
output_size=(H, W),
kernel_size=patch_size,
stride=stride,
)
folded_norm = folded_sum / norm_mask
Without the overlap mask, the reconstructed image would be unnaturally bright where patches overlap, as shown below.
In this flow-matching setup, the model receives a corrupted image rather than a clean one. Let x be a clean image and z(t) its corrupted form, where t represents time. z(0) is pure noise and z(1) equals x.
Following the Back to Basics formulation, the Transformer predicts x in image space, while the loss is computed in velocity space (v).
Given t and the corrupted input z:
Using xpredicted, the loss is:
During training, we sample t to generate the corrupted images used for denoising. The sampling distribution is important because it determines which corruption levels the network observes. This matters particularly as image resolution increases: larger images contain more spatial information than smaller ones. For example, corrupting 42 pixels in a 64×64 image affects roughly 1% of its pixels, whereas the same number in a 256×256 image affects only about 0.06%.
As the output resolution increases, we therefore need to adjust the sampling distribution for t. Sampling too many values close to 1 biases the model toward lightly corrupted images. I used the schedule suggested by Back to Basics for 256x256 images. For more detail on noise schedules, Simple Diffusion is a useful reference.
import math
import torch
def sample_logit_t(dim, resolution=256, base_res=256, sig=0.8, mu_base=-0.8):
"""
Adjusts the logit-normal sampling mean based on resolution.
Shifts the distribution to higher noise for higher resolutions.
"""
# 1. Calculate the resolution shift (from Simple Diffusion paper)
# As resolution increases, we shift the mean to sample higher noise levels
mu = mu_base - math.log(resolution / base_res)
# 2. Sample from logit-normal distribution
u_sample = torch.randn(dim) * sig + mu
t = torch.sigmoid(u_sample)
return t
The graph below shows the distribution of t produced by this sampling schedule.
Flow matching requires us to solve an ordinary differential equation (ODE) iteratively. The model defines a velocity field, or rate of change over time, which moves the image step by step from noise toward a clean sample.
While many implementations use adaptive solvers, I chose Euler’s method to keep the implementation simple and avoid relying on a solver library.
Suppose we know a function x(t), its derivative dx/dt, and its value at time t. We can approximate x(t + epsilon) by taking a small step in the direction of the derivative:
We can repeat this process until we reach t2, producing an estimate of x(t2).
A smaller step size, epsilon, generally produces a more accurate approximation at the cost of more steps.
@torch.no_grad()
def euler_method(y0, x0, x1, dfdx, step_size):
x = x0
x_tensor = torch.tensor(x, device=y0.device, dtype=y0.dtype)
y = y0
B = y.shape[0]
while x < x1:
dfdx_val = dfdx(y, x_tensor.repeat(B))
y = y + step_size * dfdx_val
x += step_size
x_tensor = torch.tensor(x, device=y0.device, dtype=y0.dtype)
return y
To verify the implementation, the visualization below shows Euler’s method approximating a sine wave with different step sizes. Smaller steps correspond to a smaller epsilon and lower approximation error.
With 100 steps, Euler’s method closely matches the sine wave.
I ran the training loop on the full celebrity dataset using a 12-layer Transformer with approximately 114 million parameters. With an embedding size of 768, patch_size=16, and stride=12, the model reached 100 epochs in about 6 hours on an RTX 5090.
Here are a few samples generated after this modest training run:
While far from perfect, these images show that the model learned the overall structure of faces. The overlapping patches also mitigated grid-like artifacts, a common failure mode I encountered during this exercise.
Noticeable denoising artifacts appear at high values of t, likely because the sampling schedule is biased toward lower values. An adjusted schedule may improve this, which I will leave for future exploration.
This learning path has been highly nonlinear. It involved repeatedly asking debugging questions (“Is my RoPE implementation correct?” “Is my attention implementation correct?”) and reading different papers to understand the essence of flow matching.
In an era where a single prompt to an LLM can generate anything, one might ask: what is the point of spending all this time writing a post to document the journey? For me, it comes down to the pure joy of building. It is much like hiking: there is a deep, quiet satisfaction in setting a goal and seeing it through, regardless of how many others have walked the same path. Writing these posts remains rewarding because each one represents a small milestone in my learning journey.