Energy-Guided Flow Matching

No new backbone, no additional training data, and no inference-time spectral computation. EG-FM changes the probability path itself, teaching the model to establish global structure before synthesizing fine-grained details.

paper: https://arxiv.org/abs/2608.05811

code: https://github.com/ysng123/EG-FM

Figure 1. FID versus training epochs. EG-FM variants move toward the desirable lower-left region, achieving lower FID with fewer epochs.

Takeaway

Pixel-space generative models operate directly on RGB values, bypassing the information loss introduced by latent compression. This makes them attractive for preserving precise boundaries, textures, and other high-frequency details. However, direct pixel-space generation also requires the model to coordinate global structure and local appearance in a high-dimensional space.

Standard Flow Matching connects Gaussian noise to a fixed clean-image endpoint. At every time step, the model is directed toward the same full-spectrum image, leaving the temporal evolution of different frequency components to be learned implicitly.

We introduce Energy-Guided Flow Matching, or EG-FM, which explicitly builds a coarse-to-fine ordering into the generative trajectory. Instead of using a fixed clean endpoint, EG-FM constructs a moving spectral endpoint that evolves smoothly from a low-frequency image to the complete clean image. The rate at which frequency information is released is determined separately for every training sample according to its spectral energy.

This path-level modification does not require changes to the backbone architecture or training data. On ImageNet at $256\times 256256\backslash times256$, EG-FM reaches an FID of 1.55 after 200 epochs and 1.45 after 600 epochs with PixelDiT-XL. At $512\times 512512\backslash times512$, HyperDiT-H with EG-FM obtains an FID of 1.58 after only 40 additional high-resolution adaptation epochs. When transferred to text-to-image generation, EG-FM achieves 0.85 on GenEval and 83.9 on DPG-Bench.

---

1. Why Does This Problem Matter?

Latent-space generative models first compress images using an autoencoder and then learn the distribution of the resulting latent representations. This substantially reduces computational cost, but lossy compression can discard information that the subsequent generative model can no longer recover exactly.

Pixel-space models avoid this reconstruction bottleneck by learning the image distribution directly in RGB space. Their challenge is different: the model must operate in a much higher-dimensional space and learn complex dependencies between global layout, object structure, local boundaries, and fine textures.

The standard conditional path used in Flow Matching is

$$z_t=tx+(1-t)ϵ,z\_t\; =\; t\backslash ,x\; +\; (1-t)\backslash epsilon,$$

where $xx$ is a clean image and $ϵ\sim \mathcal{N}(0,I)\backslash epsilon\backslash sim\backslash mathcal\{N\}(0,I)$ is Gaussian noise.

The corresponding velocity target is

$$v_t=x-ϵ\mathrm{.}v\_t\; =\; x-\backslash epsilon.$$

Because $xx$ is fixed, the model points toward the same complete, full-spectrum image at every time step. The formulation is mathematically simple and computationally efficient, but it does not explicitly represent the coarse-to-fine evolution commonly associated with image generation:

• Low-frequency components primarily determine global layout and large-scale structure.
• High-frequency components encode edges, textures, and small visual details.
• A structured generation process should ideally establish the former before resolving the latter.

Under standard Flow Matching, the network must discover this ordering entirely through optimization. It is effectively asked to reason about composition, object shape, boundaries, and texture from the same full-spectrum target throughout training. The paper argues that this unified full-frequency objective can make pixel-space generation unnecessarily difficult.

The motivation behind EG-FM is therefore straightforward:

Instead of requiring the model to discover frequency ordering on its own, we can encode that ordering directly into the probability path.

---

2. How EG-FM Builds a Coarse-to-Fine Generation Path

EG-FM turns the fixed clean-image endpoint of standard Flow Matching into a moving spectral endpoint:

$$z_t=t{y}_t(x)+(1-t)ϵ\mathrm{.}z\_t=t\backslash ,y\_t(x)+(1-t)\backslash epsilon.$$

At the beginning of the trajectory, $y_t(x)y\_t(x)$ is a low-pass version of the training image. As time advances, it gradually restores edges, textures, and other high-frequency information until it becomes the complete clean image. The trajectory therefore still starts from Gaussian noise and ends at the real image, but the intermediate targets now express an explicit visual order: establish global structure first, then fill in fine detail.

Figure 2. Overview of Energy-Guided Flow Matching. EG-FM builds a sample-adaptive frequency response and trains on a moving spectral endpoint that evolves from low to full frequencies.

The transition is implemented with a heat-kernel filter. One scalar controls how strongly high frequencies are suppressed: a larger value produces a smoother endpoint, while decreasing it continuously reveals more of the original spectrum. This provides a stable path between the initial coarse image and the final full-resolution target, rather than switching frequency bands on abruptly.

A shared filtering schedule, however, would not mean the same thing for every image. A portrait with broad smooth regions and a scene filled with foliage or text can have very different spectral-energy distributions. EG-FM therefore measures how much information each individual image has recovered relative to its own initial spectral gap. At the same training time, every sample has regained the same proportion of its missing energy, even though the filter strength required to reach that point differs from image to image.

Figure 3. Sample-adaptive heat-time schedules. Under the same release clock, images with different spectral compositions follow different heat-time trajectories.

In practice, the method finds the corresponding filter strength with a short bisection procedure and uses a smooth release clock so that frequency information enters gently at both ends of the trajectory. This makes the model’s time variable more comparable across samples: it represents relative progress from coarse structure to full detail, rather than a single blur level imposed on every image.

Because the endpoint itself changes over time, the correct training target must describe both motion toward the endpoint and motion of the endpoint. Its velocity is

$$v_t=y_t(x)-ϵ+t{\mathrm{\partial }}_t{y}_t(x)\mathrm{.}v\_t=y\_t(x)-\backslash epsilon+t\backslash ,\backslash partial\_t\; y\_t(x).$$

The first term is the familiar transport from noise toward the current image target. The second corrects for the fact that the target is simultaneously revealing new frequency content. EG-FM computes this correction analytically from the energy schedule, so it does not need to backpropagate through the bisection procedure.

Taken together, these choices form one coherent trajectory design: the heat kernel creates a smooth coarse-to-fine endpoint, the energy schedule adapts its pace to each image, and the velocity target follows the resulting path exactly. The backbone and the usual velocity-regression objective remain unchanged; no auxiliary loss is introduced.

---

3. Experimental Results

The results show that EG-FM not only accelerates convergence but also achieves state-of-the-art performance for pixel-space generation on ImageNet $256\times 256256\backslash times256$, reaching 1.45 FID. This demonstrates that redesigning the probability path alone can advance pixel-space generation to a new state of the art without increasing model size or inference cost.

3.1 Faster Convergence and State-of-the-Art Pixel-Space Generation

The ImageNet $256\times 256256\backslash times256$ experiments evaluate EG-FM with three architecturally different pixel-space backbones.

Table 1. Class-conditional generation on ImageNet $256\times 256256\backslash times256$. Each Energy-Guided FM row uses the same backbone and ADM evaluation protocol. PixelDiT additionally exposes convergence at multiple training budgets. NFE denotes the number of function evaluations; ×2 accounts for conditional and unconditional classifier-free-guidance evaluations.

The PixelDiT comparison is especially informative. EG-FM reaches an FID of 1.55 after 200 epochs, while the standard model reaches a comparable 1.54 after 800 epochs.

The advantage also appears early in training. At the same 80-epoch budget, EG-FM lowers PixelDiT-XL’s FID from 2.36 to 1.99.

After 600 epochs, EG-FM further reduces PixelDiT-XL’s FID to 1.45, establishing the best result among the pixel-space methods in the table. In other words, EG-FM matches the standard model with one quarter of the training epochs and ultimately reaches state-of-the-art performance among comparable pixel-space approaches.

These are controlled within-backbone comparisons: the architecture, model size, sampler, number of function evaluations, and evaluation protocol are held fixed. The principal experimental variable is the probability trajectory. The SOTA improvement can therefore be attributed directly to the redesigned generative path rather than to a larger model or a more expensive sampler.

3.2 Transfer from $256\times 256256\backslash times256$ to $512\times 512512\backslash times512$

The high-resolution experiment tests whether the same frequency-release trajectory remains effective when the number of pixels is quadrupled.

PixelDiT-XL and HyperDiT-H are initialized from their respective $256\times 256256\backslash times256$ EG-FM checkpoints and then adapted at $512\times 512512\backslash times512$ for only 40 additional epochs.

Table 2. Detailed comparison for class-conditional ImageNet generation at $512\times 512512\backslash times512$. Metrics are computed on 50K generated samples with the ADM evaluation suite. NFE includes conditional and unconditional classifier-free-guidance evaluations; † denotes continued training from a $256\times 256256\backslash times256$ checkpoint. A dash indicates an unreported item.

PixelDiT-XL with EG-FM reaches 1.68 FID, compared with 1.81 for the reported standard PixelDiT result after substantially more continued training. HyperDiT-H with EG-FM obtains an FID of 1.58, the best result among the methods listed in the paper’s $512\times 512512\backslash times512$ comparison.

These results suggest that the energy-guided path is not tied to one image resolution. The same trajectory principle remains effective after scaling to a denser pixel space, with relatively little high-resolution adaptation.

3.3 Transfer to Text-to-Image Generation

Text-to-image generation is a stronger transfer test than changing resolution alone. The model must preserve text-conditioned semantics while progressively introducing pixel-space detail.

Using the same PixelDiT text-to-image backbone, EG-FM produces the following improvements in the paper’s full $512\times 512512\backslash times512$ comparison:

Table 3. Text-to-image generation at $512\times 512512\backslash times512$. † denotes a pixel-space diffusion model. Because EG-FM changes only the training path, it retains the PixelDiT-T2I backbone, parameter count, and inference throughput.

The GenEval improvement is distributed across several compositional abilities. Compared with PixelDiT-T2I, EG-FM-T2I improves counting from 0.70 to 0.74, spatial position from 0.53 to 0.72, and color-attribute binding from 0.65 to 0.77.

On DPG-Bench, EG-FM-T2I raises the overall score from 83.7 to 83.9, with further gains in global consistency, attributes, relations, and other dense-prompt requirements.

Figure 4. Qualitative results from EG-FM, including ImageNet class-conditional and text-to-image samples.

Figure 5. Additional $512\times 512512\backslash times512$ text-to-image samples generated by EG-FM-T2I, spanning architecture, portraits, animals, natural scenes, products, and artistic styles.

3.4 Why EG-FM Is More Than “Adding Blur”

The ablation studies show that performance depends on the complete trajectory design, not merely on applying low-pass filtering.

Initial endpoint strength

The parameter ${\sigma }_0\backslash sigma\_0$ controls the degradation strength of the initial endpoint.

A moderate low-pass endpoint performs best. If filtering is too weak, the trajectory approaches standard Flow Matching. If filtering is too strong, essential spatial structure is removed along with premature detail.

The resulting U-shaped trend indicates that the initial endpoint must balance two objectives: suppressing fine details that appear too early while retaining enough structure to guide generation.

Granularity of heat-time adaptation

Performance improves as the schedule becomes more specific. Dataset-level and class-level averages help, but the image-specific schedule performs best. This directly supports the central hypothesis that images with different spectra require different frequency-release rates.

Shape of the release clock

All four functions use the same initial and final endpoint and release the same total spectral energy. They differ only in when that energy is introduced.

The result demonstrates that generation quality depends not only on what frequency content is released, but also on when it enters the trajectory.

---

4. Training and Inference Efficiency

EG-FM performs Fourier transforms, spectral-energy reductions, and bisection while constructing its training targets. The natural question is whether this additional computation offsets the convergence advantage.

The paper compares Standard FM and EG-FM under matched hardware, batch size, numerical precision, data pipeline, and gradient-accumulation settings.

The largest measured increase in per-sample computation is only 0.0256%. Across the evaluated configurations, per-step wall-clock overhead ranges from 0.41% to 4.81%. The 4.81% case occurs with the smallest PixelDiT-B/16 model; all other tested configurations remain within approximately 1% of Standard FM.

For the direct velocity-prediction models used in the main experiments, the spectral operations are needed only when constructing training supervision. At inference time, the sampler directly integrates the learned velocity field.

No FFT, inverse FFT, energy calculation, or bisection is performed during sampling. EG-FM uses the same backbone, solver, time grid, and number of function evaluations as Standard FM. It therefore leaves inference FLOPs and wall-clock time unchanged in this setting.

This distinction is important:

EG-FM moves complexity into the definition of the training path, not into the deployed generative model.

Because the per-epoch overhead is small, the reduction in epochs required to reach a target FID translates into lower end-to-end training cost.

---

5. How Should the Research Contribution Be Assessed?

The contribution of EG-FM can be evaluated at several levels.

A clearly defined problem

The paper identifies an underexplored issue in pixel-space generation. Existing work often focuses on architecture design, patchification, capacity allocation, or additional objectives, while treating the probability path as fixed.

EG-FM asks whether the path itself should reflect the multiscale structure of natural images. This is a meaningful question because pixel-space models must retain the high-frequency information that latent compression often discards.

A coherent technical mechanism

The three main components form a closed technical loop:

• The moving spectral endpoint determines how the target changes.
• Energy-guided heat-time scheduling determines how quickly it changes for each sample.
• The path-consistent velocity ensures that the supervision matches the derivative of the actual trajectory.

Removing any one of these components would leave the formulation incomplete. A moving endpoint without adaptive scheduling would ignore spectral variation across images. A moving endpoint without the endpoint-motion velocity term would provide inconsistent supervision.

Backbone-independent improvements

EG-FM is evaluated with DeCo, HyperDiT, and PixelDiT. The consistent improvement across these different architectures is important evidence that the method is not narrowly tailored to one backbone.

The strongest empirical evidence comes from the paired Standard-FM versus EG-FM comparisons using the same architecture and sampling configuration. Comparisons with unrelated methods remain useful for context, but those methods can differ in model size, training duration, and sampling budget.

Evaluation beyond a single setting

The experiments cover:

• ImageNet class-conditional generation at two resolutions;
• three distinct backbone families;
• multiple model capacities;
• text-to-image transfer;
• initial-endpoint strength;
• sample-adaptation granularity;
• release-clock curvature;
• classifier-free-guidance scale;
• matched training and inference cost.

This creates a relatively complete chain of evidence from the proposed trajectory design to generation quality, convergence, transfer, and efficiency.

---

Overall Assessment

Energy-Guided Flow Matching offers a simple but consequential perspective:

The difficulty of pixel-space generation does not have to be addressed only by building a more complicated network. The probability path itself can carry a useful prior about how visual information should emerge.

Standard Flow Matching uses a fixed full-spectrum image as its endpoint and leaves the ordering from global structure to local detail implicit. EG-FM turns the endpoint into a time-dependent spectral object and uses each image’s own energy distribution to make the shared time variable represent comparable relative progress across samples.

Its value lies at three levels.

First, it converts an intuitive coarse-to-fine phenomenon into an explicit and differentiable probability path.

Second, it replaces a shared frequency schedule with sample-adaptive energy matching, respecting the fact that different images contain different spectral compositions.

Third, it improves generation quality and convergence without modifying the backbone, adding an auxiliary loss, or increasing inference cost for the main velocity-prediction setting.

EG-FM is therefore more than a frequency-filtering technique. It is a systematic study of how a generative trajectory should organize visual information. The results suggest that, in high-dimensional pixel space, establishing structure first and progressively releasing detail is not merely an intuitive description of generation—it can be formulated directly as the path that a Flow Matching model is trained to follow.