Tag: Operator Fusion

Affiliate Disclosure: This post contains AliExpress affiliate links. If you make a purchase through these links, the author may receive a commission.

---

1. Can Partitioned DiT Blocks Be Recombined?

When deploying generative Diffusion Transformers (DiT) in embedded NPU environments, runtime memory limits and operator support constraints are critical factors directly dictating architectural design. This in-depth experiment was conducted on 2026-8-20 on a Radxa ROCK 5C 8GB single-board computer.

The experiment used a directly purchased ROCK 5C 8GB board.

The test board is a RK3588S2-based ROCK 5C, and the RKNN target is rk3588. It should be distinguished from the ROCK 5C Lite listed on the same product page.

In Part 1, the Anima configuration partitioned the 28 DiT blocks into attention, cross-attention, and FFN components, yielding 84 FP16 RKNN models, with AdaLN and residual additions handled on the CPU. Here, we examined whether the 3 models per block could be fused into a single model.

The baseline pipeline processed AdaLN (Adaptive LayerNorm) operations and residual additions in the host CPU Python runtime. However, repeatedly loading, executing, and releasing 3 subgraphs per step within a single DiT block—round-tripping intermediate tensors to the CPU host—incurs substantial driver overhead. If self-attention, cross-attention, FFN, and AdaLN within a block could be fused into a single RKNN graph (a fused 1-model), we expected to reduce the lifecycle invocations per block from 3 to 1 while eliminating host tensor transfer overhead.

However, compiling a graph containing AdaLN operators under RKNN Toolkit 2.3.2—the default compiler at the time—triggered a fatal compiler segmentation fault (SIGSEGV, exit code 139). We first had to determine whether this crash stemmed from fundamental hardware non-support for the Anima DiT architecture or a compiler bug.

---

2. Building the Same AdaLN Graph with Different Toolkits

To isolate the compiler crash, we extracted a minimal ONNX reproducer containing only the crashing AdaLN operator instead of the full DiT block and ran compilation experiments.

We compiled the identical AdaLN reproducer graph using RKNN Toolkit 2.3.2 and 2.3.0. All other build conditions were kept constant to compare results across compiler versions.

Using the identical reproducer ONNX model, target (rk3588), and single-core optimization settings, the build results were as follows:

Toolkit Version	AdaLN Graph Build	Exit Code
2.3.2	Failed: SIGSEGV	139
2.3.0	Success	0

In 2.3.2, the build aborted with a SIGSEGV immediately following LayerNorm graph optimization. In 2.3.0, the identical graph built without errors.

Because the outcome differed solely by version on identical input graphs and targets, we determined this failure to be a compiler regression. The older 2.3.0 release provided a functional workaround for this graph, rather than resolving the issue by upgrading to a newer release.

---

3. Verifying Fused Model Execution at 64px and 512px

Using Toolkit 2.3.0, we compiled a fused model for the first DiT block and verified both a 64px test configuration and a 512px production inference configuration.

Fused Block Resolution	RKNN Conversion	On-Board Execution
64px	Success	Init and inference verified
512px	Success	Init, inference, and release verified

The 512px model build completed without triggering system swap. However, because certain Transpose and Erf operations were offloaded to the CPU, fusing into a single RKNN file does not mean every operator executes entirely on the NPU.

A more critical validation checkpoint was binary compatibility with the target board’s local runtime. Even if compiled with 2.3.0, deployment complexity escalates sharply if the board OS’s RKNPU kernel driver or runtime libraries must be forcibly replaced.

The compiled model was executed under runtime API 2.3.0 and driver 0.9.8. Without modifying the kernel or driver, we verified loading, initialization, inference, and context release across 3 repeated rounds of conditional and unconditional inference.

In this single-block validation, switching only the compiler environment was sufficient to execute on the existing board runtime.

---

4. Numerical Differences in Single-Block Output

Once hardware execution of the fused model was verified, we performed a single-block correctness gate to confirm whether the NPU fused operation numerically matched the original floating-point reference tensors. The references used were the original PyTorch FP16 block outputs and ONNX Runtime (ORT) outputs.

Target Architecture	Reference Tensor	Cosine Similarity	Max Absolute Error (Max Abs)	Mean Absolute Error (Mean Abs)	Mean Squared Error (MSE)	NaN/Inf Detection
64px Fused	PyTorch	0.99998315	0.28125	0.0018879	4.2748e-05	0 / 0
64px Fused	ONNX Runtime	0.99998608	0.28125	0.0018948	3.5387e-05	0 / 0
512px Split (Baseline)	PyTorch	0.99999881	0.14937	0.0002283	1.0841e-06	0 / 0
512px Fused (Candidate)	PyTorch	0.99997673	0.25000	0.0011279	2.1100e-05	0 / 0
512px Fused (Candidate)	ONNX Runtime	0.99997590	0.28906	0.0011662	2.1854e-05	0 / 0

The 512px fused model recorded a cosine similarity of 0.99997673 and a maximum absolute error of 0.25000 against the PyTorch reference output, with no NaNs or Infs detected.

However, the fused model’s MSE was 2.1100e-05, roughly 19.5 times higher than the 1.0841e-06 of the baseline split architecture. High cosine similarity on a single block alone cannot guarantee the image quality of the final output after traversing all 28 blocks over multiple steps.

---

5. Fewer Context Invocations, but Only a 0.47% Speedup

The primary motivation for pursuing subgraph fusion was latency reduction. We anticipated measurable speedups by migrating AdaLN and residual operations from host Python into the compiled graph, thereby reducing driver context invocations.

On the ROCK 5C board, we executed 3 rounds of a 2-pass Classifier-Free Guidance (CFG) computation stream for a 512px single block and precisely measured the average execution time of each stage.

The table below compares measured latency stage-by-stage between the baseline 3-model split configuration and the candidate Fused 1-model configuration.

Execution Stage & Component	Baseline 3-Model Split	Candidate Fused 1-Model	Fused Delta	Cause of Change & Analysis
Model Load	0.4863 s	0.4814 s	-0.0049 s	Weight file loading time (marginal reduction for a single block)
Runtime Init	0.4464 s	0.4240 s	-0.0224 s	Slight reduction in NPU runtime context initialization time
NPU Inference (×2)	3.1177 s	3.2972 s	+0.1795 s (+5.76%)	NPU execution slowdown due to operator fusion
Context Release	0.1694 s	0.1623 s	-0.0071 s	Context memory release time
Host AdaLN & Residual	0.1658 s	0.0000 s	-0.1658 s (-100%)	Elimination of separate Python host AdaLN path
Net Component Sum	4.3856 s	4.3650 s	-0.0206 s (-0.47%)	Actual net time saved was merely 0.0206 s
Measured Wall Time	4.7379 s	4.4981 s	-0.2398 s (-5.06%)	Includes instrumentation distortion from multiple memory snapshots in split mode

The benchmark data revealed bottlenecks that ran counter to intuitive expectations. First, host compute time of 0.1658 s was completely eliminated to 0 s in the Fused model. However, pure NPU forward inference time (Inference ×2) increased by +0.1795 s (+5.76%) from 3.1177 s to 3.2972 s. The increased inference latency in the fused configuration completely offset the savings gained by removing host compute (0.1658 s).

Second, while context invocations dropped by 66.7% from 3 to 1, lifecycle overhead (Load, Init, Release) did not decrease as expected. Total single-block lifecycle time fell only by 3.1% from 1.1021 s to 1.0678 s. This revealed that practical RKNPU driver initialization and loading overhead is governed by total model file weight footprint (approximately 140~155 MB) and graph complexity rather than raw C API call counts.

Third, while measured wall time appeared to decrease by 0.2398 s (5.06%), this figure included profiling overhead from collecting memory snapshots 6 times across 3 sub-contexts in split mode. Excluding snapshot measurement distortion, the actual net speedup based on pure component totals was merely 0.0206 s (-0.47%) per block.

---

6. NPU Device Mapping Memory Increased by 94.30 MiB

While the latency reduction was negligible, runtime memory suffered significant penalties. The memory metrics measured at the board level were as follows:

Memory Metric	Baseline 3-Model Split	Candidate Fused 1-Model	Difference & Rate of Change
NPU Device-Mapped Device PSS	127,144 KiB	223,704 KiB	+96,560 KiB (+94.30 MiB, +75.9%)
Peak Process PSS During Execution	648,282 KiB	694,882 KiB	+46,600 KiB (+45.51 MiB, +7.2%)
New System Swap Generated	0	0	0 (Safe for single block)

The split architecture executes each branch sequentially before releasing it. The peak NPU device mapping maintained at any one time was 127,144 KiB, or approximately 124.16 MiB.

The fused architecture retains the entire block within a single runtime context. Device mapping increased to 223,704 KiB—approximately 218.46 MiB—consuming an additional 94.30 MiB.

On the ROCK 5C sharing 8GB of total LPDDR4X memory—where the background OS, text encoder, and VAE must coexist—a 75.9% surge in device-mapped memory introduces severe system pressure.

---

7. Why Operator Fusion Was Not Adopted

Extrapolating single-block differences across the complete pipeline suggested an estimated reduction of approximately 7.51 s. This estimate was not measured by running full image generation.

• Speed: Reductions in host compute were largely cancelled out by increased NPU inference latency.
• Memory: Executing the same block required higher device-mapped memory.
• Precision: Although single-block cosine similarity was high, MSE was larger than in the baseline split configuration.

Because speedup was capped at 0.47% while memory consumption expanded, we retained the baseline split architecture for this board.

The verification scope was limited to a single block. These findings do not reflect experiments fusing all 28 blocks to generate complete images or prolonged operational testing.

Both the residency experiment in Part 3 and this fusion experiment demonstrated that reducing call counts alone cannot account for overall performance. NPU compute time and memory overhead must be evaluated together to choose optimizations viable on a 8GB board.