InceptionNeXt

InceptionNeXt is a drop-in replacement for the ConvNeXt block that eliminates the cost of a single large $7 \times 7$ depthwise convolution by factorizing it into four parallel, low-cost Inception-style branches. The motivation is that large-kernel depthwise convs dominate ConvNeXt's wall-clock latency on modern accelerators despite their relatively small FLOP count: depthwise convolutions are memory-bandwidth bound and large kernels amplify that bottleneck.

The InceptionDWConv2d operator splits the input channels into four groups and applies in parallel: (i) an identity branch, (ii) a small $3 \times 3$ square depthwise conv, (iii) a horizontal band depthwise conv of shape $1 \times K$, and (iv) a vertical band depthwise conv of shape $K \times 1$, with $K = 11$ by default. The outputs are concatenated along the channel dimension:

The factorization preserves a large effective receptive field (the band convolutions span $K$ pixels along each axis) while cutting depthwise FLOPs and — more importantly — memory traffic. All other elements of the ConvNeXt block (LayerNorm, inverted bottleneck MLP, layer scale, residual) are kept unchanged, so the Tiny / Small / Base variants reuse the same depth and width tables as ConvNeXt while running noticeably faster at equal accuracy.