MobileNet-v4

MobileNet-v4 is built around a single searchable block, the Universal Inverted Bottleneck (UIB). It extends the inverted bottleneck of MobileNet-v2 with two optional depthwise convolutions: one before the $1 \times 1$ expansion and one between the expansion and the $1 \times 1$ projection,

where $E$ is the pointwise expansion, $P$ the linear pointwise projection, and each $D$ is either a depthwise convolution or the identity. The four on/off combinations recover four familiar blocks: the classic inverted bottleneck (IB, middle depthwise only), a ConvNeXt-like block (start depthwise only, a cheap large-kernel spatial mix before the expansion), a transformer FFN (neither — two pointwise layers), and the new ExtraDW block (both), which deepens the network and widens its receptive field at almost no cost. Because the expansion and projection are shared by all four instantiations, a NAS super-network over UIB shares more than 95% of its parameters, and the search simply decides which depthwise layers to keep at each position. The stems use a fused inverted bottleneck — a dense $3 \times 3$ convolution into a $1 \times 1$ projection — which is faster than a depthwise pair at high resolution.

The hybrid variants interleave UIB blocks with Mobile MQA, a multi-query attention block tuned for accelerators: every query head shares a single key and value head, which raises operational intensity when the token count is small relative to the channel width, and the keys and values can be spatially reduced by a stride-2 $3 \times 3$ depthwise convolution,

Sharing keys and values cuts attention latency by more than 39% on mobile accelerators relative to multi-head attention at a negligible accuracy cost. Residual branches of the hybrids are scaled by a learnable per-channel layer scale $\gamma$ initialised at $10^{-5}$.

Five models come out of the search: Conv-Small, Conv-Medium and Conv-Large, built only from UIB and fused-IB blocks, and Hybrid-Medium and Hybrid-Large, which add Mobile MQA to the last two stages. Every model ends with the MobileNet-v3 style head that moves the widest $1 \times 1$ layer after global pooling. They are mostly Pareto-optimal across mobile CPUs, DSPs, GPUs, the Apple Neural Engine and the Pixel EdgeTPU at once, ranging from 73.8% ImageNet-1k top-1 at 0.2 GMACs (Conv-Small) to 83.4% (Hybrid-Large).