Activation-Free Backbones for Image Recognition

Abstract

Activations are not a representational necessity.

We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules drop into existing architectures: instantiated within MetaFormer, our PolyNeXt models match or exceed activation-based counterparts across scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness, while substantially outperforming prior polynomial networks at reduced cost.

01 · Key Idea

Multiplication in place of activation

The Hadamard product of two learned linear projections, $({\mathbf{W}}_a\mathbf{x})\ast ({\mathbf{W}}_b\mathbf{x})$, is already a second-degree polynomial in the input, with no ReLU or GELU required. Compose these across layers and the polynomial degree grows as $2^L$ while parameters grow only linearly. Drag the weights below to see two lines become a curve.

02 · The Modules

Three polynomial primitives

Each module swaps a nonlinearity for a Hadamard product while preserving the input-output interface, so it stays compatible with orthogonal improvements like windowed or sparse attention.

Set beside their standard counterparts, each polynomial module replaces the activation or softmax (shown in red) with a Hadamard-product construction. This also entails supporting changes, such as parallel convolutional branches with different receptive fields and depthwise convolutions for local context.

03 · Making It Trainable

The barrier was stability, not capacity

Hadamard products amplify large values, and that compounds across depth. Three lightweight mechanisms tame it and let polynomial networks train just under 200 layers, far deeper than prior shallow-and-wide polynomial models.

04 · Architecture

PolyNeXt, end to end

A four-stage hierarchical backbone. CPolyNeXt uses PolyConv throughout; APolyNeXt switches to PolyAttn in the last two (low-resolution) stages, where global context is cheap and beneficial.

05 · Results

Matching activation-based models across scales

PolyNeXt matches or exceeds activation-based ConvFormer / CAFormer across every scale, and outperforms prior polynomial networks (MONet, DTTN) by 2–3 points at lower cost. ◆ marks activation-free models.

Why do activations hurt? The block $({\mathbf{W}}_a\mathbf{x})\ast ({\mathbf{W}}_b\mathbf{x})$ has a mutual gradient coupling, where each branch learns through its sibling's output. Inserting a GELU breaks that coupling, which is why every way of adding an activation back lowers accuracy, and replacing the product with addition collapses it (−22.3).

Citation

BibTeX

If you find this work useful, please cite our paper.

@inproceedings{wang2026polynext,
  title     = {Activation-Free Backbones for Image Recognition: Polynomial
               Alternatives within MetaFormer-Style Vision Models},
  author    = {Wang, Jeffrey and Gregory, Jonathan and Chrysos, Grigorios G.},
  booktitle = {Proceedings of the 43rd International Conference on
               Machine Learning (ICML)},
  year      = {2026}
}

The BibTeX entry will be updated with the official PMLR proceedings reference once published.