Fusion: A Framework for Unified Sequential Token Adaptation in Vision Transformers
Token merging, early exiting and token pruning each cut a Vision Transformer's inference cost alone, but combined they destabilise the intermediate representations. Fusion stages them so they cooperate (merge first, check confidence, prune only what continues), with routing modules that adapt compression per input and expose the accuracy/latency trade-off at inference time, no retraining.
In the first half of the network, the most similar tokens are averaged together, a little at each of three steps.
At two checkpoints further in, an image stops as soon as the model is confident enough in its answer.
In the last layers, tokens that barely matter are dropped from the images that continue.
The order is the point: running pruning and early exit in parallel cost 1.62 pp of accuracy, and staging them recovers 1.58 pp. One trained model serves two settings, balanced and power-save, with no retraining.
The benchmark reproduces on one validation shard