Transformers have become the primary backbone of the computer vision community due to their impressive perfor- mance. However, the unfriendly computation cost impedes their potential in the video recognition domain. To opti- mize the speed-accuracy trade-off, we propose Semantic- aware Temporal Accumulation score (STA) to prune spatio- temporal tokens integrally. STA score considers two crit- ical factors: temporal redundancy and semantic impor- tance. The former depicts a specific region based on whether it is a new occurrence or a seen entity by ag- gregating token-to-token similarity in consecutive frames while the latter evaluates each token based on its contri- bution to the overall prediction. As a result, tokens with higher scores of STA carry more temporal redundancy as well as lower semantics thus being pruned. Based on the STA score, we are able to progressively prune the tokens without introducing any additional parameters or requir- ing further re-training. We directly apply the STA module to off-the-shelf ViT and VideoSwin backbones, and the em- pirical results on Kinetics-400 and Something-Something V2 achieve over 30% computation reduction with a neg- ligible ∼ 0.2% accuracy drop. The code is released at here.

@inproceedings{ding2023prune,
  title={Prune spatio-temporal tokens by semantic-aware temporal accumulation},
  author={Ding, Shuangrui and Zhao, Peisen and Zhang, Xiaopeng and Qian, Rui and Xiong, Hongkai and Tian, Qi},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={16945--16956},
  year={2023}
}