Sliced-Recursive-Transformer (SReT)

Sliced Recursive Transformer

Zhiqiang Shen, Zechun Liu, Eric Xing

This paper presents a neat yet effective recursive operation on transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across the depth of transformer networks. The proposed method can obtain a substantial gain (~2%) simply using a naive recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimal computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximation method through multiple sliced group self-attentions across recursive layers, which can reduce the cost consumption by 10~30% with minimal performance loss.

@misc{arxiv2111052972021,
  title = {Sliced Recursive Transformer},
  author = {Zhiqiang Shen and Zechun Liu and Eric Xing},
  year = {2021},
  eprint = {2111.05297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.05297}
}