He, X. O. (2024). Mixture of A Million Experts. arXiv preprint arXiv:2407.04153.
This paper introduces PEER (parameter efficient expert retrieval), a novel layer design utilizing the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). PEER aims to improve the performance-compute trade-off in transformer models, addressing limitations of existing sparse mixture-of-experts (MoE) architectures. Experiments on language modeling tasks show that PEER layers outperform dense feedforward layers and coarse-grained MoEs, enabling efficient scaling of transformer models while maintaining computational efficiency.