Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., … & Cobbe, K. (2023). Let’s verify step by step. arXiv preprint arXiv:2305.20050.

This paper, titled “Let’s Verify Step by Step,” critiques the reliability of large language models (LLMs) in performing complex multi-step reasoning tasks. The authors explore two methods of supervision—outcome supervision, which provides feedback only on the final result, and process supervision, which provides feedback on each intermediate step. The study finds that process supervision significantly improves the reliability of models, particularly in solving problems from the MATH dataset. The authors also demonstrate that active learning enhances the effectiveness of process supervision, making data collection more efficient. They release a large dataset, PRM800K, to support further research in this area.

https://arxiv.org/abs/2305.20050