Speculative decoding can help AI chatbots improve throughput and reduce hardware demand by using a smaller model to draft tokens that a larger model validates.
In a paper published in Proceedings of the National Academy of Sciences, researchers from Technion and Tel Aviv University ...