Orca 2: Teaching Small Language Models How to Reason


Orca 1 learns from rich signals, such as explanation traces, allowing it to outperform conventional instruction-tuned models on benchmarks like BigBench Hard and AGIEval. In Orca 2, we continue exploring how improved training signals can enhance smaller LMs' reasoning abilities. Research on training small LMs has often relied on imitation learning to replicate the output of…

This story appeared on arxiv.org, 2023-11-21.
Read the Entire Internet on a Single Page. Join Now →