Large Language Models (LLMs) have shown remarkable abilities in solving complex tasks, including mathematical reasoning. But their massive size makes them impractical for many applications. Smaller, more efficient models are needed, but they often struggle with the multi-step reasoning that LLMs excel at. New research introduces a clever technique called "SIKeD" (Self-guided Iterative Knowledge Distillation) to address this challenge. Imagine an LLM as a seasoned math tutor and a smaller model as its student. Traditionally, the LLM would show the student various problem-solving strategies, and the student would try to mimic them. However, smaller models tend to get stuck on one strategy, even when it's not the most effective approach. SIKeD changes this dynamic by letting the student actively participate in the learning process. The LLM still provides initial guidance, but the student also tries to solve problems independently, choosing its own strategies. The key is that the student focuses on its *successful* attempts. These self-generated solutions are then combined with the LLM's examples, creating a more tailored and effective training dataset. This iterative process repeats, with the student continually refining its understanding and strategy selection. Results show that SIKeD significantly boosts the math performance of smaller models across various datasets, even closing the gap with much larger LLMs. This iterative, self-guided learning approach represents a significant step towards developing more efficient and capable AI models for a wider range of applications. It also raises intriguing questions about how similar learning paradigms might enhance other AI skills beyond mathematical reasoning.