Reinforcement learning (RL), a powerful technique for training AI agents, often struggles in complex environments with sparse rewards. Imagine an agent trying to navigate a maze where it only receives a reward upon reaching the exit. With limited feedback, the agent can take a long time to stumble upon the correct path. This is where reward shaping comes in, providing additional incentives to guide the agent's learning process. However, designing effective reward functions is a challenge in itself. This research paper explores a novel approach: using large language models (LLMs) to generate heuristics for reward shaping. LLMs, known for their language processing capabilities, can be surprisingly effective at providing high-level guidance. The researchers investigate two types of abstractions for representing the RL problem to the LLM: deterministic and hierarchical. In the deterministic approach, the LLM receives a simplified, deterministic version of the environment. In the hierarchical approach, the LLM works with a higher-level representation of the task, focusing on subgoals. The results are promising, showing significant improvements in sample efficiency across various environments, including maze navigation, household tasks, and even Minecraft. The LLMs, acting as heuristic generators, help the RL agents learn much faster by providing valuable insights. This research opens up exciting possibilities for using LLMs to enhance RL in complex, real-world scenarios. While challenges remain, such as the need for effective verifiers to ensure the LLM's guidance is valid, this approach offers a new perspective on how we can leverage the power of language models to improve AI learning.