Large language models (LLMs) are incredibly powerful, but sometimes they exhibit undesirable behaviors like generating false information or being stubbornly uncooperative. Researchers are actively developing techniques to “steer” these models towards preferred behaviors, like truthfulness and helpfulness. But how effective are these methods, really? A new research paper argues that current evaluation techniques aren't up to scratch, leading to inflated claims about the success of behavior steering. The researchers identify four key properties missing from many evaluations: testing in realistic, open-ended generation contexts, accounting for the model’s confidence in its responses, enabling comparisons across different target behaviors, and comparing against a proper baseline. They propose a more rigorous evaluation pipeline and put popular steering methods to the test. The results? Some interventions are less effective than previously thought, highlighting the importance of better evaluation practices. This research sheds light on the complexities of controlling LLM behavior and emphasizes the need for robust, standardized evaluation methods. As LLMs become increasingly integrated into our lives, ensuring they behave reliably is paramount, and this study provides a crucial step toward that goal. It also opens the door for more refined steering methods that can truly align LLMs with human values and intentions, paving the way for more responsible and beneficial AI.