Large language models (LLMs) are powering increasingly sophisticated chatbots, but how do we ensure they can handle complex, multi-turn conversations? A new research paper introduces MORTAR, a clever method for finding hidden bugs in these AI systems. Imagine shuffling a conversation's back-and-forth, dropping parts of it, or even repeating certain questions—that's essentially what MORTAR does. By creating these altered dialogues, researchers can test whether the chatbot still provides consistent and correct answers. This approach goes beyond simply checking if a chatbot gives the right response to a single question. It delves into the chatbot's ability to understand context and maintain coherence throughout a conversation. MORTAR uses a knowledge graph to track the information exchanged in the dialogue, acting as a memory of sorts. This helps in identifying when a chatbot stumbles in its reasoning due to missing information or inconsistencies introduced by the perturbations. Experiments show that MORTAR is surprisingly effective at uncovering flaws in various LLM-based chatbots, some of which go unnoticed by traditional testing methods. Interestingly, bigger isn't always better. While larger LLMs are generally more robust, MORTAR revealed that they can sometimes rely too heavily on memorized information rather than reasoning about the conversation's flow. This research is a step towards building more reliable and trustworthy AI conversational agents. As chatbots become integrated into more aspects of our lives, rigorous testing like this is crucial for ensuring a positive and error-free user experience. The challenges ahead involve developing even more sophisticated perturbations and refining the knowledge graph approach to capture the nuances of human conversation better. This work opens up exciting possibilities for not only improving chatbot testing but also for developing techniques to enhance LLM training and create synthetic dialogue data for more comprehensive AI development.