Large language models (LLMs) like ChatGPT are increasingly used for finding information, but they sometimes confidently generate false statements—a phenomenon known as "hallucination." This raises a crucial question: how can we determine the boundaries of an LLM's actual knowledge? New research explores this by testing LLMs with "semi-open-ended questions"—questions with many possible right answers, some common and some obscure. These questions are tricky because LLMs might know *some* answers but not others, blurring the line between what they know and don't know. The researchers used a clever method to probe these boundaries. They first had an LLM generate semi-open-ended questions and some initial answers. Then, they used a second LLM to suggest additional, less common answers, pushing beyond the first LLM's readily available knowledge. Finally, they checked the accuracy of these less common answers using both the LLM's self-assessment and by cross-referencing with reliable online sources. The results? Even advanced LLMs like GPT-4 struggle with these questions. They often give incorrect or unverifiable answers, especially when pushed beyond common knowledge. What's more, LLMs are often unaware of their own limitations, confidently asserting incorrect information. This research highlights the importance of developing better methods to detect the limits of AI knowledge. It also suggests that simply asking an LLM if it's sure about an answer isn't enough—we need more sophisticated ways to verify the information they provide. As LLMs become more integrated into our lives, understanding their knowledge boundaries is critical for building trust and ensuring that we're not misled by AI's confident hallucinations.