Large language models (LLMs) are revolutionizing how we interact with technology, but how reliable are they really? Their outputs can vary due to the random sampling process inherent in their design, making it hard to tell if a change in the output is a meaningful response to a prompt change or just random fluctuation. Researchers are tackling this challenge with a new framework called Distribution-Based Perturbation Analysis (DBPA). Instead of looking at single LLM outputs, DBPA analyzes distributions of outputs within a semantic similarity space. This allows researchers to distinguish true changes from random noise. Imagine asking an LLM a medical question. Subtle changes in wording might lead to different answers. DBPA can determine if these differences are statistically significant, revealing the LLM's sensitivity to specific phrasing. This method is model-agnostic, meaning it works on any LLM without needing to know its internal workings. It can also assess various types of perturbations, from rephrasing questions to evaluating the impact of different training iterations. Early experiments show promising results. DBPA reveals that advanced models like GPT-4 are more robust to irrelevant prompt changes than smaller models. Furthermore, DBPA can even measure how closely different LLMs align with each other by comparing their output distributions. This research has significant real-world implications. As LLMs are increasingly used in critical applications like healthcare and legal document drafting, understanding their reliability is paramount. DBPA offers a powerful tool for evaluating and improving the stability of these models, paving the way for more trustworthy and consistent AI systems. However, choosing the right similarity metrics and translating these findings into practical strategies for improvement remain open challenges. This research highlights the ongoing evolution of evaluating and refining LLM behavior, pushing us closer to truly dependable AI.