Large language models (LLMs) are impressive, but they're not without their quirks. One of the most concerning is their tendency to 'hallucinate'—generate incorrect or nonsensical information. Researchers are constantly exploring new architectural approaches to building LLMs, like recurrent models, hoping to address limitations of the dominant transformer-based models. But do these architectural changes actually influence the problem of hallucinations? A fascinating new study delves into this question, specifically examining how architectural biases affect an LLM's propensity for generating false information. The research suggests that while no single architecture completely avoids hallucinations, the *types* of hallucinations and the ease with which they occur differ significantly based on the model's design. For instance, in tests involving recalling obscure facts, recurrent models struggled compared to their transformer counterparts. However, when tricked with misleading prompts designed to test memorization, recurrent models proved more resilient to the deception, suggesting they focus more on the immediate context than simply retrieving memorized facts. Interestingly, the size of the model and the application of instruction tuning also played a role. While larger models generally performed better with factual recall, this wasn't always the case for faithfulness to the given context. Recurrent models, in particular, seemed to plateau in their faithfulness regardless of size increases, unlike transformer models which showed continued improvement. Similarly, instruction tuning, a common technique to improve LLM performance, had inconsistent results. It significantly improved the faithfulness of transformer models but had negligible effects on models with recurrent layers, indicating fundamental differences in how these models learn. This research highlights the complex interplay between LLM architecture and hallucination. There’s no silver bullet, but understanding these architectural nuances is crucial for designing more reliable and trustworthy AI systems in the future. Different architectures have different strengths, and future research will need to develop tailored mitigation techniques to address the specific types of hallucinations each architecture is prone to.