Large language models (LLMs) excel at text, but struggle with numbers. Predicting probabilities, essential for tasks like decision-making and forecasting, reveals this weakness. Existing LLM architectures, using techniques like softmax over discretized bins, often fail to capture the continuous nature of probabilities, leading to noisy outputs and suboptimal performance. A novel approach, the "Fourier head," integrates Fourier series into the LLM architecture to address this. The Fourier head learns a continuous probability density function and then discretizes it. This process smooths out predictions, filtering noise and enhancing accuracy. Imagine trying to predict the next move in a game like Atari's Seaquest. A standard LLM might give jerky, random probabilities to different actions. The Fourier head, however, understands the continuous relationship between actions (like moving left versus slightly up-left) and provides smoother, more intuitive probabilities. Tests on Seaquest showed a remarkable 46% improvement in performance. Similar gains appeared in time series forecasting, where the Fourier head boosted the accuracy of a state-of-the-art model by 3.5%. The magic of the Fourier head lies in its ability to filter high-frequency noise while preserving crucial low-frequency signals. This aligns more closely with how humans understand probabilities, focusing on broad trends rather than tiny fluctuations. While promising, the Fourier head faces challenges. Balancing expressiveness (ability to model complex distributions) with smoothness is crucial. Too much smoothness, and the model becomes simplistic; too little, and it overfits to noise. Further research will explore optimizing this balance and integrating the Fourier head into general LLM training. This innovation brings us closer to LLMs that can reason probabilistically like humans, unlocking their potential for advanced decision-making and forecasting across various fields.