Large language models (LLMs) have surprised the AI community with their unexpected ability to perform symbolic reasoning, a task traditionally considered beyond the reach of neural networks. This ability is most apparent in "in-context learning" (ICL), where LLMs can generalize from a few examples provided in a prompt. This blog post explores research that delves into the mechanisms behind this symbolic thinking by creating a fully interpretable transformer network designed for a specific type of ICL: templatic text generation. Imagine teaching an AI to translate English passive voice into a logical form, simply by providing a few examples. That's the power of ICL. This research uses the "swap" task as a case study: given a prompt like "Q B C V D E A D E V B C Q F G V J K L A", the AI should generate "J K L V F G", effectively swapping the strings around the delimiter "V". This seemingly simple task requires the network to parse the input, identify the template, and then apply it to new data – a process mirroring core aspects of symbolic computation. To uncover how this is possible, the researchers introduce the "Transformer Production Framework" (TPF). TPF describes a system at three levels: functional (defining the task), algorithmic (specifying the procedure), and implementational (building the network). The algorithmic level uses a "Production System Language" (PSL), inspired by cognitive science models of human thought. PSL programs consist of "productions" – rules that trigger actions based on conditions. For example, a production might say, "If you see symbol 'B' followed by 'C' in the question, swap them in the answer." These productions are then translated into a form understandable by a transformer, using queries, keys, and values. The final level is the "Discrete-Attention-only Transformer" (DAT), a simplified transformer that uses discrete attention and state normalization. DAT uses “registers” in the hidden state to store the values of symbolic variables, a kind of disentangled residual stream. By compiling PSL programs into DAT weights, the researchers create a network whose every neuron and connection contribute transparently to the symbolic computation. Astonishingly, the researchers prove that PSL, and therefore this specialized transformer, is Turing complete, meaning it can theoretically perform any computation a regular computer can. This research offers a powerful framework for understanding how transformers can perform symbolic reasoning. It suggests that LLMs might be implicitly learning something similar to these productions, parsing prompts and applying learned templates. The framework also proposes testable hypotheses about how these symbolic structures might be encoded in the residual stream of trained transformers, opening new avenues for improving the interpretability and control of these powerful models. Future directions include extending the framework to handle more complex tasks involving embedded templates and recursive structures, blurring the lines further between neural networks and symbolic computation, and bridging the gap to continuous, distributed embeddings that are likely closer to the inner workings of real LLMs.