Large language models (LLMs) have revolutionized various fields, but their general-purpose nature often requires customization for specific real-world applications. However, customizing LLMs through fine-tuning can lead to catastrophic forgetting, where the model loses its original capabilities. In this blog post, we explore Q-Adapter, a novel method that addresses this challenge by enabling customization while mitigating forgetting. The core idea behind Q-Adapter is to reframe LLM customization as a reward maximization problem. Instead of simply fine-tuning the LLM on new data, Q-Adapter aims to optimize the sum of two reward functions: one representing the original capabilities of the LLM (r1) and another characterizing the new preferences (r2). The challenge lies in the fact that both reward functions are typically unknown. Q-Adapter leverages the residual Q-learning framework to overcome this obstacle. It learns a residual Q-function that captures the difference between the optimal Q-functions for the two reward functions. This allows Q-Adapter to customize the LLM without explicitly knowing r1. Furthermore, Q-Adapter directly learns the residual Q-function from human preference data using the Bradley-Terry model, bypassing the need to estimate the reward function r2. In essence, Q-Adapter introduces a small adapter module that approximates the residual Q-function. During inference, the adapter's output is combined with the pre-trained LLM's output to produce responses that satisfy both the new preferences and the original capabilities. This approach ensures that the LLM retains its general knowledge and skills while adapting to new tasks and domains. Experiments on the Llama 3.1 model demonstrate Q-Adapter's effectiveness in retaining existing knowledge while learning new preferences. Results on various benchmarks, including MMLU, GSM8k, and IFEval, show that Q-Adapter outperforms other customization methods in terms of both knowledge retention and preference alignment. Q-Adapter offers a promising solution for customizing LLMs without the risk of catastrophic forgetting. This approach opens up new possibilities for tailoring LLMs to diverse applications while preserving their valuable pre-trained knowledge. Future research could extend Q-Adapter to other LLM architectures and explore continuous customization scenarios with multiple adapter modules.