FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users

Generally, preference fine-tuning algorithms aim to optimize a reward function that captures the preferences of a user. To do so, these approaches aggregate preferences from a user into a single preference dataset, which is then used to fine-tune a language model through RLHF or Preference Optimization. Can we instead learn a personalized reward function directly from a few preferences of a user?

To instantiate this idea, we propose Few-Shot Preference Optimization (FSPO), where with just the weak additional requirement of a scorer id to differentiate users, we can learn a personalized reward function for each user. In FSPO, we consider each user as a new task instance in meta-learning, where each user has a preference dataset comprising of prompts, preferred responses, and dispreferred responses. Then, leveraging the strong in-context learning capabilities of LLMs, we condition the model on N preference examples from the user and fine-tune the model to maximize the likelihood of the preferred responses and minimize the likelihood of the dispreferred response using preference optimization loss such as DPO or IPO. This allows us to learn a personalized reward function for each user.

When a user description is provided (even if synthetically generated), FSPO can be reframed as a two-step prediction process, as illustrated above. In the first stage, the system generates a user description based on the user’s few-shot preferences. In the subsequent stage, it produces a response conditioned on the original prompt, the few-shot preferences, and the generated user description. This intermediate step not only offers an interpretable summary of the few-shot preferences but also serves as a better representation for reward modeling and response generation.

In all, FSPO can be easily instantiated. First, a minibatch of training users is sampled. For each user, we sample a few-shot preference dataset and (potentially) a user description. We then fine-tune the model on the few-shot preferences and the user description using a preference optimization loss such as DPO or IPO (equations 5 + 6). This allows us to learn a personalized reward function for each user. If we want to do COT, we train with teacher-forcing, first predicting the loss of the user description and then conditioned on the ground truth user description and the few-shot preferences, predict the loss on the responses. We then update with gradient descent. We summarize FSPO in Algorithm 1.

From an information-theoretic perspective, the few-shot binary preferences can be seen as a N-bit representation of the user, representing up to 2^N different personas or reward functions. There are several ways to represent users: surveys, chat histories, or other forms of interaction that reveal hidden preferences. We restrict our study to such a N-bit user representation, as such a constrained representation can improve the performance when transferring reward models learned on synthetic personalities to real users. We defer the study of less constrained representations to future work.

To evaluate the performance of FSPO in generating personalized responses, we require benchmark tasks that capture a wide range of user preferences to evaluate the performance of FSPO and baselines in generating personalized responses. We propose a benchmark consisting of 3 domains, where personalization can be studied: (1) Reviews, studying the generation ability of models for reviews of movies, TV shows, and books that are consistent with a user’s writing style, (2) Explain Like I'm X (ELIX): studying the generation ability of models for responses that are consistent with a user’s education level, and (3) Roleplay: studying the generation ability of models for responses that are consistent with a user's description, with effective transferability to a real human-study. We open source the benchmark tasks with the synthetic data and the evaluation scripts for the community to use.

Collecting personalized data at scale presents significant challenges, primarily due to the high cost and inherent unreliability of human annotation. Curating a diverse set of users to capture the full spectrum of real-world variability further complicates the process, often limiting the scope and representativeness of the data. Synthetically generating data using a LLM is a promising alternative, since it can both reduce costly human data generation and annotation and streamline the data curation process. But how do we go about generating such a preference data using language models in a way that transfers to real people?

Following work on task construction from Meta-Learning, we propose two design decisions to effectively encourage transfer from synthetic to real users: (1) Encouraging Diversity and (2) Structured Task Construction. Encouraging diversity in the synthetic data allows us to capture a wide range of user preferences, while structured task construction allows us to capture coherent, self-consistent user preferences. For diversity, we sample a wide range of prompts from human participants and data sources, and augment these prompts by few-shot prompting with an LLM. For responses, we either used Persona-Steering or View Conditioning to encourage diversity. For structured task construction, we labeled preferences in a consistent manner, utilizing a modified version of the Alpaca Eval prompt, and reduced underspecification of the user description by iteratively refining the user description. We summarize the task construction process in the flowchart below.

Below, we evaluate FSPO against 4 baselines: (1) a base model generating responses without any personalization, (2) few-shot prompting the base model, (3) few-shot supervised fine-tuning (Pref-FT) based off GPO, and (4) prompting with the ground truth persona. We evaluate FSPO on the 3 benchmark tasks and find that FSPO outperforms all baselines in generating personalized responses. We also find that COT enables us to close the gap to the oracle method, where we prompt with the ground truth persona. For Reviews, we have 4 trained users (concise, verbose, positive, negative) and 4 interpolated users (concise + positive, verbose + positive, concise + negative, verbose + negative) to evaluate how FSPO generalizes to unseen users. Additionally, for ELIX, we have 2 tasks: ELIX-easy and ELIX-hard, where ELIX-easy is a simple task with 5 educational levels and ELIX-hard is a more challenging task with specializated educational levels such as a PhD in Physics. For Roleplay, we have 1500 total users and evaluate the performance of FSPO on held-out users and prompts for generating personalized responses. Additionally, we run a preliminary, controlled human study using Prolific, with a data collection app, that was built to collect human preferences from real human participants.

Table 1: Automatic Winrates on Roleplay (1500 users)

Table 2: GPT-4o Winrates on ELIX-easy and ELIX-hard

Table 3: Review Winrates - Trained and Interpolated Users

Table 4: Roleplay: Human Eval Winrates