While LLMs have achieved impressive results, they lack domain expertise and understanding of user expectations which may lead them to miss out on the nuances in how humans speak or misinterpret cultural context and be deeply offensive. Aligning allows us to leverage human expertise to teach the models and result in more accurate and reliable text generation.
LLMs, displaying emergent properties akin to agents in reinforcement learning, can have their outputs shaped through human feedback, thus fostering a dynamic learning environment. By integrating human-annotated feedback, models can be fine-tuned for superior, tailored outputs in areas like customer support, financial research, or customer relationship management, including a better understanding of industry shorthand or slang. (e.g.: ICICI price vs. give today’s price of ICICI).
<aside> 💡 In finance, due to regulations, financial data can't be shared with third-party annotators. Also, annotators may not share the user's financial perspective, and seeking explicit user feedback will reflect poorly on the product's readiness and the platform’s trustworthiness.
</aside>
There are many reasons to perform alignment. For example, OpenAI’s goal with human feedback alignment is for safety and fairness. Similarly, in finance, you need to focus on aligning the model for 2 different factors: compliance and usefulness. When done correctly, it allows the model to understand the user’s financial preferences with a very limited number of tokens in inference. While compliance can be handled post-answer generation, model awareness of regulation certainly helps.
Fig 1: Our RLHF approach using a rewards model to predict the most likely user feedback
In the proposal, a rewards model is added to the RLHF process, trained using 1 million Mixpanel events from our bot's usage, with training samples comprising a question, answer, user’s financial profile, and relevant chat history. This model, simulating 80% proximate user feedback, enables the LLM to refine responses to new tasks without human intervention and fostering adaptive behavior.
However, one challenge we have is the curation of the above-mentioned dataset of user feedback that needs to be provided to the LLM. Balancing the need for accuracy, safety, and fairness while ensuring a representative dataset is crucial for training the LLM effectively. Our current selection process is based on chat history size and the number of distinct interactions and is as follows:
$$ P(SelectedChat) = \begin{cases} 1, &\quad num\_chats \geq 5 \ \ \& \ distinct\_sessions \geq 2 \\ 0, &\quad otherwise \end{cases} $$
The function we have declared above leaves significant room for improvement. This is where the concepts of self-alignment and self-curated alignment become relevant. While the former is not relevant at all, the latter will add significant value to our alignment process if achievable.