Context is king for LLMs. Too much of it, however, hurts performance significantly. Hence, appropriate processing of input data and optimizing token count become necessary steps to improve inference accuracy and cost.
Large language models are powerful tools that operate on natural language queries to perform various tasks that range from technical to creative. Due to its flexibility, it has become a valuable component for individuals and organizations, finding use in various applications, including information retrieval and document summarization. They accomplish this by organizing all relevant task details, instructions, and data to be processed into an input context. LLMs perform better when given specific and detailed instructions on performing the requested task.
However, when working on tasks that require large amounts of data as input (like retrieval of legal and financial documents,) the increasing token size of the input context quickly becomes a detriment to the model’s performance. First explored in this paper in 2023, it was discovered that the input context length and the nature of supplied data can significantly affect the accuracy of an LLM’s generated output.
<aside> 💡 This is especially important in finance as hallucination can lead to potentially far-reaching and unsavory consequences for your product from regulators.
</aside>
On testing the ability of an LLM to recall information from different positions of the input context, we observe that models are better at identifying relevant information that appears either at the very beginning or the end of its input context. In certain situations, the models that are provided reference data can perform worse than if those models were not provided any context at all (at which point the model would try to generate answers from its parametric memory.)
Fig 1: Plotting the accuracy of language models with a change in the location of relevant information that answers an input question reveals a U-shaped performance trend.
Incidentally, this trend mirrors a well-documented psychological phenomenon known as the serial position effect. This cognitive bias is the tendency of people to recall the first and last items in a series much better than the ones in the middle. Theoretically, transformer LLMs should be able to recall tokens from anywhere with equal accuracy using self-attention, but empirical evidence suggests otherwise.
Experimentation has also revealed that as input contexts get longer, the models get worse at finding the correct answer from it. This effect is not solely due to the model’s inability to find relevant context. Experimentation in a multi-document QA setting found that even if the other distractor documents are random, the same U-shaped trend persists (even though a simple lexical-overlap heuristic on the query can often identify the answer in such cases.)