Keeping transformer models up to date gets expensive quickly due to the exponential growth of compute required per unit increase in performance. Hence, retrieval-based augmentation offers a solution by querying live data and just adding it to the context of the LLM at runtime to ensure that the LLM is aware of real-time information.

Beginner approaches around retrieval involve a combination of querying a vector database and performing full-text searches on databases like Elasticsearch. Vector search involves finding similar vectors to retrieve relevant information, while full-text search matches keywords to retrieve relevant documents. So a good embedding model is necessary to ensure that synonyms are close in vector space and allow the full-text search to provide maximum coverage of relevant context.

<aside> 💡 Semi-structured and unstructured data in finance include text, emails, and financial reports. Vector databases may struggle with this data due to their numerical nature, making it hard to capture and query the rich, text-based information inherent in financial data.

</aside>

Vector databases pose challenges, particularly regarding reindexing costs. Since vectorDBs are a linear data storage mechanism, updating the database when new information gets added is very expensive because we need to reindex the entire vector column. This process of recalculating vector embeddings gets harder as the dataset grows, especially in finance where data is dynamic in nature.

The process also has a fundamental flaw: assuming cosine similarity between a question’s vectors and your datastore’s embeddings will yield relevant text. There might be relations that don't occur together frequently in the embedding model’s training text data but represent either very strong relationships or frequently changing relationships like a public company’s execs or investors.

<aside> 💡 The core approach that stands out here is integrating a knowledge graph with a supervised model for graph building and traversal along with a vector store to cache the most traveled paths in the graph along with their semantic alternatives.

</aside>

In addition, as detailed in this paper because we are mostly working with non-labeled data, we can use the LLM’s linguistic capabilities with a 0-shot answer to broaden the context of semantic search and improve handling of any latent semantic similarities introduced by the question.

Now moving on to actually building the knowledge graph (KGs), we want to rule out handcrafting because even though the graphs are very high quality they aren't scalable because even a monthly fund factsheet is 105 pages long and contains charts, tables, and so on.

         Fig 1: Cool graph and high-quality but making 8 nodes from 1 page takes $1 in labor and 30 mins

Maybe you are thinking but have you tried using LLaMaIndex or some other ETL tool to get an LLM to do this for you? To answer your question, yes anon we have tried to provide PDFs/Excels to GPT4 (the 32k one, RIP my bank balance) to get it to generate a knowledge graph, and basically, while it's fast, the graphs are super fragmented (aka useless) and probably due to its primacy-recency bias.