Embeddings models are crucial for LLMs as they convert text into numerical vectors, capturing semantic meanings. This enables LLMs to process, analyze, and generate human-like text, ensuring contextually relevant and coherent outputs.
The performance of embedding models can be evaluated using a Python framework called BEIR which is a collection of tasks(+benchmarks) and datasets for embedding models by calculating standard metrics like NDCG, Recall, Precision, etc. Similarly, a good way to boost embedding quality is confusingly also called BIER by dividing the last layer of the neural net into an ensemble and implementing online gradient boosting. (PS: This approach was originally proposed for image data only but we believe it will work with dense embeddings)
Source: [<https://openai.com/blog/new-and-improved-embedding-model>](<https://openai.com/blog/new-and-improved-embedding-model>)
<aside> 💡 Verticalized embedding models which are trained exclusively on industry jargon and shorthand knowledge are basically field experts while horizontal models like OpenAI's text-embedding-ada-002 and sentence-transformers are great generalists.
</aside>
The main issue that arises when using public information to train the embedding model is that you need to identify what is actually sponsored content as companies often feature products favorably when the owner pays for ads. Similarly, if a corporation owns a media house, it will likely negatively advertise about its competitors, e.g.: the Washington Post’s coverage of Walmart could be skewed. However, most of these could be normalized depending on dimensionality and TF-IDF attribution.
An extension of this more specific to finance extends to the fact that the information being used to train the data could include those written by people with adverse economic interests to the AMC or the end user itself. Some of these would also be due to differences in investment style and horizons alone and not necessarily malevolent. At that point, it makes sense to either normalize across different investor profiles or append these profile descriptions explicitly to the context window.
One final finance-specific issue that may require at least a partial retraining of the embedding model is new rules, compliance requirements, or guidelines by the regulator or the exchange. At this stage, you need to figure out what instruments are most affected and exclude related past content from retraining if the regulation isn't backward compatible. For eg: if there is a change that affects foreign institutions (FII), you need to exclude previous FII-related content or news data.
While there is a lot of work going into improving speech-to-text in Hindi, specifically from a finance point there are a lot of anchor words that are still in English. So we feel it is a much better use of time and money to just train better embeddings from the wrong transcript. This is because to the LLM only the vector representation matters and not human readability.
Our crawler has a domain discovery tool that is semantics-driven and helps the crawler discover new domains. We want to see if it is possible to scrape verified reviews of a particular domain to either exclude or auto-label bias ratings