How can I tell which LLM's have been trained on what?

Question

I'm really no expert and attempting to come up to speed. But...It strikes me that LLM's are a subtle combination of knowing the language and knowing some facts. If one were making a chatbot for helping (say) people understand their tax returns then you'd want a 'base' LLM that understood language and then add domain specific knowledge via RAG. So "yes" to knowing that Thursday comes after Wednesday; "no" to knowing how high Everest is; and questions regarding income tax have to reference the RAG. Is this correct?So are there stock "thin" models for doing this? Is it better to train your own?

uberman · Accepted Answer

I'm not exactly sure what you are asking. You can however construct a RAG based on a training corpus and then if you ask questions that when vecroized have no sufficiently similar match in your corpus you tell your llm to just say it cant answer that question.Concretely, index the tax documents into your data base then ask a question about Mount Everest. Vectorize that question and search your vectoized corpus for something similar. Find nothing then instruct your llm to formulate a response saying it could not answer

swrrt · Answer

Don't think RAG is the most common method for now. In my knowledge now it is mostly about post training and reenforcement learning for adding specific knowledge to base model. RAG is mostly like an add-on to a fully trained model and not so stable or inherent?