Going beyond RAG: Extended Mind Transformers - Phoebe Klett
Retrieval Augmented Generation is such a hack. Why would an embedding of your prompt coincide with the documents needed to answer it? Meanwhile Transformers already have a key/query mechanism built in! In this talk, we'll introduce Extended Mind Transformers, a new flavor of transformer that allows the model to select and attend to the most relevant information at each generation step. We demonstrate EMT's state-of-the-art performance and discuss important design decisions for long context applications.
Github: https://github.com/normal-computing/e...
HF: https://huggingface.co/collections/no...
Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/20... & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025
About Phoebe
ML Engineer with background in abstract mathematics. Building language models that natively reason.
