LLM Foundation Models (3.1)
In this video, we explore foundation models for large language models (LLMs), which serve as the base models pre-trained on extensive data, providing a foundation for specialized tasks. We delve into examples from major tech companies like OpenAI's GPT, Google's BERT, and Facebook's RoBERTa. The video also covers the significant resources required to train such models from scratch, highlighting why most academic courses focus on fine-tuning existing models rather than creating new ones.
Additionally, we discuss how to evaluate foundation models, including considerations like open vs. closed weights, the number of parameter weights, and the importance of context window size. The video explains the pros and cons of models with more weights, the environmental impact of larger models, and the crucial role of understanding tokens and context windows in managing computational costs. We also introduce specific OpenAI models used in the course, providing guidelines on their usage and associated costs.
Code for This Video:
https://github.com/jeffheaton/app_gen...
~~~~~~~~~~~~~~~ COURSE MATERIAL ~~~~~~~~~~~~~~~
📖 Textbook - Coming soon
😸🐙 GitHub - https://github.com/jeffheaton/app_gen...
▶️ Play List - • Course Overview: Applications of Gene...
~~~~~~~~~~~~~~~ CONNECT ~~~~~~~~~~~~~~~
🖥️ Website: https://www.heatonresearch.com/
🐦 Twitter - / jeffheaton
😸🐙 GitHub - https://github.com/jeffheaton
📸 Instagram - / jeffheatondotcom
🦾 Discord: / discord
▶️ Subscribe: https://www.youtube.com/c/heatonresea...
~~~~~~~~~~~~~~ SUPPORT ME 🙏~~~~~~~~~~~~~~
🅿 Patreon - / jeffheaton
🙏 Other Ways to Support (some free) - https://www.heatonresearch.com/suppor...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#FoundationModels #LargeLanguageModels #AI #MachineLearning #LLMs #OpenAI #GoogleBERT #MetaAI #AIResearch #DataScience #TechEducation #DeepLearning #ArtificialIntelligence #TechTutorial #Programming #SoftwareDevelopment
