Hugging Face Tokenizers (11.3)

Опубликовано: 22 Ноябрь 2023
на канале: Jeff Heaton

993

This video discusses how to use HuggingFace tokenizer models. These models break text down into tokens, which might be a single word, or make up multiple words. Tokenizers are the first step in natural language processing.

Code for This Video:
https://github.com/jeffheaton/app_dee...

~~~~~~~~~~~~~~~ COURSE MATERIAL ~~~~~~~~~~~~~~~
📖 Textbook - Coming soon
😸🐙 GitHub - https://github.com/jeffheaton/app_dee...
▶️ Play List -    • 2024 PyTorch Version Applications of ...
🏫 WUSTL Course Site - https://sites.wustl.edu/jeffheaton/t8...

~~~~~~~~~~~~~~~ CONNECT ~~~~~~~~~~~~~~~
🖥️ Website: https://www.heatonresearch.com/
🐦 Twitter -   / jeffheaton
😸🐙 GitHub - https://github.com/jeffheaton
📸 Instagram -   / jeffheatondotcom
🦾 Discord:   / discord
▶️ Subscribe: https://www.youtube.com/c/heatonresea...

~~~~~~~~~~~~~~ SUPPORT ME 🙏~~~~~~~~~~~~~~
🅿 Patreon -   / jeffheaton
🙏 Other Ways to Support (some free) - https://www.heatonresearch.com/suppor...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#PyTorch #nlp #huggingface