Ray Data Streaming for Large-Scale ML Training and Inference
Some of the most demanding ML use cases involve pipelines that span both CPU and GPU devices in distributed environments. Most frequently, this situation occurs in batch inference, which involves a CPU-intensive preprocessing stage (e.g., video decoding or image resizing) before utilizing a GPU-intensive model to make predictions. It also occurs in distributed training, where similar CPU-heavy transformations are required to prepare or augment the dataset prior to GPU training. In this talk, we examine how Ray data streaming works and how to use it for your own machine learning pipelines to address these common workloads utilizing all your compute resources–CPUs and GPUs–at scale.
Takeaways
• Ray Data streaming is the new execution strategy for Ray Data in Ray 2.6
• Ray Data streaming scales data preprocessing for training and batch inference to heterogeneous CPU/GPU clusters
Find the slide deck here: https://drive.google.com/file/d/1kM0k...
About Anyscale
---
Anyscale is the AI Application Platform for developing, running, and scaling AI.
https://www.anyscale.com/
If you're interested in a managed Ray service, check out:
https://www.anyscale.com/signup/
About Ray
---
Ray is the most popular open source framework for scaling and productionizing AI workloads. From Generative AI and LLMs to computer vision, Ray powers the world’s most ambitious AI workloads.
https://docs.ray.io/en/latest/
#llm #machinelearning #ray #deeplearning #distributedsystems #python #genai
