Категории

Hybrid Apache Spark Architecture with YARN and Kubernetes

Опубликовано: 01 Октябрь 2024
на канале: Databricks
1,794
22

Lyft is on the mission to improve people’s lives with the world’s best transportation. Starting 2019, Lyft has been running both Batch ETL and ML spark workloads primarily on Kubernetes with the Apache Spark on k8s operator. However, with the increasing scale of workloads in frequency and resource requirements, we started hitting numerous reliability issues related to IP allocation, container images, IAM role assignment, and Kubernetes Control Plane.

To continue supporting growing Spark usage with Lyft, the team came up with a hybrid architecture optimized for containerized and non-containerized workload based on Kubernetes and YARN. In this talk, we will also cover a dynamic runtime controller that helps with per environment config overrides and easy switchover between resource managers.

Connect with us:
Website: https://databricks.com
Facebook:   / databricksinc  
Twitter:   / databricks  
LinkedIn:   / databricks  
Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...

play_arrow
373
9

Text Drop from Signboard VFX Blender Tutorial |

Text Drop from Signboard VFX Blender Tutorial |

play_arrow
46,668
1.8 тыс

PERCHÉ FARE TEATRO | Lezione di recitazione #1

PERCHÉ FARE TEATRO | Lezione di recitazione #1
play_arrow
535
5

Asqar Ashurmatov Dadajon Аскар Ашурматов Дадажон

Asqar Ashurmatov Dadajon Аскар Ашурматов Дадажон
play_arrow
2,909
128

THE OBJECTIFIER - Blender Short Film

THE OBJECTIFIER - Blender Short Film
play_arrow
733
4

How To Change Font Color In WordPress Website | WordPress Change Text Color | Change WordPress Text

How To Change Font Color In WordPress Website | WordPress Change Text Color | Change WordPress Text
play_arrow
291
0

Белая милонга. DJ Оля Кощеева.

Белая милонга. DJ Оля Кощеева.
play_arrow
42,399
639

How to Send Your Passport for Canada Visa Stamping

How to Send Your Passport for Canada Visa Stamping
play_arrow
41,870
232

Дом-2: Ряска подстрекатель

Дом-2: Ряска подстрекатель
Похожие видео
play_arrow
Mosaic AI Gateway - Secure and Govern your AI

Mosaic AI Gateway - Secure and Govern your AI
play_arrow
Data + AI Summit 2023

Data + AI Summit 2023
play_arrow
LLM Module 6: LLMOps | 6.7 Notebook Demo

LLM Module 6: LLMOps | 6.7 Notebook Demo
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.13.1 Notebook Demo Part 1

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.13.1 Notebook Demo Part 1
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.10 Task specific Evaluations

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.10 Task specific Evaluations
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.6 Fine Tuning: LLMs as a Service

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.6 Fine Tuning: LLMs as a Service
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.3 Applying Foundation LLMs

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.3 Applying Foundation LLMs
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.4 Fine Tuning: Few-shot Learning

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.4 Fine Tuning: Few-shot Learning
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.2 Module Overview

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.2 Module Overview
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.1 Introduction

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.1 Introduction
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.7.1 Notebook Demo Part 1

LLM Module 3 - Multi-stage Reasoning | 3.7.1 Notebook Demo Part 1
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.3 Prompt Engineering

LLM Module 3 - Multi-stage Reasoning | 3.3 Prompt Engineering
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.1 Introduction

LLM Module 3 - Multi-stage Reasoning | 3.1 Introduction
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.2 Module Overview

LLM Module 3 - Multi-stage Reasoning | 3.2 Module Overview
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.2 Notebook Demo Part 2

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.2 Notebook Demo Part 2
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.1 Notebook Demo Part 1

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.1 Notebook Demo Part 1
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.6 Best Practices

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.6 Best Practices
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.5 Vector Stores

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.5 Vector Stores
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.2 Module Overview

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.2 Module Overview
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.4 Filtering

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.4 Filtering
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.3 How does Vector Search work

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.3 How does Vector Search work
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.1 Introduction

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.1 Introduction
play_arrow
LLM Module 1 - Applications with LLMs | 1.8 Summary

LLM Module 1 - Applications with LLMs | 1.8 Summary
play_arrow
LLM Module 1 - Applications with LLMs | 1.9 Notebook Demo

LLM Module 1 - Applications with LLMs | 1.9 Notebook Demo