Категории

Tangram Distributed Scheduling Framework for Apache Spark at Facebook

Опубликовано: 06 Октябрь 2024
на канале: Databricks
3,550
35

Tangram is a state-of-art resource allocator and distributed scheduling framework for Spark at Facebook with hierarchical queues and a resource based container abstraction. We support scheduling and resource management for a significant portion of Facebook's data warehouse and machine learning workloads that equates to running millions of jobs across several clusters with tens of thousands of machines. In this talk, we will describe Tangram's architecture, discuss Facebook's need for a custom scheduler, and explain how Tangram schedules Spark workloads at scale. We will specifically focus on several important features around improving Spark's efficiency, usability and reliability: 1. IO-rebalancer (Tetris) Support 2. User-Fairness Queueing 3. Heuristic-Based Backfill Scheduling Optimizations.

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...

Connect with us:
Website: https://databricks.com
Facebook:   / databricksinc  
Twitter:   / databricks  
LinkedIn:   / databricks  
Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...

play_arrow
1,013
68

00:00:00

Первое занятие онлайн-курса

Первое занятие онлайн-курса "Лечение заболеваний методом Бутейко"
play_arrow
2,070,722
4.7 тыс

Bài Ca Kỷ Niệm - Karaoke - Song Ca - Thiếu Giọng Nữ - Nhạc Sống - gia huy beat

Bài Ca Kỷ Niệm - Karaoke - Song Ca - Thiếu Giọng Nữ - Nhạc Sống - gia huy beat
play_arrow
774,235
29 тыс

The Cure for Literally Everything | Vegetarianism

The Cure for Literally Everything | Vegetarianism
play_arrow
24
0

История Железного Майка. Поездка за картой где указан динамит. Days Gone №19

История Железного Майка. Поездка за картой где указан динамит. Days Gone №19
play_arrow
525
7

Camel dance 😂____

Camel dance 😂____
play_arrow
88,390
6.5 тыс

Dragones: Una Leyenda Hecha Realidad | Reseña y Análisis

Dragones: Una Leyenda Hecha Realidad | Reseña y Análisis
play_arrow
326,933
7.6 тыс

PLANET DINOSAUR - Serie Completa | Resumen (Recopilación)

PLANET DINOSAUR - Serie Completa | Resumen (Recopilación)
play_arrow
19
1

Интерактивный пол • Игровая детская мебель в Красноярске

Интерактивный пол • Игровая детская мебель в Красноярске
Похожие видео
play_arrow
Mosaic AI Gateway - Secure and Govern your AI

Mosaic AI Gateway - Secure and Govern your AI
play_arrow
Data + AI Summit 2023

Data + AI Summit 2023
play_arrow
LLM Module 6: LLMOps | 6.7 Notebook Demo

LLM Module 6: LLMOps | 6.7 Notebook Demo
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.13.1 Notebook Demo Part 1

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.13.1 Notebook Demo Part 1
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.10 Task specific Evaluations

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.10 Task specific Evaluations
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.6 Fine Tuning: LLMs as a Service

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.6 Fine Tuning: LLMs as a Service
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.3 Applying Foundation LLMs

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.3 Applying Foundation LLMs
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.4 Fine Tuning: Few-shot Learning

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.4 Fine Tuning: Few-shot Learning
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.2 Module Overview

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.2 Module Overview
play_arrow
LLM Module 4: Fine-tuning and Evaluating LLMs | 4.1 Introduction

LLM Module 4: Fine-tuning and Evaluating LLMs | 4.1 Introduction
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.7.1 Notebook Demo Part 1

LLM Module 3 - Multi-stage Reasoning | 3.7.1 Notebook Demo Part 1
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.3 Prompt Engineering

LLM Module 3 - Multi-stage Reasoning | 3.3 Prompt Engineering
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.1 Introduction

LLM Module 3 - Multi-stage Reasoning | 3.1 Introduction
play_arrow
LLM Module 3 - Multi-stage Reasoning | 3.2 Module Overview

LLM Module 3 - Multi-stage Reasoning | 3.2 Module Overview
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.2 Notebook Demo Part 2

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.2 Notebook Demo Part 2
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.1 Notebook Demo Part 1

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.8.1 Notebook Demo Part 1
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.6 Best Practices

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.6 Best Practices
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.5 Vector Stores

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.5 Vector Stores
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.2 Module Overview

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.2 Module Overview
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.4 Filtering

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.4 Filtering
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.3 How does Vector Search work

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.3 How does Vector Search work
play_arrow
LLM Module 2 - Embeddings, Vector Databases, and Search | 2.1 Introduction

LLM Module 2 - Embeddings, Vector Databases, and Search | 2.1 Introduction
play_arrow
LLM Module 1 - Applications with LLMs | 1.8 Summary

LLM Module 1 - Applications with LLMs | 1.8 Summary
play_arrow
LLM Module 1 - Applications with LLMs | 1.9 Notebook Demo

LLM Module 1 - Applications with LLMs | 1.9 Notebook Demo