Real-Time Python Aggregations with WarpStream

Опубликовано: 10 Декабрь 2024
на канале: Quix

212

Discover how you can use Python to do real-time stateful aggregations on data coming from WarpStream—made possible with the Quix Streams Python library. I’ll walk you through the steps of a simple stream processing pipeline that uses WarpStream as the message broker.

You’ll learn how to generate a stream of synthetic user activity logs, stream the logs to Warpstream then count the number of actions per page. After that, you see how you can write the aggregations into a second topic, so that you continuously write the aggregations into DuckDB which enables you to create continuously updating page view reports.

Presenter: Merlin Carter — Quix Developer Educator

📥 This demo’s source code: https://github.com/quixio/template-wa...
👨‍💻 Quix Streams Source on Github: https://github.com/quixio/quix-streams
📓 Quix Streams Docs: https://quix.io/docs/quix-streams/qui...
⚙️ WarpStream Quick Start: https://docs.warpstream.com/warpstrea...
📺 Watch more detailed tutorials: • Quix Streams Tutorials

Timestamps:

00:00 — Intro
00:59 — Set up environment
02:14 — Producing synthetic user logs to WarpStream
03:23 — Counting user events by page using stateful aggregations
04:40 — Consuming from the aggregations topic and writing to DuckDB
05:58 — Running a sample query on DuckDB
06:25 — Outro