Session Overview
Discover how to reliably schedule heavy, hours-long batch workloads on Kubernetes using Apache Kafka without blocking queues or crashing your infrastructure.
In this talk, Alfred from Steadforce explains how his team solved massive resource bottlenecks and 503 outages caused by monolithic MATLAB calculation engines. Learn how decoupling workloads with containerized Quarkus runners, implementing a commit-on-spawn Kafka pattern, and handling Kubernetes SIGTERM signals cleanly achieved effectively-once processing and zero downtime.
What You'll Learn
- The monolith bottleneck: Why running 6-hour, 100 GB memory calculations sequentially choked smaller jobs and caused full cluster crashes
- Commit-on-spawn pattern: Pulling single messages and committing immediately so multi-hour jobs never block Kafka consumer poll intervals
- Worker-driven requeueing: Letting runner pods evaluate their own compatibility and requeue tasks with TTL countdowns and dead-letter queues
- Graceful Kubernetes shutdowns: Handling SIGTERM within a 3-minute maintenance window and preventing duplicate runs with active reconciliation
Chapters
- 00:00 — Introduction & the monolithic bottleneck
- 01:49 — The 100 GB crash & monolithic limitations
- 04:40 — Architectural goals & Quarkus runners
- 07:57 — Kafka execution topics & commit-on-spawn pattern
- 10:13 — Execution pipeline & S3 payload separation
- 12:35 — Worker declination, TTLs & dead-letter queues
- 17:58 — Achieving effectively-once execution & SIGTERM handling
- 21:06 — Production results & reusable batch blueprint
