Netflix upgrades Conductor for massive workflow scaling

Netflix has completely revised its internal Conductor workflow orchestration engine to accommodate a workload far exceeding its original design capacity. The updated version now supports approximately 200,000 workflow definitions across 150 applications, processing around 420 million workflows monthly—a volume that surpasses what earlier iterations could handle. Additionally, the redesign expands the supported workflow size from roughly 2,500 tasks to 30,000 tasks while slashing p99 workflow evaluation latency by about 40%. The evolution from versions 1.0 through 3.0 included key infrastructure changes, such as migrating execution data from Dynomite to Cassandra and moving large task inputs and outputs to Amazon S3, which improved scalability for distributed workflows.

The system manages distributed workflows for Content and Studio Engineering, Ads, and Games teams. Previous versions encountered performance issues as workflow complexity increased, particularly when loading entire workflow states into memory during evaluation. The new architecture separates workflow metadata from task and user data, storing tasks independently and loading only the necessary segments for decision-making. This change reduces memory consumption and accelerates processing. Earlier versions also relied on DynoQueues, which were replaced with Timestone to handle asynchronous task processing more efficiently.

A recurring problem in earlier versions was contention during task state coordination. Conductor 4.0 eliminates this bottleneck by isolating pending and completed task states, prioritizing finalized states. Evaluation was also moved out of the synchronous request path, relying on Timestone queues for asynchronous handling. As a result, failed lock attempts—previously as high as 2,700 per interval—have been nearly eliminated. Additionally, Kafka was introduced to decouple indexing from the execution path, with Elasticsearch and Iceberg now supporting indexing and long-term storage.

Community feedback from 2022 had already identified these scalability challenges. One user described a workflow with 55,745 tasks and 310,000 entries consuming 5 GB of JVM heap. Another noted a 1.7 MB workflow with nearly 5,000 task entries taking almost two minutes to load. These examples showed the need for a more scalable solution, which the latest version now provides. Aravind Ramkumar, a Conductor maintainer at Netflix, explained in discussions that the engine previously loaded the entire running workflow into memory for evaluation, and the redesign addresses this by loading only the necessary portions.

The update includes native concurrency controls, dynamic worker allocation, and a type-safe Java Workflow SDK. Netflix deploys the SDK for creative content ingestion and Data Clean Room workflows. The system retains its task-based structure, with workflow definitions and a separate execution engine optimized for long-running distributed processes. Earlier community discussions also highlighted worker scaling constraints, with users reporting 25,000 to 30,000 running workflows causing HTTP task queue backups, which Ramkumar advised against addressing through polling increases due to potential system overload.

Netflix anticipates further growth in workflow demand as it expands into live content, games, and podcasts, with potential demand increases of up to fivefold. In December 2023, the company stopped maintaining its public Conductor OSS repository, redirecting efforts to its internal fork. Community-developed modules and extensions remain available in a separate repository, though internal requirements now guide development.