Software Architecture & Platform SIG: Apache Helix: Building Scalable Distributed Systems just got simple!



  • Apache Helix: Building Scalable Distributed Systems just got simple!


    Distributed data systems systems are used in a variety of settings like online serving, offline analytics, data transport, and search, among other use cases. A key pattern we observe is that even though they may serve different purposes, they tend to have a lot of common functionality, and tend to use common building blocks in their architectures. One such building block that is just beginning to receive attention is cluster management, which addresses the complexity of handling a dynamic, large-scale system with many servers. Such systems must handle software and hardware failures, setup tasks such as bootstrapping data, and operational issues such as data placement, load balancing, planned upgrades, and cluster expansion.

    Helix provides an abstraction for a system developer to separate coordination and management tasks from component functional tasks of a distributed system. The developer defines the system behavior via a state model that enumerates the possible states of each component, the transitions between those states, and constraints that govern the system’s valid settings. Helix does the heavy lifting of ensuring the system satisfies that state model in the distributed setting, while also meeting the system’s goals on load balancing and throttling state changes. We detail several Helix managed production distributed systems at LinkedIn and how Helix has helped them avoid building custom management components. We describe the Helix design and implementation and present an experimental study that demonstrates its performance and functionality.


    6:30 - 7:00 p.m. Registration/Networking/Refreshments/Pizza

    7:00 - 9:00 p.m. Presentations