Apache Mesos

Mesos Research Project

Mesos started as a research project, originally named Nexus, at UC Berkeley in March 2009. The design of Mesos and a description of the original prototype were first published at HotCloud in June 2009. Mesos was open sourced on GitHub in August 2010 and joined the Apache Incubator in January 2011.

Mesos was originally inspired by the rapid explosion in cluster programming frameworks, such as MapReduce, Dryad, Pregel, and Dremel. To let organizations take advantage of these frameworks, it was increasingly necessary to share resources dynamically between them, rather than building dedicated clusters. We quickly expanded the scope of the project to also support running multiple instances of the same framework (e.g. production and experimental instances), long-running services, and web applications. Finally, we sought to enable the development of new frameworks that can coexist with current ones. We designed Spark, an in-memory computing framework for iterative and interactive analytics, to show the benefits of new frameworks.

Publications

• A Common Substrate for Cluster Computing. B. Hindman, A. Konwinski, M. Zaharia and I. Stoica, HotCloud 2009, June 2009.
• Spark: Cluster Computing with Working Sets. M. Zaharia, M. Chowdhury, M. Franklin, S. Shenker and I. Stoica, HotCloud 2010, June 2010.
• Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A.D. Joseph, R. Katz, S. Shenker and I. Stoica, NSDI 2011, March 2011.
• Dominant Resource Fairness: Fair Allocation of Multiple Resources Types. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica, NSDI 2011, March 2011.

Selected Talks

• Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center (PDF), NSDI 2011, April 2011.
• Dominant Resource Fairness: Fair Allocation of Multiple Resource Types (PDF), NSDI 2011, April 2011.
• Mesos: Efficiently Sharing the Datacenter (video), LinkedIn, December 2010.
• Spark: A Framework for Iterative and Interactive Cluster Computing (video), LinkedIn, March 2011.