Resource allocation

The Mesos resource allocation module contains the policy that the Mesos master uses to determine the type and quantity of resource offers that need to be made to each framework. Organizations can customize it to implement their own allocation policy, for example, fair sharing, priority, and so on, which allow for fine-grained resource sharing. Custom allocation modules can be developed to address specific needs.

The resource allocation module is responsible for making sure that the resources are shared in a fair manner among competing frameworks. The choice of algorithm used to determine whether the sharing policy has a great bearing on the efficiency of a cluster manager.

One of the most popular allocation algorithms, max-min fairness, works well in a homogenous environment; this is the one where resource requirements are fairly proportional between different competing users, such as the Hadoop cluster. However, scheduling resources across frameworks with heterogeneous resource demands poses a more complex challenge. What is a suitable fair share allocation policy if user A runs the tasks that require two CPUs and 8 GB RAM each and user B runs tasks that require four CPUs and 2 GB RAM each? As can be seen, user A's tasks are RAM-heavy, while user B's tasks are CPU-heavy. How, then, should a set of combined RAM + CPU resources be distributed between the two users?

The latter scenario is a common one faced by Mesos, designed as it is to manage resources primarily in a heterogeneous environment. To address this, Mesos has the Dominant Resource Fairness algorithm (DRF) as its default resource allocation policy, which is far more suitable for heterogeneous environments. The algorithm is described in detail in the following sections.

The Dominant Resource Fairness algorithm (DRF)

Job scheduling in datacenters is not limited to only CPUs but extends to other resources, such as the memory and disk, as well. In a scenario where resource demands are varied, some tasks are CPU-intensive, while some are memory- or disk-intensive; this is where the min-max fairness algorithm falls short. Herein lies the need for a resource scheduling mechanism that provides every user in a heterogeneous environment a fair share of the resources most required by it. In simple terms, DRF is an adaptation of the max-min fairness algorithm to fairly share heterogeneous resources among users.

Let's consider the following example to understand how the algorithm works.

We will assume that the resources are given in multiples of demand vectors and are divisible.

Consider a case where the total resources available are eight CPUs and 10 GB memory. User 1 runs tasks that require one CPU and 3 GB memory, and user 2 runs tasks that require three CPUs and 1 GB memory. Before we proceed to analyze how the DRF algorithm will allocate tasks, let's understand the concepts of the dominant resource and share:

The DRF allocation module tracks the dominant share of each user and makes a note of the resources allocated to each user. DRF begins allocation by offering resources (CPU or memory) to the user with the lowest dominant share among all the competing users. The user then has the option to accept the offer if it meets its requirement.

Now, let us look at each step taken by the DRF algorithm to allocate resources for users 1 and 2. For simplicity's sake, we will overlook the resources that get released back into the pool after the completion of small tasks and assume that every resource offer is accepted and that the users run an infinite number of tasks having the resource requirements. Every user 1 task would consume one-eighth of the total CPU and three-tenths of the total memory, making memory user 1's dominant resource. Every user 2 task would consume three-eighths of the total CPU and one-tenth of the total memory, making CPU user 2's dominant share.

Each row provides the following information:

To begin with, both users have a dominant share of 0% (as no resource is allocated as yet). We will assume that DRF chooses user 1 to offer resources to first, although had we assumed user 2, the final outcome would have been the same. Here are the steps it will follow:

Primarily, DRF aims to maximize the minimum dominant share across all users. As in this example, DRF worked with the users to allocate the following:

This can be diagrammatically depicted as follows:

Weighted DRF

We have so far assumed that users have an equal probability of being offered resources. There could also be a modification created in the algorithm, where one user or a set of users is favored over others in terms of resource allocation. This is referred to as Weighted DRF, wherein resources are not shared equally among users. Sharing can be weighted on a per-user and per-resource-level basis, the former being more popular.

Let's consider a per-user weighted computation of the previous example. For every user i and resource j, the weights are stated as w_1,j 3 and w_2,j = 1. This implies that user 1 will have three times the proportion of all the resources compared to user 2 in the system. If both the weights have the value 1, then allocation would be carried out in accordance with the normal DRF algorithm (as described before).

Now, let's look at each step taken by the DRF algorithm to allocate resources for users 1 and 2.

To begin with, both the users have a dominant share of 0% (as no resource is allocated as yet). We will assume that Weighted DRF chooses user 1 to offer resources to first, although had we assumed User 2, the final outcome would have been the same. Here are the steps that it will follow:

Weighted DRF aims to prioritize resource sharing based on the weight assigned to every user. In this example, Weighted DRF worked with the users to allocate the following:

This can be diagrammatically depicted as follows:

In addition to this, it is possible to create custom modules that cater to an organization or need specific resource allocation. This will be covered later in the same chapter.

Let's now look at some of the important properties that DRF follows/satisfies:

Configuring resource offers on Mesos

A common problem encountered is that sometimes, frameworks do not accept any resource offers due to improper resource configuration settings on the slaves. For example, the Elasticsearch framework requires ports 9200 and 9300, but the default port range configuration in the Mesos slaves is 31000 to 32000.

The slaves must be configured correctly so that the right resource offers are made to frameworks that can then accept them. This can be done as follows: