forked from jbarber/maui-admin-guide
-
Notifications
You must be signed in to change notification settings - Fork 0
/
3.3jobflow.html
149 lines (93 loc) · 9.4 KB
/
3.3jobflow.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta name="generator" content="HTML Tidy for Linux (vers 25 March 2009), see www.w3.org">
<title></title>
</head>
<body>
<div class="sright">
<div class="sub-content-head">
Maui Scheduler
</div>
<div id="sub-content-rpt" class="sub-content-rpt">
<div class="tab-container docs" id="tab-container">
<div class="topNav">
<div class="docsSearch"></div>
<div class="navIcons topIcons">
<a href="index.html"><img src="home.png" title="Home" alt="Home" border="0"></a> <a href="3.0basics.html"><img src="upArrow.png" title="Up" alt="Up" border="0"></a> <a href="3.2environment.html"><img src="prevArrow.png" title="Previous" alt="Previous" border="0"></a> <a href="3.4configure.html"><img src="nextArrow.png" title="Next" alt="Next" border="0"></a>
</div>
<h1>3.3 Scheduling Iterations and Job Flow</h1>
<ul>
<li>
<a href="3.3jobflow.html#SchedulingIterations">3.3.1 Scheduling Iterations</a>
<ul>
<li><a href="3.3jobflow.html#UpdateStateInformation">3.3.1.1 Update State Information</a></li>
<li><a href="3.3jobflow.html#RefreshReservations">3.3.1.2 Refresh Reservations</a></li>
<li><a href="3.3jobflow.html#ScheduleReservedJobs">3.3.1.3 Schedule Reserved Jobs</a></li>
<li><a href="3.3jobflow.html#SchedulePriorityJobs">3.3.1.4 Schedule Priority Jobs</a></li>
<li><a href="3.3jobflow.html#BackfillJobs">3.3.1.5 Backfill Jobs</a></li>
<li><a href="3.3jobflow.html#UpdateStatistics">3.3.1.6 Update Statistics</a></li>
<li><a href="3.3jobflow.html#HandleUserRequests">3.3.1.7 Handle User Requests</a></li>
<li><a href="3.3jobflow.html#PerformNextSchedulinCycle">3.3.1.8 Perform Next Scheduling Cycle</a></li>
</ul>
</li>
<li>
<a href="3.3jobflow.html#DetailedJobFlow">3.3.2 Detailed Job Flow</a>
<ul>
<li><a href="3.3jobflow.html#BasicJobFeasibility">3.3.2.1 Determine Basic Job Feasibility</a></li>
<li><a href="3.3jobflow.html#PrioritizeJobs">3.3.2.2 Prioritize Jobs</a></li>
<li><a href="3.3jobflow.html#EnforceThrottlingPolicies">3.3.2.3 Enforce Configured Throttling Policies</a></li>
<li><a href="3.3jobflow.html#ResourceAvailability">3.3.2.4 Determine Resource Availability</a></li>
<li><a href="3.3jobflow.html#AllocateResources">3.3.2.5 Allocate Resources to Job</a></li>
<li><a href="3.3jobflow.html#DistributeJobs">3.3.2.6 Distribute Jobs Tasks Across Allocated Resources</a></li>
<li><a href="3.3jobflow.html#LaunchJob">3.3.2.7 Launch Job</a></li>
</ul>
</li>
</ul>
<h2><a name="SchedulingIterations" id="SchedulingIterations"></a>3.3.1 Scheduling Iterations</h2>In any given scheduling iteration, many activities take place. These are broken into the following major categories:
<p>Update State Information<br>
Refresh Reservations<br>
Schedule Reserved Jobs<br>
Schedule Priority Jobs<br>
Backfill Jobs<br>
Update Statistics<br>
Handle User Requests</p>
<h3><a name="UpdateStateInformation" id="UpdateStateInformation"></a>3.3.1.1 Update State Information</h3>
<p>Each iteration, the scheduler contacts the resource manager(s) and requests up to date information on compute resources, workload, and policy configuration. On most systems, these calls are to a centralized resource manager daemon which possesses all information.</p>
<h3><a name="RefreshReservations" id="RefreshReservations"></a>3.3.1.2 Refresh Reservations</h3>
<h3><a name="ScheduleReservedJobs" id="ScheduleReservedJobs"></a>3.3.1.3 Schedule Reserved Jobs</h3>
<h3><a name="SchedulePriorityJobs" id="SchedulePriorityJobs"></a>3.3.1.4 Schedule Priority Jobs</h3>
<p>In scheduling jobs, multiple steps occur.</p>
<h3><a name="BackfillJobs" id="BackfillJobs"></a>3.3.1.5 Backfill Jobs</h3>
<h3><a name="UpdateStatistics" id="UpdateStatistics"></a>3.3.1.6 Update Statistics</h3>
<h3><a name="HandleUserRequests" id="HandleUserRequests"></a>3.3.1.7 Handle User Requests</h3>
<p>User requests include any call requesting state information, configuration changes, or job or resource manipulation commands. These requests may come in the form of user client calls, peer daemon calls, or process signals.</p>
<h3><a name="PerformNextSchedulinCycle" id="PerformNextSchedulinCycle"></a>3.3.1.8 Perform Next Scheduling Cycle</h3>
<p>Maui operates on a polling/event driven basis. When all scheduling activities are complete, Maui will process user requests until a new resource manager event is received or an internal event is generated. Resource manager events include activities such as a new job submission or completion of an active job, addition of new node resources, or changes in resource manager policies. Internal events include admin '<a href="commands/schedctl.html">schedule</a>' requests, reservation activation/deactivation, or the expiration of the <a href="a.fparameters.html#rmpollinterval">RMPOLLINTERVAL</a> timer.<br></p>
<h2><a name="DetailedJobFlow" id="DetailedJobFlow"></a>3.3.2 Detailed Job Flow</h2>
<h3><a name="BasicJobFeasibility" id="BasicJobFeasibility"></a>3.3.2.1 Determine Basic Job Feasibility</h3>
<p>The first step in scheduling is determining which jobs are feasible. This step eliminates jobs which have job holds in place, invalid job states (i.e., Completed, Not Queued, Defered, etc), or unsatisfied preconditions. Preconditions may include stage-in files or completion of preliminary job steps.</p>
<h3><a name="PrioritizeJobs" id="PrioritizeJobs"></a>3.3.2.2 Prioritize Jobs</h3>
<p>With a list of feasible jobs created, the next step involves <a href="5.1jobprioritization.html">determining the relative priority</a> of all jobs within that list. A priority for each job is calculated based on job attributes such as job owner, job size, length of time the job has been queued, and so forth.</p>
<h3><a name="EnforceThrottlingPolicies" id="EnforceThrottlingPolicies"></a>3.3.2.3 Enforce Configured Throttling Policies</h3>
<p>Any configured <a href="6.2throttlingpolicies.html">throttling policies</a> are then applied constraining how many jobs, nodes, processors, etc are allowed on a per credential basis. Jobs which violate these policies are not considered for scheduling.</p>
<h3><a name="ResourceAvailability" id="ResourceAvailability"></a>3.3.2.4 Determine Resource Availability</h3>
<p>For each job, Maui attempts to locate the required compute resources needed by the job. In order for a match to be made, the node must possess all node attributes specified by the job and possess adequate available resources to meet the <i>TasksPerNode</i> job constraint (Default TasksPerNode is 1) Normally, Maui determine a node to have adequate resources if the resources are <i>neither utilized by nor dedicated to</i> another job using the calculation</p>
<p>R.Available = R.Configured - MAX(R.Dedicated,R.Utilized).</p>
<p>The <a href="a.fparameters.html#resourceavailabilitypolicy">RESOURCEAVAILABILITYPOLICY</a> parameter can be modified to adjust this behavior.</p>
<h3><a name="AllocateResources" id="AllocateResources"></a>3.3.2.5 Allocate Resources to Job</h3>
<p>If adequate resources can be found for a job, the <a href="5.2nodeallocation.html">node allocation policy</a> is then applied to select the best set of resources. These allocation policies allow selection criteria such as speed of node, type of reservations, or excess node resources to be figured into the allocation decision to improve the performance of the job and/or maximize the freedom of the scheduler in making future scheduling decisions.</p>
<h3><a name="DistributeJobs" id="DistributeJobs"></a>3.3.2.6 Distribute Jobs Tasks Across Allocated Resources</h3>
<p>With the resources selected, Maui then maps job tasks to the actual resources. This distribution of tasks is typically based on simple task distribution algorithms such as round-robin or max blocking, but can also incorporate parallel language library (i.e., MPI, PVM, etc) specific patterns used to minimize interprocess communication overhead.</p>
<h3><a name="LaunchJob" id="LaunchJob"></a>3.3.2.7 Launch Job</h3>
<p>With the resources selected and task distribution mapped, the scheduler then contacts the resource manager and informs it where and how to launch the job. The resource manager then initiates the actual job executable.</p>
<div class="navIcons bottomIcons">
<a href="index.html"><img src="home.png" title="Home" alt="Home" border="0"></a> <a href="3.0basics.html"><img src="upArrow.png" title="Up" alt="Up" border="0"></a> <a href="3.2environment.html"><img src="prevArrow.png" title="Previous" alt="Previous" border="0"></a> <a href="3.4configure.html"><img src="nextArrow.png" title="Next" alt="Next" border="0"></a>
</div>
</div>
</div>
</div>
<div class="sub-content-btm"></div>
</div>
</body>
</html>