1. photon
  2. Commit d00518dd15249f9b134abbad4b630b4b42698528
Browse code

scheduler: Rewrite core data-structures and graph algorithms for efficiency

The existing implementation of the scheduler takes 24 minutes to build
the package dependency graph on a system with ~660 packages. This
rewrite of the scheduler's graph algorithms reduces the graph build
time to 0.085 seconds (85 milliseconds), which is an improvement of
over 17000x! :-)


Some of the other scheduler enhancements in this rewrite include the
following:

- A dependency graph with actual graph nodes, which help maintain
per-node information (unlike the current graph implementation which
is a 2-dimensional array of package-names).

- Optimal package dependencies in the graph:

We know that install-requires dependencies are weak [ Eg: if a
package 'A' only install-requires 'B', then it is not essential to
build 'B' before 'A'; hence the A->B dependency is weak. However,
if a package 'C' build-requires 'A', then we will need to build 'A'
as well as 'B' before 'C' ].

We use this knowledge to optimize the graph further, such that all
the dependencies are strong. In other words, install-requires
dependencies are moved up in the graph and turned into build-requires
dependencies of packages that actually need them. (In the above
example, 'B' would cease to be a dependency of 'A', and would instead
become a build-requires dependency of 'C'). This optimization also
ends up reducing the effective number of package dependencies in the
graph, which enables a greater level of build-parallelism.

- Cleaner algorithm to calculate the critical chain weights of
packages.

- Copious amounts of code documentation to make sure that it is
understandable.


Notes:

- The current implementation of _getRequiredPackages() returns *all*
dependencies of the given package, i.e., both build-requires and
install-requires. However, for the rewritten scheduler, we want this
function to just return the install-time dependencies. So we modify
the function accordingly, but retain the original behavior at the old
callsites of this function -- i.e., in
_getListNextPackagesReadyToBuild(), we use a list that has the merged
output of both _getRequiredPackages() and _getBuildRequiredPackages().
This will be cleaned up in a later patch.

- The existing publishBuildDependencies logic doesn't seem very useful
as it is, and is hence left out from the new scheduler. It will be
rewritten for the new scheduler in a subsequent patch.

Change-Id: Ide580ccd83ab95e65703dc39034486d8a8f8d72c
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/6162
Reviewed-by: Alexey Makhalov <amakhalov@vmware.com>
Tested-by: Anish Swaminathan <anishs@vmware.com>

Srivatsa S. Bhat (VMware) authored on 2018/11/15 05:39:01
Showing 1 changed files
support/package-builder/Scheduler.py
History View file @ d00518d
... ... 
@@ -7,6 +7,52 @@ from Logger import Logger
7 7 
 from SpecData import SPECS
8 8 
 from StringUtils import StringUtils
9 9
10 
+
11 
+class DependencyGraphNode(object):
12 
+
13 
+    def __init__(self, packageName, packageVersion, pkgWeight):
14 
+
15 
+        self.packageName = packageName
16 
+        self.packageVersion = packageVersion
17 
+
18 
+        self.buildRequiresPkgNodes = set() # Same as in spec file
19 
+        self.installRequiresPkgNodes = set() # Same as in spec file
20 
+
21 
+        # Auxiliary build-requires packages.
22 
+        #
23 
+        # This is the result of "moving up" the (weak)
24 
+        # install-requires package dependencies to their ancestors
25 
+        # that actually need them as a build dependency (more details
26 
+        # in the code below). This is used to optimize the dependency
27 
+        # graph by reorganizing parent-child relationships based on
28 
+        # strong dependencies.
29 
+        self.auxBuildRequiresPkgNodes = set()
30 
+
31 
+        # Accumulated install-requires packages.
32 
+        #
33 
+        # This is mostly used as a helper when building the graph
34 
+        # (specifically, as an intermediate step when computing
35 
+        # auxBuildRequiredPkgNodes), and is later unused.
36 
+        self.accumInstallRequiresPkgNodes = set()
37 
+
38 
+        self.childPkgNodes = set() # Packages that I depend on.
39 
+        self.parentPkgNodes = set() # Packages that depend on me.
40 
+
41 
+        self.selfWeight = pkgWeight # Own package weight.
42 
+
43 
+        # Critical-chain-weight: The key scheduling metric.
44 
+        #
45 
+        # Weight of the critical chain that can be built starting from
46 
+        # this package. Higher the criticalChainWeight, more the
47 
+        # benefit from building this package as early as possible.
48 
+        self.criticalChainWeight = 0
49 
+
50 
+        # Internal data-structure used to perform controlled
51 
+        # traversals of the dependency graph, as well as certain
52 
+        # sanity checks.
53 
+        self.numVisits = 0
54 
+
55 
+
10 56 
 class Scheduler(object):
11 57
12 58 
     lock = threading.Lock()
... ... 
@@ -21,6 +67,7 @@ class Scheduler(object):
21 21 
     logger = None
22 22 
     event = None
23 23 
     stopScheduling = False
24 
+    mapPackagesToGraphNodes = {}
24 25
25 26 
     @staticmethod
26 27 
     def setEvent(event):
... ... 
@@ -33,6 +80,7 @@ class Scheduler(object):
33 33 
     @staticmethod
34 34 
     def setParams(sortedList, listOfAlreadyBuiltPackages):
35 35 
         Scheduler.sortedList = sortedList
36 
+
36 37 
         Scheduler.listOfAlreadyBuiltPackages = listOfAlreadyBuiltPackages
37 38 
         for x in Scheduler.sortedList:
38 39 
             if x not in Scheduler.listOfAlreadyBuiltPackages or x in constants.testForceRPMS:
... ... 
@@ -98,6 +146,7 @@ class Scheduler(object):
98 98 
     def getDoneList():
99 99 
         return list(Scheduler.listOfAlreadyBuiltPackages)
100 100
101 
+
101 102 
     @staticmethod
102 103 
     def _getBuildRequiredPackages(pkg):
103 104 
         listRequiredRPMPackages = []
... ... 
@@ -112,6 +161,321 @@ class Scheduler(object):
112 112
113 113 
         return listRequiredPackages
114 114
115 
+
116 
+    def _createGraphNodes():
117 
+
118 
+        # GRAPH-BUILD STEP 1: Initialize graph nodes for each package.
119 
+        #
120 
+        # Create a graph with a node to represent every package and all
121 
+        # its dependent packages in the given list.
122 
+        for package in Scheduler.sortedList:
123 
+            packageName, packageVersion = StringUtils.splitPackageNameAndVersion(package)
124 
+            node = DependencyGraphNode(packageName, packageVersion,
125 
+                                       Scheduler._getWeight(package))
126 
+            Scheduler.mapPackagesToGraphNodes[package] = node
127 
+
128 
+        for package in Scheduler.sortedList:
129 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
130 
+            for childPackage in Scheduler._getBuildRequiredPackages(package):
131 
+                childPkgNode = Scheduler.mapPackagesToGraphNodes[childPackage]
132 
+                pkgNode.buildRequiresPkgNodes.add(childPkgNode)
133 
+
134 
+            for childPackage in Scheduler._getRequiredPackages(package):
135 
+                childPkgNode = Scheduler.mapPackagesToGraphNodes[childPackage]
136 
+                pkgNode.installRequiresPkgNodes.add(childPkgNode)
137 
+
138 
+        # GRAPH-BUILD STEP 2: Mark package dependencies in the graph.
139 
+        #
140 
+        # Add parent-child relationships between dependent packages.
141 
+        # If a package 'A' build-requires or install-requires package 'B', then:
142 
+        #   - Mark 'B' as a child of 'A' in the graph.
143 
+        #   - Mark 'A' as a parent of 'B' in the graph.
144 
+        #
145 
+        #                     A
146 
+        #
147 
+        #                  /     \
148 
+        #                 v       v
149 
+        #
150 
+        #                B         C
151 
+        #
152 
+        for package in Scheduler.sortedList:
153 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
154 
+            for childPkgNode in pkgNode.buildRequiresPkgNodes:
155 
+                pkgNode.childPkgNodes.add(childPkgNode)
156 
+                childPkgNode.parentPkgNodes.add(pkgNode)
157 
+
158 
+            for childPkgNode in pkgNode.installRequiresPkgNodes:
159 
+                pkgNode.childPkgNodes.add(childPkgNode)
160 
+                childPkgNode.parentPkgNodes.add(pkgNode)
161 
+
162 
+    def _optimizeGraph():
163 
+
164 
+        # GRAPH-BUILD STEP 3: Convert weak (install-requires) dependencies
165 
+        #                     into strong (aux-build-requires) dependencies.
166 
+        #
167 
+        # Consider the following graph on the left, where package 'A'
168 
+        # install-requires 'B' and build-requires 'C'.  Package 'C'
169 
+        # install-requires 'D'. Package 'D' build-requires 'E' and
170 
+        # install-requires 'F'.
171 
+        #
172 
+        #  b     : build-requires dependency
173 
+        #  i     : install-requires dependency
174 
+        #  aux-b : auxiliary build-requires dependency (explained later)
175 
+        #
176 
+        # Now, we know that install-requires dependencies are weaker
177 
+        # than build-requires dependencies. That is, for example, in the
178 
+        # original graph below, package 'B' does not need to be built
179 
+        # before package 'A', but package 'C' must be built before
180 
+        # package 'A'.
181 
+        #
182 
+        # Using this knowledge, we optimize the graph by re-organizing
183 
+        # the dependencies such that all of them are strong (we call
184 
+        # these newly computed build-dependencies as "auxiliary build
185 
+        # dependencies"). The optimized graph for the example below is
186 
+        # presented on the right -- the key property of the optimized
187 
+        # graph is that every child package *MUST* be built before its
188 
+        # parent(s). This process helps relax package dependencies to
189 
+        # a great extent, by giving us the flexibility to delay
190 
+        # building certain packages until they are actually needed.
191 
+        # Another important benefit of this optimization is that it
192 
+        # nullifies certain dependencies altogether (eg: A->B), thereby
193 
+        # enabling a greater level of build-parallelism.
194 
+        #
195 
+        #      Original Graph                  Optimized Graph
196 
+        #                             +
197 
+        #          A                  |       B              A
198 
+        #                             +
199 
+        #       i / \ b               |                b/    |aux-b  \aux-b
200 
+        #        /   \                +                /     |        \
201 
+        #       v     v               |               v      v         v
202 
+        #                             +
203 
+        #      B        C             |              C       D          F
204 
+        #                             +
205 
+        #                \i           |                   b/
206 
+        #                 \           +                   /
207 
+        #                  v          |                  v
208 
+        #                             +
209 
+        #                  D          |                 E
210 
+        #                             +
211 
+        #                b/  \i       |
212 
+        #                /    \       +
213 
+        #               v      v      |
214 
+        #                             +
215 
+        #              E        F     |
216 
+        #
217 
+        #
218 
+        # In the code below, we use 'accumulated-install-requires' set
219 
+        # as a placeholder to bubble-up install-requires dependencies of
220 
+        # each package to all its ancestors. In each such path, we look
221 
+        # for the nearest ancestor that has a build-requires dependency
222 
+        # on that path going up from the given package to that ancestor.
223 
+        # If we find such an ancestor, we convert the bubbled-up
224 
+        # install-requires packages accumulated so far into the
225 
+        # auxiliary-build-requires set at that ancestor. (This is how
226 
+        # 'D' and 'F' become aux-build-requires of 'A' in the optimized
227 
+        # graph above).
228 
+        #
229 
+        # Graph Traversal : Bottom-up (starting with packages that
230 
+        #                   have no children).
231 
+        #
232 
+        nodesToVisit = set()
233 
+        for package in Scheduler.sortedList:
234 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
235 
+            if len(pkgNode.childPkgNodes) == 0:
236 
+                nodesToVisit.add(pkgNode)
237 
+
238 
+        while nodesToVisit:
239 
+            pkgNode = nodesToVisit.pop()
240 
+
241 
+            pkgNode.accumInstallRequiresPkgNodes |= pkgNode.installRequiresPkgNodes
242 
+
243 
+            if len(pkgNode.childPkgNodes) == 0:
244 
+                # Count self-visit if you don't expect any other
245 
+                # visitors.
246 
+                pkgNode.numVisits += 1
247 
+
248 
+            for parentPkgNode in pkgNode.parentPkgNodes:
249 
+                if (pkgNode not in parentPkgNode.buildRequiresPkgNodes) and \
250 
+                   (pkgNode not in parentPkgNode.installRequiresPkgNodes):
251 
+                    raise Exception ("Visitor to parent is not its child " + \
252 
+                                     " Visitor: " + pkgNode.packageName + \
253 
+                                     " Parent:  " + parentPkgNode.packageName)
254 
+
255 
+                if pkgNode in parentPkgNode.buildRequiresPkgNodes:
256 
+                    parentPkgNode.auxBuildRequiresPkgNodes |= pkgNode.accumInstallRequiresPkgNodes
257 
+                else:
258 
+                    parentPkgNode.accumInstallRequiresPkgNodes |= pkgNode.accumInstallRequiresPkgNodes
259 
+
260 
+                parentPkgNode.numVisits += 1
261 
+                # Each child is expected to visit the parent once.
262 
+                # Note that a package might have the same packages as
263 
+                # both build-requires and install-requires children.
264 
+                # They don't count twice.
265 
+                numExpectedVisits = len(parentPkgNode.childPkgNodes)
266 
+                if parentPkgNode.numVisits == numExpectedVisits:
267 
+                    nodesToVisit.add(parentPkgNode)
268 
+                elif parentPkgNode.numVisits > numExpectedVisits:
269 
+                    raise Exception ("Parent node visit count > num of children " + \
270 
+                                     " Parent node: " + parentPkgNode.packageName + \
271 
+                                     " Visit count: " + str(parentPkgNode.numVisits) + \
272 
+                                     " Num of children: " + str(numExpectedVisits))
273 
+
274 
+            pkgNode.accumInstallRequiresPkgNodes.clear()
275 
+
276 
+        # Clear out the visit counter for reuse.
277 
+        for package in Scheduler.sortedList:
278 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
279 
+            if pkgNode.numVisits == 0:
280 
+                raise Exception ("aux-build-requires calculation never visited " \
281 
+                                 "package " + pkgNode.packageName)
282 
+            else:
283 
+                pkgNode.numVisits = 0
284 
+
285 
+        # GRAPH-BUILD STEP 4: Re-organize the dependencies in the graph based on
286 
+        #                     the above optimization.
287 
+        #
288 
+        # Now re-arrange parent-child relationships between packages using the
289 
+        # following criteria:
290 
+        # If a package 'A' build-requires or aux-build-requires package 'B', then:
291 
+        #   - Mark 'B' as a child of 'A' in the graph.
292 
+        #   - Mark 'A' as a parent of 'B' in the graph.
293 
+        # If a package 'A' only install-requires package 'B', then:
294 
+        #   - Remove 'B' as a child of 'A' in the graph.
295 
+        #   - Remove 'A' as a parent of 'B' in the graph.
296 
+        # No node should have a non-zero accum-install-requires set.
297 
+
298 
+        for package in Scheduler.sortedList:
299 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
300 
+            childPkgNodesToRemove = set()
301 
+            for childPkgNode in pkgNode.childPkgNodes:
302 
+                if (childPkgNode not in pkgNode.buildRequiresPkgNodes) and \
303 
+                   (childPkgNode not in pkgNode.auxBuildRequiresPkgNodes):
304 
+                       # We can't modify a set during iteration, so we
305 
+                       # accumulate the set of children we want to
306 
+                       # remove, and delete them after the for-loop.
307 
+                       childPkgNodesToRemove.add(childPkgNode)
308 
+                       childPkgNode.parentPkgNodes.remove(pkgNode)
309 
+
310 
+            pkgNode.childPkgNodes = pkgNode.childPkgNodes - \
311 
+                                    childPkgNodesToRemove
312 
+
313 
+            for newChildPkgNode in pkgNode.auxBuildRequiresPkgNodes:
314 
+                pkgNode.childPkgNodes.add(newChildPkgNode)
315 
+                newChildPkgNode.parentPkgNodes.add(pkgNode)
316 
+
317 
+
318 
+    def _calculateCriticalChainWeights():
319 
+
320 
+        # GRAPH-BUILD STEP 5: Calculate critical-chain-weight of packages.
321 
+        #
322 
+        # Calculation of critical-chain-weight (the key scheduling
323 
+        # metric):
324 
+        # --------------------------------------------------------
325 
+        # Let us define a "chain" of a given package to be the
326 
+        # sequence of parent packages that can be built starting from
327 
+        # that package. For example, if a package 'A' build-requires
328 
+        # 'B', which in turn build-requires 'C', then one of the
329 
+        # chains of 'C' is C->B->A. Now, if there are
330 
+        # multiple such chains possible from 'C', then we define the
331 
+        # "critical-chain" of 'C' to be the longest of those chains,
332 
+        # where "longest" is determined by the time it takes to build
333 
+        # all the packages in that chain. The build-times of any two
334 
+        # chains can be compared based on the sum of the
335 
+        # individual weights of each package in their respective
336 
+        # chains.
337 
+        #
338 
+        # Below, we calculate the critical-chain-weight of each
339 
+        # package (which is the maximum weight of all the paths
340 
+        # leading up to that package). Later on, we will schedule
341 
+        # package-builds by the decreasing order of the packages'
342 
+        # critical-chain-weight.
343 
+        #
344 
+        #
345 
+        #               ...  ...        ...
346 
+        #                 \   |         /
347 
+        #                  v  v        v
348 
+        #
349 
+        #                     A        B        C
350 
+        #
351 
+        #                      \       |       /
352 
+        #                       \      |      /
353 
+        #                        v     v     v
354 
+        #
355 
+        #                              D
356 
+        #
357 
+        #                            /
358 
+        #                           /
359 
+        #                          v
360 
+        #
361 
+        #                          E
362 
+        #
363 
+        #
364 
+        # In the above graph, the critical chain weight of 'D' is
365 
+        # computed as:
366 
+        # criticalChainWeight(D) = weight(D) +
367 
+        #                          max (criticalChainWeight(A),
368 
+        #                               criticalChainWeight(B),
369 
+        #                               weight(C))
370 
+        #
371 
+        # Graph Traversal : Top-down (starting with packages that
372 
+        #                   have no parents).
373 
+        #
374 
+        nodesToVisit = set()
375 
+        for package in Scheduler.sortedList:
376 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
377 
+            if len(pkgNode.parentPkgNodes) == 0:
378 
+                nodesToVisit.add(pkgNode)
379 
+
380 
+        while nodesToVisit:
381 
+            pkgNode = nodesToVisit.pop()
382 
+
383 
+            if len(pkgNode.parentPkgNodes) == 0:
384 
+                pkgNode.criticalChainWeight = pkgNode.selfWeight
385 
+                # Count self-visit if you don't expect any other
386 
+                # visitors.
387 
+                pkgNode.numVisits += 1
388 
+
389 
+            for childPkgNode in pkgNode.childPkgNodes:
390 
+                if pkgNode not in childPkgNode.parentPkgNodes:
391 
+                    raise Exception ("Visitor to child node is not its parent " + \
392 
+                                     " Visitor: " + pkgNode.packageName + \
393 
+                                     " Child node: " + childPkgNode.packageName)
394 
+
395 
+                if childPkgNode.numVisits == len(childPkgNode.parentPkgNodes):
396 
+                    raise Exception ("Child node visit count > number of parents " + \
397 
+                                     " Child node: " + childPkgNode.packageName + \
398 
+                                     " Visit count: " + childPkgNode.numVisits + \
399 
+                                     " Num of parents: " + \
400 
+                                     str(len(childPkgNode.parentPkgNodes)))
401 
+
402 
+                childPkgNode.criticalChainWeight = max(
403 
+                    childPkgNode.criticalChainWeight,
404 
+                    pkgNode.criticalChainWeight + childPkgNode.selfWeight)
405 
+
406 
+                childPkgNode.numVisits += 1
407 
+                # We can visit this package's children only after this
408 
+                # package has been visited by all its parents (thus
409 
+                # guaranteeing that its criticalChainWeight has
410 
+                # stabilized).
411 
+                if childPkgNode.numVisits == len(childPkgNode.parentPkgNodes):
412 
+                    nodesToVisit.add(childPkgNode)
413 
+
414 
+        # Clear out the visit counter for reuse.
415 
+        for package in Scheduler.sortedList:
416 
+            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
417 
+            if pkgNode.numVisits == 0:
418 
+                raise Exception ("critical-chain-weight calculation never visited " + \
419 
+                                 "package " + pkgNode.packageName)
420 
+            else:
421 
+                pkgNode.numVisits = 0
422 
+
423 
+
424 
+    def _buildGraph():
425 
+        Scheduler._createGraphNodes()
426 
+        Scheduler._optimizeGraph()
427 
+        Scheduler._calculateCriticalChainWeights()
428 
+
429 
+
115 430 
     @staticmethod
116 431 
     def _parseWeights():
117 432 
         Scheduler.pkgWeights.clear()
... ... 
@@ -150,22 +514,17 @@ class Scheduler(object):
150 150 
             Scheduler.logger.debug("Priority Scheduler disabled")
151 151 
             if constants.publishBuildDependencies:
152 152 
                 Scheduler.logger.debug("Publishing Build dependencies")
153 
-                Scheduler._makeGraph()
153 
+                Scheduler._buildGraph()
154 154 
         else:
155 155 
             Scheduler.logger.debug("Priority Scheduler enabled")
156 156 
             Scheduler._parseWeights()
157 157
158 
-            Scheduler._makeGraph()
158 
+            Scheduler._buildGraph()
159 
+
159 160 
             for package in Scheduler.sortedList:
160 
-                try:
161 
-                    Scheduler.priorityMap[package] = Scheduler._getWeight(package)
162 
-                except KeyError:
163 
-                    Scheduler.priorityMap[package] = 0
164 
-                for child_pkg in Scheduler.dependencyGraph[package].keys():
165 
-                    Scheduler.priorityMap[child_pkg] = (
166 
-                        Scheduler.priorityMap[child_pkg]
167 
-                        + (Scheduler.dependencyGraph[package][child_pkg]
168 
-                        * (Scheduler._getWeight(package))))
161 
+                pkgNode = Scheduler.mapPackagesToGraphNodes[package]
162 
+                Scheduler.priorityMap[package] = pkgNode.criticalChainWeight
163 
+
169 164 
             Scheduler.logger.debug("set Priorities: Priority of all packages")
170 165 
             Scheduler.logger.debug(Scheduler.priorityMap)
171 166
... ... 
@@ -173,7 +532,6 @@ class Scheduler(object):
173 173 
     @staticmethod
174 174 
     def _getRequiredPackages(pkg):
175 175 
         listRequiredRPMPackages = []
176 
-        listRequiredRPMPackages.extend(SPECS.getData().getBuildRequiresForPkg(pkg))
177 176 
         listRequiredRPMPackages.extend(SPECS.getData().getRequiresAllForPkg(pkg))
178 177
179 178 
         listRequiredPackages = []
... ... 
@@ -190,7 +548,9 @@ class Scheduler(object):
190 190 
         for pkg in Scheduler.listOfPackagesToBuild:
191 191 
             if pkg in Scheduler.listOfPackagesCurrentlyBuilding:
192 192 
                 continue
193 
-            listRequiredPackages = Scheduler._getRequiredPackages(pkg)
193 
+            listRequiredPackages = list(set(Scheduler._getBuildRequiredPackages(pkg) + \
194 
+                                   Scheduler._getRequiredPackages(pkg)))
195 
+
194 196 
             canBuild = True
195 197 
             for reqPkg in listRequiredPackages:
196 198 
                 if reqPkg not in Scheduler.listOfAlreadyBuiltPackages: