1. photon
  2. Commit 2f2be50d00561e2317edf5ebe62122147c5ff877
Browse code

scheduler: Skip building the dependency graph when performing `make check`

To perform `make check` on packages, we use the following steps:

Step 1. Build *all* packages.
Step 2. Invoke `make check` on selected packages.

So, in step 2, no new packages are built.

Certain packages specify additional packages to be installed in their
build environments when performing a `make check`, under %if
%{with_check}. However, these are not true (build-time) dependencies
in the usual sense, because these packages were already built in step
1. Hence, they don't introduce any build-ordering constraints in
step 2.

In step 2, all the specified packages can be "make-checked" in
parallel, since this operation is completely package-local. Thus, the
scheduler can simply skip building its dependency graph when running
`make check`. Moreover, interpreting the packages under %{with_check}
as dependencies will cause anomalies such as cycles in the graph (as
they are not true dependencies after all). Ex: make-check for gcc.

This is a preparatory patch to fix the make-check-loop issue. A
subsequent patch will complete the fix.

Change-Id: I918e768d583c27648da73cec674b4b77286ba30f
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/7346
Tested-by: gerrit-photon <photon-checkins@vmware.com>
Reviewed-by: Anish Swaminathan <anishs@vmware.com>
(cherry picked from commit db1f5400ed3f8b2a069f5f2dfeb1ba2459e9d23a)
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/7374
Reviewed-by: Srivatsa S. Bhat <srivatsab@vmware.com>

Srivatsa S. Bhat (VMware) authored on 2019/05/31 07:53:54
Showing 1 changed files
support/package-builder/Scheduler.py
History View file @ 2f2be50
... ... 
@@ -82,13 +82,33 @@ class Scheduler(object):
82 82 
         Scheduler.sortedList = sortedList
83 83
84 84 
         Scheduler.listOfAlreadyBuiltPackages = listOfAlreadyBuiltPackages
85 
-        for x in Scheduler.sortedList:
86 
-            if x not in Scheduler.listOfAlreadyBuiltPackages or x in constants.testForceRPMS:
87 
-                Scheduler.listOfPackagesToBuild.append(x)
85 
+
86 
+        for pkg in Scheduler.sortedList:
87 
+            pkgName, pkgVersion = StringUtils.splitPackageNameAndVersion(pkg)
88 
+            if (pkg not in Scheduler.listOfAlreadyBuiltPackages
89 
+               or pkgName in constants.testForceRPMS):
90 
+                Scheduler.listOfPackagesToBuild.append(pkg)
91 
+
88 92 
         Scheduler.listOfPackagesCurrentlyBuilding = set()
89 93 
         Scheduler.listOfPackagesNextToBuild = PriorityQueue()
90 94 
         Scheduler.listOfFailedPackages = []
91 
-        Scheduler._setPriorities()
95 
+
96 
+        # When performing (only) make-check, package dependencies are
97 
+        # irrelevant; i.e., all the packages can be "make-checked" in
98 
+        # parallel. So skip building the dependency graph. This is not
99 
+        # merely an optimization! A given package can define
100 
+        # additional packages to be installed in its build environment
101 
+        # when performing a make-check, under %if %{with_check}.
102 
+        # However, these are not really build-time-dependencies in the
103 
+        # usual sense; i.e., there is no ordering requirement when
104 
+        # building these packages; they only make sense when running a
105 
+        # `make check`. Hence, trying to build a dependency graph out
106 
+        # of them will result in anomalies such as cycles in the
107 
+        # graph. So skip building the graph altogether and schedule
108 
+        # all the `make check`s in parallel.
109 
+        skipGraphBuild = constants.rpmCheck
110 
+        Scheduler._setPriorities(skipGraphBuild)
111 
+
92 112 
         if constants.publishBuildDependencies:
93 113 
             # This must be called only after calling _setPriorities(),
94 114 
             # which builds the dependency graph.
... ... 
@@ -542,13 +562,17 @@ class Scheduler(object):
542 542
543 543
544 544 
     @staticmethod
545 
-    def _setPriorities():
546 
-        Scheduler._parseWeights()
547 
-        Scheduler._buildGraph()
548 
-
549 
-        for package in Scheduler.sortedList:
550 
-            pkgNode = Scheduler.mapPackagesToGraphNodes[package]
551 
-            Scheduler.priorityMap[package] = pkgNode.criticalChainWeight
545 
+    def _setPriorities(skipGraphBuild):
546 
+        if skipGraphBuild:
547 
+            for package in Scheduler.sortedList:
548 
+                Scheduler.priorityMap[package] = 0
549 
+        else:
550 
+            Scheduler._parseWeights()
551 
+            Scheduler._buildGraph()
552 
+
553 
+            for package in Scheduler.sortedList:
554 
+                pkgNode = Scheduler.mapPackagesToGraphNodes[package]
555 
+                Scheduler.priorityMap[package] = pkgNode.criticalChainWeight
552 556
553 557 
         Scheduler.logger.debug("set Priorities: Priority of all packages")
554 558 
         Scheduler.logger.debug(Scheduler.priorityMap)