Browse code

Install from bindep.txt in plugins

This allows plugins to specify their binary dependencies in bindep
format.

Some thinking on the implementation: this is in contrast to the
files/[deb|rpm] installation, which is called from the external
install_prereqs.sh script. This script being an externally callable
entry-point is really an artifact of the days when we would build
snapshot images for CI and wanted to pre-cache downloads. These days
we use the mirror system to keep packages close to CI nodes. Thus
rather than expand install_prereqs.sh to also be installing
virtualenvs and python dependencies, this seems to fit better as a
separate internal phase of stack.sh.

Documentation is updated

Change-Id: Icbdfbf97c17c906a7ae86f43e80eb2c445816228

Ian Wienand authored on 2019/01/15 16:31:05
Showing 3 changed files
... ...
@@ -222,14 +222,20 @@ dependency mechanism is beyond the scope of the current work.
222 222
 System Packages
223 223
 ===============
224 224
 
225
-Devstack provides a framework for getting packages installed at an early
226
-phase of its execution. These packages may be defined in a plugin as files
227
-that contain new-line separated lists of packages required by the plugin
228 225
 
229
-Supported packaging systems include apt and yum across multiple distributions.
230
-To enable a plugin to hook into this and install package dependencies, packages
231
-may be listed at the following locations in the top-level of the plugin
232
-repository:
226
+
227
+Devstack based
228
+--------------
229
+
230
+Devstack provides a custom framework for getting packages installed at
231
+an early phase of its execution.  These packages may be defined in a
232
+plugin as files that contain new-line separated lists of packages
233
+required by the plugin
234
+
235
+Supported packaging systems include apt and yum across multiple
236
+distributions.  To enable a plugin to hook into this and install
237
+package dependencies, packages may be listed at the following
238
+locations in the top-level of the plugin repository:
233 239
 
234 240
 - ``./devstack/files/debs/$plugin_name`` - Packages to install when running
235 241
   on Ubuntu, Debian or Linux Mint.
... ...
@@ -240,6 +246,42 @@ repository:
240 240
 - ``./devstack/files/rpms-suse/$plugin_name`` - Packages to install when
241 241
   running on SUSE Linux or openSUSE.
242 242
 
243
+Although there a no plans to remove this method of installing
244
+packages, plugins should consider it deprecated for ``bindep`` support
245
+described below.
246
+
247
+bindep
248
+------
249
+
250
+The `bindep <https://docs.openstack.org/infra/bindep>`__ project has
251
+become the defacto standard for OpenStack projects to specify binary
252
+dependencies.
253
+
254
+A plugin may provide a ``./devstack/files/bindep.txt`` file, which
255
+will be called with the *default* profile to install packages.  For
256
+details on the syntax, etc. see the bindep documentation.
257
+
258
+It is also possible to use the ``bindep.txt`` of projects that are
259
+being installed from source with the ``-bindep`` flag available in
260
+install functions.  For example
261
+
262
+.. code-block:: bash
263
+
264
+  if use_library_from_git "diskimage-builder"; then
265
+     GITREPO["diskimage-builder"]=$DISKIMAGE_BUILDER_REPO_URL
266
+     GITDIR["diskimage-builder"]=$DEST/diskimage-builder
267
+     GITBRANCH["diskimage-builder"]=$DISKIMAGE_BUILDER_REPO_REF
268
+     git_clone_by_name "diskimage-builder"
269
+     setup_dev_lib -bindep "diskimage-builder"
270
+  fi
271
+
272
+will result in any packages required by the ``bindep.txt`` of the
273
+``diskimage-builder`` project being installed.  Note however that jobs
274
+that switch projects between source and released/pypi installs
275
+(e.g. with a ``foo-dsvm`` and a ``foo-dsvm-src`` test to cover both
276
+released dependencies and master versions) will have to deal with
277
+``bindep.txt`` being unavailable without the source directory.
278
+
243 279
 
244 280
 Using Plugins in the OpenStack Gate
245 281
 ===================================
... ...
@@ -1248,6 +1248,30 @@ function get_plugin_packages {
1248 1248
     $xtrace
1249 1249
 }
1250 1250
 
1251
+# Search plugins for a bindep.txt file
1252
+#
1253
+# Uses globals ``BINDEP_CMD``, ``GITDIR``, ``DEVSTACK_PLUGINS``
1254
+#
1255
+# Note this is only valid after BINDEP_CMD is setup in stack.sh, and
1256
+# is thus not really intended to be called externally.
1257
+function _get_plugin_bindep_packages {
1258
+    local xtrace
1259
+    xtrace=$(set +o | grep xtrace)
1260
+    set +o xtrace
1261
+
1262
+    local bindep_file
1263
+    local packages
1264
+
1265
+    for plugin in ${DEVSTACK_PLUGINS//,/ }; do
1266
+        bindep_file=${GITDIR[$plugin]}/devstack/files/bindep.txt
1267
+        if [[ -f ${bindep_file} ]]; then
1268
+            packages+=$($BINDEP_CMD -b --file ${bindep_file} || true)
1269
+        fi
1270
+    done
1271
+    echo "${packages}"
1272
+    $xtrace
1273
+}
1274
+
1251 1275
 # Distro-agnostic package installer
1252 1276
 # Uses globals ``NO_UPDATE_REPOS``, ``REPOS_UPDATED``, ``RETRY_UPDATE``
1253 1277
 # install_package package [package ...]
... ...
@@ -805,6 +805,13 @@ install_infra
805 805
 $VIRTUALENV_CMD $DEST/bindep-venv
806 806
 # TODO(ianw) : optionally install from zuul checkout?
807 807
 $DEST/bindep-venv/bin/pip install bindep
808
+export BINDEP_CMD=${DEST}/bindep-venv/bin/bindep
809
+
810
+# Install packages as defined in plugin bindep.txt files
811
+pkgs="$( _get_plugin_bindep_packages )"
812
+if [[ -n "${pkgs}" ]]; then
813
+    install_package ${pkgs}
814
+fi
808 815
 
809 816
 # Extras Pre-install
810 817
 # ------------------