@@ -71,204 +71,14 @@ status information about the driver.

  *  `Metadata loop file` file attached to `Metadata file`, if loopback device is used

  *  `Library Version` from the libdevmapper used

-### options

+### About the devicemapper options

 The devicemapper backend supports some options that you can specify

 when starting the docker daemon using the `--storage-opt` flags.

 This uses the `dm` prefix and would be used something like `docker -d --storage-opt dm.foo=bar`.

-Here is the list of supported options:

-

- *  `dm.basesize`

-

-    Specifies the size to use when creating the base device, which

-    limits the size of images and containers. The default value is

-    10G. Note, thin devices are inherently "sparse", so a 10G device

-    which is mostly empty doesn't use 10 GB of space on the

-    pool. However, the filesystem will use more space for the empty

-    case the larger the device is. **Warning**: This value affects the

-    system-wide "base" empty filesystem that may already be

-    initialized and inherited by pulled images.  Typically, a change

-    to this value will require additional steps to take effect: 1)

-    stop `docker -d`, 2) `rm -rf /var/lib/docker`, 3) start `docker -d`.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.basesize=20G``

-

- *  `dm.loopdatasize`

-

-    Specifies the size to use when creating the loopback file for the

-    "data" device which is used for the thin pool. The default size is

-    100G. Note that the file is sparse, so it will not initially take

-    up this much space.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.loopdatasize=200G``

-

- *  `dm.loopmetadatasize`

-

-    Specifies the size to use when creating the loopback file for the

-    "metadadata" device which is used for the thin pool. The default size is

-    2G. Note that the file is sparse, so it will not initially take

-    up this much space.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.loopmetadatasize=4G``

-

- *  `dm.fs`

-

-    Specifies the filesystem type to use for the base device. The supported

-    options are "ext4" and "xfs". The default is "ext4"

-

-    Example use:

-

-    ``docker -d --storage-opt dm.fs=xfs``

-

- *  `dm.mkfsarg`

-

-    Specifies extra mkfs arguments to be used when creating the base device.

-

-    Example use:

-

-    ``docker -d --storage-opt "dm.mkfsarg=-O ^has_journal"``

-

- *  `dm.mountopt`

-

-    Specifies extra mount options used when mounting the thin devices.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.mountopt=nodiscard``

-

- *  `dm.thinpooldev`

-

-    Specifies a custom blockdevice to use for the thin pool.

-

-    If using a block device for device mapper storage, ideally lvm2

-    would be used to create/manage the thin-pool volume that is then

-    handed to docker to exclusively create/manage the thin and thin

-    snapshot volumes needed for its containers.  Managing the thin-pool

-    outside of docker makes for the most feature-rich method of having

-    docker utilize device mapper thin provisioning as the backing

-    storage for docker's containers.  lvm2-based thin-pool management

-    feature highlights include: automatic or interactive thin-pool

-    resize support, dynamically change thin-pool features, automatic

-    thinp metadata checking when lvm2 activates the thin-pool, etc.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.thinpooldev=/dev/mapper/thin-pool``

-

- *  `dm.datadev`

-

-    Specifies a custom blockdevice to use for data for the thin pool.

-

-    If using a block device for device mapper storage, ideally both

-    datadev and metadatadev should be specified to completely avoid

-    using the loopback device.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1``

-

- *  `dm.metadatadev`

-

-    Specifies a custom blockdevice to use for metadata for the thin

-    pool.

-

-    For best performance the metadata should be on a different spindle

-    than the data, or even better on an SSD.

-

-    If setting up a new metadata pool it is required to be valid. This

-    can be achieved by zeroing the first 4k to indicate empty

-    metadata, like this:

-

-    ``dd if=/dev/zero of=$metadata_dev bs=4096 count=1``

-

-    Example use:

-

-    ``docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1``

-

- *  `dm.blocksize`

-

-    Specifies a custom blocksize to use for the thin pool.  The default

-    blocksize is 64K.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.blocksize=512K``

-

- *  `dm.blkdiscard`

-

-    Enables or disables the use of blkdiscard when removing

-    devicemapper devices. This is enabled by default (only) if using

-    loopback devices and is required to resparsify the loopback file

-    on image/container removal.

-

-    Disabling this on loopback can lead to *much* faster container

-    removal times, but will make the space used in /var/lib/docker

-    directory not be returned to the system for other use when

-    containers are removed.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.blkdiscard=false``

-

- *  `dm.override_udev_sync_check`

-

-    Overrides the `udev` synchronization checks between `devicemapper` and `udev`.

-    `udev` is the device manager for the Linux kernel.

-

-    To view the `udev` sync support of a Docker daemon that is using the

-    `devicemapper` driver, run:

-

-        $ docker info

-	[...]

-	 Udev Sync Supported: true

-	[...]

-

-    When `udev` sync support is `true`, then `devicemapper` and udev can

-    coordinate the activation and deactivation of devices for containers.

-

-    When `udev` sync support is `false`, a race condition occurs between

-    the`devicemapper` and `udev` during create and cleanup. The race condition

-    results in errors and failures. (For information on these failures, see

-    [docker#4036](https://github.com/docker/docker/issues/4036))

-

-    To allow the `docker` daemon to start, regardless of `udev` sync not being

-    supported, set `dm.override_udev_sync_check` to true:

-

-        $ docker -d --storage-opt dm.override_udev_sync_check=true

-

-    When this value is `true`, the  `devicemapper` continues and simply warns

-    you the errors are happening.

-

-    > **Note**: The ideal is to pursue a `docker` daemon and environment that

-    > does support synchronizing with `udev`. For further discussion on this

-    > topic, see [docker#4036](https://github.com/docker/docker/issues/4036).

-    > Otherwise, set this flag for migrating existing Docker daemons to a

-    > daemon with a supported environment.

-

- *  `dm.use_deferred_removal`

-

-    Enables use of deferred device removal if libdm and kernel driver

-    support the mechanism.

-

-    Deferred device removal means that if device is busy when devices is

-    being removed/deactivated, then a deferred removal is scheduled on

-    device. And devices automatically goes away when last user of device

-    exits.

-

-    For example, when container exits, its associated thin device is

-    removed. If that devices has leaked into some other mount namespace

-    can can't be removed now, container exit will still be successful

-    and this option will just schedule device for deferred removal and

-    will not wait in a loop trying to remove a busy device.

-

-    Example use:

-

-    ``docker -d --storage-opt dm.use_deferred_removal=true``

-

+These options are currently documented both in [the man

+page](../../../man/docker.1.md) and in [the online

+documentation](https://docs.docker.com/reference/commandline/daemon/#docker-

+execdriver-option).  If you add an options, update both the `man` page and the

+documentation.

@@ -178,7 +178,24 @@ Particular storage-driver can be configured with options specified with

 `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm` and

 options for `zfs` start with `zfs`.

-Currently supported options of `devicemapper`:

+*  `dm.thinpooldev`

+

+     Specifies a custom block storage device to use for the thin pool.

+

+     If using a block device for device mapper storage, it is best to use `lvm`

+     to create and manage the thin-pool volume. This volume is then handed to Docker

+     to exclusively create snapshot volumes needed for images and containers.  

+

+     Managing the thin-pool outside of Docker makes for the most feature-rich

+     method of having Docker utilize device mapper thin provisioning as the

+     backing storage for Docker's containers. The highlights of the lvm-based

+     thin-pool management feature include: automatic or interactive thin-pool

+     resize support, dynamically changing thin-pool features, automatic thinp

+     metadata checking when lvm activates the thin-pool, etc.

+

+     Example use:

+

+        docker -d --storage-opt dm.thinpooldev=/dev/mapper/thin-pool

  *  `dm.basesize`

@@ -188,9 +205,9 @@ Currently supported options of `devicemapper`:

     10 GB of space on the pool. However, the filesystem will use more space for

     the empty case the larger the device is.

-     **Warning:** This value affects the system-wide "base" empty filesystem

-     that may already be initialized and inherited by pulled images. Typically,

-     a change to this value will require additional steps to take effect:

+    This value affects the system-wide "base" empty filesystem

+    that may already be initialized and inherited by pulled images. Typically,

+    a change to this value requires additional steps to take effect:

         $ sudo service docker stop

         $ sudo rm -rf /var/lib/docker

@@ -202,9 +219,12 @@ Currently supported options of `devicemapper`:

  *  `dm.loopdatasize`

-    Specifies the size to use when creating the loopback file for the "data"

-    device which is used for the thin pool. The default size is 100G. Note that

-    the file is sparse, so it will not initially take up this much space.

+    >**Note**: This option configures devicemapper loopback, which should not be used in production.

+		

+    Specifies the size to use when creating the loopback file for the

+    "data" device which is used for the thin pool. The default size is

+    100G. The file is sparse, so it will not initially take up this

+    much space.

     Example use:

@@ -212,10 +232,12 @@ Currently supported options of `devicemapper`:

  *  `dm.loopmetadatasize`

+    >**Note**: This option configures devicemapper loopback, which should not be used in production.

+

     Specifies the size to use when creating the loopback file for the

-    "metadata" device which is used for the thin pool. The default size is 2G.

-    Note that the file is sparse, so it will not initially take up this much

-    space.

+    "metadadata" device which is used for the thin pool. The default size

+    is 2G. The file is sparse, so it will not initially take up

+    this much space.

     Example use:

@@ -248,6 +270,8 @@ Currently supported options of `devicemapper`:

  *  `dm.datadev`

+    (Deprecated, use `dm.thinpooldev`)

+

     Specifies a custom blockdevice to use for data for the thin pool.

     If using a block device for device mapper storage, ideally both datadev and

@@ -256,12 +280,12 @@ Currently supported options of `devicemapper`:

     Example use:

-        $ docker -d \

-            --storage-opt dm.datadev=/dev/sdb1 \

-            --storage-opt dm.metadatadev=/dev/sdc1

+        $ docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1

  *  `dm.metadatadev`

+    (Deprecated, use `dm.thinpooldev`)

+

     Specifies a custom blockdevice to use for metadata for the thin pool.

     For best performance the metadata should be on a different spindle than the

@@ -270,13 +294,11 @@ Currently supported options of `devicemapper`:

     If setting up a new metadata pool it is required to be valid. This can be

     achieved by zeroing the first 4k to indicate empty metadata, like this:

-        $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1

+	$ dd if=/dev/zero of=$metadata_dev bs=4096 count=1

     Example use:

-        $ docker -d \

-            --storage-opt dm.datadev=/dev/sdb1 \

-            --storage-opt dm.metadatadev=/dev/sdc1

+        $ docker -d --storage-opt dm.datadev=/dev/sdb1 --storage-opt dm.metadatadev=/dev/sdc1

  *  `dm.blocksize`

@@ -325,7 +347,7 @@ Currently supported options of `devicemapper`:

     To allow the `docker` daemon to start, regardless of `udev` sync not being

     supported, set `dm.override_udev_sync_check` to true:

-        $ docker -d --storage-opt dm.override_udev_sync_check=true

+	$ docker -d --storage-opt dm.override_udev_sync_check=true

     When this value is `true`, the  `devicemapper` continues and simply warns

     you the errors are happening.

@@ -337,6 +359,7 @@ Currently supported options of `devicemapper`:

     > Otherwise, set this flag for migrating existing Docker daemons to

     > a daemon with a supported environment.

+

 ## Docker execdriver option

 Currently supported options of `zfs`:

@@ -297,79 +297,198 @@ inside it)

 # STORAGE DRIVER OPTIONS

-Options to storage backend can be specified with **--storage-opt** flags. The

-only backend which currently takes options is *devicemapper*. Therefore use these

+Docker uses storage backends (known as "graphdrivers" in the Docker

+internals) to create writable containers from images.  Many of these

+backends use operating system level technologies and can be

+configured.

+

+Specify options to the storage backend with **--storage-opt** flags. The only

+backend that currently takes options is *devicemapper*. Therefore use these

 flags with **-s=**devicemapper.

+Specifically for devicemapper, the default is a "loopback" model which

+requires no pre-configuration, but is extremely inefficient.  Do not

+use it in production.

+

+To make the best use of Docker with the devicemapper backend, you must

+have a recent version of LVM.  Use `lvm` to create a thin pool; for

+more information see `man lvmthin`.  Then, use `--storage-opt

+dm.thinpooldev` to tell the Docker engine to use that pool for

+allocating images and container snapshots.

+

 Here is the list of *devicemapper* options:

+#### dm.thinpooldev

+

+Specifies a custom block storage device to use for the thin pool.

+

+If using a block device for device mapper storage, it is best to use

+`lvm` to create and manage the thin-pool volume. This volume is then

+handed to Docker to create snapshot volumes needed for images and

+containers.

+

+Managing the thin-pool outside of Docker makes for the most feature-rich method

+of having Docker utilize device mapper thin provisioning as the backing storage

+for Docker's containers. The highlights of the LVM-based thin-pool management

+feature include: automatic or interactive thin-pool resize support, dynamically

+changing thin-pool features, automatic thinp metadata checking when lvm activates

+the thin-pool, etc.

+

+Example use: `docker -d --storage-opt dm.thinpooldev=/dev/mapper/thin-pool`

+

 #### dm.basesize

-Specifies the size to use when creating the base device, which limits the size

-of images and containers. The default value is 10G. Note, thin devices are

-inherently "sparse", so a 10G device which is mostly empty doesn't use 10 GB

-of space on the pool. However, the filesystem will use more space for the empty

-case the larger the device is. **Warning**: This value affects the system-wide

-"base" empty filesystem that may already be initialized and inherited by pulled

-images.

-#### dm.loopdatasize

-Specifies the size to use when creating the loopback file for the "data"

-device which is used for the thin pool. The default size is 100G. Note that the

-file is sparse, so it will not initially take up this much space.

+Specifies the size to use when creating the base device, which limits

+the size of images and containers. The default value is 10G. Note,

+thin devices are inherently "sparse", so a 10G device which is mostly

+empty doesn't use 10 GB of space on the pool. However, the filesystem

+will use more space for base images the larger the device

+is. 

-#### dm.loopmetadatasize

-Specifies the size to use when creating the loopback file for the "metadadata"

-device which is used for the thin pool. The default size is 2G. Note that the

-file is sparse, so it will not initially take up this much space.

+This value affects the system-wide "base" empty filesystem that may already

+be initialized and inherited by pulled images. Typically, a change to this

+value requires additional steps to take effect:

+

+        $ sudo service docker stop

+        $ sudo rm -rf /var/lib/docker

+        $ sudo service docker start

+

+Example use: `docker -d --storage-opt dm.basesize=20G`

 #### dm.fs

-Specifies the filesystem type to use for the base device. The supported

-options are "ext4" and "xfs". The default is "ext4"

+

+Specifies the filesystem type to use for the base device. The

+supported options are `ext4` and `xfs`. The default is `ext4`.

+

+Example use: `docker -d --storage-opt dm.fs=xfs`

 #### dm.mkfsarg

+

 Specifies extra mkfs arguments to be used when creating the base device.

+Example use: `docker -d --storage-opt "dm.mkfsarg=-O ^has_journal"`

+

 #### dm.mountopt

+

 Specifies extra mount options used when mounting the thin devices.

+Example use: `docker -d --storage-opt dm.mountopt=nodiscard`

+

+#### dm.use_deferred_removal

+

+Enables use of deferred device removal if `libdm` and the kernel driver

+support the mechanism.

+

+Deferred device removal means that if device is busy when devices are

+being removed/deactivated, then a deferred removal is scheduled on

+device. And devices automatically go away when last user of the device

+exits.

+

+For example, when a container exits, its associated thin device is removed. If

+that device has leaked into some other mount namespace and can't be removed,

+the container exit still succeeds and this option causes the system to schedule

+the device for deferred removal. It does not wait in a loop trying to remove a busy

+device.

+

+Example use: `docker -d --storage-opt dm.use_deferred_removal=true`

+

+#### dm.loopdatasize

+

+**Note**: This option configures devicemapper loopback, which should not be used in production.

+

+Specifies the size to use when creating the loopback file for the

+"data" device which is used for the thin pool. The default size is

+100G. The file is sparse, so it will not initially take up

+this much space.

+

+Example use: `docker -d --storage-opt dm.loopdatasize=200G`

+

+#### dm.loopmetadatasize

+

+**Note**: This option configures devicemapper loopback, which should not be used in production.

+

+Specifies the size to use when creating the loopback file for the

+"metadadata" device which is used for the thin pool. The default size

+is 2G. The file is sparse, so it will not initially take up

+this much space.

+

+Example use: `docker -d --storage-opt dm.loopmetadatasize=4G`

+

 #### dm.datadev

-Specifies a custom blockdevice to use for data for the thin pool.

-If using a block device for device mapper storage, ideally both datadev and

-metadatadev should be specified to completely avoid using the loopback device.

+(Deprecated, use `dm.thinpooldev`)

-#### dm.metadatadev

-Specifies a custom blockdevice to use for metadata for the thin pool.

+Specifies a custom blockdevice to use for data for a

+Docker-managed thin pool.  It is better to use `dm.thinpooldev` - see

+the documentation for it above for discussion of the advantages.

-For best performance the metadata should be on a different spindle than the

-data, or even better on an SSD.

+#### dm.metadatadev

-If setting up a new metadata pool it is required to be valid. This can be

-achieved by zeroing the first 4k to indicate empty metadata, like this:

+(Deprecated, use `dm.thinpooldev`)

-    dd if=/dev/zero of=/dev/metadata_dev bs=4096 count=1

+Specifies a custom blockdevice to use for metadata for a

+Docker-managed thin pool.  See `dm.datadev` for why this is

+deprecated.

 #### dm.blocksize

-Specifies a custom blocksize to use for the thin pool. The default blocksize

-is 64K.

+

+Specifies a custom blocksize to use for the thin pool.  The default

+blocksize is 64K.

+

+Example use: `docker -d --storage-opt dm.blocksize=512K`

 #### dm.blkdiscard

-Enables or disables the use of blkdiscard when removing devicemapper devices.

-This is enabled by default (only) if using loopback devices and is required to

-resparsify the loopback file on image/container removal.

-

-Disabling this on loopback can lead to *much* faster container removal times,

-but will prevent the space used in `/var/lib/docker` directory from being returned to

-the system for other use when containers are removed.

-

-# EXAMPLES

-Launching docker daemon with *devicemapper* backend with particular block devices

-for data and metadata:

-

-    docker -d -s=devicemapper \

-      --storage-opt dm.datadev=/dev/vdb \

-      --storage-opt dm.metadatadev=/dev/vdc \

-      --storage-opt dm.basesize=20G

+

+Enables or disables the use of `blkdiscard` when removing devicemapper

+devices.  This is disabled by default due to the additional latency,

+but as a special case with loopback devices it will be enabled, in

+order to re-sparsify the loopback file on image/container removal.

+

+Disabling this on loopback can lead to *much* faster container removal

+times, but it also prevents the space used in `/var/lib/docker` directory

+from being returned to the system for other use when containers are

+removed.

+

+Example use: `docker -d --storage-opt dm.blkdiscard=false`

+

+#### dm.override_udev_sync_check

+

+By default, the devicemapper backend attempts to synchronize with the

+`udev` device manager for the Linux kernel.  This option allows

+disabling that synchronization, to continue even though the

+configuration may be buggy.

+

+To view the `udev` sync support of a Docker daemon that is using the

+`devicemapper` driver, run:

+

+        $ docker info

+	[...]

+	 Udev Sync Supported: true

+	[...]

+

+When `udev` sync support is `true`, then `devicemapper` and `udev` can

+coordinate the activation and deactivation of devices for containers.

+

+When `udev` sync support is `false`, a race condition occurs between

+the`devicemapper` and `udev` during create and cleanup. The race

+condition results in errors and failures. (For information on these

+failures, see

+[docker#4036](https://github.com/docker/docker/issues/4036))

+

+To allow the `docker` daemon to start, regardless of whether `udev` sync is

+`false`, set `dm.override_udev_sync_check` to true:

+

+        $ docker -d --storage-opt dm.override_udev_sync_check=true

+

+When this value is `true`, the driver continues and simply warns you

+the errors are happening.

+

+**Note**: The ideal is to pursue a `docker` daemon and environment

+that does support synchronizing with `udev`. For further discussion on

+this topic, see

+[docker#4036](https://github.com/docker/docker/issues/4036).

+Otherwise, set this flag for migrating existing Docker daemons to a

+daemon with a supported environment.

 # EXEC DRIVER OPTIONS