@@ -195,17 +195,17 @@ options for `zfs` start with `zfs`.

      to create and manage the thin-pool volume. This volume is then handed to Docker

      to exclusively create snapshot volumes needed for images and containers.

-     Managing the thin-pool outside of Docker makes for the most feature-rich

+     Managing the thin-pool outside of Engine makes for the most feature-rich

      method of having Docker utilize device mapper thin provisioning as the

-     backing storage for Docker's containers. The highlights of the lvm-based

+     backing storage for Docker containers. The highlights of the lvm-based

      thin-pool management feature include: automatic or interactive thin-pool

      resize support, dynamically changing thin-pool features, automatic thinp

      metadata checking when lvm activates the thin-pool, etc.

-     As a fallback if no thin pool is provided, loopback files will be

+     As a fallback if no thin pool is provided, loopback files are

      created. Loopback is very slow, but can be used without any

      pre-configuration of storage. It is strongly recommended that you do

-     not use loopback in production. Ensure your Docker daemon has a

+     not use loopback in production. Ensure your Engine daemon has a

      `--storage-opt dm.thinpooldev` argument provided.

      Example use:

@@ -441,29 +441,33 @@ options for `zfs` start with `zfs`.

 *  `dm.min_free_space`

-    Specifies the min free space percent in thin pool require for new device

+    Specifies the min free space percent in a thin pool require for new device

     creation to succeed. This check applies to both free data space as well

     as free metadata space. Valid values are from 0% - 99%. Value 0% disables

-    free space checking logic. If user does not specify a value for this optoin,

-    then default value for this option is 10%.

+    free space checking logic. If user does not specify a value for this option,

+    the Engine uses a default value of 10%.

-    Whenever a new thin pool device is created (during docker pull or

-    during container creation), docker will check minimum free space is

-    available as specified by this parameter. If that is not the case, then

-    device creation will fail and docker operation will fail.

+    Whenever a new a thin pool device is created (during `docker pull` or during

+    container creation), the Engine checks if the minimum free space is

+    available. If sufficient space is unavailable, then device creation fails

+    and any relevant `docker` operation fails.

-    One will have to create more free space in thin pool to recover from the

-    error. Either delete some of the images and containers from thin pool and

-    create free space or add more storage to thin pool.

+    To recover from this error, you must create more free space in the thin pool

+    to recover from the error. You can create free space by deleting some images

+    and containers from the thin pool. You can also add more storage to the thin

+    pool.

-    For lvm thin pool, one can add more storage to volume group container thin

-    pool and that should automatically resolve it. If loop devices are being

-    used, then stop docker, grow the size of loop files and restart docker and

-    that should resolve the issue.

+    To add more space to a LVM (logical volume management) thin pool, just add

+    more storage to the volume group container thin pool; this should automatically

+    resolve any errors. If your configuration uses loop devices, then stop the

+    Engine daemon, grow the size of loop files and restart the daemon to resolve

+    the issue.

     Example use:

-        $ docker daemon --storage-opt dm.min_free_space=10%

+    ```bash

+    $ docker daemon --storage-opt dm.min_free_space=10%

+    ```

 Currently supported options of `zfs`:

@@ -17,7 +17,9 @@ for image and container management. This article refers to the Device Mapper

 storage driver as `devicemapper`, and the kernel framework as `Device Mapper`.

->**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires that you use the `devicemapper` storage driver.

+>**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL

+and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires

+that you use the `devicemapper` storage driver.

 ## An alternative to AUFS

@@ -206,103 +208,155 @@ mode uses block devices to create the thin pool. The following procedure shows

 you how to configure a Docker host to use the `devicemapper` storage driver in

 a `direct-lvm` configuration.

-> **Caution:** If you have already run the Docker daemon on your Docker host

+> **Caution:** If you have already run the Engine daemon on your Docker host

 > and have images you want to keep, `push` them Docker Hub or your private

 > Docker Trusted Registry before attempting this procedure.

 The procedure below will create a 90GB data volume and 4GB metadata volume to

 use as backing for the storage pool. It assumes that you have a spare block

-device at `/dev/xvdf` with enough free space to complete the task. The device

+device at `/dev/sdd` with enough free space to complete the task. The device

 identifier and volume sizes may be be different in your environment and you

-should substitute your own values throughout the procedure. The procedure also

-assumes that the Docker daemon is in the `stopped` state.

+should substitute your own values throughout the procedure.

-1. Log in to the Docker host you want to configure and stop the Docker daemon.

+The procedure also assumes that the Engine daemon is in the `stopped` state.

+Any existing images or data are lost by this process.

-2. If it exists, delete your existing image store by removing the

-`/var/lib/docker` directory.

+1. Log in to the Docker host you want to configure.

+2. If it is running, stop the Engine daemon.

+3. Install the logical volume management version 2.

-        $ sudo rm -rf /var/lib/docker

+    ```bash

+    $ yum install lvm2

+    ```

+4. Create a physical volume replacing `/dev/sdd` with your block device.

-3. Create an LVM physical volume (PV) on your spare block device using the

-`pvcreate` command.

+    ```bash

+    $ pvcreate /dev/sdd

+  ```

-        $ sudo pvcreate /dev/xvdf

-        Physical volume `/dev/xvdf` successfully created

+5. Create a 'docker' volume group.

-    The device identifier may be different on your system. Remember to

-substitute your value in the command above.

+    ```bash

+    $ vgcreate docker /dev/sdd

+    ```

-4. Create a new volume group (VG) called `vg-docker` using the PV created in

-the previous step.

+6. Create a thin pool named `thinpool`.

-        $ sudo vgcreate vg-docker /dev/xvdf

-        Volume group `vg-docker` successfully created

+    In this example, the data logical is 95% of the 'docker' volume group size.

+    Leaving this free space allows for auto expanding of either the data or

+    metadata if space runs low as a temporary stopgap.

-5. Create a new 90GB logical volume (LV) called `data` from space in the

-`vg-docker` volume group.

+    ```bash

+   $ lvcreate --wipesignatures y -n thinpool docker -l 95%VG

+   $ lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG

+   ```

-        $ sudo lvcreate -L 90G -n data vg-docker

-        Logical volume `data` created.

+7. Convert the pool to a thin pool.

-    The command creates an LVM logical volume called `data` and an associated

-block device file at `/dev/vg-docker/data`. In a later step, you instruct the

-`devicemapper` storage driver to use this block device to store image and

-container data.

+    ```bash

+    $ lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta

+    ```

-    If you receive a signature detection warning, make sure you are working on

-the correct devices before continuing. Signature warnings indicate that the

-device you're working on is currently in use by LVM or has been used by LVM in

-the past.

+8. Configure autoextension of thin pools via an `lvm` profile.

-6. Create a new logical volume (LV) called `metadata` from space in the

-`vg-docker` volume group.

+    ```bash

+    $ vi /etc/lvm/profile/docker-thinpool.profile

+    ```

-        $ sudo lvcreate -L 4G -n metadata vg-docker

-        Logical volume `metadata` created.

+9. Specify 'thin_pool_autoextend_threshold' value.

-    This creates an LVM logical volume called `metadata` and an associated

-block device file at `/dev/vg-docker/metadata`. In the next step you instruct

-the `devicemapper` storage driver to use this block device to store image and

-container metadata.

+    The value should be the percentage of space used before `lvm` attempts

+    to autoextend the available space (100 = disabled).

-7. Start the Docker daemon with the `devicemapper` storage driver and the

-`--storage-opt` flags.

+    ```

+    thin_pool_autoextend_threshold = 80

+    ```

-    The `data` and `metadata` devices that you pass to the `--storage-opt`

-options were created in the previous steps.

+10. Modify the `thin_pool_autoextend_percent` for when thin pool autoextension occurs.

-          $ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata &

-          [1] 2163

-          [root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)

-          INFO[0027] Option DefaultDriver: bridge

-          INFO[0027] Option DefaultNetwork: bridge

-          <output truncated>

-          INFO[0027] Daemon has completed initialization

-          INFO[0027] Docker daemon commit=1b09a95-unsupported graphdriver=aufs version=1.11.0-dev

+    The value's setting is the perentage of space to increase the thin pool (100 =

+    disabled)

-    It is also possible to set the `--storage-driver` and `--storage-opt` flags

- in the Docker config file and start the daemon normally using the `service` or

- `systemd` commands.

+    ```

+    thin_pool_autoextend_percent = 20

+    ```

-8. Use the `docker info` command to verify that the daemon is using `data` and

-`metadata` devices you created.

+11. Check your work, your `docker-thinpool.profile` file should appear similar to the following:

-        $ sudo docker info

-        INFO[0180] GET /v1.20/info

-        Containers: 0

-        Images: 0

-        Storage Driver: devicemapper

-         Pool Name: docker-202:1-1032-pool

-         Pool Blocksize: 65.54 kB

-         Backing Filesystem: xfs

-         Data file: /dev/vg-docker/data

-         Metadata file: /dev/vg-docker/metadata

-        [...]

+    An example `/etc/lvm/profile/docker-thinpool.profile` file:

+

+    ```

+     activation {

+         thin_pool_autoextend_threshold=80

+         thin_pool_autoextend_percent=20

+     }

+     ```

+

+12. Apply your new lvm profile

+

+    ```bash

+    $ lvchange --metadataprofile docker-thinpool docker/thinpool

+  ```

+

+13. Verify the `lv` is monitored.

+

+    ```bash

+    $ lvs -o+seg_monitor

+    ```

+

+14. If Engine was previously started, clear your graph driver directory.

+

+    Clearing your graph driver removes any images and containers in your Docker

+    installation.

+

+    ```bash

+    $ rm -rf /var/lib/docker/*

+    ```

+

+14. Configure the Engine daemon with specific devicemapper options.

+

+    There are two ways to do this. You can set options on the commmand line if you start the daemon there:

+

+    ```bash

+    --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true

+    ```

+

+    You can also set them for startup in the `daemon.json` configuration, for example:

+

+    ```json

+     {

+             "storage-driver": "devicemapper",

+             "storage-opts": [

+                     "dm.thinpooldev=/dev/mapper/docker-thinpool",

+                     "dm.use_deferred_removal=true"

+             ]

+     }

+    ```

+15. Start the Engine daemon.

+

+    ```bash

+    $ systemctl start docker

+    ```

+

+After you start the Engine daemon, ensure you monitor your thin pool and volume

+group free space. While the volume group will auto-extend, it can still fill

+up. To monitor logical volumes, use `lvs` without options or `lvs -a` to see tha

+data and metadata sizes. To monitor volume group free space, use the `vgs` command.

+

+Logs can show the auto-extension of the thin pool when it hits the threshold, to

+view the logs use:

+

+```bash

+journalctl -fu dm-event.service

+```

+

+If you run into repeated problems with thin pool, you can use the

+`dm.min_free_space` option to tune the Engine behavior. This value ensures that

+operations fail with a warning when the free space is at or near the minimum.

+For information, see <a

+href="https://docs.docker.com/engine/reference/commandline/daemon/#storage-driver-options"

+target="_blank">the storage driver options in the Engine daemon reference</a>.

-    The output of the command above shows the storage driver as `devicemapper`.

- The last two lines also confirm that the correct devices are being used for

-the `Data file` and the `Metadata file`.

 ### Examine devicemapper structures on the host

@@ -336,7 +390,7 @@ Docker-MAJ:MIN-INO-pool

 Because Device Mapper operates at the block level it is more difficult to see

 diffs between image layers and containers. Docker 1.10 and later no longer

-matches image layer IDs with directory names in `/var/lib/docker`.  However,

+matches image layer IDs with directory names in `/var/lib/docker`. However,

 there are two key directories. The `/var/lib/docker/devicemapper/mnt` directory

  contains the mount points for image and container layers. The

 `/var/lib/docker/devicemapper/metadata`directory contains one file for every

@@ -410,3 +464,4 @@ data volumes.

 * [Select a storage driver](selectadriver.md)

 * [AUFS storage driver in practice](aufs-driver.md)

 * [Btrfs storage driver in practice](btrfs-driver.md)

+* [daemon reference](../../reference/commandline/daemon#storage-driver-options)

@@ -269,19 +269,27 @@ allocating images and container snapshots.

 Specifies a custom block storage device to use for the thin pool.

-If using a block device for device mapper storage, it is best to use

-`lvm` to create and manage the thin-pool volume. This volume is then

-handed to Docker to create snapshot volumes needed for images and

-containers.

+If using a block device for device mapper storage, it is best to use `lvm`

+to create and manage the thin-pool volume. This volume is then handed to Docker

+to exclusively create snapshot volumes needed for images and containers.

-Managing the thin-pool outside of Docker makes for the most feature-rich method

-of having Docker utilize device mapper thin provisioning as the backing storage

-for Docker's containers. The highlights of the LVM-based thin-pool management

-feature include: automatic or interactive thin-pool resize support, dynamically

-changing thin-pool features, automatic thinp metadata checking when lvm activates

-the thin-pool, etc.

+Managing the thin-pool outside of Engine makes for the most feature-rich

+method of having Docker utilize device mapper thin provisioning as the

+backing storage for Docker containers. The highlights of the lvm-based

+thin-pool management feature include: automatic or interactive thin-pool

+resize support, dynamically changing thin-pool features, automatic thinp

+metadata checking when lvm activates the thin-pool, etc.

-Example use: `docker daemon --storage-opt dm.thinpooldev=/dev/mapper/thin-pool`

+As a fallback if no thin pool is provided, loopback files are

+created. Loopback is very slow, but can be used without any

+pre-configuration of storage. It is strongly recommended that you do

+not use loopback in production. Ensure your Engine daemon has a

+`--storage-opt dm.thinpooldev` argument provided.

+

+Example use:

+

+   $ docker daemon \

+         --storage-opt dm.thinpooldev=/dev/mapper/thin-pool

 #### dm.basesize

@@ -471,27 +479,29 @@ daemon with a supported environment.

 #### dm.min_free_space

-Specifies the min free space percent in thin pool require for new device

+Specifies the min free space percent in a thin pool require for new device

 creation to succeed. This check applies to both free data space as well

 as free metadata space. Valid values are from 0% - 99%. Value 0% disables

-free space checking logic. If user does not specify a value for this optoin,

-then default value for this option is 10%.

-

-Whenever a new thin pool device is created (during docker pull or

-during container creation), docker will check minimum free space is

-available as specified by this parameter. If that is not the case, then

-device creation will fail and docker operation will fail.

-

-One will have to create more free space in thin pool to recover from the

-error. Either delete some of the images and containers from thin pool and

-create free space or add more storage to thin pool.

-

-For lvm thin pool, one can add more storage to volume group container thin

-pool and that should automatically resolve it. If loop devices are being

-used, then stop docker, grow the size of loop files and restart docker and

-that should resolve the issue.

-

-Example use: `docker daemon --storage-opt dm.min_free_space=10%`

+free space checking logic. If user does not specify a value for this option,

+the Engine uses a default value of 10%.

+

+Whenever a new a thin pool device is created (during `docker pull` or during

+container creation), the Engine checks if the minimum free space is

+available. If the space is unavailable, then device creation fails and any

+relevant `docker` operation fails.

+

+To recover from this error, you must create more free space in the thin pool to

+recover from the error. You can create free space by deleting some images

+and containers from tge thin pool. You can also add

+more storage to the thin pool.

+

+To add more space to an LVM (logical volume management) thin pool, just add

+more storage to the  group container thin pool; this should automatically

+resolve any errors. If your configuration uses loop devices, then stop the

+Engine daemon, grow the size of loop files and restart the daemon to resolve

+the issue.

+

+Example use:: `docker daemon --storage-opt dm.min_free_space=10%`

 ## ZFS options