@@ -10,184 +10,203 @@ parent = "engine_driver"

 # Docker and AUFS in practice

-AUFS was the first storage driver in use with Docker. As a result, it has a long and close history with Docker, is very stable, has a lot of real-world deployments, and has strong community support.  AUFS has several features that make it a good choice for Docker. These features enable:

+AUFS was the first storage driver in use with Docker. As a result, it has a

+long and close history with Docker, is very stable, has a lot of real-world

+deployments, and has strong community support. AUFS has several features that

+make it a good choice for Docker. These features enable:

 - Fast container startup times.

 - Efficient use of storage.

 - Efficient use of memory.

-Despite its capabilities and long history with Docker, some Linux distributions do not support AUFS. This is usually because AUFS is not included in the mainline (upstream) Linux kernel.

+Despite its capabilities and long history with Docker, some Linux distributions

+ do not support AUFS. This is usually because AUFS is not included in the

+mainline (upstream) Linux kernel.

-The following sections examine some AUFS features and how they relate to Docker.

+The following sections examine some AUFS features and how they relate to

+Docker.

 ## Image layering and sharing with AUFS

-AUFS is a *unification filesystem*. This means that it takes multiple directories on a single Linux host, stacks them on top of each other, and provides a single unified view. To achieve this, AUFS uses *union mount*.

+AUFS is a *unification filesystem*. This means that it takes multiple

+directories on a single Linux host, stacks them on top of each other, and

+provides a single unified view. To achieve this, AUFS uses a *union mount*.

-AUFS stacks multiple directories and exposes them as a unified view through a single mount point. All of the directories in the stack, as well as the union mount point, must all exist on the same Linux host. AUFS refers to each directory that it stacks as a *branch*.

+AUFS stacks multiple directories and exposes them as a unified view through a

+single mount point. All of the directories in the stack, as well as the union

+mount point, must all exist on the same Linux host. AUFS refers to each

+directory that it stacks as a *branch*.

-Within Docker, AUFS union mounts enable image layering. The AUFS storage driver implements Docker image layers using this union mount system. AUFS branches correspond to Docker image layers. The diagram below shows a Docker container based on the `ubuntu:latest` image.

+Within Docker, AUFS union mounts enable image layering. The AUFS storage driver

+ implements Docker image layers using this union mount system. AUFS branches

+correspond to Docker image layers. The diagram below shows a Docker container

+based on the `ubuntu:latest` image.

 ![](images/aufs_layers.jpg)

-This diagram shows the relationship between the Docker image layers and the AUFS branches (directories) in `/var/lib/docker/aufs`. Each image layer  and the container layer correspond to an AUFS branch (directory) in the Docker host's local storage area. The union mount point gives the unified view of all layers.

+This diagram shows that each image layer, and the container layer, is

+represented in the Docker hosts filesystem as a directory under

+`/var/lib/docker/`. The union mount point provides the unified view of all

+layers. As of Docker 1.10, image layer IDs do not correspond to the names of

+the directories that contain their data.

-AUFS also supports the copy-on-write technology (CoW). Not all storage drivers do.

+AUFS also supports the copy-on-write technology (CoW). Not all storage drivers

+do.

 ## Container reads and writes with AUFS

-Docker leverages AUFS CoW technology to enable image sharing and minimize the use of disk space. AUFS works at the file level. This means that all AUFS CoW operations copy entire files - even if only a small part of the file is being modified. This behavior can have a noticeable impact on container performance, especially if the files being copied are large, below a lot of image layers, or the CoW operation must search a deep directory tree.

-

-Consider, for example, an application running in a container needs to add a single new value to a large key-value store (file). If this is the first time the file is modified it does not yet exist in the container's top writable layer. So, the CoW must *copy up* the file from the underlying image. The AUFS storage driver searches each image layer for the file. The search order is from top to bottom. When it is found, the entire file is *copied up* to the container's top writable layer. From there, it can be opened and modified.

-

-Larger files obviously take longer to *copy up* than smaller files, and files that exist in lower image layers take longer to locate than those in higher layers. However, a *copy up* operation only occurs once per file on any given container. Subsequent reads and writes happen against the file's copy already *copied-up* to the container's top layer.

-

+Docker leverages AUFS CoW technology to enable image sharing and minimize the

+use of disk space. AUFS works at the file level. This means that all AUFS CoW

+operations copy entire files - even if only a small part of the file is being

+modified. This behavior can have a noticeable impact on container performance,

+ especially if the files being copied are large, below a lot of image layers,

+or the CoW operation must search a deep directory tree.

+

+Consider, for example, an application running in a container needs to add a

+single new value to a large key-value store (file). If this is the first time

+the file is modified, it does not yet exist in the container's top writable

+layer. So, the CoW must *copy up* the file from the underlying image. The AUFS

+storage driver searches each image layer for the file. The search order is from

+ top to bottom. When it is found, the entire file is *copied up* to the

+container's top writable layer. From there, it can be opened and modified.

+

+Larger files obviously take longer to *copy up* than smaller files, and files

+that exist in lower image layers take longer to locate than those in higher

+layers. However, a *copy up* operation only occurs once per file on any given

+container. Subsequent reads and writes happen against the file's copy already

+*copied-up* to the container's top layer.

 ## Deleting files with the AUFS storage driver

 The AUFS storage driver deletes a file from a container by placing a *whiteout

 file* in the container's top layer. The whiteout file effectively obscures the

-existence of the file in image's lower, read-only layers.  The simplified

+existence of the file in the read-only image layers below. The simplified

 diagram below shows a container based on an image with three image layers.

 ![](images/aufs_delete.jpg)

 The `file3` was deleted from the container. So, the AUFS storage driver  placed

 a whiteout file in the container's top layer. This whiteout file effectively

-"deletes"  `file3` from the container by obscuring any of the original file's

-existence in the image's read-only base layer. Of course, the image could have

-been in any of the other layers instead or in addition depending on how the

-layers are built.

+"deletes" `file3` from the container by obscuring any of the original file's

+existence in the image's read-only layers. This works the same no matter which

+of the image's read-only layers the file exists in.

 ## Configure Docker with AUFS

-You can only use the AUFS storage driver on Linux systems with AUFS installed. Use the following command to determine if your system supports AUFS.

+You can only use the AUFS storage driver on Linux systems with AUFS installed.

+Use the following command to determine if your system supports AUFS.

-```bash

-$ grep aufs /proc/filesystems

-nodev   aufs

-```

+    $ grep aufs /proc/filesystems

+    nodev   aufs

-This output indicates the system supports AUFS.  Once you've verified your

+This output indicates the system supports AUFS. Once you've verified your

 system supports AUFS, you can must instruct the Docker daemon to use it. You do

 this from the command line with the `docker daemon` command:

-```bash

-$ sudo docker daemon --storage-driver=aufs &

-```

+    $ sudo docker daemon --storage-driver=aufs &

+

 Alternatively, you can edit the Docker config file and add the

 `--storage-driver=aufs` option to the `DOCKER_OPTS` line.

-```bash

-# Use DOCKER_OPTS to modify the daemon startup options.

-DOCKER_OPTS="--storage-driver=aufs"

-```

+    # Use DOCKER_OPTS to modify the daemon startup options.

+    DOCKER_OPTS="--storage-driver=aufs"

-Once your daemon is running, verify the storage driver with the `docker info` command.

+Once your daemon is running, verify the storage driver with the `docker info`

+command.

-```bash

-$ sudo docker info

-Containers: 1

-Images: 4

-Storage Driver: aufs

- Root Dir: /var/lib/docker/aufs

- Backing Filesystem: extfs

- Dirs: 6

- Dirperm1 Supported: false

-Execution Driver: native-0.2

-...output truncated...

-```

+    $ sudo docker info

+    Containers: 1

+    Images: 4

+    Storage Driver: aufs

+     Root Dir: /var/lib/docker/aufs

+     Backing Filesystem: extfs

+     Dirs: 6

+     Dirperm1 Supported: false

+    Execution Driver: native-0.2

+    ...output truncated...

-The output above shows that the Docker daemon is running the AUFS storage driver on top of an existing ext4 backing filesystem.

+The output above shows that the Docker daemon is running the AUFS storage

+driver on top of an existing `ext4` backing filesystem.

 ## Local storage and AUFS

-As the `docker daemon` runs with the AUFS driver, the driver stores images and containers on within the Docker host's local storage area in the `/var/lib/docker/aufs` directory.

+As the `docker daemon` runs with the AUFS driver, the driver stores images and

+containers within the Docker host's local storage area under

+`/var/lib/docker/aufs/`.

 ### Images

 Image layers and their contents are stored under

-`/var/lib/docker/aufs/diff/<image-id>` directory. The contents of an image

-layer in this location includes all the files and directories belonging in that

-image layer.

+`/var/lib/docker/aufs/diff/`. With Docker 1.10 and higher, image layer IDs do

+not correspond to directory names.

 The `/var/lib/docker/aufs/layers/` directory contains metadata about how image

 layers are stacked. This directory contains one file for every image or

-container layer on the Docker host. Inside each file are the image layers names

-that exist below it. The diagram below shows an image with 4 layers.

-

-![](images/aufs_metadata.jpg)

-

-Inspecting the contents of the file relating to the top layer of the image

-shows the three image layers below it. They are listed in the order they are

-stacked.

+container layer on the Docker host (though file names no longer match image

+layer IDs). Inside each file are the names of the directories that exist below

+it in the stack

-```bash

-$ cat /var/lib/docker/aufs/layers/91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c

+The command below shows the contents of a metadata file in

+`/var/lib/docker/aufs/layers/` that lists the the three directories that are

+stacked below it in the union mount. Remember, these directory names do no map

+to image layer IDs with Docker 1.10 and higher.

-d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82

-

-c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87

-

-d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391

-```

+    $ cat /var/lib/docker/aufs/layers/91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c

+    d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82

+    c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87

+    d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391

 The base layer in an image has no image layers below it, so its file is empty.

 ### Containers

-Running containers are mounted at locations in the

-`/var/lib/docker/aufs/mnt/<container-id>` directory. This is the AUFS union

-mount point that exposes the container and all underlying image layers as a

-single unified view. If a container is not running, its directory still exists

-but is empty. This is because containers are only mounted when they are running.

+Running containers are mounted below `/var/lib/docker/aufs/mnt/<container-id>`.

+ This is where the AUFS union mount point that exposes the container and all

+underlying image layers as a single unified view exists. If a container is not

+running, it still has a directory here but it is empty. This is because AUFS

+only mounts a container when it is running. With Docker 1.10 and higher,

+ container IDs no longer correspond to directory names under

+`/var/lib/docker/aufs/mnt/<container-id>`.

 Container metadata and various config files that are placed into the running

-container are stored in `/var/lib/containers/<container-id>`. Files in this

-directory exist for all containers on the system, including ones that are

-stopped. However, when a container is running the container's log files are also

-in this directory.

-

-A container's thin writable layer is stored under

-`/var/lib/docker/aufs/diff/<container-id>`. This directory is stacked by AUFS as

-the containers top writable layer and is where all changes to the container are

-stored. The directory exists even if the container is stopped. This means that

-restarting a container will not lose changes made to it. Once a container is

-deleted this directory is deleted.

-

-Information about which image layers are stacked below a container's top

-writable layer is stored in the following file

-`/var/lib/docker/aufs/layers/<container-id>`. The command below shows that the

-container with ID `b41a6e5a508d` has 4 image layers below it:

-

-```bash

-$ cat /var/lib/docker/aufs/layers/b41a6e5a508dfa02607199dfe51ed9345a675c977f2cafe8ef3e4b0b5773404e-init

-91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c

-d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82

-c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87

-d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391

-```

-

-The image layers are shown in order. In the output above, the layer starting

-with image ID "d3a1..." is the image's base  layer. The image layer starting

-with "91e5..." is the image's topmost layer.

-

+container are stored in `/var/lib/docker/containers/<container-id>`. Files in

+this directory exist for all containers on the system, including ones that are

+stopped. However, when a container is running the container's log files are

+also in this directory.

+

+A container's thin writable layer is stored in a directory under

+`/var/lib/docker/aufs/diff/`. With Docker 1.10 and higher, container IDs no

+longer correspond to directory names. However, the containers thin writable

+layer still exists under here and is stacked by AUFS as the top writable layer

+and is where all changes to the container are stored. The directory exists even

+ if the container is stopped. This means that restarting a container will not

+lose changes made to it. Once a container is deleted, it's thin writable layer

+in this directory is deleted.

 ## AUFS and Docker performance

 To summarize some of the performance related aspects already mentioned:

-- The AUFS storage driver is a good choice for PaaS and other similar use-cases where container density is important. This is because AUFS efficiently shares images between multiple running containers, enabling fast container start times and minimal use of disk space.

-

-- The underlying mechanics of how AUFS shares files between image layers and containers uses the systems page cache very efficiently.

-

-- The AUFS storage driver can introduce significant latencies into container write performance. This is because the first time a container writes to any file, the file has be located and copied into the containers top writable layer. These latencies increase and are compounded when these files exist below many image layers and the files themselves are large.

-

-One final point. Data volumes provide the best and most predictable performance.

-This is because they bypass the storage driver and do not incur any of the

-potential overheads introduced by thin provisioning and copy-on-write. For this

-reason, you may want to place heavy write workloads on data volumes.

+- The AUFS storage driver is a good choice for PaaS and other similar use-cases

+ where container density is important. This is because AUFS efficiently shares

+images between multiple running containers, enabling fast container start times

+ and minimal use of disk space.

+

+- The underlying mechanics of how AUFS shares files between image layers and

+containers uses the systems page cache very efficiently.

+

+- The AUFS storage driver can introduce significant latencies into container

+write performance. This is because the first time a container writes to any

+file, the file has be located and copied into the containers top writable

+layer. These latencies increase and are compounded when these files exist below

+ many image layers and the files themselves are large.

+

+One final point. Data volumes provide the best and most predictable

+performance. This is because they bypass the storage driver and do not incur 

+any of the potential overheads introduced by thin provisioning and

+copy-on-write. For this reason, you may want to place heavy write workloads on

+data volumes.

 ## Related information

@@ -13,126 +13,118 @@ parent = "engine_driver"

 Btrfs is a next generation copy-on-write filesystem that supports many advanced

 storage technologies that make it a good fit for Docker. Btrfs is included in

 the mainline Linux kernel and its on-disk-format is now considered stable.

-However, many of its features are still under heavy development and users should

-consider it a fast-moving target.

+However, many of its features are still under heavy development and users 

+should consider it a fast-moving target.

 Docker's `btrfs` storage driver leverages many Btrfs features for image and

-container management. Among these features are thin provisioning, copy-on-write,

-and snapshotting.

+container management. Among these features are thin provisioning, 

+copy-on-write, and snapshotting.

-This article refers to Docker's Btrfs storage driver as `btrfs` and the overall Btrfs Filesystem as Btrfs.

+This article refers to Docker's Btrfs storage driver as `btrfs` and the overall

+ Btrfs Filesystem as Btrfs.

 >**Note**: The [Commercially Supported Docker Engine (CS-Engine)](https://www.docker.com/compatibility-maintenance) does not currently support the `btrfs` storage driver.

 ## The future of Btrfs

-Btrfs has been long hailed as the future of Linux filesystems. With full support in the mainline Linux kernel, a stable on-disk-format, and active development with a focus on stability, this is now becoming more of a reality.

+Btrfs has been long hailed as the future of Linux filesystems. With full 

+support in the mainline Linux kernel, a stable on-disk-format, and active 

+development with a focus on stability, this is now becoming more of a reality.

-As far as Docker on the Linux platform goes, many people see the `btrfs` storage driver as a potential long-term replacement for the `devicemapper` storage driver. However, at the time of writing, the `devicemapper` storage driver should be considered safer, more stable, and more *production ready*. You should only consider the `btrfs` driver for production deployments if you understand it well and have existing experience with Btrfs.

+As far as Docker on the Linux platform goes, many people see the `btrfs` 

+storage driver as a potential long-term replacement for the `devicemapper` 

+storage driver. However, at the time of writing, the `devicemapper` storage 

+driver should be considered safer, more stable, and more *production ready*. 

+You should only consider the `btrfs` driver for production deployments if you 

+understand it well and have existing experience with Btrfs.

 ## Image layering and sharing with Btrfs

-Docker leverages Btrfs *subvolumes* and *snapshots* for managing the on-disk components of image and container layers.  Btrfs subvolumes look and feel like a normal Unix filesystem. As such, they can have their own internal directory structure that hooks into the wider Unix filesystem.

+Docker leverages Btrfs *subvolumes* and *snapshots* for managing the on-disk 

+components of image and container layers.  Btrfs subvolumes look and feel like 

+a normal Unix filesystem. As such, they can have their own internal directory 

+structure that hooks into the wider Unix filesystem.

-Subvolumes are natively copy-on-write and have space allocated to them on-demand

-from an underlying storage pool. They can also be nested and snapped. The

-diagram blow shows 4 subvolumes. 'Subvolume 2' and 'Subvolume 3' are nested,

-whereas 'Subvolume 4' shows its own internal directory tree.

+Subvolumes are natively copy-on-write and have space allocated to them 

+on-demand from an underlying storage pool. They can also be nested and snapped.

+ The diagram blow shows 4 subvolumes. 'Subvolume 2' and 'Subvolume 3' are 

+nested, whereas 'Subvolume 4' shows its own internal directory tree.

 ![](images/btfs_subvolume.jpg)

-Snapshots are a point-in-time read-write copy of an entire subvolume. They exist directly below the subvolume they were created from. You can create snapshots of snapshots as shown in the diagram below.

+Snapshots are a point-in-time read-write copy of an entire subvolume. They 

+exist directly below the subvolume they were created from. You can create 

+snapshots of snapshots as shown in the diagram below.

 ![](images/btfs_snapshots.jpg)

-Btfs allocates space to subvolumes and snapshots on demand from an underlying pool of storage. The unit of allocation is referred to as a *chunk* and *chunks* are normally ~1GB in size.

+Btfs allocates space to subvolumes and snapshots on demand from an underlying 

+pool of storage. The unit of allocation is referred to as a *chunk*, and 

+*chunks* are normally ~1GB in size.

-Snapshots are first-class citizens in a Btrfs filesystem. This means that they look, feel, and operate just like regular subvolumes. The technology required to create them is built directly into the Btrfs filesystem thanks to its native copy-on-write design. This means that Btrfs snapshots are space efficient with little or no performance overhead. The diagram below shows a subvolume and its snapshot sharing the same data.

+Snapshots are first-class citizens in a Btrfs filesystem. This means that they 

+look, feel, and operate just like regular subvolumes. The technology required 

+to create them is built directly into the Btrfs filesystem thanks to its 

+native copy-on-write design. This means that Btrfs snapshots are space 

+efficient with little or no performance overhead. The diagram below shows a 

+subvolume and its snapshot sharing the same data.

 ![](images/btfs_pool.jpg)

-Docker's `btrfs` storage driver stores every image layer and container in its own Btrfs subvolume or snapshot. The base layer of an image is stored as a subvolume whereas child image layers and containers are stored as snapshots. This is shown in the diagram below.

+Docker's `btrfs` storage driver stores every image layer and container in its 

+own Btrfs subvolume or snapshot. The base layer of an image is stored as a 

+subvolume whereas child image layers and containers are stored as snapshots. 

+This is shown in the diagram below.

 ![](images/btfs_container_layer.jpg)

-The high level process for creating images and containers on Docker hosts running the `btrfs` driver is as follows:

+The high level process for creating images and containers on Docker hosts 

+running the `btrfs` driver is as follows:

-1. The image's base layer is stored in a Btrfs subvolume under

+1. The image's base layer is stored in a Btrfs *subvolume* under

 `/var/lib/docker/btrfs/subvolumes`.

-    The image ID is used as the subvolume name. E.g., a base layer with image ID

-    "f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b" will be

-    stored in

-    `/var/lib/docker/btrfs/subvolumes/f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b`

+2. Subsequent image layers are stored as a Btrfs *snapshot* of the parent 

+layer's subvolume or snapshot.

-2. Subsequent image layers are stored as a Btrfs snapshot of the parent layer's subvolume or snapshot.

-

-    The diagram below shows a three-layer image. The base layer is a subvolume. Layer 1 is a snapshot of the base layer's subvolume. Layer 2 is a snapshot of Layer 1's snapshot.

+    The diagram below shows a three-layer image. The base layer is a subvolume.

+ Layer 1 is a snapshot of the base layer's subvolume. Layer 2 is a snapshot of 

+Layer 1's snapshot.

     ![](images/btfs_constructs.jpg)

+As of Docker 1.10, image layer IDs no longer correspond to directory names 

+under `/var/lib/docker/`.

+

 ## Image and container on-disk constructs

 Image layers and containers are visible in the Docker host's filesystem at

-`/var/lib/docker/btrfs/subvolumes/<image-id> OR <container-id>`. Directories for

+`/var/lib/docker/btrfs/subvolumes/`. However, as previously stated, directory 

+names no longer correspond to image layer IDs. That said, directories for

 containers are present even for containers with a stopped status. This is

 because the `btrfs` storage driver mounts a default, top-level subvolume at

 `/var/lib/docker/subvolumes`. All other subvolumes and snapshots exist below

 that as Btrfs filesystem objects and not as individual mounts.

-The following example shows a single Docker image with four image layers.

-

-```bash

-$ sudo docker images -a

-REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

-ubuntu              latest              0a17decee413        2 weeks ago         188.3 MB

-<none>              <none>              3c9a9d7cc6a2        2 weeks ago         188.3 MB

-<none>              <none>              eeb7cb91b09d        2 weeks ago         188.3 MB

-<none>              <none>              f9a9f253f610        2 weeks ago         188.1 MB

-```

-

-Each image layer exists as a Btrfs subvolume or snapshot with the same name as its image ID as illustrated by the `btrfs subvolume list` command shown below:

-

-```bash

-$ sudo btrfs subvolume list /var/lib/docker

-ID 257 gen 9 top level 5 path btrfs/subvolumes/f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b

-ID 258 gen 10 top level 5 path btrfs/subvolumes/eeb7cb91b09d5de9edb2798301aeedf50848eacc2123e98538f9d014f80f243c

-ID 260 gen 11 top level 5 path btrfs/subvolumes/3c9a9d7cc6a235eb2de58ca9ef3551c67ae42a991933ba4958d207b29142902b

-ID 261 gen 12 top level 5 path btrfs/subvolumes/0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751

-```

-

-Under the `/var/lib/docker/btrfs/subvolumes` directory, each of these subvolumes and snapshots are visible as a normal Unix directory:

-

-```bash

-$ ls -l /var/lib/docker/btrfs/subvolumes/

-total 0

-drwxr-xr-x 1 root root 132 Oct 16 14:44 0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751

-drwxr-xr-x 1 root root 132 Oct 16 14:44 3c9a9d7cc6a235eb2de58ca9ef3551c67ae42a991933ba4958d207b29142902b

-drwxr-xr-x 1 root root 132 Oct 16 14:44 eeb7cb91b09d5de9edb2798301aeedf50848eacc2123e98538f9d014f80f243c

-drwxr-xr-x 1 root root 132 Oct 16 14:44 f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b

-```

-

 Because Btrfs works at the filesystem level and not the block level, each image

-and container layer can be browsed in the filesystem using normal Unix commands.

-The example below shows a truncated output of an `ls -l` command against the

-image's top layer:

-

-```bash

-$ ls -l /var/lib/docker/btrfs/subvolumes/0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751/

-total 0

-drwxr-xr-x 1 root root 1372 Oct  9 08:39 bin

-drwxr-xr-x 1 root root    0 Apr 10  2014 boot

-drwxr-xr-x 1 root root  882 Oct  9 08:38 dev

-drwxr-xr-x 1 root root 2040 Oct 12 17:27 etc

-drwxr-xr-x 1 root root    0 Apr 10  2014 home

-...output truncated...

-```

+and container layer can be browsed in the filesystem using normal Unix 

+commands. The example below shows a truncated output of an `ls -l` command an 

+image layer:

+

+    $ ls -l /var/lib/docker/btrfs/subvolumes/0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751/

+    total 0

+    drwxr-xr-x 1 root root 1372 Oct  9 08:39 bin

+    drwxr-xr-x 1 root root    0 Apr 10  2014 boot

+    drwxr-xr-x 1 root root  882 Oct  9 08:38 dev

+    drwxr-xr-x 1 root root 2040 Oct 12 17:27 etc

+    drwxr-xr-x 1 root root    0 Apr 10  2014 home

+    ...output truncated...

 ## Container reads and writes with Btrfs

 A container is a space-efficient snapshot of an image. Metadata in the snapshot

-points to the actual data blocks in the storage pool. This is the same as with a

-subvolume. Therefore, reads performed against a snapshot are essentially the

+points to the actual data blocks in the storage pool. This is the same as with 

+a subvolume. Therefore, reads performed against a snapshot are essentially the

 same as reads performed against a subvolume. As a result, no performance

 overhead is incurred from the Btrfs driver.

@@ -145,28 +137,34 @@ new files to a container's snapshot operate at native Btrfs speeds.

 Updating an existing file in a container causes a copy-on-write operation

 (technically *redirect-on-write*). The driver leaves the original data and

 allocates new space to the snapshot. The updated data is written to this new

-space. Then, the driver updates the filesystem metadata in the snapshot to point

-to this new data. The original data is preserved in-place for subvolumes and

-snapshots further up the tree. This behavior is native to copy-on-write

+space. Then, the driver updates the filesystem metadata in the snapshot to 

+point to this new data. The original data is preserved in-place for subvolumes 

+and snapshots further up the tree. This behavior is native to copy-on-write

 filesystems like Btrfs and incurs very little overhead.

-With Btfs, writing and updating lots of small files can result in slow performance. More on this later.

+With Btfs, writing and updating lots of small files can result in slow 

+performance. More on this later.

 ## Configuring Docker with Btrfs

-The `btrfs` storage driver only operates on a Docker host where `/var/lib/docker` is mounted as a Btrfs filesystem. The following procedure shows  how to configure Btrfs on Ubuntu 14.04 LTS.

+The `btrfs` storage driver only operates on a Docker host where 

+`/var/lib/docker` is mounted as a Btrfs filesystem. The following procedure 

+shows  how to configure Btrfs on Ubuntu 14.04 LTS.

 ### Prerequisites

-If you have already used the Docker daemon on your Docker host and have images you want to keep, `push` them to Docker Hub or your private Docker Trusted Registry before attempting this procedure.

+If you have already used the Docker daemon on your Docker host and have images 

+you want to keep, `push` them to Docker Hub or your private Docker Trusted 

+Registry before attempting this procedure.

-Stop the Docker daemon. Then, ensure that you have a spare block device at `/dev/xvdb`. The device identifier may be different in your environment and you should substitute your own values throughout the procedure.

+Stop the Docker daemon. Then, ensure that you have a spare block device at 

+`/dev/xvdb`. The device identifier may be different in your environment and you

+ should substitute your own values throughout the procedure.

-The procedure also assumes your kernel has the appropriate Btrfs modules loaded. To verify this, use the following command:

+The procedure also assumes your kernel has the appropriate Btrfs modules 

+loaded. To verify this, use the following command:

-```bash

-$ cat /proc/filesystems | grep btrfs

-```

+    $ cat /proc/filesystems | grep btrfs

 ### Configure Btrfs on Ubuntu 14.04 LTS

@@ -181,7 +179,9 @@ Assuming your system meets the prerequisites, do the following:

 2. Create the Btrfs storage pool.

-    Btrfs storage pools are created with the `mkfs.btrfs` command. Passing multiple devices to the `mkfs.btrfs` command creates a pool across all of those devices. Here you create a pool with a single device at `/dev/xvdb`.

+    Btrfs storage pools are created with the `mkfs.btrfs` command. Passing 

+multiple devices to the `mkfs.btrfs` command creates a pool across all of those

+ devices. Here you create a pool with a single device at `/dev/xvdb`.

         $ sudo mkfs.btrfs -f /dev/xvdb

         WARNING! - Btrfs v3.12 IS EXPERIMENTAL

@@ -199,7 +199,8 @@ Assuming your system meets the prerequisites, do the following:

     noted earlier, Btrfs is not currently recommended for production deployments

     unless you already have extensive experience.

-3. If it does not already exist, create a directory for the Docker host's local storage area at `/var/lib/docker`.

+3. If it does not already exist, create a directory for the Docker host's local

+ storage area at `/var/lib/docker`.

         $ sudo mkdir /var/lib/docker

@@ -210,7 +211,10 @@ Assuming your system meets the prerequisites, do the following:

         $ sudo blkid /dev/xvdb

         /dev/xvdb: UUID="a0ed851e-158b-4120-8416-c9b072c8cf47" UUID_SUB="c3927a64-4454-4eef-95c2-a7d44ac0cf27" TYPE="btrfs"

-    b. Create a `/etc/fstab` entry to automatically mount `/var/lib/docker` each time the system boots.

+    b. Create an `/etc/fstab` entry to automatically mount `/var/lib/docker` 

+each time the system boots. Either of the following lines will work, just 

+remember to substitute the UUID value with the value obtained from the previous

+ command.

         /dev/xvdb /var/lib/docker btrfs defaults 0 0

         UUID="a0ed851e-158b-4120-8416-c9b072c8cf47" /var/lib/docker btrfs defaults 0 0

@@ -223,10 +227,11 @@ Assuming your system meets the prerequisites, do the following:

         <output truncated>

         /dev/xvdb on /var/lib/docker type btrfs (rw)

-    The last line in the output above shows the `/dev/xvdb` mounted at `/var/lib/docker` as Btrfs.

-

+    The last line in the output above shows the `/dev/xvdb` mounted at 

+`/var/lib/docker` as Btrfs.

-Now that you have a Btrfs filesystem mounted at `/var/lib/docker`, the daemon should automatically load with the `btrfs` storage driver.

+Now that you have a Btrfs filesystem mounted at `/var/lib/docker`, the daemon 

+should automatically load with the `btrfs` storage driver.

 1. Start the Docker daemon.

@@ -236,9 +241,10 @@ Now that you have a Btrfs filesystem mounted at `/var/lib/docker`, the daemon sh

     The procedure for starting the Docker daemon may differ depending on the

     Linux distribution you are using.

-    You can start the Docker daemon with the `btrfs` storage driver by passing

-    the `--storage-driver=btrfs` flag to the `docker daemon` command or you can

-    add the `DOCKER_OPTS` line to the Docker config file.

+    You can force the the Docker daemon to start with the `btrfs` storage 

+driver by either passing the `--storage-driver=btrfs` flag to the `docker 

+daemon` at startup, or adding it to the `DOCKER_OPTS` line to the Docker config

+ file.

 2. Verify the storage driver with the `docker info` command.

@@ -252,25 +258,54 @@ Your Docker host is now configured to use the `btrfs` storage driver.

 ## Btrfs and Docker performance

-There are several factors that influence Docker's performance under the  `btrfs` storage driver.

-

-- **Page caching**. Btrfs does not support page cache sharing. This means that *n* containers accessing the same file require *n* copies to be cached. As a result, the `btrfs` driver may not be the best choice for PaaS and other high density container use cases.

-

-- **Small writes**. Containers performing lots of small writes (including Docker hosts that start and stop many containers) can lead to poor use of Btrfs chunks. This can ultimately lead to out-of-space conditions on your Docker host and stop it working. This is currently a major drawback to using current versions of Btrfs.

-

-    If you use the `btrfs` storage driver, closely monitor the free space on your Btrfs filesystem using the `btrfs filesys show` command. Do not trust the output of normal Unix commands such as `df`; always use the Btrfs native commands.

-

-- **Sequential writes**. Btrfs writes data to disk via journaling technique. This can impact sequential writes, where performance can be up to half.

-

-- **Fragmentation**. Fragmentation is a natural byproduct of copy-on-write filesystems like Btrfs. Many small random writes can compound this issue. It can manifest as CPU spikes on Docker hosts using SSD media and head thrashing on Docker hosts using spinning media. Both of these result in poor performance.

-

-    Recent versions of Btrfs allow you to specify `autodefrag` as a mount option. This mode attempts to detect random writes and defragment them. You should perform your own tests before enabling this option on your Docker hosts. Some tests have shown this option has a negative performance impact on Docker hosts performing lots of small writes (including systems that start and stop many containers).

-

-- **Solid State Devices (SSD)**. Btrfs has native optimizations for SSD media. To enable these, mount with the `-o ssd` mount option. These optimizations include enhanced SSD write performance by avoiding things like *seek optimizations* that have no use on SSD media.

-

-    Btfs also supports the TRIM/Discard primitives. However, mounting with the `-o discard` mount option can cause performance issues. Therefore, it is recommended you perform your own tests before using this option.

-

-- **Use Data Volumes**. Data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.

+There are several factors that influence Docker's performance under the `btrfs`

+ storage driver.

+

+- **Page caching**. Btrfs does not support page cache sharing. This means that 

+*n* containers accessing the same file require *n* copies to be cached. As a 

+result, the `btrfs` driver may not be the best choice for PaaS and other high 

+density container use cases.

+

+- **Small writes**. Containers performing lots of small writes (including 

+Docker hosts that start and stop many containers) can lead to poor use of Btrfs

+ chunks. This can ultimately lead to out-of-space conditions on your Docker 

+host and stop it working. This is currently a major drawback to using current 

+versions of Btrfs.

+

+    If you use the `btrfs` storage driver, closely monitor the free space on 

+your Btrfs filesystem using the `btrfs filesys show` command. Do not trust the 

+output of normal Unix commands such as `df`; always use the Btrfs native 

+commands.

+

+- **Sequential writes**. Btrfs writes data to disk via journaling technique. 

+This can impact sequential writes, where performance can be up to half.

+

+- **Fragmentation**. Fragmentation is a natural byproduct of copy-on-write 

+filesystems like Btrfs. Many small random writes can compound this issue. It 

+can manifest as CPU spikes on Docker hosts using SSD media and head thrashing 

+on Docker hosts using spinning media. Both of these result in poor performance.

+

+    Recent versions of Btrfs allow you to specify `autodefrag` as a mount 

+option. This mode attempts to detect random writes and defragment them. You 

+should perform your own tests before enabling this option on your Docker hosts.

+ Some tests have shown this option has a negative performance impact on Docker 

+hosts performing lots of small writes (including systems that start and stop 

+many containers).

+

+- **Solid State Devices (SSD)**. Btrfs has native optimizations for SSD media. 

+To enable these, mount with the `-o ssd` mount option. These optimizations 

+include enhanced SSD write performance by avoiding things like *seek 

+optimizations* that have no use on SSD media.

+

+    Btfs also supports the TRIM/Discard primitives. However, mounting with the 

+`-o discard` mount option can cause performance issues. Therefore, it is 

+recommended you perform your own tests before using this option.

+

+- **Use Data Volumes**. Data volumes provide the best and most predictable 

+performance. This is because they bypass the storage driver and do not incur 

+any of the potential overheads introduced by thin provisioning and 

+copy-on-write. For this reason, you should place heavy write workloads on data 

+volumes.

 ## Related Information

@@ -51,56 +51,84 @@ Device Mapper technology works at the block level rather than the file level.

 This means that `devicemapper` storage driver's thin provisioning and

 copy-on-write operations work with blocks rather than entire files.

->**Note**: Snapshots are also referred to as *thin devices* or *virtual devices*. They all mean the same thing in the context of the `devicemapper` storage driver.

+>**Note**: Snapshots are also referred to as *thin devices* or *virtual 

+>devices*. They all mean the same thing in the context of the `devicemapper` 

+>storage driver.

-With the `devicemapper` the high level process for creating images is as follows:

+With `devicemapper` the high level process for creating images is as follows:

 1. The `devicemapper` storage driver creates a thin pool.

-    The pool is created from block devices or loop mounted sparse files (more on this later).

+    The pool is created from block devices or loop mounted sparse files (more 

+on this later).

 2. Next it creates a *base device*.

-    A base device is a thin device with a filesystem. You can see which filesystem is in use by running the `docker info` command and checking the `Backing filesystem` value.

+    A base device is a thin device with a filesystem. You can see which 

+filesystem is in use by running the `docker info` command and checking the 

+`Backing filesystem` value.

 3. Each new image (and image layer) is a snapshot of this base device.

-    These are thin provisioned copy-on-write snapshots. This means that they are initially empty and only consume space from the pool when data is written to them.

+    These are thin provisioned copy-on-write snapshots. This means that they 

+are initially empty and only consume space from the pool when data is written 

+to them.

-With `devicemapper`, container layers are snapshots of the image they are created from. Just as with images, container snapshots are thin provisioned copy-on-write snapshots. The container snapshot stores all updates to the container. The `devicemapper` allocates space to them on-demand from the pool as and when data is written to the container.

+With `devicemapper`, container layers are snapshots of the image they are 

+created from. Just as with images, container snapshots are thin provisioned 

+copy-on-write snapshots. The container snapshot stores all updates to the 

+container. The `devicemapper` allocates space to them on-demand from the pool 

+as and when data is written to the container.

-The high level diagram below shows a thin pool with a base device and two images.

+The high level diagram below shows a thin pool with a base device and two 

+images.

 ![](images/base_device.jpg)

-If you look closely at the diagram you'll see that it's snapshots all the way down. Each image layer is a snapshot of the layer below it. The lowest layer of each image is a snapshot of the the base device that exists in the pool. This base device is a `Device Mapper` artifact and not a Docker image layer.

+If you look closely at the diagram you'll see that it's snapshots all the way 

+down. Each image layer is a snapshot of the layer below it. The lowest layer of

+ each image is a snapshot of the the base device that exists in the pool. This 

+base device is a `Device Mapper` artifact and not a Docker image layer.

-A container is a snapshot of the image it is created from. The diagram below shows two containers - one based on the Ubuntu image and the other based on the Busybox image.

+A container is a snapshot of the image it is created from. The diagram below 

+shows two containers - one based on the Ubuntu image and the other based on the

+ Busybox image.

 ![](images/two_dm_container.jpg)

 ## Reads with the devicemapper

-Let's look at how reads and writes occur using the `devicemapper` storage driver. The diagram below shows the high level process for reading a single block (`0x44f`) in an example container.

+Let's look at how reads and writes occur using the `devicemapper` storage 

+driver. The diagram below shows the high level process for reading a single 

+block (`0x44f`) in an example container.

 ![](images/dm_container.jpg)

-1. An application makes a read request for block 0x44f in the container.

+1. An application makes a read request for block `0x44f` in the container.

-    Because the container is a thin snapshot of an image it does not have the data. Instead, it has a pointer (PTR) to where the data is stored in the  image snapshot lower down in the image stack.

+    Because the container is a thin snapshot of an image it does not have the 

+data. Instead, it has a pointer (PTR) to where the data is stored in the image 

+snapshot lower down in the image stack.

-2. The storage driver follows the pointer to block `0xf33` in the snapshot relating to image layer `a005...`.

+2. The storage driver follows the pointer to block `0xf33` in the snapshot 

+relating to image layer `a005...`.

-3. The `devicemapper` copies the contents of block `0xf33` from the image snapshot to memory in the container.

+3. The `devicemapper` copies the contents of block `0xf33` from the image 

+snapshot to memory in the container.

 4. The storage driver returns the data to the requesting application.

 ### Write examples

-With the `devicemapper` driver, writing new data to a container is accomplished by an *allocate-on-demand* operation. Updating existing data uses a copy-on-write operation. Because Device Mapper is a block-based technology these operations occur at the block level.

+With the `devicemapper` driver, writing new data to a container is accomplished

+ by an *allocate-on-demand* operation. Updating existing data uses a 

+copy-on-write operation. Because Device Mapper is a block-based technology 

+these operations occur at the block level.

-For example, when  making a small change to a large file in a container, the `devicemapper` storage driver does not copy the entire file. It only copies the blocks to be modified. Each block is 64KB.

+For example, when making a small change to a large file in a container, the 

+`devicemapper` storage driver does not copy the entire file. It only copies the

+ blocks to be modified. Each block is 64KB.

 #### Writing new data

@@ -108,9 +136,11 @@ To write 56KB of new data to a container:

 1. An application makes a request to write 56KB of new data to the container.

-2. The allocate-on-demand operation allocates a single new 64KB block to the containers snapshot.

+2. The allocate-on-demand operation allocates a single new 64KB block to the 

+container's snapshot.

-    If the write operation is larger than 64KB, multiple new blocks are allocated to the container snapshot.

+    If the write operation is larger than 64KB, multiple new blocks are 

+allocated to the container's snapshot.

 3. The data is written to the newly allocated block.

@@ -122,7 +152,8 @@ To modify existing data for the first time:

 2. A copy-on-write operation locates the blocks that need updating.

-3. The operation allocates new blocks to the container snapshot and copies the data into those blocks.

+3. The operation allocates new empty blocks to the container snapshot and 

+copies the data into those blocks.

 4. The modified data is written into the newly allocated blocks.

@@ -133,7 +164,8 @@ to the application's read and write operations.

 ## Configuring Docker with Device Mapper

 The `devicemapper` is the default Docker storage driver on some Linux

-distributions. This includes RHEL and most of its forks. Currently, the following distributions support the driver:

+distributions. This includes RHEL and most of its forks. Currently, the 

+following distributions support the driver:

 * RHEL/CentOS/Fedora

 * Ubuntu 12.04          

@@ -142,9 +174,9 @@ distributions. This includes RHEL and most of its forks. Currently, the followin

 Docker hosts running the `devicemapper` storage driver default to a

 configuration mode known as `loop-lvm`. This mode uses sparse files to build

-the thin pool used by image and container snapshots. The mode is designed to work out-of-the-box

-with no additional configuration. However, production deployments should not run

-under `loop-lvm` mode.

+the thin pool used by image and container snapshots. The mode is designed to 

+work out-of-the-box with no additional configuration. However, production 

+deployments should not run under `loop-lvm` mode.

 You can detect the mode by viewing the `docker info` command:

@@ -161,56 +193,84 @@ You can detect the mode by viewing the `docker info` command:

      Library Version: 1.02.93-RHEL7 (2015-01-28)

      ...

-The output above shows a Docker host running with the `devicemapper` storage driver operating in `loop-lvm` mode. This is indicated by the fact that the `Data loop file` and a `Metadata loop file` are on files under `/var/lib/docker/devicemapper/devicemapper`. These are loopback mounted sparse files.

+The output above shows a Docker host running with the `devicemapper` storage 

+driver operating in `loop-lvm` mode. This is indicated by the fact that the 

+`Data loop file` and a `Metadata loop file` are on files under 

+`/var/lib/docker/devicemapper/devicemapper`. These are loopback mounted sparse 

+files.

 ### Configure direct-lvm mode for production

 The preferred configuration for production deployments is `direct lvm`. This

 mode uses block devices to create the thin pool. The following procedure shows

-you how to configure a Docker host to use the `devicemapper` storage driver in a

-`direct-lvm` configuration.

+you how to configure a Docker host to use the `devicemapper` storage driver in 

+a `direct-lvm` configuration.

-> **Caution:** If you have already run the Docker daemon on your Docker host and have images you want to keep, `push` them Docker Hub or your private Docker Trusted Registry before attempting this procedure.

+> **Caution:** If you have already run the Docker daemon on your Docker host 

+> and have images you want to keep, `push` them Docker Hub or your private 

+> Docker Trusted Registry before attempting this procedure.

-The procedure below will create a 90GB data volume and 4GB metadata volume to use as backing for the storage pool. It assumes that you have a spare block device at `/dev/xvdf` with enough free space to complete the task. The device identifier and volume sizes may be be different in your environment and you should substitute your own values throughout the procedure. The procedure also assumes that the Docker daemon is in the `stopped` state.

+The procedure below will create a 90GB data volume and 4GB metadata volume to 

+use as backing for the storage pool. It assumes that you have a spare block 

+device at `/dev/xvdf` with enough free space to complete the task. The device 

+identifier and volume sizes may be be different in your environment and you 

+should substitute your own values throughout the procedure. The procedure also 

+assumes that the Docker daemon is in the `stopped` state.

 1. Log in to the Docker host you want to configure and stop the Docker daemon.

-2. If it exists, delete your existing image store by removing the `/var/lib/docker` directory.

+2. If it exists, delete your existing image store by removing the 

+`/var/lib/docker` directory.

         $ sudo rm -rf /var/lib/docker

-3. Create an LVM physical volume (PV) on your spare block device using the `pvcreate` command.

+3. Create an LVM physical volume (PV) on your spare block device using the 

+`pvcreate` command.