1. docker
  2. Commit c50bd847a72c5eafd6274e67a612dc1e48a8b84b
Browse code

Add special memory management file updating after assignment for Nigel Adding in some notes from Nigel work Updating with the storage driver content Nigel added Updating with Nigel's polishing tech Adding in Nigel graphics First pass of aufs material Capturing Nigel's latest Comments back to Nigel on devicemapper Incorporating Nigel's comments v3 Converting images for dm Entering comments into aufs page Adding the btfs storage driver Moving to userguide Adding in two new driver articles from Nigel Optimized images Updating with comments

Signed-off-by: Mary Anthony <mary@docker.com>
(cherry picked from commit 950fbf99b1214e77c0dcc27e8f9a16e01b2c630a)

Mary Anthony authored on 2015/09/17 02:09:10
Showing 32 changed files
docs/userguide/dockervolumes.md
History View file @ c50bd84
... ... 
@@ -25,9 +25,7 @@ Docker.
25 25 
 ## Data volumes
26 26
27 27 
 A *data volume* is a specially-designated directory within one or more
28 
-containers that bypasses the [*Union File
29 
-System*](../reference/glossary.md#union-file-system). Data volumes provide several
30 
-useful features for persistent or shared data:
28 
+containers that bypasses the [*Union File System*](../reference/glossary.md#union-file-system). Data volumes provide several useful features for persistent or shared data:
31 29
32 30 
 - Volumes are initialized when a container is created. If the container's
33 31 
   base image contains data at the specified mount point, that existing data is
... ... 
@@ -272,6 +270,12 @@ Then un-tar the backup file in the new container's data volume.
272 272 
 You can use the techniques above to automate backup, migration and
273 273 
 restore testing using your preferred tools.
274 274
275 
+## Important tips on using shared volumes
276 
+
277 
+Multiple containers can also share one or more data volumes. However, multiple containers writing to a single shared volume can cause data corruption. Make sure you're applications are designed to write to shared data stores.
278 
+
279 
+Data volumes are directly accessible from the Docker host. This means you can read and write to them with normal Linux tools. In most cases you should not do this as it can cause data corruption if your containers and applications are unaware of your direct access.
280 
+
275 281 
 # Next steps
276 282
277 283 
 Now we've learned a bit more about how to use Docker we're going to see how to
docs/userguide/storagedriver/aufs-driver.md
History View file @ c50bd84
278 284 
new file mode 100644
... ... 
@@ -0,0 +1,197 @@
0 
+<!--[metadata]>
1 
+title = "AUFS storage driver in practice"
2 
+description = "Learn how to optimize your use of AUFS driver."
3 
+keywords = ["container, storage, driver, AUFS "]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+<![end-metadata]-->
7 
+
8 
+# Docker and AUFS in practice
9 
+
10 
+AUFS was the first storage driver in use with Docker. As a result, it has a long and close history with Docker, is very stable, has a lot of real-world deployments, and has strong community support.  AUFS has several features that make it a good choice for Docker. These features enable:
11 
+
12 
+- Fast container startup times.
13 
+- Efficient use of storage.
14 
+- Efficient use of memory.
15 
+
16 
+Despite its capabilities and long history with Docker, some Linux distributions do not support AUFS. This is usually because AUFS is not included in the mainline (upstream) Linux kernel.
17 
+
18 
+The following sections examine some AUFS features and how they relate to Docker.
19 
+
20 
+## Image layering and sharing with AUFS
21 
+
22 
+AUFS is a *unification filesystem*. This means that it takes multiple directories on a single Linux host, stacks them on top of each other, and provides a single unified view. To achieve this, AUFS uses *union mount*.
23 
+
24 
+AUFS stacks multiple directories and exposes them as a unified view through a single mount point. All of the directories in the stack, as well as the union mount point, must all exist on the same Linux host. AUFS refers to each directory that it stacks as a *branch*.
25 
+
26 
+Within Docker, AUFS union mounts enable image layering. The AUFS storage driver implements Docker image layers using this union mount system. AUFS branches correspond to Docker image layers. The diagram below shows a Docker container based on the `ubuntu:latest` image.
27 
+
28 
+![](images/aufs_layers.jpg)
29 
+
30 
+This diagram shows the relationship between the Docker image layers and the AUFS branches (directories) in `/var/lib/docker/aufs`. Each image layer  and the container layer correspond to an AUFS branch (directory) in the Docker host's local storage area. The union mount point gives the unified view of all layers.
31 
+
32 
+AUFS also supports the copy-on-write technology (CoW). Not all storage drivers do.
33 
+
34 
+## Container reads and writes with AUFS
35 
+
36 
+Docker leverages AUFS CoW technology to enable image sharing and minimize the use of disk space. AUFS works at the file level. This means that all AUFS CoW operations copy entire files - even if only a small part of the file is being modified. This behavior can have a noticeable impact on container performance, especially if the files being copied are large, below a lot of image layers, or the CoW operation must search a deep directory tree.
37 
+
38 
+Consider, for example, an application running in a container needs to add a single new value to a large key-value store (file). If this is the first time the file is modified it does not yet exist in the container's top writable layer. So, the CoW must *copy up* the file from the underlying image. The AUFS storage driver searches each image layer for the file. The search order is from top to bottom. When it is found, the entire file is *copied up* to the container's top writable layer. From there, it can be opened and modified.
39 
+
40 
+Larger files obviously take longer to *copy up* than smaller files, and files that exist in lower image layers take longer to locate than those in higher layers. However, a *copy up* operation only occurs once per file on any given container. Subsequent reads and writes happen against the file's copy already *copied-up* to the container's top layer.
41 
+
42 
+
43 
+## Deleting files with the AUFS storage driver
44 
+
45 
+The AUFS storage driver deletes a file from a container by placing a *whiteout
46 
+file* in the container's top layer. The whiteout file effectively obscures the
47 
+existence of the file in image's lower, read-only layers.  The simplified
48 
+diagram below shows a container based on an image with three image layers.
49 
+
50 
+![](images/aufs_delete.jpg)
51 
+
52 
+The `file3` was deleted from the container. So, the AUFS storage driver  placed
53 
+a whiteout file in the container's top layer. This whiteout file effectively
54 
+"deletes"  `file3` from the container by obscuring any of the original file's
55 
+existence in the image's read-only base layer. Of course, the image could have
56 
+been in any of the other layers instead or in addition depending on how the
57 
+layers are built.
58 
+
59 
+## Configure Docker with AUFS
60 
+
61 
+You can only use the AUFS storage driver on Linux systems with AUFS installed. Use the following command to determine if your system supports AUFS.
62 
+
63 
+```bash
64 
+$ grep aufs /proc/filesystems
65 
+nodev   aufs
66 
+```
67 
+
68 
+This output indicates the system supports AUFS.  Once you've verified your
69 
+system supports AUFS, you can must instruct the Docker daemon to use it. You do
70 
+this from the command line with the `docker daemon` command:
71 
+
72 
+```bash
73 
+$ sudo docker daemon --storage-driver=aufs &
74 
+```
75 
+
76 
+Alternatively, you can edit the Docker config file and add the
77 
+`--storage-driver=aufs` option to the `DOCKER_OPTS` line.
78 
+
79 
+```bash
80 
+# Use DOCKER_OPTS to modify the daemon startup options.
81 
+DOCKER_OPTS="--storage-driver=aufs"
82 
+```
83 
+
84 
+Once your daemon is running, verify the storage driver with the `docker info` command.
85 
+
86 
+```bash
87 
+$ sudo docker info
88 
+Containers: 1
89 
+Images: 4
90 
+Storage Driver: aufs
91 
+ Root Dir: /var/lib/docker/aufs
92 
+ Backing Filesystem: extfs
93 
+ Dirs: 6
94 
+ Dirperm1 Supported: false
95 
+Execution Driver: native-0.2
96 
+...output truncated...
97 
+````
98 
+
99 
+The output above shows that the Docker daemon is running the AUFS storage driver on top of an existing ext4 backing filesystem.
100 
+
101 
+## Local storage and AUFS
102 
+
103 
+As the `docker daemon` runs with the AUFS driver, the driver stores images and containers on within the Docker host's local storage area in the `/var/lib/docker/aufs` directory.
104 
+
105 
+### Images
106 
+
107 
+Image layers and their contents are stored under
108 
+`/var/lib/docker/aufs/mnt/diff/<image-id>` directory. The contents of an image
109 
+layer in this location includes all the files and directories belonging in that
110 
+image layer.
111 
+
112 
+The `/var/lib/docker/aufs/layers/` directory contains metadata about how image
113 
+layers are stacked. This directory contains one file for every image or
114 
+container layer on the Docker host. Inside each file are the image layers names
115 
+that exist below it. The diagram below shows an image with 4 layers.
116 
+
117 
+![](images/aufs_metadata.jpg)
118 
+
119 
+Inspecting the contents of the file relating to the top layer of the image
120 
+shows the three image layers below it. They are listed in the order they are
121 
+stacked.
122 
+
123 
+```bash
124 
+$ cat /var/lib/docker/aufs/layers/91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c
125 
+
126 
+d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82
127 
+
128 
+c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87
129 
+
130 
+d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391
131 
+```
132 
+
133 
+The base layer in an image has no image layers below it, so its file is empty.
134 
+
135 
+### Containers
136 
+
137 
+Running containers are mounted at locations in the
138 
+`/var/lib/docker/aufs/mnt/<container-id>` directory. This is the AUFS union
139 
+mount point that exposes the container and all underlying image layers as a
140 
+single unified view. If a container is not running, its directory still exists
141 
+but is empty. This is because containers are only mounted when they are running.
142 
+
143 
+Container metadata and various config files that are placed into the running
144 
+container are stored in `/var/lib/containers/<container-id>`. Files in this
145 
+directory exist for all containers on the system, including ones that are
146 
+stopped. However, when a container is running the container's log files are also
147 
+in this directory.
148 
+
149 
+A container's thin writable layer is stored under
150 
+`/var/lib/docker/aufs/diff/<container-id>`. This directory is stacked by AUFS as
151 
+the containers top writable layer and is where all changes to the container are
152 
+stored. The directory exists even if the container is stopped. This means that
153 
+restarting a container will not lose changes made to it. Once a container is
154 
+deleted this directory is deleted.
155 
+
156 
+Information about which image layers are stacked below a container's top
157 
+writable layer is stored in the following file
158 
+`/var/lib/docker/aufs/layers/<container-id>`. The command below shows that the
159 
+container with ID `b41a6e5a508d` has 4 image layers below it:
160 
+
161 
+```bash
162 
+$ cat /var/lib/docker/aufs/layers/b41a6e5a508dfa02607199dfe51ed9345a675c977f2cafe8ef3e4b0b5773404e-init
163 
+91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c
164 
+d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82
165 
+c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87
166 
+d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391
167 
+```
168 
+
169 
+The image layers are shown in order. In the output above, the layer starting
170 
+with image ID "d3a1..." is the image's base  layer. The image layer starting
171 
+with "91e5..." is the image's topmost layer.
172 
+
173 
+
174 
+## AUFS and Docker performance
175 
+
176 
+To summarize some of the performance related aspects already mentioned:
177 
+
178 
+- The AUFS storage driver is a good choice for PaaS and other similar use-cases where container density is important. This is because AUFS efficiently shares images between multiple running containers, enabling fast container start times and minimal use of disk space.
179 
+
180 
+- The underlying mechanics of how AUFS shares files between image layers and containers uses the systems page cache very efficiently.
181 
+
182 
+- The AUFS storage driver can introduce significant latencies into container write performance. This is because the first time a container writes to any file, the file has be located and copied into the containers top writable layer. These latencies increase and are compounded when these files exist below many image layers and the files themselves are large.
183 
+
184 
+One final point. Data volumes provide the best and most predictable performance.
185 
+This is because they bypass the storage driver and do not incur any of the
186 
+potential overheads introduced by thin provisioning and copy-on-write. For this
187 
+reason, you may want to place heavy write workloads on data volumes.
188 
+
189 
+## Related information
190 
+
191 
+* [Understand images, containers, and storage drivers](imagesandcontainers.md)
192 
+* [Select a storage driver](selectadriver.md)
193 
+* [BTRFS storage driver in practice](btrfs-driver.md)
194 
+* [Device Mapper storage driver in practice](device-mapper-driver.md)
docs/userguide/storagedriver/btrfs-driver.md
History View file @ c50bd84
0 195 
new file mode 100644
... ... 
@@ -0,0 +1,280 @@
0 
+<!--[metadata]>
1 
+title = "BTRFS storage in practice"
2 
+description = "Learn how to optimize your use of BTRFS driver."
3 
+keywords = ["container, storage, driver, BTRFS "]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+<![end-metadata]-->
7 
+
8 
+# Docker and BTRFS in practice
9 
+
10 
+Btrfs is a next generation copy-on-write filesystem that supports many advanced
11 
+storage technologies that make it a good fit for Docker. Btrfs is included in
12 
+the mainline Linux kernel and it's on-disk-format is now considered stable.
13 
+However, many of its features are still under heavy development and users should
14 
+consider it a fast-moving target.
15 
+
16 
+Docker's `btrfs` storage driver leverages many Btrfs features for image and
17 
+container management. Among these features are thin provisioning, copy-on-write,
18 
+and snapshotting.
19 
+
20 
+This article refers to Docker's Btrfs storage driver as `btrfs` and the overall Btrfs Filesystem as Btrfs.
21 
+
22 
+>**Note**: The [Commercially Supported Docker Engine (CS-Engine)](https://www.docker.com/compatibility-maintenance) does not currently support the `btrfs` storage driver.
23 
+
24 
+## The future of Btrfs
25 
+
26 
+Btrfs has been long hailed as the future of Linux filesystems. With full support in the mainline Linux kernel, a stable on-disk-format, and active development with a focus on stability, this is now becoming more of a reality.
27 
+
28 
+As far as Docker on the Linux platform goes, many people see the `btrfs` storage driver as a potential long-term replacement for the `devicemapper` storage driver. However, at the time of writing, the `devicemapper` storage driver should be considered safer, more stable, and more *production ready*. You should only consider the `btrfs` driver for production deployments if you understand it well and have existing experience with Btrfs.
29 
+
30 
+## Image layering and sharing with Btrfs
31 
+
32 
+Docker leverages Btrfs *subvolumes* and *snapshots* for managing the on-disk components of image and container layers.  Btrfs subvolumes look and feel like a normal Unix filesystem. As such, they can have their own internal directory structure that hooks into the wider Unix filesystem.
33 
+
34 
+Subvolumes are natively copy-on-write and have space allocated to them on-demand
35 
+from an underlying storage pool. They can also be nested and snapped. The
36 
+diagram blow shows 4 subvolumes. 'Subvolume 2' and 'Subvolume 3' are nested,
37 
+whereas 'Subvolume 4' shows its own internal directory tree.
38 
+
39 
+![](images/btfs_subvolume.jpg)
40 
+
41 
+Snapshots are a point-in-time read-write copy of an entire subvolume. They exist directly below the subvolume they were created from. You can create snapshots of snapshots as shown in the diagram below.
42 
+
43 
+![](images/btfs_snapshots.jpg)
44 
+
45 
+Btfs allocates space to subvolumes and snapshots on demand from an underlying pool of storage. The unit of allocation is referred to as a *chunk* and *chunks* are normally ~1GB in size.
46 
+
47 
+Snapshots are first-class citizens in a Btrfs filesystem. This means that they look, feel, and operate just like regular subvolumes. The technology required to create them is built directly into the Btrfs filesystem thanks to its native copy-on-write design. This means that Btrfs snapshots are space efficient with little or no performance overhead. The diagram below shows a subvolume and it's snapshot sharing the same data.
48 
+
49 
+![](images/btfs_pool.jpg)
50 
+
51 
+Docker's `btrfs` storage driver stores every image layer and container in its own Btrfs subvolume or snapshot. The base layer of an image is stored as a subvolume whereas child image layers and containers are stored as snapshots. This is shown in the diagram below.
52 
+
53 
+![](images/btfs_container_layer.jpg)
54 
+
55 
+The high level process for creating images and containers on Docker hosts running the `btrfs` driver is as follows:
56 
+
57 
+1. The image's base layer is stored in a Btrfs subvolume under
58 
+`/var/lib/docker/btrfs/subvolumes`.
59 
+
60 
+    The image ID is used as the subvolume name. E.g., a base layer with image ID
61 
+    "f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b" will be
62 
+    stored in
63 
+    `/var/lib/docker/btrfs/subvolumes/f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b`
64 
+
65 
+2. Subsequent image layers are stored as a Btrfs snapshot of the parent layer's subvolume or snapshot.
66 
+
67 
+    The diagram below shows a three-layer image. The base layer is a subvolume. Layer 1 is a snapshot of the base layer's subvolume. Layer 2 is a snapshot of Layer 1's snapshot.
68 
+
69 
+    ![](images/btfs_constructs.jpg)
70 
+
71 
+## Image and container on-disk constructs
72 
+
73 
+Image layers and containers are visible in the Docker host's filesystem at
74 
+`/var/lib/docker/btrfs/subvolumes/<image-id> OR <container-id>`. Directories for
75 
+containers are present even for containers with a stopped status. This is
76 
+because the `btrfs` storage driver mounts a default, top-level subvolume at
77 
+`/var/lib/docker/subvolumes`. All other subvolumes and snapshots exist below
78 
+that as Btrfs filesystem objects and not as individual mounts.
79 
+
80 
+The following example shows a single Docker image with four image layers.
81 
+
82 
+```bash
83 
+$ sudo docker images -a
84 
+REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
85 
+ubuntu              latest              0a17decee413        2 weeks ago         188.3 MB
86 
+<none>              <none>              3c9a9d7cc6a2        2 weeks ago         188.3 MB
87 
+<none>              <none>              eeb7cb91b09d        2 weeks ago         188.3 MB
88 
+<none>              <none>              f9a9f253f610        2 weeks ago         188.1 MB
89 
+```
90 
+
91 
+Each image layer exists as a Btrfs subvolume or snapshot with the same name as it's image ID as illustrated by the `btrfs subvolume list` command shown below:
92 
+
93 
+```bash
94 
+$ sudo btrfs subvolume list /var/lib/docker
95 
+ID 257 gen 9 top level 5 path btrfs/subvolumes/f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b
96 
+ID 258 gen 10 top level 5 path btrfs/subvolumes/eeb7cb91b09d5de9edb2798301aeedf50848eacc2123e98538f9d014f80f243c
97 
+ID 260 gen 11 top level 5 path btrfs/subvolumes/3c9a9d7cc6a235eb2de58ca9ef3551c67ae42a991933ba4958d207b29142902b
98 
+ID 261 gen 12 top level 5 path btrfs/subvolumes/0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751
99 
+```
100 
+
101 
+Under the `/var/lib/docker/btrfs/subvolumes` directoy, each of these subvolumes and snapshots are visible as a normal Unix directory:
102 
+
103 
+```bash
104 
+$ ls -l /var/lib/docker/btrfs/subvolumes/
105 
+total 0
106 
+drwxr-xr-x 1 root root 132 Oct 16 14:44 0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751
107 
+drwxr-xr-x 1 root root 132 Oct 16 14:44 3c9a9d7cc6a235eb2de58ca9ef3551c67ae42a991933ba4958d207b29142902b
108 
+drwxr-xr-x 1 root root 132 Oct 16 14:44 eeb7cb91b09d5de9edb2798301aeedf50848eacc2123e98538f9d014f80f243c
109 
+drwxr-xr-x 1 root root 132 Oct 16 14:44 f9a9f253f6105141e0f8e091a6bcdb19e3f27af949842db93acba9048ed2410b
110 
+```
111 
+
112 
+Because Btrfs works at the filesystem level and not the block level, each image
113 
+and container layer can be browsed in the filesystem using normal Unix commands.
114 
+The example below shows a truncated output of an `ls -l` command against the
115 
+image's top layer:
116 
+
117 
+```bash
118 
+$ ls -l /var/lib/docker/btrfs/subvolumes/0a17decee4139b0de68478f149cc16346f5e711c5ae3bb969895f22dd6723751/
119 
+total 0
120 
+drwxr-xr-x 1 root root 1372 Oct  9 08:39 bin
121 
+drwxr-xr-x 1 root root    0 Apr 10  2014 boot
122 
+drwxr-xr-x 1 root root  882 Oct  9 08:38 dev
123 
+drwxr-xr-x 1 root root 2040 Oct 12 17:27 etc
124 
+drwxr-xr-x 1 root root    0 Apr 10  2014 home
125 
+...output truncated...
126 
+```
127 
+
128 
+## Container reads and writes with Btrfs
129 
+
130 
+A container is a space-efficient snapshot of an image. Metadata in the snapshot
131 
+points to the actual data blocks in the storage pool. This is the same as with a
132 
+subvolume. Therefore, reads performed against a snapshot are essentially the
133 
+same as reads performed against a subvolume. As a result, no performance
134 
+overhead is incurred from the Btrfs driver.
135 
+
136 
+Writing a new file to a container invokes an allocate-on-demand operation to
137 
+allocate new data block to the container's snapshot. The file is then written to
138 
+this new space. The allocate-on-demand operation is native to all writes with
139 
+Btrfs and is the same as writing new data to a subvolume. As a result, writing
140 
+new files to a container's snapshot operate at native Btrfs speeds.
141 
+
142 
+Updating an existing file in a container causes a copy-on-write operation
143 
+(technically *redirect-on-write*). The driver leaves the original data and
144 
+allocates new space to the snapshot. The updated data is written to this new
145 
+space. Then, the driver updates the filesystem metadata in the snapshot to point
146 
+to this new data. The original data is preserved in-place for subvolumes and
147 
+snapshots further up the tree. This behavior is native to copy-on-write
148 
+filesystems like Btrfs and incurs very little overhead.
149 
+
150 
+With Btfs, writing and updating lots of small files can result in slow performance. More on this later.
151 
+
152 
+## Configuring Docker with Btrfs
153 
+
154 
+The `btrfs` storage driver only operates on a Docker host where `/var/lib/docker` is mounted as a Btrfs filesystem. The following procedure shows  how to configure Btrfs on Ubuntu 14.04 LTS.
155 
+
156 
+### Prerequisites
157 
+
158 
+If you have already used the Docker daemon on your Docker host and have images you want to keep, `push` them to Docker Hub or your private Docker Trusted Registry before attempting this procedure.
159 
+
160 
+Stop the Docker daemon. Then, ensure that you have a spare block device at `/dev/xvdb`. The device identifier may be different in your environment and you should substitute your own values throughout the procedure.
161 
+
162 
+The procedure also assumes your kernel has the appropriate Btrfs modules loaded. To verify this, use the following command:
163 
+
164 
+```bash
165 
+$ cat /proc/filesystems | grep btrfs`
166 
+```
167 
+
168 
+### Configure Btrfs on Ubuntu 14.04 LTS
169 
+
170 
+Assuming your system meets the prerequisites, do the following:
171 
+
172 
+1. Install the "btrfs-tools" package.
173 
+
174 
+        $ sudo apt-get install btrfs-tools
175 
+        Reading package lists... Done
176 
+        Building dependency tree
177 
+        <output truncated>
178 
+
179 
+2. Create the Btrfs storage pool.
180 
+
181 
+    Btrfs storage pools are created with the `mkfs.btrfs` command. Passing multiple devices to the `mkfs.btrfs` command creates a pool across all of those devices. Here you create a pool with a single device at `/dev/xvdb`.
182 
+
183 
+        $ sudo mkfs.btrfs -f /dev/xvdb
184 
+        WARNING! - Btrfs v3.12 IS EXPERIMENTAL
185 
+        WARNING! - see http://btrfs.wiki.kernel.org before using
186 
+
187 
+        Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
188 
+        fs created label (null) on /dev/xvdb
189 
+            nodesize 16384 leafsize 16384 sectorsize 4096 size 4.00GiB
190 
+        Btrfs v3.12
191 
+
192 
+    Be sure to substitute `/dev/xvdb` with the appropriate device(s) on your
193 
+    system.
194 
+
195 
+    > **Warning**: Take note of the warning about Btrfs being experimental. As
196 
+    noted earlier, Btrfs is not currently recommended for production deployments
197 
+    unless you already have extensive experience.
198 
+
199 
+3. If it does not already exist, create a directory for the Docker host's local storage area at `/var/lib/docker`.
200 
+
201 
+        $ sudo mkdir /var/lib/docker
202 
+
203 
+4. Configure the system to automatically mount the Btrfs filesystem each time the system boots.
204 
+
205 
+    a. Obtain the Btrfs filesystem's UUID.
206 
+
207 
+        $ sudo blkid /dev/xvdb
208 
+        /dev/xvdb: UUID="a0ed851e-158b-4120-8416-c9b072c8cf47" UUID_SUB="c3927a64-4454-4eef-95c2-a7d44ac0cf27" TYPE="btrfs"
209 
+
210 
+    b. Create a `/etc/fstab` entry to automatically mount `/var/lib/docker` each time the system boots.
211 
+
212 
+        /dev/xvdb /var/lib/docker btrfs defaults 0 0
213 
+        UUID="a0ed851e-158b-4120-8416-c9b072c8cf47" /var/lib/docker btrfs defaults 0 0
214 
+
215 
+5. Mount the new filesystem and verify the operation.
216 
+
217 
+        $ sudo mount -a
218 
+        $ mount
219 
+        /dev/xvda1 on / type ext4 (rw,discard)
220 
+        <output truncated>
221 
+        /dev/xvdb on /var/lib/docker type btrfs (rw)
222 
+
223 
+    The last line in the output above shows the `/dev/xvdb` mounted at `/var/lib/docker` as Btrfs.
224 
+
225 
+
226 
+Now that you have a Btrfs filesystem mounted at `/var/lib/docker`, the daemon should automatically load with the `btrfs` storage driver.
227 
+
228 
+1. Start the Docker daemon.
229 
+
230 
+        $ sudo service docker start
231 
+        docker start/running, process 2315
232 
+
233 
+    The procedure for starting the Docker daemon may differ depending on the
234 
+    Linux distribution you are using.
235 
+
236 
+    You can start the Docker daemon with the `btrfs` storage driver by passing
237 
+    the `--storage-driver=btrfs` flag to the `docker daemon` command or you can
238 
+    add the `DOCKER_OPTS` line to the Docker config file.
239 
+
240 
+2. Verify the storage driver with the `docker info` command.
241 
+
242 
+        $ sudo docker info
243 
+        Containers: 0
244 
+        Images: 0
245 
+        Storage Driver: btrfs
246 
+        [...]
247 
+
248 
+Your Docker host is now configured to use the `btrfs` storage driver.
249 
+
250 
+## BTRFS and Docker performance
251 
+
252 
+There are several factors that influence Docker's performance under the  `btrfs` storage driver.
253 
+
254 
+- **Page caching**. Btrfs does not support page cache sharing. This means that *n* containers accessing the same file require *n* copies to be cached. As a result, the `btrfs` driver may not be the best choice for PaaS and other high density container use cases.
255 
+
256 
+- **Small writes**. Containers performing lots of small writes (including Docker hosts that start and stop many containers) can lead to poor use of Btrfs chunks. This can ultimately lead to out-of-space conditions on your Docker host and stop it working. This is currently a major drawback to using current versions of Btrfs.
257 
+
258 
+    If you use the `btrfs` storage driver, closely monitor the free space on your Btrfs filesystem using the `btrfs filesys show` command. Do not trust the output of normal Unix commands such as `df`; always use the Btrfs native commands.
259 
+
260 
+- **Sequential writes**. Btrfs writes data to disk via journaling technique. This can impact sequential writes, where performance can be up to half.
261 
+
262 
+- **Fragmentation**. Fragmentation is a natural byproduct of copy-on-write filesystems like Btrfs. Many small random writes can compound this issue. It can manifest as CPU spikes on Docker hosts using SSD media and head thrashing on Docker hosts using spinning media. Both of these result in poor performance.
263 
+
264 
+    Recent versions of Btrfs allow you to specify `autodefrag` as a mount option. This mode attempts to detect random writes and defragment them. You should perform your own tests before enabling this option on your Docker hosts. Some tests have shown this option has a negative performance impact on Docker hosts performing lots of small writes (including systems that start and stop many containers).
265 
+
266 
+- **Solid State Devices (SSD)**. Btrfs has native optimizations for SSD media. To enable these, mount with the `-o ssd` mount option. These optimizations include enhanced SSD write performance by avoiding things like *seek optimizations* that have no use on SSD media.
267 
+
268 
+    Btfs also supports the TRIM/Discard primitives. However, mounting with the `-o discard` mount option can cause performance issues. Therefore, it is recommended you perform your own tests before using this option.
269 
+
270 
+- **Use Data Volumes**. Data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.
271 
+
272 
+## Related Information
273 
+
274 
+* [Understand images, containers, and storage drivers](imagesandcontainers.md)
275 
+* [Select a storage driver](selectadriver.md)
276 
+* [AUFS storage driver in practice](aufs-driver.md)
277 
+* [Device Mapper storage driver in practice](device-mapper-driver.md)
docs/userguide/storagedriver/device-mapper-driver.md
History View file @ c50bd84
0 278 
new file mode 100644
... ... 
@@ -0,0 +1,310 @@
0 
+<!--[metadata]>
1 
+title="Device mapper storage in practice"
2 
+description="Learn how to optimize your use of device mapper driver."
3 
+keywords=["container, storage, driver, device mapper"]
4 
+[menu.main]
5 
+parent="mn_storage_docker"
6 
+<![end-metadata]-->
7 
+
8 
+# Docker and the Device Mapper storage driver
9 
+
10 
+Device Mapper is a kernel-based framework that underpins many advanced
11 
+volume management technologies on Linux. Docker's `devicemapper` storage driver
12 
+leverages the thin provisioning and snapshotting capabilities of this framework
13 
+for image and container management. This article refers to the Device Mapper
14 
+storage driver as `devicemapper`, and the kernel framework as `Device Mapper`.
15 
+
16 
+
17 
+>**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires that you use the `devicemapper` storage driver.
18 
+
19 
+
20 
+## An alternative to AUFS
21 
+
22 
+Docker originally ran on Ubuntu and Debian Linux and used AUFS for its storage
23 
+backend. As Docker became popular, many of the companies that wanted to use it
24 
+were using Red Hat Enterprise Linux (RHEL). Unfortunately, because the upstream
25 
+mainline Linux kernel did not include AUFS, RHEL did not use AUFS either.
26 
+
27 
+To correct this Red Hat developers investigated getting AUFS into the mainline
28 
+kernel. Ultimately, though, they decided a better idea was to develop a new
29 
+storage backend. Moreover, they would base this new storage backend on existing
30 
+`Device Mapper` technology.
31 
+
32 
+Red Hat collaborated with Docker Inc. to contribute this new driver. As a result
33 
+of this collaboration, Docker's Engine was re-engineered to make the storage
34 
+backend pluggable. So it was that the `devicemapper` became the second storage
35 
+driver Docker supported.
36 
+
37 
+Device Mapper has been included in the mainline Linux kernel since version
38 
+2.6.9. It is a core part of RHEL family of Linux distributions. This means that
39 
+the `devicemapper` storage driver is based on stable code that has a lot of
40 
+real-world production deployments and strong community support.
41 
+
42 
+
43 
+## Image layering and sharing
44 
+
45 
+The `devicemapper` driver stores every image and container on its own virtual
46 
+device. These devices are thin-provisioned copy-on-write snapshot devices.
47 
+Device Mapper technology works at the block level rather than the file level.
48 
+This means that `devicemapper` storage driver's thin provisioning and
49 
+copy-on-write operations work with blocks rather than entire files.
50 
+
51 
+>**Note**: Snapshots are also referred to as *thin devices* or *virtual devices*. They all mean the same thing in the context of the `devicemapper` storage driver.
52 
+
53 
+With the `devicemapper` the high level process for creating images is as follows:
54 
+
55 
+1. The `devicemapper` storage driver creates a thin pool.
56 
+
57 
+    The pool is created from block devices or loop mounted sparse files (more on this later).
58 
+
59 
+2. Next it creates a *base device*.
60 
+
61 
+    A base device is a thin device with a filesystem. You can see which filesystem is in use by running the `docker info` command and checking the `Backing filesystem` value.
62 
+
63 
+3. Each new image (and image layer) is a snapshot of this base device.
64 
+
65 
+    These are thin provisioned copy-on-write snapshots. This means that they are initially empty and only consume space from the pool when data is written to them.
66 
+
67 
+With `devicemapper`, container layers are snapshots of the image they are created from. Just as with images, container snapshots are thin provisioned copy-on-write snapshots. The container snapshot stores all updates to the container. The `devicemapper` allocates space to them on-demand from the pool as and when data is written to the container.
68 
+
69 
+The high level diagram below shows a thin pool with a base device and two images.
70 
+
71 
+![](images/base_device.jpg)
72 
+
73 
+If you look closely at the diagram you'll see that it's snapshots all the way down. Each image layer is a snapshot of the layer below it. The lowest layer of each image is a snapshot of the the base device that exists in the pool. This base device is a `Device Mapper` artifact and not a Docker image layer.
74 
+
75 
+A container is a snapshot of the image it is created from. The diagram below shows two containers - one based on the Ubuntu image and the other based on the Busybox image.
76 
+
77 
+![](images/two_dm_container.jpg)
78 
+
79 
+
80 
+## Reads with the devicemapper
81 
+
82 
+Let's look at how reads and writes occur using the `devicemapper` storage driver. The diagram below shows the high level process for reading a single block (`0x44f`) in an example container.
83 
+
84 
+![](images/dm_container.jpg)
85 
+
86 
+1. An application makes a read request for block 0x44f in the container.
87 
+
88 
+    Because the container is a thin snapshot of an image it does not have the data. Instead, it has a pointer (PTR) to where the data is stored in the  image snapshot lower down in the image stack.
89 
+
90 
+2. The storage driver follows the pointer to block `0xf33` in the snapshot relating to image layer `a005...`.
91 
+
92 
+3. The `devicemapper` copies the contents of block `0xf33` from the image snapshot to memory in the container.
93 
+
94 
+4. The storage driver returns the data to the requesting application.
95 
+
96 
+### Write examples
97 
+
98 
+With the `devicemapper` driver, writing new data to a container is accomplished by an *allocate-on-demand* operation. Updating existing data uses a copy-on-write operation. Because Device Mapper is a block-based technology these operations occur at the block level.
99 
+
100 
+For example, when  making a small change to a large file in a container, the `devicemapper` storage driver does not copy the entire file. It only copies the blocks to be modified. Each block is 64KB.
101 
+
102 
+#### Writing new data
103 
+
104 
+To write 56KB of new data to a container:
105 
+
106 
+1. An application makes a request to write 56KB of new data to the container.
107 
+
108 
+2. The allocate-on-demand operation allocates a single new 64KB block to the containers snapshot.
109 
+
110 
+    If the write operation is larger than 64KB, multiple new blocks are allocated to the container snapshot.
111 
+
112 
+3. The data is written to the newly allocated block.
113 
+
114 
+#### Overwriting existing data
115 
+
116 
+To modify existing data for the first time:
117 
+
118 
+1. An application makes a request to modify some data in the container.
119 
+
120 
+2. A copy-on-write operation locates the blocks that need updating.
121 
+
122 
+3. The operation allocates new blocks to the container snapshot and copies the data into those blocks.
123 
+
124 
+4. The modified data is written into the newly allocated blocks.
125 
+
126 
+The application in the container is unaware of any of these
127 
+allocate-on-demand and copy-on-write operations. However, they may add latency
128 
+to the application's read and write operations.
129 
+
130 
+## Configuring Docker with Device Mapper
131 
+
132 
+The `devicemapper` is the default Docker storage driver on some Linux
133 
+distributions. This includes RHEL and most of its forks. Currently, the following distributions support the driver:
134 
+
135 
+* RHEL/CentOS/Fedora
136 
+* Ubuntu 12.04
137 
+* Ubuntu 14.04
138 
+* Debian
139 
+
140 
+Docker hosts running the `devicemapper` storage driver default to a
141 
+configuration mode known as `loop-lvm`. This mode uses sparse files to build
142 
+the thin pool used by image and container snapshots. The mode is designed to work out-of-the-box
143 
+with no additional configuration. However, production deployments should not run
144 
+under `loop-lvm` mode.
145 
+
146 
+You can detect the mode by viewing the `docker info` command:
147 
+
148 
+    $ sudo docker info
149 
+    Containers: 0
150 
+    Images: 0
151 
+    Storage Driver: devicemapper
152 
+     Pool Name: docker-202:2-25220302-pool
153 
+     Pool Blocksize: 65.54 kB
154 
+     Backing Filesystem: xfs
155 
+     ...
156 
+     Data loop file: /var/lib/docker/devicemapper/devicemapper/data
157 
+     Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
158 
+     Library Version: 1.02.93-RHEL7 (2015-01-28)
159 
+     ...
160 
+
161 
+The output above shows a Docker host running with the `devicemapper` storage driver operating in `loop-lvm` mode. This is indicated by the fact that the `Data loop file` and a `Metadata loop file` are on files under `/var/lib/docker/devicemapper/devicemapper`. These are loopback mounted sparse files.
162 
+
163 
+### Configure direct-lvm mode for production
164 
+
165 
+The preferred configuration for production deployments is `direct lvm`. This
166 
+mode uses block devices to create the thin pool. The following procedure shows
167 
+you how to configure a Docker host to use the `devicemapper` storage driver in a
168 
+`direct-lvm` configuration.
169 
+
170 
+> **Caution:** If you have already run the Docker daemon on your Docker host and have images you want to keep, `push` them Docker Hub or your private Docker Trusted Registry before attempting this procedure.
171 
+
172 
+The procedure below will create a 90GB data volume and 4GB metadata volume to use as backing for the storage pool. It assumes that you have a spare block device at `/dev/xvdf` with enough free space to complete the task. The device identifier and volume sizes may be be different in your environment and you should substitute your own values throughout the procedure. The procedure also assumes that the Docker daemon is in the `stopped` state.
173 
+
174 
+1. Log in to the Docker host you want to configure and stop the Docker daemon.
175 
+
176 
+2. If it exists, delete your existing image store by removing the `/var/lib/docker` directory.
177 
+
178 
+        $ sudo rm -rf /var/lib/docker
179 
+
180 
+3. Create an LVM physical volume (PV) on your spare block device using the `pvcreate` command.
181 
+
182 
+        $ sudo pvcreate /dev/xvdf
183 
+        Physical volume `/dev/xvdf` successfully created
184 
+
185 
+    The device identifier may be different on your system. Remember to substitute your value in the command above.
186 
+
187 
+4. Create a new volume group (VG) called `vg-docker` using the PV created in the previous step.
188 
+
189 
+        $ sudo vgcreate vg-docker /dev/xvdf
190 
+        Volume group `vg-docker` successfully created
191 
+
192 
+5. Create a new 90GB logical volume (LV) called `data` from space in the `vg-docker` volume group.
193 
+
194 
+        $ sudo lvcreate -L 90G -n data vg-docker
195 
+        Logical volume `data` created.
196 
+
197 
+    The command creates an LVM logical volume called `data` and an associated block device file at `/dev/vg-docker/data`. In a later step, you instruct the `devicemapper` storage driver to use this block device to store image and container data.
198 
+
199 
+    If you receive a signature detection warning, make sure you are working on the correct devices before continuing. Signature warnings indicate that the device you're working on is currently in use by LVM or has been used by LVM in the past.
200 
+
201 
+6. Create a new logical volume (LV) called `metadata` from space in the `vg-docker` volume group.
202 
+
203 
+        $ sudo lvcreate -L 4G -n metadata vg-docker
204 
+        Logical volume `metadata` created.
205 
+
206 
+    This creates an LVM logical volume called `metadata` and an associated block device file at `/dev/vg-docker/metadata`. In the next step you instruct the `devicemapper` storage driver to use this block device to store image and container metadata.
207 
+
208 
+5. Start the Docker daemon with the `devicemapper` storage driver and the `--storage-opt` flags.
209 
+
210 
+  The `data` and `metadata` devices that you pass to the `--storage-opt` options were created in the previous steps.
211 
+
212 
+        $ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata &
213 
+        [1] 2163
214 
+        [root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
215 
+        INFO[0027] Option DefaultDriver: bridge
216 
+        INFO[0027] Option DefaultNetwork: bridge
217 
+        <output truncated>
218 
+        INFO[0027] Daemon has completed initialization
219 
+        INFO[0027] Docker daemon                                 commit=0a8c2e3 execdriver=native-0.2 graphdriver=devicemapper version=1.8.2
220 
+
221 
+    It is also possible to set the `--storage-driver` and `--storage-opt` flags in the Docker config file and start the daemon normally using the `service` or `systemd` commands.
222 
+
223 
+6. Use the `docker info` command to verify that the daemon is using `data` and `metadata` devices you created.
224 
+
225 
+        $ sudo docker info
226 
+        INFO[0180] GET /v1.20/info
227 
+        Containers: 0
228 
+        Images: 0
229 
+        Storage Driver: devicemapper
230 
+         Pool Name: docker-202:1-1032-pool
231 
+         Pool Blocksize: 65.54 kB
232 
+         Backing Filesystem: xfs
233 
+         Data file: /dev/vg-docker/data
234 
+         Metadata file: /dev/vg-docker/metadata
235 
+        [...]
236 
+
237 
+    The output of the command above shows the storage driver as `devicemapper`. The last two lines also confirm that the correct devices are being used for the `Data file` and the `Metadata file`.
238 
+
239 
+### Examine devicemapper structures on the host
240 
+
241 
+You can use the `lsblk` command to see the device files created above and the `pool` that the `devicemapper` storage driver creates on top of them.
242 
+
243 
+    $ sudo lsblk
244 
+    NAME                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
245 
+    xvda                       202:0    0    8G  0 disk
246 
+    └─xvda1                    202:1    0    8G  0 part /
247 
+    xvdf                       202:80   0  100G  0 disk
248 
+    ├─vg--docker-data          253:0    0   90G  0 lvm
249 
+    │ └─docker-202:1-1032-pool 253:2    0  100G  0 dm
250 
+    └─vg--docker-metadata      253:1    0    4G  0 lvm
251 
+      └─docker-202:1-1032-pool 253:2    0  100G  0 dm
252 
+
253 
+The diagram below shows the image from prior examples updated with the detail from the `lsblk` command above.
254 
+
255 
+![](http://farm1.staticflickr.com/703/22116692899_0471e5e160_b.jpg)
256 
+
257 
+In the diagram, the pool is named `Docker-202:1-1032-pool` and spans the `data` and `metadata` devices created earlier. The `devicemapper` constructs the pool name as follows:
258 
+
259 
+```
260 
+Docker-MAJ:MIN-INO-pool
261 
+```
262 
+
263 
+`MAJ`, `MIN` and `INO` refer to the major and minor device numbers and inode.
264 
+
265 
+Because Device Mapper operates at the block level it is more difficult to see
266 
+diffs between image layers and containers. However, there are two key
267 
+directories. The `/var/lib/docker/devicemapper/mnt` directory contains the mount
268 
+points for images and containers. The `/var/lib/docker/devicemapper/metadata`
269 
+directory contains one file for every image and container snapshot. The files
270 
+contain metadata about each snapshot in JSON format.
271 
+
272 
+## Device Mapper and Docker performance
273 
+
274 
+It is important to understand the impact that allocate-on-demand and copy-on-write operations can have on overall container performance.
275 
+
276 
+### Allocate-on-demand performance impact
277 
+
278 
+The `devicemapper` storage driver allocates new blocks to a container via an allocate-on-demand operation. This means that each time an app writes to somewhere new inside a container, one or more empty blocks has to be located from the pool and mapped into the container.
279 
+
280 
+All blocks are 64KB. A write that uses less than 64KB still results in a single 64KB block being allocated. Writing more than 64KB of data uses multiple 64KB blocks. This can impact container performance, especially in containers that perform lots of small writes. However, once a block is allocated to a container subsequent reads and writes can operate directly on that block.
281 
+
282 
+### Copy-on-write performance impact
283 
+
284 
+Each time a container updates existing data for the first time, the `devicemapper` storage driver has to perform a copy-on-write operation. This copies the data from the image snapshot to the container's snapshot. This process can have a noticeable impact on container performance.
285 
+
286 
+All copy-on-write operations have a 64KB granularity. As a results, updating 32KB of a 1GB file causes the driver to copy a single 64KB block into the container's snapshot. This has obvious performance advantages over file-level copy-on-write operations which would require copying the entire 1GB file into the container layer.
287 
+
288 
+In practice, however, containers that perform lots of small block writes (<64KB) can perform worse with `devicemapper` than with AUFS.
289 
+
290 
+### Other device mapper performance considerations
291 
+
292 
+There are several other things that impact the performance of the `devicemapper` storage driver..
293 
+
294 
+- **The mode.** The default mode for Docker running the `devicemapper` storage driver is `loop-lvm`. This mode uses sparse files and suffers from poor performance. It is **not recommended for production**. The recommended mode for production environments is `direct-lvm` where the storage driver writes directly to raw block devices.
295 
+
296 
+- **High speed storage.** For best performance you should place the `Data file` and `Metadata file` on high speed storage such as SSD. This can be direct attached storage or from a SAN or NAS array.
297 
+
298 
+- **Memory usage.** `devicemapper` is not the most memory efficient Docker storage driver. Launching *n* copies of the same container loads *n* copies of its files into memory. This can have a memory impact on your Docker host. As a result, the `devicemapper` storage driver may not be the best choice for PaaS and other high density use cases.
299 
+
300 
+One final point, data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.
301 
+
302 
+## Related Information
303 
+
304 
+* [Understand images, containers, and storage drivers](imagesandcontainers.md)
305 
+* [Select a storage driver](selectadriver.md)
306 
+* [AUFS storage driver in practice](aufs-driver.md)
307 
+* [BTRFS storage driver in practice](btrfs-driver.md)
docs/userguide/storagedriver/images/aufs_delete.jpg
History View file @ c50bd84
0 308 
new file mode 100644
1 309 
Binary files /dev/null and b/docs/userguide/storagedriver/images/aufs_delete.jpg differ
docs/userguide/storagedriver/images/aufs_layers.jpg
History View file @ c50bd84
2 310 
new file mode 100644
3 311 
Binary files /dev/null and b/docs/userguide/storagedriver/images/aufs_layers.jpg differ
docs/userguide/storagedriver/images/aufs_metadata.jpg
History View file @ c50bd84
4 312 
new file mode 100644
5 313 
Binary files /dev/null and b/docs/userguide/storagedriver/images/aufs_metadata.jpg differ
docs/userguide/storagedriver/images/base_device.jpg
History View file @ c50bd84
6 314 
new file mode 100644
7 315 
Binary files /dev/null and b/docs/userguide/storagedriver/images/base_device.jpg differ
docs/userguide/storagedriver/images/btfs_constructs.jpg
History View file @ c50bd84
8 316 
new file mode 100644
9 317 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_constructs.jpg differ
docs/userguide/storagedriver/images/btfs_container_layer.jpg
History View file @ c50bd84
10 318 
new file mode 100644
11 319 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_container_layer.jpg differ
docs/userguide/storagedriver/images/btfs_layers.png
History View file @ c50bd84
12 320 
new file mode 100644
13 321 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_layers.png differ
docs/userguide/storagedriver/images/btfs_pool.jpg
History View file @ c50bd84
14 322 
new file mode 100644
15 323 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_pool.jpg differ
docs/userguide/storagedriver/images/btfs_snapshots.jpg
History View file @ c50bd84
16 324 
new file mode 100644
17 325 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_snapshots.jpg differ
docs/userguide/storagedriver/images/btfs_subvolume.jpg
History View file @ c50bd84
18 326 
new file mode 100644
19 327 
Binary files /dev/null and b/docs/userguide/storagedriver/images/btfs_subvolume.jpg differ
docs/userguide/storagedriver/images/container-layers.jpg
History View file @ c50bd84
20 328 
new file mode 100644
21 329 
Binary files /dev/null and b/docs/userguide/storagedriver/images/container-layers.jpg differ
docs/userguide/storagedriver/images/dm_container.jpg
History View file @ c50bd84
22 330 
new file mode 100644
23 331 
Binary files /dev/null and b/docs/userguide/storagedriver/images/dm_container.jpg differ
docs/userguide/storagedriver/images/image-layers.jpg
History View file @ c50bd84
24 332 
new file mode 100644
25 333 
Binary files /dev/null and b/docs/userguide/storagedriver/images/image-layers.jpg differ
docs/userguide/storagedriver/images/overlay_constructs.jpg
History View file @ c50bd84
26 334 
new file mode 100644
27 335 
Binary files /dev/null and b/docs/userguide/storagedriver/images/overlay_constructs.jpg differ
docs/userguide/storagedriver/images/overlay_constructs2.jpg
History View file @ c50bd84
28 336 
new file mode 100644
29 337 
Binary files /dev/null and b/docs/userguide/storagedriver/images/overlay_constructs2.jpg differ
docs/userguide/storagedriver/images/saving-space.jpg
History View file @ c50bd84
30 338 
new file mode 100644
31 339 
Binary files /dev/null and b/docs/userguide/storagedriver/images/saving-space.jpg differ
docs/userguide/storagedriver/images/shared-uuid.jpg
History View file @ c50bd84
32 340 
new file mode 100644
33 341 
Binary files /dev/null and b/docs/userguide/storagedriver/images/shared-uuid.jpg differ
docs/userguide/storagedriver/images/shared-volume.jpg
History View file @ c50bd84
34 342 
new file mode 100644
35 343 
Binary files /dev/null and b/docs/userguide/storagedriver/images/shared-volume.jpg differ
docs/userguide/storagedriver/images/sharing-layers.jpg
History View file @ c50bd84
36 344 
new file mode 100644
37 345 
Binary files /dev/null and b/docs/userguide/storagedriver/images/sharing-layers.jpg differ
docs/userguide/storagedriver/images/two_dm_container.jpg
History View file @ c50bd84
38 346 
new file mode 100644
39 347 
Binary files /dev/null and b/docs/userguide/storagedriver/images/two_dm_container.jpg differ
docs/userguide/storagedriver/images/zfs_clones.jpg
History View file @ c50bd84
40 348 
new file mode 100644
41 349 
Binary files /dev/null and b/docs/userguide/storagedriver/images/zfs_clones.jpg differ
docs/userguide/storagedriver/images/zfs_zpool.jpg
History View file @ c50bd84
42 350 
new file mode 100644
43 351 
Binary files /dev/null and b/docs/userguide/storagedriver/images/zfs_zpool.jpg differ
docs/userguide/storagedriver/images/zpool_blocks.jpg
History View file @ c50bd84
44 352 
new file mode 100644
45 353 
Binary files /dev/null and b/docs/userguide/storagedriver/images/zpool_blocks.jpg differ
docs/userguide/storagedriver/imagesandcontainers.md
History View file @ c50bd84
46 354 
new file mode 100644
... ... 
@@ -0,0 +1,255 @@
0 
+<!--[metadata]>
1 
+title = "Understand images, containers, and storage drivers"
2 
+description = "Learn the technologies that support storage drivers."
3 
+keywords = ["container, storage, driver, AUFS, btfs, devicemapper,zvfs"]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+weight = -2
7 
+<![end-metadata]-->
8 
+
9 
+
10 
+# Understand images, containers, and storage drivers
11 
+
12 
+To use storage drivers effectively, you must understand how Docker builds and
13 
+stores images. Then, you need an understanding of how these images are used in  containers. Finally, you'll need a short introduction to the technologies that enable both images and container operations.
14 
+
15 
+## Images and containers rely on layers
16 
+
17 
+Docker images are a series of read-only layers that are stacked
18 
+on top of each other to form a single unified view. The first image in the stack
19 
+is called a *base image* and all the other layers are stacked on top of this
20 
+layer. The diagram below shows the Ubuntu 15:04 image comprising 4 stacked image layers.
21 
+
22 
+![](images/image-layers.jpg)
23 
+
24 
+When you make a change inside a container by, for example, adding a new file to the Ubuntu 15.04 image, you add a new layer on top of the underlying image stack. This change creates a new image layer containing the newly added file. Each image layer has its own universal unique identifier (UUID) and each successive image layer builds on top of the image layer below it.
25 
+
26 
+Containers (in the storage context) are a combination of a Docker image with a
27 
+thin writable layer added to the top known as the *container layer*.  The diagram below shows a container running the Ubuntu 15.04 image.
28 
+
29 
+![](images/container-layers.jpg)
30 
+
31 
+The major difference between a container and an image is this writable layer. All writes to the container that add new or modifying existing data are stored in this writable layer. When the container is deleted the writeable layer is also deleted. The image remains unchanged.
32 
+
33 
+Because each container has its own thin writable container layer and all data is stored this container layer, this means that multiple containers can share access to the same underlying image and yet have their own data state. The diagram below shows multiple containers sharing the same Ubuntu 15.04 image.
34 
+
35 
+![](images/sharing-layers.jpg)
36 
+
37 
+A storage driver is responsible for enabling and managing both the image layers and the writeable container layer. How a storage driver accomplishes these behaviors can vary. Two key technologies behind Docker image and container management are stackable image layers and copy-on-write (CoW).
38 
+
39 
+
40 
+## The copy-on-write strategy
41 
+
42 
+Sharing is a good way to optimize resources. People do this instinctively in
43 
+daily life. For example, twins Jane and Joseph taking an Algebra class at
44 
+different times from different teachers can share the same exercise book by
45 
+passing it between each other. Now, suppose Jane gets an assignment to complete
46 
+the homework on page 11 in the book. At that point, Jane copy page 11, complete the homework, and hand in her copy. The original exercise book is unchanged and only Jane has a copy of the changed page 11.
47 
+
48 
+Copy-on-write is a similar strategy of sharing and copying. In this strategy,
49 
+system processes that need the same data share the same instance of that data
50 
+rather than having their own copy. At some point, if one process needs to modify
51 
+or write to the data, only then does the operating system make a copy of the
52 
+data for that process to use. Only the process that needs to write has access to
53 
+the data copy. All the other processes continue to use the original data.
54 
+
55 
+Docker uses a copy-on-write technology with both images and containers. This CoW
56 
+strategy optimizes both image disk space usage and the performance of container
57 
+start times. The next sections look at how copy-on-write is leveraged with
58 
+images and containers thru sharing and copying.
59 
+
60 
+### Sharing promotes smaller images
61 
+
62 
+This section looks at image layers and copy-on-write technology.  All image and container layers exist inside the Docker host's *local storage area* and are managed by the storage driver. It is a location on the host's
63 
+filesystem.
64 
+
65 
+The Docker client reports on image layers when instructed to pull and push
66 
+images with `docker pull` and `docker push`. The command below pulls the
67 
+`ubuntu:15.04` Docker image from Docker Hub.
68 
+
69 
+    $ docker pull ubuntu:15.04
70 
+    15.04: Pulling from library/ubuntu
71 
+    6e6a100fa147: Pull complete
72 
+    13c0c663a321: Pull complete
73 
+    2bd276ed39d5: Pull complete
74 
+    013f3d01d247: Pull complete
75 
+    Digest: sha256:c7ecf33cef00ae34b131605c31486c91f5fd9a76315d075db2afd39d1ccdf3ed
76 
+    Status: Downloaded newer image for ubuntu:15.04
77 
+
78 
+From the output, you'll see  that the command actually pulls 4 image layers.
79 
+Each of the above lines lists an image layer and its UUID. The combination of
80 
+these four layers makes up the `ubuntu:15.04` Docker image.
81 
+
82 
+The image layers are stored in the Docker host's local storage area. Typically,
83 
+the local storage area is in the host's `/var/lib/docker` directory. Depending
84 
+on which storage driver the local storage area may be in a different location.  You can list the layers in the local storage area. The following example shows the storage as it appears under the AUFS storage driver:
85 
+
86 
+    $ sudo ls /var/lib/docker/aufs/layers
87 
+    013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778  2bd276ed39d5fcfd3d00ce0a190beeea508332f5aec3c6a125cc619a3fdbade6
88 
+    13c0c663a321cd83a97f4ce1ecbaf17c2ba166527c3b06daaefe30695c5fcb8c  6e6a100fa147e6db53b684c8516e3e2588b160fd4898b6265545d5d4edb6796d
89 
+
90 
+If you `pull` another image that shares some of the same image layers as the `ubuntu:15.04` image, the Docker daemon recognize this, and only pull the layers it hasn't already stored. After the second pull, the two images will share any common image layers.
91 
+
92 
+You can illustrate this now for yourself. Starting the `ubuntu:15.04` image that
93 
+you just pulled, make a change to it, and build a new image based on the change.
94 
+One way to do this is using a Dockerfile and the `docker build` command.
95 
+
96 
+1. In an empty directory, create a simple `Dockerfile` that starts with the ubuntu:15.04 image.
97 
+
98 
+        FROM ubuntu:15.04
99 
+
100 
+2. Add a new file called "newfile" in the image's `/tmp` directory with the text "Hello world" in it.
101 
+
102 
+  When you are done, the `Dockerfile` contains two lines:
103 
+
104 
+        FROM ubuntu:15.04
105 
+
106 
+        RUN echo "Hello world" > /tmp/newfile
107 
+
108 
+3. Save and close the file.
109 
+
110 
+2. From a terminal in the same folder as your Dockerfile, run the following command:
111 
+
112 
+        $ docker build -t changed-ubuntu .
113 
+        Sending build context to Docker daemon 2.048 kB
114 
+        Step 0 : FROM ubuntu:15.04
115 
+         ---> 013f3d01d247
116 
+        Step 1 : RUN echo "Hello world" > /tmp/newfile
117 
+         ---> Running in 2023460815df
118 
+         ---> 03b964f68d06
119 
+        Removing intermediate container 2023460815df
120 
+        Successfully built 03b964f68d06
121 
+
122 
+    > **Note:** The period (.) at the end of the above command is important. It tells the `docker build` command to use the current working directory as its build context.
123 
+
124 
+    The output above shows a new image with image ID `03b964f68d06`.
125 
+
126 
+3. Run the `docker images` command to verify the new image is in the Docker host's local storage area.
127 
+
128 
+        REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
129 
+        changed-ubuntu      latest              03b964f68d06        33 seconds ago      131.4 MB
130 
+        ubuntu
131 
+
132 
+4. Run the `docker history` command to see which image layers were used to create the new `changed-ubuntu` image.
133 
+
134 
+        $ docker history changed-ubuntu
135 
+        IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
136 
+        03b964f68d06        About a minute ago   /bin/sh -c echo "Hello world" > /tmp/newfile    12 B
137 
+        013f3d01d247        6 weeks ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
138 
+        2bd276ed39d5        6 weeks ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.879 kB
139 
+        13c0c663a321        6 weeks ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   701 B
140 
+        6e6a100fa147        6 weeks ago          /bin/sh -c #(nop) ADD file:49710b44e2ae0edef4   131.4 MB
141 
+
142 
+    The `docker history` output shows the new `03b964f68d06` image layer at the
143 
+    top. You know that the `03b964f68d06` layer was added because it was created
144 
+    by the `echo "Hello world" > /tmp/newfile` command in your `Dockerfile`.
145 
+    The 4 image layers below it are the exact same image layers the make up the
146 
+    ubuntu:15.04 image as their UUIDs match.
147 
+
148 
+5. List the contents of the local storage area to further confirm.
149 
+
150 
+        $ sudo ls /var/lib/docker/aufs/layers
151 
+        013f3d01d24738964bb7101fa83a926181d600ebecca7206dced59669e6e6778  2bd276ed39d5fcfd3d00ce0a190beeea508332f5aec3c6a125cc619a3fdbade6
152 
+        03b964f68d06a373933bd6d61d37610a34a355c168b08dfc604f57b20647e073  6e6a100fa147e6db53b684c8516e3e2588b160fd4898b6265545d5d4edb6796d
153 
+        13c0c663a321cd83a97f4ce1ecbaf17c2ba166527c3b06daaefe30695c5fcb8c
154 
+
155 
+      Where before you had four layers stored, you now have 5.
156 
+
157 
+Notice the new `changed-ubuntu` image does not have its own copies of every layer. As can be seen in the diagram below, the new image is sharing it's four underlying layers with the `ubuntu:15.04` image.
158 
+
159 
+![](images/saving-space.jpg)
160 
+
161 
+The `docker history` command also shows the size of each image layer. The `03b964f68d06` is only consuming 13 Bytes of disk space. Because all of the layers below it already exist on the Docker host and are shared with the `ubuntu15:04` image, this means the entire `changed-ubuntu` image only consumes 13 Bytes of disk space.
162 
+
163 
+This sharing of image layers is what makes Docker images and containers so space
164 
+efficient.
165 
+
166 
+### Copying makes containers efficient
167 
+
168 
+You learned earlier that a container a Docker image with a thin writable, container layer added. The diagram below shows the layers of a container based on the `ubuntu:15.04` image:
169 
+
170 
+![](images/container-layers.jpg)
171 
+
172 
+All writes made to a container are stored in the thin writable container layer. The other layers are read-only (RO) image layers and can't be changed. This means that multiple containers can safely share a single underlying image. The diagram below shows multiple containers sharing a single copy of the `ubuntu:15.04` image. Each container has its own thin RW layer, but they all share a single instance of the ubuntu:15.04 image:
173 
+
174 
+![](images/sharing-layers.jpg)
175 
+
176 
+When a write operation occurs in a container, Docker uses the storage driver to perform a copy-on-write operation. The type of operation depends on the storage driver. For AUFS and OverlayFS storage drivers the copy-on-write operation is pretty much as follows:
177 
+
178 
+*  Search through the layers for the file to update. The process starts at the top, newest layer and works down to the base layer one-at-a-time.
179 
+*  Perform a "copy-up" operation on the first copy of the file that is found. A "copy up" copies the file up to the container's own thin writable layer.
180 
+* Modify the *copy of the file* in container's thin writable layer.
181 
+
182 
+BTFS, ZFS, and other drivers handle the copy-on-write differently. You can read more about the methods of these drivers later in their detailed descriptions.
183 
+
184 
+Containers that write a lot of data will consume more space than containers that do not. This is because most write operations consume new space in the containers thin writable top layer. If your container needs to write a lot of data, you can use a data volume.
185 
+
186 
+A copy-up operation can incur a noticeable performance overhead. This overhead is different depending on which storage driver is in use. However, large files, lots of layers, and deep directory trees can make the impact more noticeable. Fortunately, the operation only occurs the first time any particular file is modified. Subsequent modifications to the same file do not cause a copy-up operation and can operate directly on the file's existing copy already present in container layer.
187 
+
188 
+Let's see what happens if we spin up 5 containers based on our `changed-ubuntu` image we built earlier:
189 
+
190 
+1. From a terminal on your Docker host, run the following `docker run` command 5 times.
191 
+
192 
+        $ docker run -dit changed-ubuntu bash
193 
+        75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4
194 
+        $ docker run -dit changed-ubuntu bash
195 
+        9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47
196 
+        $ docker run -dit changed-ubuntu bash
197 
+        a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a
198 
+        $ docker run -dit changed-ubuntu bash
199 
+        8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373
200 
+        $ docker run -dit changed-ubuntu bash
201 
+        0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef
202 
+
203 
+  This launches 5 containers based on the `changed-ubuntu` image.  As the container is created, Docker adds a writable layer and assigns it a UUID. This is the value returned from the `docker run` command.
204 
+
205 
+2. Run the `docker ps` command to verify the 5 containers are running.
206 
+
207 
+        $ docker ps
208 
+        CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
209 
+        0ad25d06bdf6        changed-ubuntu      "bash"              About a minute ago   Up About a minute                       stoic_ptolemy
210 
+        8eb24b3b2d24        changed-ubuntu      "bash"              About a minute ago   Up About a minute                       pensive_bartik
211 
+        a651680bd6c2        changed-ubuntu      "bash"              2 minutes ago        Up 2 minutes                            hopeful_turing
212 
+        9280e777d109        changed-ubuntu      "bash"              2 minutes ago        Up 2 minutes                            backstabbing_mahavira
213 
+        75bab0d54f3c        changed-ubuntu      "bash"              2 minutes ago        Up 2 minutes                            boring_pasteur
214 
+
215 
+    The output above shows 5 running containers, all sharing the `changed-ubuntu` image. Each `CONTAINER ID` is derived from the UUID when creating each container.
216 
+
217 
+3. List the contents of the local storage area.
218 
+
219 
+        $ sudo ls containers
220 
+        0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef  9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47
221 
+        75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4  a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a
222 
+        8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373
223 
+
224 
+Docker's copy-on-write strategy not only reduces the amount of space consumed by containers, it also reduces the time required to start a container. At start time, Docker only has to create the thin writable layer for each container. The diagram below shows these 5 containers sharing a single read-only (RO) copy of the `changed-ubuntu` image.
225 
+
226 
+![](images/shared-uuid.jpg)
227 
+
228 
+If Docker had to make an entire copy of the underlying image stack each time it
229 
+started a new container, container start times and disk space used would be
230 
+significantly increased.
231 
+
232 
+## Data volumes and the storage driver
233 
+
234 
+When a container is deleted, any data written to the container that is not stored in a *data volume* is deleted along with the container. A data volume is directory or file that is mounted directly into a container.
235 
+
236 
+Data volumes are not controlled by the storage driver. Reads and writes to data
237 
+volumes bypass the storage driver and operate at native host speeds. You can mount any number of data volumes into a container. Multiple containers can also share one or more data volumes.
238 
+
239 
+The diagram below shows a single Docker host running two containers. Each container exists inside of its own address space within the Docker host's local storage area. There is also a single shared data volume located at `/data` on the Docker host. This is mounted directly into both containers.
240 
+
241 
+![](images/shared-volume.jpg)
242 
+
243 
+The data volume resides outside of the local storage area on the Docker host further reinforcing its independence from the storage driver's control. When a container is deleted, any data stored in shared data volumes persists on the Docker host.
244 
+
245 
+For detailed information about data volumes [Managing data in containers](https://docs.docker.com/userguide/dockervolumes/).
246 
+
247 
+## Related information
248 
+
249 
+* [Select a storage driver](selectadriver.md)
250 
+* [AUFS storage driver in practice](aufs-driver.md)
251 
+* [BTRFS storage driver in practice](btrfs-driver.md)
252 
+* [Device Mapper storage driver in practice](device-mapper-driver.md)
docs/userguide/storagedriver/index.md
History View file @ c50bd84
0 253 
new file mode 100644
... ... 
@@ -0,0 +1,28 @@
0 
+<!--[metadata]>
1 
+title = "Docker storage drivers"
2 
+description = "Learn how select the proper storage driver for your container."
3 
+keywords = ["container, storage, driver, AUFS, btfs, devicemapper,zvfs"]
4 
+[menu.main]
5 
+identifier = "mn_storage_docker"
6 
+parent = "mn_use_docker"
7 
+weight = 7
8 
+<![end-metadata]-->
9 
+
10 
+
11 
+# Docker storage drivers
12 
+
13 
+Docker relies on driver technology to manage the storage and interactions associated with images and they containers that run them. This section contains the following pages:
14 
+
15 
+* [Understand images, containers, and storage drivers](imagesandcontainers.md)
16 
+* [Select a storage driver](selectadriver.md)
17 
+* [AUFS storage driver in practice](aufs-driver.md)
18 
+* [BTRFS storage driver in practice](btrfs-driver.md)
19 
+* [Device Mapper storage driver in practice](device-mapper-driver.md)
20 
+* [OverlayFS in practice](overlayfs-driver.md)
21 
+* [FS storage in practice](zfs-driver.md)
22 
+
23 
+If you are new to Docker containers make sure you read ["Understand images, containers, and storage drivers"](imagesandcontainers.md) first. It explains key concepts and technologies that can help you when working with storage drivers.
24 
+
25 
+&nbsp;
docs/userguide/storagedriver/overlayfs-driver.md
History View file @ c50bd84
0 26 
new file mode 100644
... ... 
@@ -0,0 +1,190 @@
0 
+<!--[metadata]>
1 
+title = "OverlayFS storage in practice"
2 
+description = "Learn how to optimize your use of OverlayFS driver."
3 
+keywords = ["container, storage, driver, OverlayFS "]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+<![end-metadata]-->
7 
+
8 
+# Docker and OverlayFS in practice
9 
+
10 
+OverlayFS is a modern *union filesystem* that is similar to AUFS. In comparison to AUFS, OverlayFS:
11 
+
12 
+* has a simpler design
13 
+* has been in the mainline Linux kernel since version 3.18
14 
+* is potentially faster
15 
+
16 
+As a result, OverlayFS is rapidly gaining popularity in the Docker community and is seen by many as a natural successor to AUFS. As promising as OverlayFS is, it is still relatively young. Therefore caution should be taken before using it in production Docker environments.
17 
+
18 
+Docker's `overlay` storage driver leverages several OverlayFS features to build and manage the on-disk structures of images and containers.
19 
+
20 
+>**Note**: Since it was merged into the mainline kernel, the OverlayFS *kernel module* was renamed from "overlayfs" to "overlay". As a result you may see the two terms used interchangeably in some documentation. However, this document uses  "OverlayFS" to refer to the overall filesystem, and `overlay` to refer to Docker's storage-driver.
21 
+
22 
+
23 
+## Image layering and sharing with OverlayFS
24 
+
25 
+OverlayFS takes two directories on a single Linux host, layers one on top of the other, and provides a single unified view. These directories are often referred to as *layers* and the technology used to layer them is is known as a *union mount*. The OverlayFS terminology is "lowerdir" for the bottom layer and "upperdir" for the top layer. The unified view is exposed through its own directory called "merged".
26 
+
27 
+The diagram below shows how a Docker image and a Docker container are layered. The image layer is the "lowerdir" and the container layer is the "upperdir". The unified view is exposed through a directory called "merged" which is effectively the containers mount point. The diagram shows how Docker constructs map to OverlayFS constructs.
28 
+
29 
+![](images/overlay_constructs.jpg)
30 
+
31 
+Notice how the image layer and container layer can contain the same files. When this happens, the files in the container layer ("upperdir") are dominant and obscure the existence of the same files in the image layer ("lowerdir"). The container mount ("merged") presents the unified view.
32 
+
33 
+OverlayFS only works with two layers. This means that multi-layered images cannot be implemented as multiple OverlayFS layers. Instead, each image layer is implemented as its own directory under `/var/lib/docker/overlay`. Hard links are then used as a space-efficient way to reference data shared with lower layers. The diagram below shows a four-layer image and how it is represented in the Docker host's filesystem.
34 
+
35 
+![](images/overlay_constructs2.jpg)
36 
+
37 
+To create a container, the `overlay` driver combines the directory representing the image's top layer plus a new directory for the container. The image's top layer is the "lowerdir" in the overlay and read-only. The new directory for the container is the "upperdir" and is writable.
38 
+
39 
+## Example: Image and container on-disk constructs
40 
+
41 
+The following `docker images -a` command shows a Docker host with a single image. As can be seen, the image consists of four layers.
42 
+
43 
+    $ docker images -a
44 
+    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
45 
+    ubuntu              latest              1d073211c498        7 days ago          187.9 MB
46 
+    <none>              <none>              5a4526e952f0        7 days ago          187.9 MB
47 
+    <none>              <none>              99fcaefe76ef        7 days ago          187.9 MB
48 
+    <none>              <none>              c63fb41c2213        7 days ago          187.7 MB
49 
+
50 
+Below, the command's output illustrates that each of the four image layers has it's own directory under `/var/lib/docker/overlay/`.
51 
+
52 
+    $ ls -l /var/lib/docker/overlay/
53 
+    total 24
54 
+    drwx------ 3 root root 4096 Oct 28 11:02 1d073211c498fd5022699b46a936b4e4bdacb04f637ad64d3475f558783f5c3e
55 
+    drwx------ 3 root root 4096 Oct 28 11:02 5a4526e952f0aa24f3fcc1b6971f7744eb5465d572a48d47c492cb6bbf9cbcda
56 
+    drwx------ 5 root root 4096 Oct 28 11:06 99fcaefe76ef1aa4077b90a413af57fd17d19dce4e50d7964a273aae67055235
57 
+    drwx------ 3 root root 4096 Oct 28 11:01 c63fb41c2213f511f12f294dd729b9903a64d88f098c20d2350905ac1fdbcbba
58 
+
59 
+Each directory is named after the image layer IDs in the previous `docker images -a` command. The image layer directories contain the files unique to that layer as well as hard links to the data that is shared with lower layers. This allows for efficient use of disk space.
60 
+
61 
+The following `docker ps` command shows the same Docker host running a single container. The container ID is "73de7176c223".
62 
+
63 
+    $ docker ps
64 
+    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
65 
+    73de7176c223        ubuntu              "bash"              2 days ago          Up 2 days                               stupefied_nobel
66 
+
67 
+This container exists on-disk in the Docker host's filesystem under `/var/lib/docker/overlay/73de7176c223...`.  If you inspect this directory using the `ls -l` command you find the following file and directories.
68 
+
69 
+    $ ls -l /var/lib/docker/overlay/73de7176c223a6c82fd46c48c5f152f2c8a7e49ecb795a7197c3bb795c4d879e
70 
+    total 16
71 
+    -rw-r--r-- 1 root root   64 Oct 28 11:06 lower-id
72 
+    drwxr-xr-x 1 root root 4096 Oct 28 11:06 merged
73 
+    drwxr-xr-x 4 root root 4096 Oct 28 11:06 upper
74 
+    drwx------ 3 root root 4096 Oct 28 11:06 work
75 
+
76 
+These four filesystem objects are all artifacts of OverlayFS. The "lower-id" file contains the ID of the top layer of the image the container is based on. This is used by OverlayFS as the "lowerdir".
77 
+
78 
+    $ cat /var/lib/docker/overlay/73de7176c223a6c82fd46c48c5f152f2c8a7e49ecb795a7197c3bb795c4d879e/lower-id
79 
+    1d073211c498fd5022699b46a936b4e4bdacb04f637ad64d3475f558783f5c3e
80 
+
81 
+The "upper" directory is the containers read-write layer. Any changes made to the container are written to this directory.
82 
+
83 
+The "merged" directory is effectively the containers mount point. This is where the unified view of the image ("lowerdir") and container ("upperdir") is exposed. Any changes written to the container are immediately reflected in this directory.
84 
+
85 
+The "work" directory is required for OverlayFS to function. It is used for things such as *copy_up* operations.
86 
+
87 
+You can verify all of these constructs from the output of the `mount` command. (Ellipses and line breaks are used in the output below to enhance readability.)
88 
+
89 
+    $ mount | grep overlay
90 
+    overlay on /var/lib/docker/overlay/73de7176c223.../merged
91 
+    type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/1d073211c498.../root,
92 
+    upperdir=/var/lib/docker/overlay/73de7176c223.../upper,
93 
+    workdir=/var/lib/docker/overlay/73de7176c223.../work)
94 
+
95 
+The output reflects the overlay is mounted as read-write ("rw").
96 
+
97 
+## Container reads and writes with overlay
98 
+
99 
+Consider three scenarios where a container opens a file for read access with overlay.
100 
+
101 
+- **The file does not exist in the container layer**. If a container opens a file for read access and the file does not already exist in the container ("upperdir") it is read from the image ("lowerdir"). This should incur very little performance overhead.
102 
+
103 
+- **The file only exists in the container layer**. If a container opens a file for read access and the file exists in the container ("upperdir") and not in the image ("lowerdir"), it is read directly from the container.
104 
+
105 
+- **The file exists in the container layer and the image layer**. If a container opens a file for read access and the file exists in the image layer and the container layer, the file's version in the container layer is read. This is because files in the container layer ("upperdir") obscure files with the same name in the image layer ("lowerdir").
106 
+
107 
+Consider some scenarios where files in a container are modified.
108 
+
109 
+- **Writing to a file for the first time**. The first time a container writes to an existing file, that file does not exist in the container ("upperdir"). The `overlay` driver performs a *copy_up* operation to copy the file from the image ("lowerdir") to the container ("upperdir"). The container then writes the changes to the new copy of the file in the container layer.
110 
+
111 
+    However, OverlayFS works at the file level not the block level. This means that all OverlayFS copy-up operations copy entire files, even if the file is very large and only a small part of it is being modified. This can have a noticeable impact on container write performance. However, two things are worth noting:
112 
+
113 
+    * The copy_up operation only occurs the first time any given file is written to. Subsequent writes to the same file will operate against the copy of the file already copied up to the container.
114 
+
115 
+    * OverlayFS only works with two layers. This means that performance should be better than AUFS which can suffer noticeable latencies when searching for files in images with many layers.
116 
+
117 
+- **Deleting files and directories**. When files are deleted within a container a *whiteout* file is created in the containers "upperdir". The version of the file in the image layer ("lowerdir") is not deleted. However, the whiteout file in the container obscures it.
118 
+
119 
+    Deleting a directory in a container results in *opaque directory* being created in the "upperdir". This has the same effect as a whiteout file and effectively masks the existence of the directory in the image's "lowerdir".
120 
+
121 
+## Configure Docker with the overlay storage driver
122 
+
123 
+To configure Docker to use the overlay storage driver your Docker host must be running version 3.18 of the Linux kernel (preferably newer) with the overlay kernel module loaded.  OverlayFS can operate on top of most supported Linux filesystems. However, ext4 is currently recommended for use in production environments.
124 
+
125 
+The following procedure shows you how to configure your Docker host to use OverlayFS. The procedure assumes that the Docker daemon is in a stopped state.
126 
+
127 
+> **Caution:** If you have already run the Docker daemon on your Docker host and have images you want to keep, `push` them Docker Hub or your private Docker Trusted Registry before attempting this procedure.
128 
+
129 
+1. If it is running, stop the Docker `daemon`.
130 
+
131 
+2. Verify your kernel version and that the overlay kernel module is loaded.
132 
+
133 
+        $ uname -r
134 
+        3.19.0-21-generic
135 
+
136 
+        $ lsmod | grep overlay
137 
+        overlay
138 
+
139 
+3. Start the Docker daemon with the `overlay` storage driver.
140 
+
141 
+        $ docker daemon --storage-driver=overlay &
142 
+        [1] 29403
143 
+        root@ip-10-0-0-174:/home/ubuntu# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
144 
+        INFO[0000] Option DefaultDriver: bridge
145 
+        INFO[0000] Option DefaultNetwork: bridge
146 
+        <output truncated>
147 
+
148 
+    Alternatively, you can force the Docker daemon to automatically start with
149 
+    the `overlay` driver by editing the Docker config file and adding the
150 
+    `--storage-driver=overlay` flag to the `DOCKER_OPTS` line. Once this option
151 
+    is set you can start the daemon using normal startup scripts without having
152 
+    to manually pass in the `--storage-driver` flag.
153 
+
154 
+4. Verify that the daemon is using the `overlay` storage driver
155 
+
156 
+        $ docker info
157 
+        Containers: 0
158 
+        Images: 0
159 
+        Storage Driver: overlay
160 
+         Backing Filesystem: extfs
161 
+        <output truncated>
162 
+
163 
+    Notice that the *Backing filesystem* in the output above is showing as `extfs`. Multiple backing filesystems are supported but `extfs` (ext4) is recommended for production use cases.
164 
+
165 
+Your Docker host is now using the `overlay` storage driver. If you run the `mount` command, you'll find Docker has automatically created the `overlay` mount with the required "lowerdir", "upperdir", "merged" and "workdir" constructs.
166 
+
167 
+## OverlayFS and Docker Performance
168 
+
169 
+As a general rule, the `overlay` driver should be fast. Almost certainly faster than `aufs` and `devicemapper`. In certain circumstances it may also be faster than `btrfs`. That said, there are a few things to be aware of relative to the performance of Docker using the `overlay` storage driver.
170 
+
171 
+- **Page Caching**. OverlayFS supports page cache sharing. This means multiple containers accessing the same file can share a single page cache entry (or entries). This makes the `overlay` driver efficient with memory and a good option for PaaS and other high density use cases.
172 
+
173 
+- **copy_up**. As with AUFS, OverlayFS has to perform copy-up operations any time a container writes to a file for the first time. This can insert latency into the write operation &mdash; especially if the file being copied up is large. However, once the file has been copied up, all subsequent writes to that file occur without the need for further copy-up operations.
174 
+
175 
+    The OverlayFS copy_up operation should be faster than the same operation with AUFS. This is because AUFS supports more layers than OverlayFS and it is possible to incur far larger latencies if searching through many AUFS layers.
176 
+
177 
+- **RPMs and Yum**. OverlayFS only implements a subset of the POSIX standards. This can result in certain OverlayFS operations breaking POSIX standards. One such operation is the *copy-up* operation. Therefore, using `yum` inside of a container on a Docker host using the `overlay` storage driver is unlikely to work without implementing workarounds.
178 
+
179 
+- **Inode limits**. Use of the `overlay` storage driver can cause excessive inode consumption. This is especially so as the number of images and containers on the Docker host grows. A Docker host with a large number of images and lots of started and stopped containers can quickly run out of inodes.
180 
+
181 
+  Unfortunately you can only specify the number of inodes in a filesystem at the time of creation. For this reason, you may wish to consider putting `/var/lib/docker` on a separate device with its own filesystem or manually specifying the number of inodes when creating the filesystem.
182 
+
183 
+The following generic performance best practices also apply to OverlayFS.
184 
+
185 
+- **Solid State Devices (SSD)**. For best performance it is always a good idea to use fast storage media such as solid state devices (SSD).
186 
+
187 
+- **Use Data Volumes**. Data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.
docs/userguide/storagedriver/selectadriver.md
History View file @ c50bd84
0 188 
new file mode 100644
... ... 
@@ -0,0 +1,119 @@
0 
+<!--[metadata]>
1 
+title = "Select a storage driver"
2 
+description = "Learn how select the proper storage driver for your container."
3 
+keywords = ["container, storage, driver, AUFS, btfs, devicemapper,zvfs"]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+weight = -1
7 
+<![end-metadata]-->
8 
+
9 
+# Select a storage driver
10 
+
11 
+This page describes Docker's storage driver feature. It lists the storage
12 
+driver's that Docker supports and the basic commands associated with managing them. Finally, this page provides guidance on choosing a storage driver.
13 
+
14 
+The material on this page is intended for readers who already have an [understanding of the storage driver technology](imagesandcontainers.md).
15 
+
16 
+## A pluggable storage driver architecture
17 
+
18 
+The Docker has a pluggable storage driver architecture.  This gives you the flexibility to "plug in" the storage driver is best for your environment and use-case. Each Docker storage driver is based on a Linux filesystem or volume manager. Further, each storage driver is free to implement the management of image layers and the container layer in it's own unique way. This means some storage drivers perform better than others in different circumstances.
19 
+
20 
+Once you decide which driver is best, you set this driver on the Docker daemon at start time. As a result, the Docker daemon can only run one storage driver, and all containers created by that daemon instance use the same storage driver. The table below shows the supported storage driver technologies and the driver names:
21 
+
22 
+|Technology    |Storage driver name  |
23 
+|--------------|---------------------|
24 
+|OverlayFS     |`overlay`            |
25 
+|AUFS          |`aufs`               |
26 
+|BTRFS         |`btrfs`              |
27 
+|Device Maper  |`devicemapper`       |
28 
+|VFS*          |`vfs`                |
29 
+|ZFS           |`zfs`                |
30 
+
31 
+To find out which storage driver is set on the daemon , you use the `docker info` command:
32 
+
33 
+    $ docker info
34 
+    Containers: 0
35 
+    Images: 0
36 
+    Storage Driver: overlay
37 
+     Backing Filesystem: extfs
38 
+    Execution Driver: native-0.2
39 
+    Logging Driver: json-file
40 
+    Kernel Version: 3.19.0-15-generic
41 
+    Operating System: Ubuntu 15.04
42 
+    ... output truncated ...
43 
+
44 
+The `info` subcommand reveals that the Docker daemon is using the `overlay` storage driver with a `Backing Filesystem` value of `extfs`. The `extfs` value means that the `overlay` storage driver is operating on top of an existing (ext) filesystem. The backing filesystem refers to the filesystem that was used to create the Docker host's local storage area under `/var/lib/docker`.
45 
+
46 
+Which storage driver you use, in part, depends on the backing filesystem you plan to use for your Docker host's local storage area. Some storage drivers can operate on top of different backing filesystems. However, other storage drivers require the backing filesystem to be the same as the storage driver. For example, the `btrfs` storage driver on a `btrfs` backing filesystem. The following table lists each storage driver and whether it must match the host's backing file system:
47 
+
48 
+    |Storage driver |Must match backing filesystem |
49 
+    |---------------|------------------------------|
50 
+    |overlay        |No                            |
51 
+    |aufs           |No                            |
52 
+    |btrfs          |Yes                           |
53 
+    |devicemapper   |No                            |
54 
+    |vfs*           |No                            |
55 
+    |zfs            |Yes                           |
56 
+
57 
+
58 
+You pass the `--storage-driver=<name>` option to the `docker daemon` command line or by setting the option on the `DOCKER_OPTS` line in `/etc/defaults/docker` file.
59 
+
60 
+The following command shows how to start the Docker daemon with the `devicemapper` storage driver using the `docker daemon` command:
61 
+
62 
+    $ docker daemon --storage-driver=devicemapper &
63 
+
64 
+    $ docker info
65 
+    Containers: 0
66 
+    Images: 0
67 
+    Storage Driver: devicemapper
68 
+     Pool Name: docker-252:0-147544-pool
69 
+     Pool Blocksize: 65.54 kB
70 
+     Backing Filesystem: extfs
71 
+     Data file: /dev/loop0
72 
+     Metadata file: /dev/loop1
73 
+     Data Space Used: 1.821 GB
74 
+     Data Space Total: 107.4 GB
75 
+     Data Space Available: 3.174 GB
76 
+     Metadata Space Used: 1.479 MB
77 
+     Metadata Space Total: 2.147 GB
78 
+     Metadata Space Available: 2.146 GB
79 
+     Udev Sync Supported: true
80 
+     Deferred Removal Enabled: false
81 
+     Data loop file: /var/lib/docker/devicemapper/devicemapper/data
82 
+     Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
83 
+     Library Version: 1.02.90 (2014-09-01)
84 
+    Execution Driver: native-0.2
85 
+    Logging Driver: json-file
86 
+    Kernel Version: 3.19.0-15-generic
87 
+    Operating System: Ubuntu 15.04
88 
+    <output truncated>
89 
+
90 
+Your choice of storage driver can affect the performance of your containerized applications. So it's important to understand the different storage driver options available and select the right one for your application. Later, in this page you'll find some advice for choosing an appropriate driver.
91 
+
92 
+## Shared storage systems and the storage driver
93 
+
94 
+Many enterprises consume storage from shared storage systems such as SAN and NAS arrays. These often provide increased performance and availability, as well as advanced features such as thin provisioning, deduplication and compression.
95 
+
96 
+The Docker storage driver and data volumes can both operate on top of storage provided by shared storage systems. This allows Docker to leverage the increased performance and availability these systems provide. However, Docker does not integrate with these underlying systems.
97 
+
98 
+Remember that each Docker storage driver is based on a Linux filesystem or volume manager. Be sure to follow existing best practices for operating your storage driver (filesystem or volume manager) on top of your shared storage system. For example, if using the ZFS storage driver on top of *XYZ* shared storage system, be sure to follow best practices for operating ZFS filesystems on top of XYZ shared storage system.
99 
+
100 
+## Which storage driver should you choose?
101 
+
102 
+As you might expect, the answer to this question is "it depends". While there are some clear cases where one particular storage driver outperforms other for certain workloads, you should factor all of the following into your decision:
103 
+
104 
+Choose a storage driver that you and your team/organization are comfortable with.  Consider how much experience you have with a particular storage driver. There is no substitute for experience and it is rarely a good idea to try something brand new in production. That's what labs and laptops are for!
105 
+
106 
+If your Docker infrastructure is under support contracts, choose an option that will get you good support. You probably don't want to go with a solution that your support partners have little or no experience with.
107 
+
108 
+Whichever driver you choose, make sure it has strong community support and momentum. This is important because storage driver development in the Docker project relies on the community as much as the Docker staff to thrive.
109 
+
110 
+
111 
+## Related information
112 
+
113 
+* [Understand images, containers, and storage drivers](imagesandcontainers.md)
114 
+* [AUFS storage driver in practice](aufs-driver.md)
115 
+* [BTRFS storage driver in practice](btrfs-driver.md)
116 
+* [Device Mapper storage driver in practice](device-mapper-driver.md)
docs/userguide/storagedriver/zfs-driver.md
History View file @ c50bd84
0 117 
new file mode 100644
... ... 
@@ -0,0 +1,218 @@
0 
+<!--[metadata]>
1 
+title = "ZFS storage in practice"
2 
+description = "Learn how to optimize your use of ZFS driver."
3 
+keywords = ["container, storage, driver, ZFS "]
4 
+[menu.main]
5 
+parent = "mn_storage_docker"
6 
+<![end-metadata]-->
7 
+
8 
+# Docker and ZFS in practice
9 
+
10 
+ZFS is a next generation filesystem that supports many advanced storage technologies such as volume management, snapshots, checksumming, compression and deduplication, replication and more.
11 
+
12 
+It was created by Sun Microsystems (now Oracle Corporation) and is open sourced under the CDDL license. Due to licensing incompatibilities between the CDDL and GPL, ZFS cannot be shipped as part of the mainline Linux kernel. However, the ZFS On Linux (ZoL) project provides an out-of-tree kernel module and userspace tools which can be installed separately.
13 
+
14 
+The ZFS on Linux (ZoL) port is healthy and maturing. However, at this point in time it is not recommended to use the `zfs` Docker storage driver for production use unless you have substantial experience with ZFS on Linux.
15 
+
16 
+> **Note:** There is also a FUSE implementation of ZFS on the Linux platform. This should work with Docker but is not recommended. The native ZFS driver (ZoL) is more tested, more performant, and is more widely used. The remainder of this document will relate to the native ZoL port.
17 
+
18 
+
19 
+## Image layering and sharing with ZFS
20 
+
21 
+The Docker `zfs` storage driver makes extensive use of three ZFS datasets:
22 
+
23 
+- filesystems
24 
+- snapshots
25 
+- clones
26 
+
27 
+ZFS filesystems are thinly provisioned and have space allocated to them from a ZFS pool (zpool) via allocate on demand operations. Snapshots and clones are space-efficient point-in-time copies of ZFS filesystems. Snapshots are read-only. Clones are read-write. Clones can only be created from snapshots. This simple relationship is shown in the diagram below.
28 
+
29 
+![](images/zfs_clones.jpg)
30 
+
31 
+The solid line in the diagram shows the process flow for creating a clone. Step 1 creates a snapshot of the filesystem, and step two creates the clone from the snapshot. The dashed line shows the relationship between the clone and the filesystem, via the snapshot. All three ZFS datasets draw space form the same underlying zpool.
32 
+
33 
+On Docker hosts using the `zfs` storage driver, the base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it's created from. All ZFS datasets draw their space from a common zpool. The diagram below shows how this is put together with a running container based on a two-layer image.
34 
+
35 
+![](images/zfs_zpool.jpg)
36 
+
37 
+The following process explains how images are layered and containers created. The process is based on the diagram above.
38 
+
39 
+1. The base layer of the image exists on the Docker host as a ZFS filesystem.
40 
+
41 
+    This filesystem consumes space from the zpool used to create the Docker host's local storage area at `/var/lib/docker`.
42 
+
43 
+2. Additional image layers are clones of the dataset hosting the image layer directly below it.
44 
+
45 
+    In the diagram, "Layer 1" is added by making a ZFS snapshot of the base layer and then creating a clone from that snapshot. The clone is writable and consumes space on-demand from the zpool. The snapshot is read-only, maintaining the base layer as an immutable object.
46 
+
47 
+3. When the container is launched, a read-write layer is added above the image.
48 
+
49 
+    In the diagram above, the container's read-write layer is created by making a snapshot of the top layer of the image (Layer 1) and creating a clone from that snapshot.
50 
+
51 
+    As changes are made to the container, space is allocated to it from the zpool via allocate-on-demand operations. By default, ZFS will allocate space in blocks of 128K.
52 
+
53 
+This process of creating child layers and containers from *read-only* snapshots allows images to be maintained as immutable objects.
54 
+
55 
+## Container reads and writes with ZFS
56 
+
57 
+Container reads with the `zfs` storage driver are very simple. A newly launched container is based on a ZFS clone. This clone initially shares all of its data with the dataset it was created from. This means that read operations with the `zfs` storage driver are fast &ndash; even if the data being read was copied into the container yet. This sharing of data blocks is shown in the diagram below.
58 
+
59 
+![](images/zpool_blocks.jpg)
60 
+
61 
+Writing new data to a container is accomplished via an allocate-on-demand operation. Every time a new area of the container needs writing to, a new block is allocated from the zpool. This means that containers consume additional space as new data is written to them. New space is allocated to the container (ZFS Clone) from the underlying zpool.
62 
+
63 
+Updating *existing data* in a container is accomplished by allocating new blocks to the containers clone and storing the changed data in those new blocks. The original are unchanged, allowing the underlying image dataset to remain immutable. This is the same as writing to a normal ZFS filesystem and is an implementation of copy-on-write semantics.
64 
+
65 
+## Configure Docker with the ZFS storage driver
66 
+
67 
+The `zfs` storage driver is only supported on a Docker host where `/var/lib/docker` is mounted as a ZFS filesystem. This section shows you how to install and configure native ZFS on Linux (ZoL) on an Ubuntu 14.04 system.
68 
+
69 
+### Prerequisites
70 
+
71 
+If you have already used the Docker daemon on your Docker host and have images you want to keep, `push` them Docker Hub or your private Docker Trusted Registry before attempting this procedure.
72 
+
73 
+Stop the Docker daemon. Then, ensure that you have a spare block device at `/dev/xvdb`. The device identifier may be be different in your environment and you should substitute your own values throughout the procedure.
74 
+
75 
+### Install Zfs on Ubuntu 14.04 LTS
76 
+
77 
+1. If it is running, stop the Docker `daemon`.
78 
+
79 
+1. Install `the software-properties-common` package.
80 
+
81 
+    This is required for the `add-apt-repository` command.
82 
+
83 
+        $ sudo apt-get install software-properties-common
84 
+        Reading package lists... Done
85 
+        Building dependency tree
86 
+        <output truncated>
87 
+
88 
+2. Add the `zfs-native` package archive.
89 
+
90 
+        $ sudo add-apt-repository ppa:zfs-native/stable
91 
+         The native ZFS filesystem for Linux. Install the ubuntu-zfs package.
92 
+        <output truncated>
93 
+        gpg: key F6B0FC61: public key "Launchpad PPA for Native ZFS for Linux" imported
94 
+        gpg: Total number processed: 1
95 
+        gpg:               imported: 1  (RSA: 1)
96 
+        OK
97 
+
98 
+3. Get the latest package lists for all registered repositories and package archives.
99 
+
100 
+        $ sudo apt-get update
101 
+        Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
102 
+        Get:1 http://us-west-2.ec2.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
103 
+        <output truncated>
104 
+        Fetched 10.3 MB in 4s (2,370 kB/s)
105 
+        Reading package lists... Done
106 
+
107 
+4. Install the `ubuntu-zfs` package.
108 
+
109 
+        $ sudo apt-get install -y ubuntu-zfs
110 
+        Reading package lists... Done
111 
+        Building dependency tree
112 
+        <output truncated>
113 
+
114 
+5. Load the `zfs` module.
115 
+
116 
+        $ sudo modprobe zfs
117 
+
118 
+6. Verify that it loaded correctly.
119 
+
120 
+        $ lsmod | grep zfs
121 
+        zfs                  2768247  0
122 
+        zunicode              331170  1 zfs
123 
+        zcommon                55411  1 zfs
124 
+        znvpair                89086  2 zfs,zcommon
125 
+        spl                    96378  3 zfs,zcommon,znvpair
126 
+        zavl                   15236  1 zfs
127 
+
128 
+## Configure ZFS for Docker
129 
+
130 
+Once ZFS is installed and loaded, you're ready to configure ZFS for Docker.
131 
+
132 
+
133 
+1. Create a new `zpool`.
134 
+
135 
+        $ sudo zpool create -f zpool-docker /dev/xvdb
136 
+
137 
+  The command creates the `zpool` and gives it the name "zpool-docker". The name is arbitrary.
138 
+
139 
+2. Check that the `zpool` exists.
140 
+
141 
+        $ sudo zfs list
142 
+        NAME            USED  AVAIL    REFER  MOUNTPOINT
143 
+        zpool-docker    55K   3.84G    19K    /zpool-docker
144 
+
145 
+3. Create and mount a new ZFS filesystem to `/var/lib/docker`.
146 
+
147 
+        $ sudo zfs create -o mountpoint=/var/lib/docker zpool-docker/docker
148 
+
149 
+4. Check that the previous step worked.
150 
+
151 
+        $ sudo zfs list -t all
152 
+        NAME                 USED  AVAIL  REFER  MOUNTPOINT
153 
+        zpool-docker         93.5K  3.84G    19K  /zpool-docker
154 
+        zpool-docker/docker  19K    3.84G    19K  /var/lib/docker
155 
+
156 
+  Now that you have a ZFS filesystem mounted to `/var/lib/docker`, the daemon should automatically load with the `zfs` storage driver.
157 
+
158 
+5. Start the Docker daemon.
159 
+
160 
+        $ sudo service docker start
161 
+        docker start/running, process 2315
162 
+
163 
+    The procedure for starting the Docker daemon may differ depending on the
164 
+    Linux distribution you are using. It is possible to force the Docker daemon
165 
+    to start with the `zfs` storage driver by passing the `--storage-driver=zfs`
166 
+    flag to the `docker daemon` command, or to the `DOCKER_OPTS` line in the
167 
+    Docker config file.
168 
+
169 
+6. Verify that the daemon is using the `zfs` storage driver.
170 
+
171 
+        $ sudo docker info
172 
+        Containers: 0
173 
+        Images: 0
174 
+        Storage Driver: zfs
175 
+         Zpool: zpool-docker
176 
+         Zpool Health: ONLINE
177 
+         Parent Dataset: zpool-docker/docker
178 
+         Space Used By Parent: 27648
179 
+         Space Available: 4128139776
180 
+         Parent Quota: no
181 
+         Compression: off
182 
+        Execution Driver: native-0.2
183 
+        [...]
184 
+
185 
+  The output of the command above shows that the Docker daemon is using the
186 
+  `zfs` storage driver and that the parent dataset is the `zpool-docker/docker`
187 
+  filesystem created earlier.
188 
+
189 
+Your Docker host is now using ZFS to store to manage its images and containers.
190 
+
191 
+## ZFS and Docker performance
192 
+
193 
+There are several factors that influence the performance of Docker using the `zfs` storage driver.
194 
+
195 
+- **Memory**. Memory has a major impact on ZFS performance. This goes back to the fact that ZFS was originally designed for use on big Sun Solaris servers with large amounts of memory. Keep this in mind when sizing your Docker hosts.
196 
+
197 
+- **ZFS Features**. Using ZFS features, such as deduplication, can significantly increase the amount
198 
+of memory ZFS uses. For memory consumption and performance reasons it is
199 
+recommended to turn off ZFS deduplication. However, deduplication at other
200 
+layers in the stack (such as SAN or NAS arrays) can still be used as these do
201 
+not impact ZFS memory usage and performance. If using SAN, NAS or other hardware
202 
+RAID technologies you should continue to follow existing best practices for
203 
+using them with ZFS.
204 
+
205 
+* **ZFS Caching**. ZFS caches disk blocks in a memory structure called the adaptive replacement cache (ARC). The *Single Copy ARC* feature of ZFS allows a single cached copy of a block to be shared by multiple clones of a filesystem. This means that multiple running containers can share a single copy of cached block. This means that ZFS is a good option for PaaS and other high density use cases.
206 
+
207 
+- **Fragmentation**. Fragmentation is a natural byproduct of copy-on-write filesystems like ZFS. However, ZFS writes in 128K blocks and allocates *slabs* (multiple 128K blocks) to CoW operations in an attempt to reduce fragmentation. The ZFS intent log (ZIL) and the coalescing of writes (delayed writes) also help to reduce fragmentation.
208 
+
209 
+- **Use the native ZFS driver for Linux**. Although the Docker `zfs` storage driver supports the ZFS FUSE implementation, it is not recommended when high performance is required. The native ZFS on Linux driver tends to perform better than the FUSE implementation.
210 
+
211 
+The following generic performance best practices also apply to ZFS.
212 
+
213 
+- **Use of SSD**. For best performance it is always a good idea to use fast storage media such as solid state devices (SSD). However, if you only have a limited amount of SSD storage available it is recommended to place the ZIL on SSD.
214 
+
215 
+- **Use Data Volumes**. Data volumes provide the best and most predictable performance. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. For this reason, you may want to place heavy write workloads on data volumes.