The CFQ I/O scheduler has known performance issues when used in
certain OS configurations. For example, we can see between 10x - 30x
drop in I/O throughput running the following command, with the CFQ I/O
scheduler:

dd if=/dev/zero of=/root/test.img bs=512 count=10000 oflags=dsync

Throughput with CFQ: 60 KB/s
Throughput with noop or deadline: 1.5 MB/s - 2 MB/s

This performance drop is caused by the undesirable interaction between
4 different components:

- blkio cgroup controller enabled
- ext4 with the jbd2 kthread running in the root blkio cgroup
- dd running on ext4, in any other blkio cgroup than that of jbd2
- CFQ I/O scheduler with defaults for slice_idle and group_idle

When docker is enabled, systemd creates a blkio cgroup called
system.slice to run system services (and docker) under it, and a
separate blkio cgroup called user.slice for user processes. So, when
dd is invoked, it runs under user.slice.

The dd command above includes the dsync flag, which performs an
fdatasync after every write to the output file. Since dd is writing to
a file on ext4, jbd2 will be active, committing transactions
corresponding to those fdatasync requests from dd. (In other words, dd
depends on jdb2, in order to make forward progress). But jdb2 being a
kernel thread, runs in the root blkio cgroup, as opposed to dd, which
runs under user.slice.

Now, if the I/O scheduler in use for the underlying block device is
CFQ, then its inter-queue/inter-group idling takes effect (via the
slice_idle and group_idle parameters, both of which default to 8ms).
Therefore, everytime CFQ switches between processing requests from dd
vs jbd2, this 8ms idle time is injected, which slows down the overall
throughput tremendously!

Unfortunately, the pre-conditions that cause this performance drop
correspond to most of the common configurations of Photon OS! Fixing
CFQ itself is challenging (and is still being discussed on the linux
kernel mailing list [1]), so switch the default I/O scheduler to
'deadline' in the meantime.

For more details on this problem, as well as the ongoing discussion
around its fix, refer to [1].

[1]. https://lore.kernel.org/lkml/8d72fcf7-bbb4-2965-1a06-e9fc177a8938@csail.mit.edu/

Change-Id: I257deacbfd15cfe35f99072440da4e09472e2ed3
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/7323
Tested-by: gerrit-photon <photon-checkins@vmware.com>
Reviewed-by: Srinidhi Rao <srinidhir@vmware.com>
Reviewed-by: Alexey Makhalov <amakhalov@vmware.com>
(cherry picked from commit 7974ca9f70e37dee132758ead578700f7369c1c9)
Reviewed-on: http://photon-jenkins.eng.vmware.com:8082/7341
Reviewed-by: Srivatsa S. Bhat <srivatsab@vmware.com>