/ SysAdmin

Install Docker on a ZFS pool

Why? Why would you run Docker on ZFS?

I discovered ZFS when I was playing with LXD, because it's the recommended storage driver for it.

While ZFS has a lot of great features, the ones I like the most are the RAM cache, compression, and snapshots.

After moving from LXD to Docker (so leaving ZFS), I felt the difference in speed the cache gave me, and I saw some files like databases being 3 times bigger.

Well, running Docker on ZFS shouldn't be difficult, right? No, it really isn't!

In fact, there is even a page dedicated in the docs: https://docs.docker.com/storage/storagedriver/zfs-driver/.

Install Docker

First, we need Docker, obviously.

According the the docs:

curl -fsSL get.docker.com -o get-docker.sh
sh get-docker.sh

Stop it while we mess with ZFS:

service docker stop

Remove the docker data:

rm -rf /var/lib/docker

Create a zpool

You can help yourself with this article of mine to setup a ZFS pool.

You have two choices. If you intend to only store Docker data on your pool, mount it on /var/lib/docker.
If you intent to store other stuff, just mount it anywhere you want, we'll user an option later to specify the Docker data directory.

As for me, my zpool is structured like this:

  • /zpool
    • docker
    • srv

The equivalent of /var/lib/docker is in /zpool/docker, and my data is in /zpool/srv (bind mounts etc).

Use ZFS as Docker's storage driver

Let's check what storage driver Docker uses by default:

root@hinata ~ # docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file

To use ZFS as the storage driver, we need to specify it to the deamon.

Create /etc/docker/daemon.json and add:

{
  "storage-driver": "zfs"
}

If your zpool is not mounted on /var/lib/docker your need to use the "graph" option like this in my case:

{
  "storage-driver": "zfs",
  "graph": "/zpool/docker"
}

You can now start docker. Docker will copy basic data into the data folder:

root@hinata ~ # service docker start
root@hinata ~ # ls /zpool/docker/
builder  containerd  containers  image  network  plugins  runtimes  swarm  tmp  trust  volumes  zfs

Make sure Docker is using your zpool:

root@hinata ~ # docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.03.0-ce
Storage Driver: zfs
 Zpool: zpool-docker
 Zpool Health: ONLINE
 Parent Dataset: zpool-docker
 Space Used By Parent: 81920
 Space Available: 143545778176
 Parent Quota: no
 Compression: ons

When we launch a new container, we can see it's mounted into the zpool:

root@hinata ~# zfs list -r zpool-docker
NAME                                                                                 USED  AVAIL  REFER  MOUNTPOINT
zpool-docker                                                                        10.7G   123G  10.7G  /zpool
zpool-docker/41fb183f4bdc6c4a965261a8700e0c9f9faf6777a35f8ed5fc5040ebef735518       20.5K   123G  20.5K  legacy
zpool-docker/accec25e0803671ce053645c93efcab5afcd6f290115d3583a54adfb3c99acee         10K   123G  22.5K  legacy
zpool-docker/accec25e0803671ce053645c93efcab5afcd6f290115d3583a54adfb3c99acee-init    19K   123G  22.5K  legacy

That's it, you can now enjoy Docker on ZFS!

As I said, I have all my data and Docker data in my zpool. I saw a performance improvement by moving back my containers to ZFS, especially when they're IO intensive (obviously), like databases.

The documentation says you should use Ubuntu for ZFS, but I knew it worked fine on Debian because I tested with LXD, and I confirm it works fine with Docker too.

Again, you should really read the documentation because it's more detailed and goes more in depth on how ZFS works.

Angristan

Angristan

I'm an 18 years old French sysadmin studying at a IT school and working for a web hosting company.

Read More