Using persistent volumes with Docker

Sometimes we need persistent data during the time, even when a container is deleted, in these cases we use persistent volumes with docker.

By default all files created inside a container are stored on a writable container layer. This means that:

  • The data doesn’t persist when that container no longer exists, and it can be difficult to get the data out of the container if another process needs it.
  • A container’s writable layer is tightly coupled to the host machine where the container is running. You can’t easily move the data somewhere else.
  • Writing into a container’s writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem.

Docker has two options for containers to store files in the host machine, so that the files are persisted even after the container stops: volumes, and bind mounts. If you’re running Docker on Linux you can also use a tmpfs mount. If you’re running Docker on Windows you can also use a named pipe.

In this entry we will use the persistent volumes with Docker, keep reading for understand how they work.

First of all we need a directory to store our yaml files and docker-compose installed, obviously we need a docker-engine in our system.

[[email protected] ~]# mkdir -p /srv/docker-files/mysql && touch /srv/docker-files/mysql/docker-compose.yml

Fill the file with the following code.

version: '2'
services:
  mysql:
    container_name: mysql-server
    restart: always
    image: mysql/mysql-server
    environment:
      MYSQL_ROOT_PASSWORD: 'root'
      MYSQL_USER: 'user'
      MYSQL_PASS: 'user'
    volumes:
      - mysql-volume:/var/lib/mysql
volumes:
  mysql-volume:

Use the docker compose command inside the folder:

[[email protected] ~]# docker-compose up -d
Creating volume “mysql_persistent-volume” with default driver
Creating mysql-server

If we list our volumes some data will be prompted:

[[email protected] ~]# docker volume ls
DRIVER VOLUME NAME
local mysql_persistent-volume
[[email protected] ~]# docker volume inspect mysql_persistent-volume
[
{
“CreatedAt”: “2019-11-08T22:17:29+01:00”,
“Driver”: “local”,
“Labels”: null,
“Mountpoint”: “/var/lib/docker/volumes/mysql_persistent-volume/_data”,
“Name”: “mysql_persistent-volume”,
“Options”: null,
“Scope”: “local”
}
]

Now connect to the mysql through the command exec and create a database.

[[email protected] ~]# docker exec -it 05bc7019e294  /bin/bash
bash-4.2# mysql -p
Enter password:
[[email protected] ~]# mysql> create database codinginfrastructure;
Query OK, 1 row affected (0.02 sec)

Leave from the container and remove it, first you must stop the container.

[[email protected] ~]# docker stop 05bc7019e294
docker rm 05bc7019e294

Now the docker ps -a command will not return us any container. Now try to build again your container with docker-compose up -d and check again your database. You will see the database that we created before. Even after to remove the container. Thats fantastic!

You can read more information about the volumes at docker page.

Installing vault

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.

A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Understanding who is accessing what secrets is already very difficult and platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.

The key features of Vault are:

  • Secure Secret Storage: Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets. Vault can write to disk, Consul, and more.
  • Dynamic Secrets: Vault can generate secrets on-demand for some systems, such as AWS or SQL databases. For example, when an application needs to access an S3 bucket, it asks Vault for credentials, and Vault will generate an AWS keypair with valid permissions on demand. After creating these dynamic secrets, Vault will also automatically revoke them after the lease is up.
  • Data Encryption: Vault can encrypt and decrypt data without storing it. This allows security teams to define encryption parameters and developers to store encrypted data in a location such as SQL without having to design their own encryption methods.
  • Leasing and Renewal: All secrets in Vault have a lease associated with them. At the end of the lease, Vault will automatically revoke that secret. Clients are able to renew leases via built-in renew APIs.
  • Revocation: Vault has built-in support for secret revocation. Vault can revoke not only single secrets, but a tree of secrets, for example all secrets read by a specific user, or all secrets of a particular type. Revocation assists in key rolling as well as locking down systems in the case of an intrusion.

Vault installation in EL/7

We will download the binaries from vaulproject.io:

[[email protected] ~]#  wget https://releases.hashicorp.com/vault/1.2.3/vault_1.2.3_linux_amd64.zip

Unzip the file:

[[email protected] ~]#  unzip vault_1.2.3_linux_amd64.zip

Place the binari inside the PATH folder

[[email protected] ~]#  mv vault /usr/local/bin/

Run the autocomplete install command, export the following env to your profile and reload the shell

[[email protected] ~]#  vault -autocomplete-install
[[email protected] ~]# echo “export VAULT_ADDR=http://127.0.0.1:8200” >> ~/.bashrc
[[email protected] ~]# exec $SHELL

Run the dev server with the command:

[[email protected] ~]#  vault server -dev

Now we are able to save a secret, try to enter the followign command

[[email protected] ~]#  vault kv put secret/hello foo=world
Key Value
— —–
created_time 2019-11-07T21:34:03.909652779Z
deletion_time n/a
destroyed false
version 1

Now retrieve your secret with the following command

[[email protected] ~]#  vault kv get secret/hello
====== Metadata ======
Key Value
— —–
created_time 2019-11-07T21:34:03.909652779Z
deletion_time n/a
destroyed false
version 1

=== Data ===
Key Value
— —–
foo world