Docker Deep Dive: Understanding the Fundamentals of containers

As a developer, you know how important it is to have a reliable and efficient workflow. That's why many developers love Docker - it's a powerful containerization platform that makes it easy to package, deploy, and manage applications and their dependencies. by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package.

Docker uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable file system such as OverlayFS and others to allow independent "containers" to run within a single Linux instance, avoiding the overhead of starting and maintaining virtual machines.

cgroups

Cgroups are a Linux kernel feature that allows you to allocate resources (such as CPU, memory, and I/O bandwidth) to processes or groups of processes. Docker uses cgroups to limit the resources that each container can use, ensuring that containers do not interfere with each other or with the host system.

To demonstrate how Docker uses cgroups (Control Groups) in Linux, we can create a simple Docker container that limits the CPU and memory usage of a process. Here is an example of how to do this:

Run the Docker container with the docker run command, using the --cpu-shares and --memory options to limit the CPU and memory usage of the container and the --device-read-bps and --device-write-bps options to specify the maximum read and write bandwidth in bytes per second that the container is allowed to use for all block devices.

To get the block device name Use the df command

df

docker run -d --name demo-cgroups --device-read-bps /dev/your-device:500m --device-write-bps /dev/your-device:500m  --cpu-shares 512 --memory 512m ubuntu:latest sleep infinity

This will run the Docker container with a CPU share of 512 and a memory limit of 512 MB. limit the read and write bandwidth of a Docker container to 500 MB/s for the block device specified /dev/your-device

You can use the docker stats command to view the I/O bandwidth usage of the container

docker stats demo-cgroups

Playing with cgroups in Linux:

You can use the cgroup command line utility to manage cgroups in Linux you can install it on debian based distro using apt-get install cgroup-tools . Here is how we can use cgroup to manipulate cgroups:

To create a new cgroup, use the cgcreate command:

sudo cgcreate -g memory:/mycgroup

This will create a new cgroup named "mycgroup" in the "memory" hierarchy.

Create a demo process to test the cgroup

sleep infinity &

To move a process into a cgroup, use the cgclassify command:

sudo cgclassify -g memory:/mycgroup YOUR_PROCESS_ID

This will move process with PID YOUR_PROCESS_ID into the "mycgroup" cgroup.

To set a resource limit for a cgroup, use the cgset command:

sudo cgset -r memory.limit_in_bytes=256M mycgroup

This will set a memory limit of 256MB for the "mycgroup" cgroup.

To view the status of a cgroup, use the cgget command:

sudo cgget -r memory.usage_in_bytes mycgroup

This will display the current memory usage for the "mycgroup" cgroup.

More information about cgroups can be found here

Kernel namespaces

Kernel namespaces are a Linux kernel feature that allow you to create isolated environments within a single Linux instance. Docker uses kernel namespaces to create separate namespaces for each container, allowing each container to have its own view of the system. This isolation ensures that containers do not interfere with each other or with the host system.

To demonstrate how Docker uses namespaces in Linux, we can create a simple Docker container that runs a process in a separate namespace. Here is an example of how to do this:

Run a container for the test with  :

docker run -it --name demo-ns ubuntu:latest /bin/bash

You can verify that the process is running in a separate namespace by using the ps command inside and outside the container:

# Inside the container
ps aux

# Outside the container
docker exec demo ps aux

The output of the ps command will show the PID of the process running in the container. You should see that the PID is different inside and outside the container, indicating that the process is running in a separate namespace.

More information about how to manipulate docker container pid namespace can be  here

Useful resources to dig deep into Linux namespaces : https://blog.quarkslab.com/digging-into-linux-namespaces-part-1.html

Union file systems

Union file systems, such as OverlayFS, allow you to overlay one file system on top of another, creating a single logical file system that combines the contents of both. Docker uses union file systems to create lightweight containers that share a common base image, reducing the disk space and bandwidth required to store and transfer containers.

OverlayFS  allows you to overlay one file system on top of another, creating a single logical file system that combines the contents of both. This can be useful in a variety of situations, such as creating lightweight containers that share a common base image or allowing multiple users to share the same file system without modifying the underlying files

To demonstrate how Docker uses union file systems in Linux, we can create a simple Docker container that uses an OverlayFS union file system to combine two different filesystems

Create a temporary directory and add a file to it:

mkdir temp
echo "overlay" > temp/hostname

This will create a temporary directory and add a file named hostname to it, containing the text "overlay".

Run the Docker container with the docker run command, using the --mount option to mount the temporary directory as an OverlayFS layer on top of the base image:

docker run -it --name demo-ufs -v $(pwd)/temp/hostname:/etc/hostname ubuntu:latest cat /etc/hostname

This will run the Docker container with the temporary directory mounted as an OverlayFS layer on top of the base image. The /etc/hostname file in the base image will be overlaid with the hostname file in the temporary directory, and the cat command will display the contents of the overlaid file.

You can verify that the overlay is being used

Useful resources to dig deep into OverlayFS : https://adil.medium.com/container-filesystem-under-the-hood-overlayfs-5a8053fe3a0a

conclusion

In conclusion, Docker is a powerful containerization platform that has revolutionized the way developers build, deploy, and manage applications These Linux kernel features motioned on that post are essential for the functionality and performance of Docker, and they play a crucial role in enabling developers to build, deploy, and manage applications in containers. Docker has become an industry standard for containerization, and it is used by millions of developers and organizations around the world to build, deploy, and manage applications in a fast, efficient, and scalable manner.