• In this post, I will describe how to set up a standalone pipeline for monitoring Docker containers using cAdvisor, Prometheus, and Grafana. We use the following file system in this work.

docker-compose.yml
run_moni.sh
prometheus.yml
grafana_data
prometheus_data

• docker-compose.yml file for setting up cAdvisor, Prometheus, and Grafana containers can be given as follows.

version: '3.9'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
    - 9090:9090
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - type: bind
      source: ./prometheus_data
      target: /prometheus
    depends_on:
    - cadvisor
    restart: always
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    privileged: true
    restart: always
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
    - 3000:3000
    volumes:
    - type: bind
      source: ./grafana_data
      target: /var/lib/grafana
    depends_on:
    - prometheus
    restart: always

• In the above configuration, cAdvisor will monitor the volumes /, /var/run, /sys, and /var/lib/docker of the host machine. Prometheus will scrape cAdvisor. Finally, Grafana will be used to visualize the data collected by Prometheus.
• prometheus.yml file for Prometheus configuration can be given as follows. Here, Prometheus will scrape cAdvisor every 5 seconds.

scrape_configs:
- job_name: cadvisor
  scrape_interval: 5s
  static_configs:
  - targets:
    - cadvisor:8080

• Use run_moni.sh to start the monitoring pipeline. This is illustrated as follows. Here, docker_data and prometheus_data are the folders for storing data generated by Grafana and Prometheus, respectively.

# If there is no grafana_data folder, create it and change permissions to 777
if [ ! -d grafana_data ]; then
    mkdir grafana_data
    chmod -R 777 grafana_data
fi

# If there is no prometheus_data folder, create it and change permissions to 777
if [ ! -d prometheus_data ]; then
    mkdir prometheus_data
    chmod -R 777 prometheus_data
fi

echo killing old docker processes
docker-compose rm -fs

echo building docker containers
docker-compose up --build --remove-orphans -d

• At this point, the Grafana dashboard is available at http://localhost:3000. Log in to Grafana and add the prometheus data source at http://prometheus:9090 and create dashboards of your choice using different queries.
  • Example query for Bytes received per second: rate(container_network_receive_bytes_total{name=~"container1|container2|container3"}[1m]), where container1, container2, and container3 are the names of the containers to be monitored.
  • In container monitoring, when containers are stopped and started again, another time series is created since the container IDs change. Therefore, to aggregate the different time series using container name, you can use a query as follows: label_replace(sum(rate(container_network_receive_bytes_total{image!="",name=~"container1"}[1m])) by (name), "name", "$1", "container_id", "(.*)").