Day 76 : Monitoring Docker Logs Using Grafana and Prometheus-ll (Feb 10, 2024)

Introduction

In this comprehensive tutorial, we'll guide you through the process of sending Docker logs to Grafana for effective monitoring and visualization. This setup involves launching two EC2 instances, installing Grafana on Instance 1, and Prometheus with Docker on Instance 2. By the end of this guide, you'll have a fully functional Grafana dashboard displaying Docker container metrics.

Prerequisites

• AWS account with EC2 instances

• Basic knowledge of Linux commands

• SSH access to both EC2 instances

Step 1: Launch EC2 Instances

Launch two EC2 instances on AWS, naming them Instance 1 and Instance 2. Ensure you have the necessary security groups, key pairs, and IAM roles configured for smooth communication.

Step 2: Install Grafana on Instance 1

SSH into Instance 1 and install Grafana using the following commands:

sudo apt-get update
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Access Grafana's web interface by navigating to http://<instance_1_ip>:3000 in your browser. Default login credentials are usually admin/admin.

Step 3: Install Prometheus and Docker on Instance 2

SSH into Instance 2 and install Prometheus and Docker:

sudo apt-get update
sudo apt-get install -y prometheus docker.io

Step 4: Configure Docker Metrics Collection

Create a file named daemon.json in /etc/docker/ with the following content:

{
    "metrics-addr": "0.0.0.0:9323",
    "experimental": true
}

Restart Docker:

sudo systemctl restart docker

You should now see your docker metrics by going to <instance_2_ip>:9323/metrics.

Step 5: Configure Prometheus for Docker Metrics

Edit the prometheus.yml file in /etc/prometheus/ and add a new job:

- job_name: 'docker_metrics'
  static_configs:
    - targets: ['<instance_2_ip>:9323']

Restart Prometheus:

sudo systemctl restart prometheus

Verify the new job is visible in Prometheus targets.

Step 6: Configure Grafana Data Source

1. Log in to Grafana.

2. Navigate to "Settings" > "Data Sources."

3. Click "Add your first data source" and select Prometheus.

4. Set the URL to <instance_2_ip>:9090 and click "Save & Test."

Step 7: Create Grafana Dashboard

1. Go to the "+" icon on the left sidebar and select "Dashboard."

2. Click "Add new panel" and choose Prometheus as the data source.

3. Use the sample query:

      engine_daemon_container_states_containers{state="running"}
   

4. Customize the panel and dashboard as needed.