Day 4: Monitoring Setup Journey: Checkmk, Prometheus, Grafana, and More on Ubuntu 22.04

Checkmk Installation and Client Integration

✅ Environment Setup

• Host Machine: Laptop (where Checkmk web UI is accessed)

• Remote Machine: Ubuntu 22.04 server (Checkmk server setup)

• Client Machines: Ubuntu servers to be monitored

⚠ Important: Ensure TCP port 6556 is open on all client machines.

---

Step 1: Install Checkmk on Remote Machine

sudo apt update && sudo apt upgrade -y

Download Checkmk Raw Edition:

wget https://download.checkmk.com/checkmk/2.3.0p27/check-mk-raw-2.3.0p27_0.jammy_amd64.deb

Ensure sources.list has the required Ubuntu repositories:

deb http://archive.ubuntu.com/ubuntu/ jammy main
deb http://archive.ubuntu.com/ubuntu/ jammy universe
deb http://archive.ubuntu.com/ubuntu jammy-security main
deb http://archive.ubuntu.com/ubuntu jammy-security universe

Install Checkmk:

sudo apt install ./check-mk-raw-2.3.0p27_0.jammy_amd64.deb -y
omd version

---

Step 2: Create and Start a Checkmk Site

sudo omd create osfp
sudo omd start osfp

Enable external access:

sudo omd config osfp set APACHE_TCP_ADDR 0.0.0.0

Access the web interface:

http://<server-public-ip>/osfp

---

Step 3: Install Checkmk Agent on Client Machines

• Download the agent from Setup > Agents > Linux in the Checkmk UI.

• Transfer the agent to the client:

scp -i <pem-file> check-mk-agent_2.3.0p27-1_all.deb <user>@<client-ip>:~

• Install on the client:

sudo dpkg -i check-mk-agent_2.3.0p27-1_all.deb
check_mk_agent

---

Step 4: Add Client as a Host in Checkmk

1. Navigate: Setup > Hosts > Add Host

2. Add the hostname and private IP

3. Save → Run Service Discovery → Activate Changes

4. Validate via Monitor > All Hosts

---

❌ Checkmk Didn’t Work for Me

Even after following the guide, my Checkmk server kept showing the DOWN status. I couldn’t resolve it after several retries and decided to switch to Prometheus + Grafana for monitoring.

---

✅ Prometheus & Grafana Setup (Optimized)

Final Docker Compose

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-storage:/prometheus
    ports:
      - "9090:9090"
    restart: always
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
    volumes:
      - grafana-storage:/var/lib/grafana
    restart: always
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  grafana-storage:
  prometheus-storage:

---

Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets:
        - '10.122.0.3:9100'
        - '10.122.0.2:9200'
        - '10.122.0.6:9200'
        - '10.122.0.5:9100'

  - job_name: 'cadvisor'
    static_configs:
      - targets:
        - '10.122.0.3:8080'

  - job_name: 'kube-state-metrics'
    static_configs:
      - targets:
        - '139.59.43.181:31776'

---

✅ Node Exporter Installation (v1.9.0)

VERSION=1.9.0  
wget https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-amd64.tar.gz
tar xvf node_exporter-${VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${VERSION}.linux-amd64/node_exporter /usr/local/bin/

Create systemd service:

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Start and enable:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify:

curl http://localhost:9100/metrics

---

✅ cAdvisor for Container Monitoring

If cAdvisor isn’t running, start it:

docker run -d --name=cadvisor \
  --restart always \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/run/docker.sock:/var/run/docker.sock:ro \
  --volume=/etc/machine-id:/etc/machine-id:ro \
  --publish=8080:8080 \
  gcr.io/cadvisor/cadvisor:latest

Check if running:

ss -tulnp | grep 8080

Expected:

tcp   LISTEN   0   128   0.0.0.0:8080   0.0.0.0:*   users:(("cadvisor",pid,fd))

---

✅ Kubernetes Metrics Monitoring

For Kubernetes:

• Use kube-state-metrics

• Add the correct target in prometheus.yml

---

Setting Up Metrics Server and Kube-State-Metrics in MicroK8s for Prometheus Monitoring

Monitoring is a crucial part of running any Kubernetes cluster. In this guide, we’ll walk through setting up the metrics-server and kube-state-metrics in a MicroK8s environment and making them accessible for Prometheus scraping.

---

✅ Step 1: Install Metrics Server

The metrics-server provides resource usage metrics (CPU, memory) for Kubernetes objects. It's required for kubectl top commands and is commonly used by autoscalers.

Download and apply the metrics-server YAML:

microk8s.kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Verify if the metrics-server deployment is running:

microk8s.kubectl get deployment -n kube-system | grep metrics-server

---

✅ Step 2: Expose Metrics Server as NodePort (Optional but Useful for External Access)

By default, the metrics-server runs as a ClusterIP service. If you want to scrape it externally or from Prometheus, you can expose it as a NodePort:

microk8s.kubectl patch svc metrics-server -n kube-system -p '{"spec": {"type": "NodePort"}}'

Check the NodePort details:

microk8s.kubectl get svc -n kube-system metrics-server

---

✅ Step 3: Enable Prometheus in MicroK8s

MicroK8s provides a built-in Prometheus addon that simplifies the setup.

microk8s enable prometheus

Once enabled, Prometheus and Grafana will be running inside your cluster.

---

✅ Step 4: Locate Kube-State-Metrics

The kube-state-metrics is automatically deployed as part of the Prometheus addon in MicroK8s. It provides cluster state information (deployments, pods, nodes, etc.) that Prometheus scrapes.

Check if the kube-state-metrics pod is running:

microk8s.kubectl get pods --all-namespaces | grep kube-state-metrics

Get the kube-state-metrics service information:

microk8s.kubectl get svc -n observability | grep kube-state-metrics

You should see the NodePort or ClusterIP assigned to this service.

---

✅ Step 5: Configure Prometheus to Scrape Kube-State-Metrics

In your Prometheus configuration, you need to add a job for kube-state-metrics. Replace the IP and port with the actual NodePort or ClusterIP you retrieved in the previous step.

 - job_name: 'kube-state-metrics'
    static_configs:
      - targets:
          - '139.59.43.181:31479'  # Replace with your kube-state-metrics NodePort