# Day 4: Monitoring Setup Journey: Checkmk, Prometheus, Grafana, and More on Ubuntu 22.04

## **Checkmk Installation and Client Integration**

### ✅ **Environment Setup**

* **Host Machine:** Laptop (where Checkmk web UI is accessed)
    
* **Remote Machine:** Ubuntu 22.04 server (Checkmk server setup)
    
* **Client Machines:** Ubuntu servers to be monitored
    

⚠ **Important:** Ensure TCP port `6556` is open on all client machines.

---

### **Step 1: Install Checkmk on Remote Machine**

```plaintext
sudo apt update && sudo apt upgrade -y
```

Download Checkmk Raw Edition:

```plaintext
wget https://download.checkmk.com/checkmk/2.3.0p27/check-mk-raw-2.3.0p27_0.jammy_amd64.deb
```

Ensure `sources.list` has the required Ubuntu repositories:

```plaintext
deb http://archive.ubuntu.com/ubuntu/ jammy main
deb http://archive.ubuntu.com/ubuntu/ jammy universe
deb http://archive.ubuntu.com/ubuntu jammy-security main
deb http://archive.ubuntu.com/ubuntu jammy-security universe
```

Install Checkmk:

```plaintext
sudo apt install ./check-mk-raw-2.3.0p27_0.jammy_amd64.deb -y
omd version
```

---

### **Step 2: Create and Start a Checkmk Site**

```plaintext
sudo omd create osfp
sudo omd start osfp
```

Enable external access:

```plaintext
sudo omd config osfp set APACHE_TCP_ADDR 0.0.0.0
```

Access the web interface:

```plaintext
http://<server-public-ip>/osfp
```

---

### **Step 3: Install Checkmk Agent on Client Machines**

* Download the agent from **Setup &gt; Agents &gt; Linux** in the Checkmk UI.
    
* Transfer the agent to the client:
    

```plaintext
scp -i <pem-file> check-mk-agent_2.3.0p27-1_all.deb <user>@<client-ip>:~
```

* Install on the client:
    

```plaintext
sudo dpkg -i check-mk-agent_2.3.0p27-1_all.deb
check_mk_agent
```

---

### **Step 4: Add Client as a Host in Checkmk**

1. Navigate: **Setup &gt; Hosts &gt; Add Host**
    
2. Add the hostname and private IP
    
3. Save → Run Service Discovery → Activate Changes
    
4. Validate via **Monitor &gt; All Hosts**
    

---

## ❌ **Checkmk Didn’t Work for Me**

Even after following the guide, my Checkmk server kept showing the **DOWN** status. I couldn’t resolve it after several retries and decided to switch to **Prometheus + Grafana** for monitoring.

---

## ✅ **Prometheus & Grafana Setup (Optimized)**

### **Final Docker Compose**

```plaintext
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-storage:/prometheus
    ports:
      - "9090:9090"
    restart: always
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
    volumes:
      - grafana-storage:/var/lib/grafana
    restart: always
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  grafana-storage:
  prometheus-storage:
```

---

### **Prometheus Configuration (prometheus.yml)**

```plaintext
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets:
        - '10.122.0.3:9100'
        - '10.122.0.2:9200'
        - '10.122.0.6:9200'
        - '10.122.0.5:9100'

  - job_name: 'cadvisor'
    static_configs:
      - targets:
        - '10.122.0.3:8080'

  - job_name: 'kube-state-metrics'
    static_configs:
      - targets:
        - '139.59.43.181:31776'
```

---

## ✅ **Node Exporter Installation (v1.9.0)**

```plaintext
VERSION=1.9.0  
wget https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-amd64.tar.gz
tar xvf node_exporter-${VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${VERSION}.linux-amd64/node_exporter /usr/local/bin/
```

Create systemd service:

```plaintext
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF
```

Start and enable:

```plaintext
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
```

Verify:

```plaintext
curl http://localhost:9100/metrics
```

---

## ✅ **cAdvisor for Container Monitoring**

If cAdvisor isn’t running, start it:

```plaintext
docker run -d --name=cadvisor \
  --restart always \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/run/docker.sock:/var/run/docker.sock:ro \
  --volume=/etc/machine-id:/etc/machine-id:ro \
  --publish=8080:8080 \
  gcr.io/cadvisor/cadvisor:latest
```

Check if running:

```plaintext
ss -tulnp | grep 8080
```

Expected:

```plaintext
tcp   LISTEN   0   128   0.0.0.0:8080   0.0.0.0:*   users:(("cadvisor",pid,fd))
```

---

## ✅ **Kubernetes Metrics Monitoring**

For Kubernetes:

* Use `kube-state-metrics`
    
* Add the correct target in `prometheus.yml`
    

---

Setting Up Metrics Server and Kube-State-Metrics in MicroK8s for Prometheus Monitoring

Monitoring is a crucial part of running any Kubernetes cluster. In this guide, we’ll walk through setting up the **metrics-server** and **kube-state-metrics** in a **MicroK8s** environment and making them accessible for Prometheus scraping.

---

## ✅ Step 1: Install Metrics Server

The **metrics-server** provides resource usage metrics (CPU, memory) for Kubernetes objects. It's required for `kubectl top` commands and is commonly used by autoscalers.

### Download and apply the metrics-server YAML:

```plaintext
microk8s.kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```

### Verify if the metrics-server deployment is running:

```plaintext
microk8s.kubectl get deployment -n kube-system | grep metrics-server
```

---

## ✅ Step 2: Expose Metrics Server as NodePort (Optional but Useful for External Access)

By default, the metrics-server runs as a ClusterIP service. If you want to scrape it externally or from Prometheus, you can expose it as a NodePort:

```plaintext
microk8s.kubectl patch svc metrics-server -n kube-system -p '{"spec": {"type": "NodePort"}}'
```

### Check the NodePort details:

```plaintext
microk8s.kubectl get svc -n kube-system metrics-server
```

---

## ✅ Step 3: Enable Prometheus in MicroK8s

MicroK8s provides a built-in Prometheus addon that simplifies the setup.

```plaintext
microk8s enable prometheus
```

Once enabled, Prometheus and Grafana will be running inside your cluster.

---

## ✅ Step 4: Locate Kube-State-Metrics

The **kube-state-metrics** is automatically deployed as part of the Prometheus addon in MicroK8s. It provides cluster state information (deployments, pods, nodes, etc.) that Prometheus scrapes.

### Check if the kube-state-metrics pod is running:

```plaintext
microk8s.kubectl get pods --all-namespaces | grep kube-state-metrics
```

### Get the kube-state-metrics service information:

```plaintext
microk8s.kubectl get svc -n observability | grep kube-state-metrics
```

You should see the NodePort or ClusterIP assigned to this service.

---

## ✅ Step 5: Configure Prometheus to Scrape Kube-State-Metrics

In your Prometheus configuration, you need to add a job for `kube-state-metrics`. Replace the IP and port with the actual NodePort or ClusterIP you retrieved in the previous step.

```plaintext
 - job_name: 'kube-state-metrics'
    static_configs:
      - targets:
          - '139.59.43.181:31479'  # Replace with your kube-state-metrics NodePort
```