Introduction

Grafana is a powerful tool for visualizing and analyzing data. In this guide, we'll walk through the step-by-step process of setting up Grafana along with Loki and Promtail to monitor and analyze error logs, focusing on 404 errors.

Setting Up Grafana

Begin by installing dependencies:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

Add the Grafana repository and update the package list:

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update the list of available packages

sudo apt-get update

Install Grafana:

sudo apt-get install grafana

Start and enable Grafana:

sudo systemctl start grafana-server.service
sudo systemctl enable grafana-server.service

Check the Grafana server status:

sudo /bin/systemctl status grafana-server

Grafana runs on port 3000.

Access it in your browser using <public>:<port>, and log in with the default username and password: admin.

Once logged in you will see the below dashboard

Adding Loki and Promtail with Docker:

Download Loki configuration:

Create a directory named grafana-config and execute the below command

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

Run Loki Docker container:

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

Ensure port 3100 is exposed.

On Browser search for <public-ip>:<port-number>/ready

Download Promtail configuration:

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Promtail Docker container:

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

1. In Grafana, navigate to Dashboard -> DataSources, and configure Promtail.

2. Modify Promtail configuration and restart it.

Now lets change the config of Promtail

Now restart Promtail

And here we go!

Monitoring 404 Errors

Now lets search for 404 error on Nginx on our dashboard

1. Open a new window and visit <public-ip>:80 to generate logs.

2. Search for 404 errors in the Grafana dashboard.

3. Refresh the page thrice with appended /app to generate 404 logs.

4. Observe the 404 logs on the Grafana dashboard.

Now go to the grafana dashboard and tap on refresh and you will see the 404 logs displayed on the panel

Now let's just search the error logs here specifically for 404

Below Dashboard shows the total 404 error found in last 15 mins

1. Customize the Grafana dashboard with additional search fields for a more detailed analysis.

2. Explore the capabilities of Grafana to create insightful visualizations.

Conclusion:

Congratulations! You've successfully set up Grafana with Loki and Promtail to monitor and analyze error logs, specifically focusing on 404 errors. Experiment with Grafana's features to create customized dashboards for a more comprehensive understanding of your system's performance. Happy monitoring!