grafana dashboard

Monitor Server with Prometheus and Grafana

Prometheus is used an open source software, that can collect metrics and alerting.

You can download latest version oof Prometheus from

https://prometheus.io/download/

Create a user

Download and Install prometheus

Create a service file

Add following content

Enable prometheus to start on boot

Start prometheus

Prometheus runs on port 9090, you can access promethus at

It will look like

Prometheus have some basic graphing features, but you can’t use it for monitoring. To create dash board and monitor, you need to use grafana.

Collecting Data

Node Exporter is used to collect data from servers. All monitored servers need Node Exporter installed.

Lets create a user for Node Exporter to run

Install Node Exporter

Create a systemd service file for node exporter

Add

Enable and start Node Exporter

Node Exporter run on port 9100 and expose system metrics on url

Node Exporter

Adding Servers to Prometheus

Once Node Exporter installed on a server, you need to tell Prometheus to get data from the Node Exporter you just installed. To do this, edit Prometheus configuration file.

Add following

To monitor multiple servers, you can dd more servers in targets line. Here is an example config

https://gist.github.com/serverok/83a622e7577da36384f87fe60c9930af/raw

Restart prometheus

Grafana

Grafana is used to visualise data collected by Prometheus. You can download Grafana from

https://grafana.com/grafana/download

Grafana offers free cloud hosted version with some limitation (1 user, 5 dashboards). Free version is suitable if you are getting started and don’t want to install your own. You can signup for cloud hosted version at

https://grafana.com/get

If you decide to install your own Grafana, you can run

Enable and start grafana

If you did your own install, grafana runs on port 3000. To access, use url

Default username and passwords are “admin”. Once logged in you will be asked to set password for grafana admin user.

Before you can use Grafana, you need to set a data source and create dash board. In our case, data source is prometheus. To connect Grafana to your Prometheus insallation, go to Settings > Data Sources

Grafana Data Sources

On next page, select Prometheus

Grafana Add data source

On next page, for URL, enter http://PROMETHUS_SERVER_IP:9090, scroll down, click on “Save & Test” button. If grafana can connect to your prometheus installation, you should see success message with “Data source is working”. If not, you need to check your firewall rules.

Creating Grafana Dashboards

Grafana displays data in dash boards. You can create your own or use pre existing dash boards. You can find pre-made dash boards at

https://grafana.com/grafana/dashboards

On my grafana installation, i used dashboard

https://grafana.com/grafana/dashboards/11074

To add this dash board to your Grafana, click on the + button, then select Import. On next screen, you can enter ID for the dash board you need to import. In this case 11074. Click “Load” button to import the dash board.

Here is a dash board for one of the server

grafana dashboard

You can edit Panels in grafana dash board to see how it is created. You can create a new dash board with panel you need. This way your dashboards only show required information.

Related Posts

Server Monitoring

Prometheus Node Exporter on non default port

Configure Munin Node

To install munin node on Ubuntu/Debian, run

To configure, edit file

Find

Below this line, you need to add IP of your Munin Master. You need to convert IP to regular expression format before adding. For example, if your IP is 88.212.32.35, you need to add

If you want to allow from any IP, add

Now restart Munin node

Add Node to Munin Master

You need to tell your Munin master about the new node you have installed. To do this, edit file

In this file, add

Now you need to wait like 5 minutes for Munin master to fetch the data. Munin master run the command /usr/bin/munin-cron to fetch data from every configured nodes.

If you have any problem with updating data, check the log files for more info.

/var/log/munin/munin-update.log => is the log file for munin master.

/var/log/munin/munin-node.log => is the log file for munin node.

Make sure you can connect to port 4949 of node server from master. If any firewall blocking, you need to open the port for munin to fetch the data.

netdata

To install netdata, run

Once installed, you will be able to see servers stats at

http://your-ip:19999

Stop/start/restart netdata