Tag: Prometheus

Prometheus init script for CentOS 6

Create init file.

touch /etc/rc.d/init.d/prometheus
chmod 755 /etc/rc.d/init.d/prometheus
vi /etc/rc.d/init.d/prometheus

Add following

#!/bin/bash
#
# /etc/rc.d/init.d/prometheus
#
# Prometheus monitoring server
#
#  chkconfig: 2345 20 80 Read
#  description: Prometheus monitoring server
#  processname: prometheus

# Source function library.
. /etc/rc.d/init.d/functions

PROGNAME=prometheus
PROG=/usr/hostonnet/prometheus/$PROGNAME
USER=prometheus
LOGFILE=/var/log/prometheus.log
DATADIR=/usr/hostonnet/prometheus/data
LOCKFILE=/var/run/$PROGNAME.pid
CONFIG_FILE=/usr/hostonnet/prometheus/prometheus.yml
ALERT_MGR_URL=localhost:9093

start() {
    echo -n "Starting $PROGNAME: "
    cd /usr/hostonnet/prometheus/
    #daemon --user $USER --pidfile="$LOCKFILE" "$PROG -config.file $CONFIG_FILE -storage.local.path $DATADIR -alertmanager.url $ALERT_MGR_URL &>$LOGFILE &"
    daemon --user $USER --pidfile="$LOCKFILE" "$PROG -config.file $CONFIG_FILE -storage.local.path $DATADIR &>$LOGFILE &"
    echo $(pidofproc $PROGNAME) >$LOCKFILE
    echo
}

stop() {
    echo -n "Shutting down $PROGNAME: "
    killproc $PROGNAME
    rm -f $LOCKFILE
    echo
}


case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status $PROGNAME
    ;;
    restart)
    stop
    start
    ;;
    reload)
    echo "Sending SIGHUP to $PROGNAME"
    kill -SIGHUP $(pidofproc $PROGNAME)
    ;;
    *)
        echo "Usage: service prometheus {start|stop|status|reload|restart}"
        exit 1
    ;;
esac

Create User For Prometheus

groupadd -r prometheus
useradd -r -g prometheus -s /sbin/nologin -d /usr/hostonnet/prometheus/ -c "prometheus Daemons" prometheus
chown -R prometheus:prometheus /usr/hostonnet/prometheus/
chown prometheus:prometheus /var/log/prometheus.log

Run prometheus on Boot

chkconfig --add prometheus
chkconfig prometheus on

Verify it is enabled

[root@backup ~]# chkconfig --list | grep prome
prometheus     	0:off	1:off	2:on	3:on	4:on	5:on	6:off
[root@backup ~]#

“3:on” will start prometheus on run level 3, that is normal boot.

Start Prometheus

[root@backup ~]# service prometheus start
Starting prometheus:                                       [  OK  ]
[root@backup ~]#

See Monitor Server with Prometheus and Grafana

October 15, 2020

Prometheus Node Exporter on non default port

To run Prometheus Node Exporter on custom port, use

/usr/local/bin/node_exporter --web.listen-address=:9101

Here is systemctl service script i used

[root@cdn 1945]# cat /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9101

[Install]
WantedBy=multi-user.target
[root@cdn 1945]#

Related Posts

Prometheus

Server Monitoring

November 23, 2019

Monitor Server with Prometheus and Grafana
Prometheus is used an open source software, that can collect metrics and alerting.

You can download latest version oof Prometheus from

https://prometheus.io/download/

Create a user
```
useradd --no-create-home --system --shell /bin/false prometheus
```
Download and Install prometheus
```
cd /usr/local/src
wget https://github.com/prometheus/prometheus/releases/download/v2.31.0-rc.1/prometheus-2.31.0-rc.1.linux-amd64.tar.gz
tar xvf prometheus-2.31.0-rc.1.linux-amd64.tar.gz
cd prometheus-2.31.0-rc.1.linux-amd64
mv prometheus /usr/local/bin/
mv promtool /usr/local/bin/
mkdir /etc/prometheus
mkdir /var/lib/prometheus
mv consoles /etc/prometheus
mv console_libraries /etc/prometheus
mv prometheus.yml /etc/prometheus
chown prometheus:prometheus /etc/prometheus
chown prometheus:prometheus /var/lib/prometheus
```
Create a service file
```
vi /etc/systemd/system/prometheus.service
```
Add following content
```
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
```
Enable prometheus to start on boot
```
systemctl enable prometheus
```
Start prometheus
```
systemctl start prometheus
systemctl status prometheus
```
Prometheus runs on port 9090, you can access promethus at
```
http://YOUR_SERVER_IP:9090/graph
```
It will look like

Prometheus have some basic graphing features, but you can’t use it for monitoring. To create dash board and monitor, you need to use grafana.

Collecting Data

Node Exporter is used to collect data from servers. All monitored servers need Node Exporter installed. You can download latest version of NodeExporter from

https://github.com/prometheus/node_exporter/releases

Lets create a user for Node Exporter to run
```
useradd --no-create-home --system --shell /bin/false node_exporter
```
Install Node Exporter
```
cd /usr/local/src
wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
cd /usr/local/src/node_exporter-1.2.2.linux-amd64/
mv node_exporter /usr/local/bin/
```
Create a systemd service file for node exporter
```
vi /etc/systemd/system/node_exporter.service
```
Add
```
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
```
Enable and start Node Exporter
```
systemctl enable node_exporter
systemctl start node_exporter
systemctl status node_exporter
```
Node Exporter run on port 9100 and expose system metrics on url
```
http://SERVER_IP:9100/metrics
```
Adding Servers to Prometheus

Once Node Exporter installed on a server, you need to tell Prometheus to get data from the Node Exporter you just installed. To do this, edit Prometheus configuration file.
```
vi /etc/prometheus/prometheus.yml
```
Add following
```
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['SERVER_IP:9100']
```
To monitor multiple servers, you can dd more servers in targets line. Here is an example config

https://gist.github.com/serverok/83a622e7577da36384f87fe60c9930af/raw

Restart prometheus
```
systemctl restart prometheus
```
Grafana

Grafana is used to visualise data collected by Prometheus. You can download Grafana from

https://grafana.com/grafana/download

Grafana offers free cloud hosted version with some limitation (1 user, 5 dashboards). Free version is suitable if you are getting started and don’t want to install your own. You can signup for cloud hosted version at

https://grafana.com/get

If you decide to install your own Grafana, you can run
```
cd /usr/local/src
wget https://dl.grafana.com/oss/release/grafana_7.3.7_amd64.deb
dpkg -i grafana_7.3.7_amd64.deb
```
Enable and start grafana
```
systemctl enable grafana-server
systemctl start grafana-server
systemctl status grafana-server
```
If you did your own install, grafana runs on port 3000. To access, use url
```
http://SERVER_IP:3000/login
```
Default username and passwords are “admin”. Once logged in you will be asked to set password for grafana admin user.

Before you can use Grafana, you need to set a data source and create dash board. In our case, data source is prometheus. To connect Grafana to your Prometheus insallation, go to Settings > Data Sources

On next page, select Prometheus

On next page, for URL, enter http://PROMETHUS_SERVER_IP:9090, scroll down, click on “Save & Test” button. If grafana can connect to your prometheus installation, you should see success message with “Data source is working”. If not, you need to check your firewall rules.

Creating Grafana Dashboards

Grafana displays data in dash boards. You can create your own or use pre existing dash boards. You can find pre-made dash boards at

https://grafana.com/grafana/dashboards

On my grafana installation, i used dashboard

https://grafana.com/grafana/dashboards/11074

To add this dash board to your Grafana, click on the + button, then select Import. On next screen, you can enter ID for the dash board you need to import. In this case 11074. Click “Load” button to import the dash board.

Here is a dash board for one of the server

You can edit Panels in grafana dash board to see how it is created. You can create a new dash board with panel you need. This way your dashboards only show required information.

Related Posts

Server Monitoring

Prometheus Node Exporter on non default port
Prometheus init script for CentOS 6
November 18, 2019

Tag: Prometheus

Prometheus init script for CentOS 6

Create User For Prometheus

Run prometheus on Boot

Prometheus Node Exporter on non default port

Monitor Server with Prometheus and Grafana

Collecting Data

Adding Servers to Prometheus

Grafana

Creating Grafana Dashboards