Tag: server monitoring

  • Monitor Server with Prometheus and Grafana

    Monitor Server with Prometheus and Grafana

    Prometheus is used an open source software, that can collect metrics and alerting.

    You can download latest version oof Prometheus from

    https://prometheus.io/download/

    Create a user

    useradd --no-create-home --system --shell /bin/false prometheus
    

    Download and Install prometheus

    cd /usr/local/src
    wget https://github.com/prometheus/prometheus/releases/download/v2.31.0-rc.1/prometheus-2.31.0-rc.1.linux-amd64.tar.gz
    tar xvf prometheus-2.31.0-rc.1.linux-amd64.tar.gz
    cd prometheus-2.31.0-rc.1.linux-amd64
    mv prometheus /usr/local/bin/
    mv promtool /usr/local/bin/
    mkdir /etc/prometheus
    mkdir /var/lib/prometheus
    mv consoles /etc/prometheus
    mv console_libraries /etc/prometheus
    mv prometheus.yml /etc/prometheus
    chown prometheus:prometheus /etc/prometheus
    chown prometheus:prometheus /var/lib/prometheus
    

    Create a service file

    vi /etc/systemd/system/prometheus.service
    

    Add following content

    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    ExecStart=/usr/local/bin/prometheus \
        --config.file /etc/prometheus/prometheus.yml \
        --storage.tsdb.path /var/lib/prometheus/ \
        --web.console.templates=/etc/prometheus/consoles \
        --web.console.libraries=/etc/prometheus/console_libraries
    
    [Install]
    WantedBy=multi-user.target
    

    Enable prometheus to start on boot

    systemctl enable prometheus
    

    Start prometheus

    systemctl start prometheus
    systemctl status prometheus
    

    Prometheus runs on port 9090, you can access promethus at

    http://YOUR_SERVER_IP:9090/graph
    

    It will look like

    Prometheus have some basic graphing features, but you can’t use it for monitoring. To create dash board and monitor, you need to use grafana.

    Collecting Data

    Node Exporter is used to collect data from servers. All monitored servers need Node Exporter installed. You can download latest version of NodeExporter from

    https://github.com/prometheus/node_exporter/releases

    Lets create a user for Node Exporter to run

    useradd --no-create-home --system --shell /bin/false node_exporter
    

    Install Node Exporter

    cd /usr/local/src
    wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
    tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
    cd /usr/local/src/node_exporter-1.2.2.linux-amd64/
    mv node_exporter /usr/local/bin/
    

    Create a systemd service file for node exporter

    vi /etc/systemd/system/node_exporter.service
    

    Add

    [Unit]
    Description=Node Exporter
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=node_exporter
    Group=node_exporter
    Type=simple
    ExecStart=/usr/local/bin/node_exporter
    
    [Install]
    WantedBy=multi-user.target
    

    Enable and start Node Exporter

    systemctl enable node_exporter
    systemctl start node_exporter
    systemctl status node_exporter
    

    Node Exporter run on port 9100 and expose system metrics on url

    http://SERVER_IP:9100/metrics
    

    Node Exporter

    Adding Servers to Prometheus

    Once Node Exporter installed on a server, you need to tell Prometheus to get data from the Node Exporter you just installed. To do this, edit Prometheus configuration file.

    vi /etc/prometheus/prometheus.yml
    

    Add following

      - job_name: 'node_exporter'
        scrape_interval: 5s
        static_configs:
          - targets: ['SERVER_IP:9100']
    

    To monitor multiple servers, you can dd more servers in targets line. Here is an example config

    https://gist.github.com/serverok/83a622e7577da36384f87fe60c9930af/raw

    Restart prometheus

    systemctl restart prometheus
    

    Grafana

    Grafana is used to visualise data collected by Prometheus. You can download Grafana from

    https://grafana.com/grafana/download

    Grafana offers free cloud hosted version with some limitation (1 user, 5 dashboards). Free version is suitable if you are getting started and don’t want to install your own. You can signup for cloud hosted version at

    https://grafana.com/get

    If you decide to install your own Grafana, you can run

    cd /usr/local/src
    wget https://dl.grafana.com/oss/release/grafana_7.3.7_amd64.deb
    dpkg -i grafana_7.3.7_amd64.deb
    

    Enable and start grafana

    systemctl enable grafana-server
    systemctl start grafana-server
    systemctl status grafana-server
    

    If you did your own install, grafana runs on port 3000. To access, use url

    http://SERVER_IP:3000/login
    

    Default username and passwords are “admin”. Once logged in you will be asked to set password for grafana admin user.

    Before you can use Grafana, you need to set a data source and create dash board. In our case, data source is prometheus. To connect Grafana to your Prometheus insallation, go to Settings > Data Sources

    Grafana Data Sources

    On next page, select Prometheus

    Grafana Add data source

    On next page, for URL, enter http://PROMETHUS_SERVER_IP:9090, scroll down, click on “Save & Test” button. If grafana can connect to your prometheus installation, you should see success message with “Data source is working”. If not, you need to check your firewall rules.

    Creating Grafana Dashboards

    Grafana displays data in dash boards. You can create your own or use pre existing dash boards. You can find pre-made dash boards at

    https://grafana.com/grafana/dashboards

    On my grafana installation, i used dashboard

    https://grafana.com/grafana/dashboards/11074

    To add this dash board to your Grafana, click on the + button, then select Import. On next screen, you can enter ID for the dash board you need to import. In this case 11074. Click “Load” button to import the dash board.

    Here is a dash board for one of the server

    grafana dashboard

    You can edit Panels in grafana dash board to see how it is created. You can create a new dash board with panel you need. This way your dashboards only show required information.

    Related Posts

    Server Monitoring

    Prometheus Node Exporter on non default port
    Prometheus init script for CentOS 6

  • Configure Munin Node

    To install munin node on Ubuntu/Debian, run

    apt install -y munin-node
    

    To configure, edit file

    vi /etc/munin/munin-node.conf
    

    Find

    allow ^127\.0\.0\.1$
    

    Below this line, you need to add IP of your Munin Master. You need to convert IP to regular expression format before adding. For example, if your IP is 88.212.32.35, you need to add

    allow ^88\.212\.32\.35$
    

    If you want to allow from any IP, add

    allow ^.*$
    

    Now restart Munin node

    systemctl restart munin-node
    

    Add Node to Munin Master

    You need to tell your Munin master about the new node you have installed. To do this, edit file

    vi /etc/munin/munin.conf
    

    In this file, add

    [hostname-of-new-node-server]
        address IP_ADDR_OF_NODE
        use_node_name yes
    

    Now you need to wait like 5 minutes for Munin master to fetch the data. Munin master run the command /usr/bin/munin-cron to fetch data from every configured nodes.

    If you have any problem with updating data, check the log files for more info.

    /var/log/munin/munin-update.log => is the log file for munin master.

    /var/log/munin/munin-node.log => is the log file for munin node.

    Make sure you can connect to port 4949 of node server from master. If any firewall blocking, you need to open the port for munin to fetch the data.

    root@monitor:~# nc 82.211.1.93 4949
    # munin node at root1272.premium-rootserver.net
    # Unknown command. Try cap, list, nodes, config, fetch, version or quit
    quit
    root@monitor:~# 
    
  • netdata

    To install netdata, run

    wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh
    sh /tmp/netdata-kickstart.sh

    Once installed, you will be able to see server stats at

    http://your-ip:19999

    Stop/start/restart netdata

    systemctl stop netdata
    systemctl start netdata
    systemctl restart netdata