Prometheus Alerting and Monitoring

Note that this is optional, but it is useful if your Prometheus server is accessible from the internet, you want it to look more professional to clients and you want to have less problems sending emails from it.

I have gone onto my domain name provider, and added an A Name record that points to the IP address of my new Prometheus server.

Example,

prometheus.sbcode.net. IN A 134.209.224.39

Your domain and IP will be different, and note that it may take some time for the DNS record to propagate across the internet.

One option to help secure our Prometheus server is to put it behind a reverse proxy so that we can later add SSL and an Authentication layer over the default unrestricted Prometheus web interface.

We can use Nginx.

# sudo apt install nginx

CD to the Nginx sites-enabled folder

# cd /etc/nginx/sites-enabled

Create a new Nginx configuration for Prometheus

# sudo nano prometheus

And copy/paste the example below

-------------------------------

server {
    listen 80;
    listen [::]:80;
    server_name  YOUR-DOMAIN-NAME;

    location / {
        proxy_pass           http://localhost:9090/;
    }
}

----------------------------

Save and test the new configuration has no errors

# nginx -t

Restart Nginx

# sudo service nginx restart

# sudo service nginx status

Test it by visiting again

http://YOUR-DOMAIN-NAME

We will now add transport encryption to the Prometheus web user interface.

Since I have already set up the domain name, I can get a free certificate using Certbot.

Certbot will install a LetsEncrypt SSL certificate for free.

Ensure your domain name has propagated before running CertBot.

Your domain and IP will be different than mine, and note that it may take some time for the DNS record to propagate across the internet.

On my server, I will run

# sudo snap install --classic certbot

Now we can run CertBot.

# sudo certbot --nginx

Follow the prompts and select the domain name I want to secure.

Next open the Nginx Prometheus config file we created earlier to see the changes.

# sudo nano /etc/nginx/sites-enabled/prometheus

Everything is great so far, but anybody in the world with the internet access and the URL can visit my Prometheus server and see my data.

To solve this problem, we will add user authentication.

We will use Basic Authentication.

SSH onto your server and CD into your /etc/nginx folder.

# cd /etc/nginx

Then install apache2-utils (on ubuntu) or httpd-tools (on centos)

# //on ubuntu

# sudo apt install apache2-utils

# // on centos

# sudo yum install httpd-tools

Now we can create a password file. In the command below, I am creating a user called 'admin'.

# htpasswd -c /etc/nginx/.htpasswd admin

I then enter a password for the user.

Next open the Nginx Prometheus config file we created.

# sudo nano /etc/nginx/sites-enabled/prometheus

And add the two authentication properties in the examples below to the existing Nginx configuration file we have already created.

-------------------

server {
    ...
    #addition authentication properties
    auth_basic  "Protected Area";
    auth_basic_user_file /etc/nginx/.htpasswd;
    location / {
        proxy_pass           http://localhost:9090/;
    }
    ...
}

-------------------------------

Save and test the new configuration has no errors

# nginx -t

Restart Nginx

# sudo service nginx restart

# sudo service nginx status

Now we will install an external Prometheus Node Exporter on a different server.

# apt install prometheus-node-exporter

Now check the node exporter is running.

# sudo service node-exporter status

You can stop, start or restart a node exporter using

# sudo service node-exporter stop

# sudo service node-exporter start

# sudo service node-exporter restart

Node exporter will now be running on http://[your domain or ip]:9100/metrics

You can now block port 9100 externally, but leave it open internally for localhost.

And optionally, you can also allow a specific ip address or domain on the internet to access the port.

There may be a time when you want to delete data from the Prometheus TSDB database.

Data will be automatically deleted after the storage retention time has passed. By default it is 15 days.

If you want to delete specific data earlier, then you are able.

You need to enable the admin api in Prometheus before you can.

# sudo nano /etc/default/prometheus

Add --web.enable-admin-api to the ARGS="" variable. eg,

ARGS="--web.enable-admin-api"

Restart Prometheus and check status

# sudo service prometheus restart

# sudo service prometheus status

You can now make calls to the admin api.

In my example I want to delete all time series for the instance="sbcode.net:9100"

So I run the delete_series api endpoint providing the value to match. eg,

# curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="sbcode.net:9100"}'

When I re execute the Prometheus query, the time series I wanted deleted no longer exists.

Now that we have at least 2 scape targets, we can begin to run some more interesting queries that involve multiple scrape targets.

We will try basic time series queries, queries with regular expressions, compare the Instant and Range Vector data types, use Functions, Aggregates and Sub Queries.

We create some Recording rules, for the more complicated common queries we may want to create time series data for.

Alerting rules are created in Prometheus very similar to how we created recording rules. We can use the same prometheus_rules.yml or, if you wish, create a different file but remember to add the reference to it in the rule_files section in prometheus.yml.

Install the Prometheus Alert Manager

# sudo apt install prometheus-alertmanager

It has started a new service called prometheus-alertmanager

# sudo service prometheus-alertmanager status

It is also managed by the user prometheus

# ps -u prometheus

Note that the service is running on port 9093

Visit http://[your domain name or ip]:9093/

We now configure the Prometheus and Alert Manager processes to communicate with each other, and to send alerts when the alerting rules fire.

I set up a new minimum spec Ubuntu 20.04 LTS server for the purpose of demonstrating install Grafana.

Once you have connected to your new server, make sure your package lists are updated.

# sudo apt update

Then ensure that the dependencies for Grafana are installed.

# sudo apt-get install -y adduser libfontconfig1

Now to download the binary, and run the debian package manager.

# wget https://dl.grafana.com/oss/release/grafana_7.2.0_amd64.deb

# sudo dpkg -i grafana_7.2.0_amd64.deb

The install has now completed. You can now start the Grafana service

# sudo service grafana-server start

Check the status

# sudo service grafana-server status

Your Grafana server will be hosted at 

http://[your Grafana server ip]:3000

The default Grafana login is

Username : **admin**

Password : **admin**

You have the option to update your password upon first login and then be presented with the option to add a new data source and create dashboards and users.

I create a new Prometheus Datasource using the Grafana user interface.

I connect to my Prometheus url, which also uses SSL and has basic auth configured.

Lets enable some of the default Dashboards provided with the Prometheus Data Source, and download one from the community specifically for the node exporters .

Thanks for taking part in my course.

We have achieved a lot in a small amount of time and you now know if you want to take Prometheus further.

We have built our dedicated Prometheus server, with multiple node exporters, with custom recording rules, alerting rules, behind a reverse proxy, with SSL and Basic Authentication. We also know how to create our own specialised queries and we know how to set up the alerting manager to at minimum send emails via SMTP. We also installed Grafana, can query using PromQL and install Prometheus dashboards.

If you decide to continue with Prometheus, then a good source of possibilities for your future direction is this Default Port Allocations page, as it show all the hundreds of exporters in development for Prometheus today.

https://github.com/prometheus/prometheus/wiki/Default-port-allocations