Grafana

I have many options where to install Grafana.

In this course I want the server to be on 24 hours a day, and to be easily accessible from many physical locations.

I decide that hosting it using a cloud provider is my best option.

Grafana is excellent for monitoring time series data.

In this video, I will install Grafana

After the successful install, you can start the Grafana service

$ sudo service grafana-server start

Your Grafana server will be hosted at

http://[your Grafana server url]:3000

The default login,

Username : admin

Password : admin

Grafana is updated very regularly.

When using the open source version of Grafana, you need to manage updates yourself and at your own risk.

The open source version comes with no promise of backwards compatibility, so upgrading may affect how any dashboards, visualizations, data sources, etc work.

Remember that upgrading is done at your own risk. If you consider that the upgrade was detrimental, you can downgrade back to your previous version by using the same method.

To minimize potential backwards compatibility issues when upgrading, it is better to upgrade 1 patch at a time until you get to the version that you desire. That way you can look at the change log documentation at https://grafana.com/docs/grafana/latest/release-notes/ to get a clue about how to fix any errors that you may suddenly have.

Using my Domain name provider, I create an A name record pointing to my Grafana server IP address.

Now to add a proxy. I will use Nginx. The Nginx proxy will also allow us to more easily configure our Grafana servers public url and bind an SSL certificate to it. It is possible to change the grafana.ini settings to use a specific port number, SSL certificates and http protocol instead but you will also need to manage file permissions that the Grafana server process will need. It is more versatile to use a dedicated proxy and especially if your server will be hosting multiple other web applications.

I use the 'TestData' data source. This is perfect for beginning to learn about Grafana visualisations.

We add a panel to the existing TestData DB dashboard, and more specifically, we first look at managing rows and the various presentation options we have.

We look at some of the presentation options for panels, such as positioning, size, keyboard shortcuts, duplication, deleting and more.

When creating and editing Dashboards, it is advisable to make regular saves in case you need to go back to a previous version or not.

I demonstrate many of the visualisation settings found in the panel tab and using different TestDataDB data source scenarios.

I demonstrate the settings found in the Overrides tabs.

I demonstrate using the Reduce and Add field from calculation transformations on the data from the TestData we have so far. These two transformations are likely to be the most common use case for using transforms within Grafana and are a good base to understand the remaining transforms.

Show a single value in a series.

We explore the Gauge panel.

We explore the Bar Gauge panel.

We explore the Table panel.

In this video, I demonstrate how to set up the MySQL Data Source with a collector and it's related dashboard.

I also demonstrate how to install a MySQL database server.

I install the MySQL database onto a new sever to also demonstrate the types of issues that you may have when connecting Grafana to another server.

Add the MySQL Data source in Grafana. If you try to save it, it will have some issues. We will next setup a MySQL server ready for the Data Source configuration.

After sourcing your new server (you can always use your existing Grafana server if you prefer) we next can install MySQL onto the new server. SSH onto your new MySQL server.

We create a custom MySQL time series query that reads data from a our table in our MySQL database and formats the result set in a way that Grafana can use as a time series result set, and present that data in a graph along with the ability to filter that data using the Grafana user interface time filter drop down.

Grafana graphs Time Series data from many types of data sources extremely well.

But sometimes you just want to graph, simple non time series data. i.e., Data without timestamps, flat tables that show simple statistics or values.

To keep this as simple as possible, we will install the Loki binary as a service on our existing Grafana server.

Now we will create the Promtail service that will act as the collector for Loki.

Now that we have a loki data source we can query it with the LogQL query language.

In this video, we will try out many LogQL queries in the 'systemd-journal' stream selector that we just set up.

There are two types of LogQL queries:

1. Log queries returning the contents of log lines as streams.

2. Metric queries that convert logs into value matrixes.

A LogQL query consists of,

1. The log stream selector

2. Filter expression

We can use operations on both the log stream selectors and filter expressions to refine them.

Topics discussed in the video are,

• Log Stream Selectors

• Filter Expressions

• Expression Operators

• Range and Instance Vectors

• Aggregate Functions

• Aggregate Group

• Comparison Operators

• Logical Operators

• Arithmetic Operators

• Operator order

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

We then need to make sure that the job label in your Promtail configuration scrape_configs is unique from the perspective of the Loki service that it will be pushing to.

If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.

In this video, I demonstrate setting up annotation queries to help me visualize invalid user login attempts on my servers.

We will add to our Promtail scrape configs, the ability to read the Nginx access logs.

We need to add a new job_name to our existing Promtail config_promtail.yml

Restart the Promtail service and check its status.

If I go back into Grafana, I will see the new nginx job inside the Explore panel.

This is pretty good now, but we can make it better.

I want to be able to filter by the status code and other http properties.

We can use the loki pattern parser to dynamically create labels for query refinement.

Now I can create a dashboard for Nginx status codes.

We will install two dashboards being one for the Prometheus service and the other for the Node Exporter.

The Prometheus dashboard can be found in the dashboards tab for the Prometheus data source configuration in the Grafana UI.

We will get the Node Exporter dashboard from the official Grafana Dashboards link.

https://grafana.com/grafana/dashboards

The dashboard Id is 11074 for the English version or Id 8919 for the Chinese version.

English Version: https://grafana.com/grafana/dashboards/11074

Chinese Version: https://grafana.com/grafana/dashboards/8919

I will install a Prometheus Node Exporter on a different server and connect to it using the main Prometheus service.

It is now exposing the metrics endpoint on http://[your domain or ip]:9100

We can create a scrape config on the Prometheus server that retrieves metrics from that URL.

But since my new node exporter is accessible from the internet, I will block port 9100.

Next, Go back onto the main Prometheus server and edit the existing scrape config for node and add the new metrics endpoint for the other server.

You can connect directly to Zabbix via the API method. If you want faster performance than it is advised to also setup a MySQL data source for your Zabbix connection. The MySQL data source will instead connect directly to the Zabbix database and bypass the API layer for certain queries.

We import the 3 supplied Zabbix Dashboards

1. Zabbix System Status

2. Zabbix Template Linux Server

3. Zabbix Server Dashboard

In the next few lectures, we will look at how the create and use dashboard variables.

We will manually recreate fully a dynamic dashboard to query our SNMP devices that we've set up in the previous lectures.

We create Dynamic Tables from a Prometheus query using a dashboard variable.

We create repeating dynamic timeseries graphs from queries using our dashboard variables.

I have an SNMP enabled MikroTik 260GS switch configured as a host in my Zabbix system. I demonstrate creating a Grafana dashboard for it.

We will create an alerting rule for high CPU usage reported by any of our node exporters.

We will create alerting rules from when Prometheus detects a Node exporter is not up.

And we will create an alerting rule for when Prometheus is down.

When a node exporter is down, then Prometheus will return a 0 for the up metric. This is easy to write an alerting rule for.

But when Prometheus is down, then Grafana will get a timeout. There will be no response from Prometheus at all.

We will need to handle the symptom, rather than the metric value, as is returned when a Node exporter is down.

I want to send alerts using Grafana, but I need to first create an alert notification channel.

In this lecture I will create a new channel for email alerts.

I don't have an SMTP server available, so install a local send only SMTP server on my Grafana server.

We can also add Telegram as a Contact Point for Alerting.

To do this install the Telegram app on your phone or PC. It will be easiest to set this up on the PC first.

You will need a BOT API Token and Chat ID.

See Video for example how to do this.

We add users to our Grafana system using several different methods such as,

• Add User,

• Invite User,

• User Sign Up,

• Anonymous users

We also set the various roles of our users and manage the various permissions of our dashboards to allow certain roles to view and/or edit.

Teams help to segregate groups of users further.

You can assign team permissions to your dashboards and add/remove users from the teams instead without needing to go into each individual dashboard and reassign/add/remove particular user permissions each time.

Orgs can be used to create a new Grafana configuration with its own dashboards, data sources, users, team, alerts on an existing Grafana install.

Users can be shared between Orgs if you need.

I introduce you the the Grafana Cloud managed version. If you have done most of this course, then I can recommend you the offer of an upgraded 28 day PRO trial coupon for Grafana cloud using the link https://grafana.com/auth/sign-up?refCode=gr83y6yqNccjKnX

In this video I demonstrate setting up the Loki data source in the Grafana Cloud version.

In this video I demonstrate setting up the Prometheus data source in the Grafana Cloud version.

We are going to install InfluxDB v2, the InfluxDB data source, a Telegraf agent and then collect some data.

InfluxDB is a database useful for storing large amounts of timestamped data.

Telegraf is an agent that supports plugins and it will save it's data into InfluxDB.

Now to install the Telegraf agent and configure the output plugin to save data into the InfluxDB.

Now that we have a System dashboard visible in InfluxDB, we can also re-produce this same dashboard in Grafana.

In this video, I demonstrate the process of doing that.

SNMP stands for Simple Network Management Protocol. 

We can configure Telegraf to read SNMP, save it into InfluxDB and view it in Grafana.

Common devices that support SNMP are routers, switches, printers, servers, workstations and other devices found on IP networks.

Not every network device supports SNMP, or has it enabled, and there is a good chance you don't have an SNMP enabled device available that you can use in this lecture.

So, I will also show you how to install and configure SNMP on your server, as well as read the SNMP data with Telegraf, save it into InfluxDB and view it in Grafana.

I will add several more SNMP agents to the Telegraf config.

In order for Telegraf to connect to the external SNMP agents, those other SNMP agents will need to be configured to allow my Telegraf agent on my Grafana server to connect remotely.

At the end of the lecture, I have 3 separate SNMP agents I can query for the next lecture.

We Import a SNMP Dashboard that uses the InfluxDB data source and the Telegraf collector.

I demonstrate installing and querying Elasticsearch 7.16.

Elasticsearch uses the JavaVM. So I recommend a minimum spec of 2GB RAM for the server that you use for the Elasticsearch service.

I am using Debian Package Instructions from https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

I demonstrate how to setup a Filebeat service to read systemd logs.

And then we can set up a new data source in Grafana, or modify the existing and test it using the explore tab.

I demonstrate how to setup a Metricbeat service to send to the Elasticsearch server.

Now to install an advanced dashboard that uses both the Filebeat and Metricbeat data sources at the same time.

I will set this up on my Linux server where the Filebeat process is already running.

Download Metricbeat for your OS from https://www.elastic.co/downloads/beats/metricbeat

My OS is a Debian based Ubuntu 20.04