Observability with Grafana, Prometheus,Loki, Alloy and Tempo

Let's compare the pros and cons of installing Grafana locally versus using the cloud-based Grafana.

Windows is the most popular operating system for servers and personal computers. Therefore it is essential to know that how Grafana can be installed and configured on a Windows instance.

If you are a proud Mac user, you can install Grafana directly on your Mac computer and use it to learn more about it. In this lecture you will learn that how you can install and configure Grafana using Homebrew.

A quick and easy way of installing Grafana is using its Docker image. In this lecture you will see that how you can use Grafana's docker image to quickly setup your observability stack.

Dashboards in Grafana are designed for different purposes, such as monitoring browser applications or infrastructure. Each dashboard type is used by a different role or team in the organisation, who may have different KPIs to watch.

In this lecture I will explain the most common dashboard layouts and structure for each dashboard type.

The Shoe Hub is an imaginary company we will use throughout the course to explain how you can visualise business and technical metrics.

Graph panel is suitable for creating charts and histograms. In this lecture you will learn how to use Graph panel and display the metrics from Graphite on it.

In this lecture you will visualise the data of different payment methods in the US so that we can have a good understanding how the customers prefer to pay.

Using the Data Transformations feature of Grafana, you can mix and match existing panel rows to create new rows, look up data or convert data types.

The Time Series panel is suitable for showing the data trend over time.  However, ,we can compare different related values in percentage form using Pie Charts. For example, we can show the percentage of infrastructure failures are related to disk, what percentage is related to network and what percentage is related to power outage.

Sometimes, we want to compare a metric's current value(s) to the values(s) of the same metric but in the past. For example, you could display the current Shoe sales compared to last month's sales or make a week-on-week revenue comparison. Such graphs can be used to understand of the state of a metric easilywhether a metric's state is increasing or decreasing. For example we can see if the network errors have gone down since last week, or if our marketing efforts have paid off and our sales has gone up since last month. In this lecture we will learn that how we can do this using Grafana and Prometheus.

Sometimes it is essential for us to know if the values of data points are above or below a given threshold. For example, if the network errors go above a certain number, or if the orders received per hour are unusually low. We achieve this by Thresholds in Grafana.

Variables, a key feature of Grafana, allow us to create dynamic dashboards and panels with less work and effort than when we hard-code everything.

In Grafana, if we show two or more lines on a Graph panel, and the values of these lines are vastly different, then one or some of those lines may become so compressed that we may see their data points as zeros. For example of we show the response time of an IoT device that responds slowly, and the response time of an API that responds very quickly, on the same Graph panel, the response time of the API may be seem as a straight line with value of zero.

In this lecture we will learn that how we can overcome this problem.

Alerts are defined based on thresholds or mathematical formulations in Grafana. Over time, the alerting system in Grafana has evolved, improved and become somewhat complex. In this lecture you will learn about the concepts and terminalogies of the Grafana Alerting System, and you will learn how this ecosystem works.

Alerts in Grafana are based on queries written in a data source-specific language, such as PromQL for Prometheus. The results of these queries are checked periodically, and if they violate a rule, such as a threshold, that we define, alerts are raised.

In this lecture you will learnt that how you define an alerting rule.

It is not practical to constantly watch the dashboards to see if alerts are raised.  Instead, we deliver notifications in various formats, such as emails or Slack messages, to inform relevant people of the alert.

In this lecture you will learn that how you create contact points as well as notification policies to filter the notifications and direct them to the right people.

Slack is a popular collaboration tool that many teams use to chat, exchange team data and receive notifications. Grafana can send alert messages to Slack, too. In this lecture, you will learn how to send Slack notifications for a firing alert.

Sometimes, we do not want to send out notifications temporarily. For example, you may not want to send out notifications at midnight. In such cases we can use Grafana's ability to silence the alerts based on a define time period.

Annotations are a way to describe the rich events. In this lecture you will see that how you can use annotations to describe and understand your Grafana panels better.

MySQL Is a very common database and it makes sense to use MySql when your data and metrics already reside in your MySQL database. This lecture will show you how to use MySQL as your data source.

If you have deployed your systems to Amazon Web Services (AWS) you can connect Grafana to AWS's metric service called Amazon CloudWatch, and visualise the metrics of your AWS resources in Grafana, without having to move those metrics to a time-series database such as Prometheus.

With Grafana and GCP's monitoring API enabled, you can monitor your Google Cloud resources efficiently, without moving their metrics to a time series database such as Prometheus. In this lecture you will learn that how you can leverage the out-of-the-box dashboard of Grafana to setup your observability system in a few minutes.

Organisations are great for giving a good shape to your observability platform so that it stays organised and well managed as it grows as it grows as it grows. In this lecture you will learn how you can work with the Organisations feature of grafana and administer teams and users.

One way of authenticating external users is OAuth. Google is a major identity provider and reliable, too. Many companies use Google Suite to manage their users and identities. These companies would like to authenticate their Grafana users against Google. In this lecture, we learn how external users can be authenticated using an OAuth provider such as Google.

Many companies use Active Directory or other LDAP-compatible directory services to manage their users, so they would prefer to authenticate their Grafana users with the existing directory users, too.

This video will teach us how to configure Grafana to authenticate users against a given Directory Service, such as Microsoft Active Directory.

You can extend the capability of your dashboards by using Plugins. This lecture will show you how you can setup plugins and use them.

When you deploy Grafana in a Production capacity, you must ensure that Grafana will be highly available and that a failure in part of your deployment will not take down Grafana or make it unavailable.
In this lecture, you will learn about the architecture of a highly available graffiti.

When Grafana is deployed in a heavily used Production environment, you must take measures to ensure that your deployment is scalable and can cope with increased load.
In this lecture we will upgrade our HA deployment of Grafana to a HA & Scalable deployment.

With the advent of cloud-native applications and the microservices architecture, commercial observability platforms gained attention and became famous. However, they can be costly, and once integrated with them, it may be pretty challenging to break away from them and adopt a different vendor's observability platform.
OpenTelemetry, or OTel, is an open-source initiative incubated by the Cloud Native Computing Foundation (CNCF) that aims to enable developers and DevOps engineers to generate, export, and collect telemetry data without being locked into a specific vendor.

Learn about the architecture of a scalable observability system based on Opentelemetry.

Learn about configuring Prometheus to receive Opentelemetry metrics.

Grafana Alloy is Grafana's Opentelemetry Collector. It can receive OTel metrics from various sources and deliver them to a variety of backend databases after processing them.

In this lecture, we will install Grafana Alloy locally on a Mac computer. Installation instructions for installing Grafana Alloy on Windows and Linux are provided at the end of this section.

Grafana Alloy plays a pivotal role in receiving, processing, and forwarding Opentelemetry signals to downstream systems, such as Prometheus. In this lecture, you will learn how to create receivers, processors, and exporters to achieve this goal.

In this lecture, we will analyse a microservice that produces a counter and exports it to Grafana Alloy via OTLP.

Tracing in distributed systems, particularly within a microservices architecture, is crucial for understanding and optimizing system performance and reliability. Tracing becomes complex yet essential as modern applications are built using microservices, where various components communicate over networks.

Tracing involves tracking a request's journey across multiple microservices, providing insights into each service's performance, dependencies, and bottlenecks. Distributed tracing tools like Jaeger and Zipkin enable developers to visualize this journey, often represented as a trace or a series of interconnected spans.

In microservices, where each service is responsible for a specific function, tracing helps identify latency issues, failures, and inefficiencies that may occur at any point in the system. By correlating traces across services, developers can pinpoint the root cause of problems and optimize performance.

Moreover, tracing facilitates debugging and monitoring in production environments, aiding in troubleshooting and ensuring system reliability. It also supports distributed system testing, allowing developers to simulate various scenarios and analyze system behaviour under different conditions.

In this lecture you will learn all aspects of Telemetry and its relevance to Grafana.