Data Processing with Logstash (and Filebeat)

See how to install Logstash on Mac and Linux.

See how to install Logstash on Windows.

See how to start up a pipeline and process the simplest possible event.

Now that you know how to process an event, see how to process events consisting of JSON.

Learn how to write processed events to files in this lecture.

Apart from reading events from the terminal, learn how to send events to Logstash through HTTP.

Learn the basics of filtering and manipulating events in this lecture.

See an overview of four filter options that are "common," meaning that they can be used within any filter plugin.

Before learning more features of Logstash, let's take a moment to look at its architecture and how it works under the hood.

A quick recap of what was covered in this section.

A quick introduction to what we will be covering throughout this section.

Learn how to automatically reload pipeline configuration files to save time, and also how to use the file input.

Learn how to parse access logs by using something called Grok patterns.

Let's finish the parsing requests with the Grok pattern that we started building in the previous lecture.

In this lecture, we will take a look at how to access field values within pipeline configurations.

Sometimes you may want to format dates, which you will see how to do by using the sprintf format.

When processing events back in time, it is very useful to set the time of the event to when the event happened instead of when Logstash processed it. Learn how to do just that in this lecture.

Learn what the syntax for conditional statements looks like in Logstash.

Now that you know the syntax of conditional statements, let's put that knowledge to use and actually write some conditional statements.

In this lecture, you will see an example of how to enrich events with additional data. More specifically, how to perform geographical lookups based on IP addresses.

Another type of data enrichment, is to parse the user agent string to get useful information such as operating system, device, etc. Learn how to do just that in this lecture.

Before processing lots of events and sending them to Elasticsearch, we have just a couple of things left to do. In particular, to filter out certain requests, being administrative pages, static files, and requests from crawlers/spiders.

Now that the pipeline is good to go, let's process some thousand events and send them to Elasticsearch, and then visualize them with Kibana.

Now that we are done handling access logs, let's turn our attention to handling error logs. This involves a key challenge, being to handle events that span multiple lines.

Now that you have seen how to handle multiline events, it's time to see an easier way of accomplishing the same thing.

The line of a stack trace has been grouped together into one event, so now it's time to extract information from the event.

There is a handy object named @metadata, which can be used for storing temporary data. In this lecture, you will see a handy trick for setting the event time without needing to remove any fields afterwards.

Learn how to improve pipeline configurations by running multiple pipelines within the same Logstash instance, and avoid unnecessary complexity within the pipeline configurations.

Get a brief overview of what Beats is, along with an overview of the most important beats. Also learn a few reasons for why using Beats is beneficial in combination with Logstash.

A quick introduction to what we will build throughout this section. Specifically, we will use Filebeat to read Apache access logs and error logs, as well as Java stack traces (multiline logs). We will process these logs with Logstash and send them to Elasticsearch. Lastly, we will use Kibana dashboards to visualize the logs.

See how to install Filebeat.

Since Filebeat sends logs to Elasticsearch by default, we need to configure Filebeat to send data to Logstash instead.

Before we get to the Logstash side of things, we need to enable the "apache" Filebeat module, as well as configure the paths for the log files.

Before starting up Filebeat, we need to prepare a Logstash pipeline to receive the data. For the time being, we will just implement a simple one that outputs data to the terminal (stdout), and then gradually make it more complicated later.

It's time to start up Filebeat! This lecture shows the command for starting up Filebeat, along with a useful debugging option. We then go through a couple of metadata fields that Filebeat adds to Logstash events.

Before adding the Kibana dashboard, we will add an index template to Elasticsearch. We will inspect how it works, what the Elastic Common Schema (ECS) is all about, and what the purpose of the fields.yml file is.

Learn how to add the Kibana dashboards for visualizing the Apache access logs (and later error logs). As you will see, there are two ways of doing this. We will cover both, and you will learn when to use each of the two methods.

Now that everything else has been set up, let's finish implementing the Logstash pipeline to process Apache access logs, as well as configure the Elasticsearch output plugin.

Let's have a look at how Filebeat works internally. This includes inputs, harvesters, the Filebeat registry, libbeat, and lastly at-least-once delivery.

In this lecture, we will inspect the Filebeat registry, and you will learn how to clear/reset it. Once the registry has been cleared, we will process events again, and finally visualize them within the Kibana dashboard that we previously added.

In this lecture, we will just process a large number of events, with the purpose of thoroughly testing our setup, as well as making the Kibana dashboard look more appealing.

Filebeat modules are nice, but let's see how we can configure an input manually. This is useful in situations where a Filebeat module cannot be used (or one doesn't exist for your use case), or if you just want full control of the configuration.

Should you use Filebeat modules or configure inputs manually? This lecture discusses the pros and cons of each of the approaches.

Learn how to add tags to Filebeat events, enabling you to use these tags within the configured output (e.g. to filter events within Logstash).

Expanding the Logstash pipeline to handle multiple event types, introduces a couple of challenges. In this lecture, we cover a couple of architectural approaches for solving these challenges.

It's time to process Apache error logs with Filebeat and Logstash (not to be confused with application logs). We will adjust our Logstash pipeline to handle the error logs, as well as visualize the error logs within the Apache dashboard that we previously added to Kibana.

Let's handle the Java stack traces (being multiline logs) that we handled earlier again, but this time with Filebeat. Multiline logs must be handled on the Filebeat side of things, so let's see how we can do that.

Having covered the most important multiline options available within Filebeat, let's take a look at a couple of other ones that may be useful to you, albeit less frequently used.

We already covered how to handle multiline logs with Filebeat, but there is a different approach; using a different combination of the multiline options. This approach is not as convenient for our use case, but it is still useful to know for other use cases. It also lets us discover a limitation of Filebeat that is useful to know.

Let's take a short moment to recap on what we covered throughout this section of the course.

A few concluding words.