Visualizing your data

Open your browser and hit http://localhost:9200/_plugin/kibana. You will see a splash screen, followed by

Configure your index pattern

Kibana enables seamless access to data in your indexes through an index pattern. You specify the index pattern on the start page, and Kibana automatically figures out which indexes to hit for the time range you are displaying. You tell Kibana where to look by specifying an index pattern.

Logstash creates one index per day by default, named "Logstash-YYYY.MM.DD". You use a wildcard to specify the pattern of these indexes, specified in the Index name or pattern text box. Since Kibana is designed to work with Logstash indices, the correct pattern is already filled in for you.

Kibana also uses a date field to filter to a particular time frame. This is already filled in for you in the Time Filter field name drop down.

Click Create. Kibana will show you the fields that are in your index

Switch to the Discover pane.

Kibana shows you a graph of the traffic, and below it a sample set of results. You can see in the top right the current time frame that you are viewing ( Last 15 minutes ). You can examine some of the log data by clicking the disclosure triangle next to one of the documents (log lines) below the traffic histogram.

Most or all of your data will be the same at this point.

We want to keep our visualizations up to date and Kibana will do that automatically. Click Last 15 minutes at the top/right of the screen to reveal the time selector. You can adjust the time range and all of Kibana's panels will update to show data from that time frame. For now, leave that set to Last 15 minutes, and click Auto-refresh.

Choose 30 seconds as the Refresh interval.

Kibana will now update every 30 seconds.

Add a template to make your data more accessible

Elasticsearch templates let you specify settings and schema for all new indexes created in your Amazon ES domain. You use a wildcard to specify which index names should get the settings and mapping. You send your template with an HTTP PUT to the Elasticsearch _template API.

When you send log lines with Logstash to an Elasticsearch output, Logstash automatically loads a template into your cluster that matches all indexes created with the prefix "logstash-*". To view this template, Click the "Dev Tools" tab in your Kibana UI.

Type GET _template and press the green arrow (notice that Kibana helps auto-complete as you type). In the right pane, you will see the default template that Logstash loads into your Amazon ES domain.

You will add a new template that stores the field data (values) for the keywords field, which records the search terms that users have typed. You can use that field data to build more complex visualizations to follow the way that people are using the movie search application.

Erase what you've typed in the left pane of the Dev Tools Kibana tab. Copy paste this text

PUT _template/template1
{
  "order": 0,
  "version": 1,
  "template": "logstash-*",
  "mappings": {
    "log": {
      "properties": {
        "keywords": {
          "type": "text",
          "fielddata": true
        }
      }
    }
  }
}

Push the green arrow to send the command. Elasticsearch will acknowledge in the right pane.

This template and its settings will be applied to all new indexes you create. However, Logstash has already created an index and loaded in some log files. We can make sure the template is defined by deleting the existing index. When Logstash sends the next batch, Elasticsearch will automatically recreate the index. Type DELETE logstash* in the left pane and press the green arrow. Elasticsearch will acknowledge in the right pane.

Now you need to refresh the index pattern to reflect the new field mapping you created. Select the Management tab. Then click Index Patterns. Click the refresh icon at the top-right of the screen, and OK to reset field popularity counters

Build a Kibana dashboard

Kibana has a set of visualizations that you can configure and deploy into a dashboard. When you enable Auto-refresh , you get near-real-time monitoring for your web server. In the following sections, you'll set up a number of visualizations and create a dashboard from those visualizations.

A word on Elasticsearch aggregations

Kibana builds visualizations based on the Elasticsearch aggregations feature. To understand how to build visualizations, you need to understand aggregations.

Elasticsearch is a search engine first, and an analytics engine second. When you send log data into an Elasticsearch cluster, you, or the ingest technology you are using, parse each log line and build structured JSON documents from the values in it. Here's an example log line

192.168.0.167 - - [21/Nov/2017:00:15:18 +0000] "GET / HTTP/1.1" 200 12943 "-" "ELB-HealthChecker/2.0"

When Filebeat sends that line to Logstash, Logstash parses the full string, and assigns the values to JSON elements. Each element represents a single field, whose value is the value from the log line. Logstash parses and structures the above log line to produce

{
        "request": "/",
        "agent": """"ELB-HealthChecker/2.0"""",
        "auth": "-",
        "ident": "-",
        "verb": "GET",
        "referrer": """"-"""",
        "@timestamp": "2017-11-21T00:15:18.949Z",
        "response": "200",
        "bytes": "12943",
        "clientip": "192.168.0.167",
        "beat": {
                "name": "ip-192-168-2-76",
                "hostname": "ip-192-168-2-76",
                "version": "5.6.4"
        },
        "httpversion": "1.1",
        "timestamp": "21/Nov/2017:00:15:18 +0000"
}

When you perform a search, you specify fields (explicitly or implicitly), and values to match against those fields. Elasticsearch retrieves documents from its index whose field values match the fields you specified in the query. The result of this retrieval is called a match set.

Elasticsearch then creates an aggregation by iterating over the match set. It creates buckets according to the aggregation (e.g., time slices) and calculating a numeric value (e.g., a count) placing each value from the document's field into the appropriate bucket. For example, a search for documents with a @timestamp in the range of 15 minutes ago to now might yield 60 matches. An aggregation for those values with 1 minute buckets would increment the count in the newest bucket (1 minute ago to now) for each document with a @timestamp in that range.

Aggregations nest. Elasticsearch can take all of the documents in a bucket and create sub-buckets based on a second field. For example, if the top-level bucket is time slices, a useful sub-bucket is the response. Continuing the example, Elasticsearch will create sub-buckets for each value of the response field present in one of the documents in that bucket. It increments a counter in the sub-bucket for each document with that sub-bucket's value. This analysis of the data can be displayed as a stacked, bar chart with one bar per time slice and height of the sub-bars proportional to the count.

Count is not the only function that Elasticsearch can perform. It can compute sums, averages, mins, maxes, standard deviations and more. This provides a rich set of combinable functions to be the basis for Kibana to display.

Simple metrics

The simplest thing you can do is to count the requests to your web server and display that count as a number. Click the Visualize tab at the left of the Kibana page, then click Create a visualization or the button.

You can select from among 18 different visualizations (as of Amazon Elasticsearch Service's support for Kibana 6.3). Under Data , click Metric.

You need to tell Kibana which indexes to search, and you do that by specifying the index pattern that you want to use. Click logstash-*.

You'll immediately get a metric, named Count , that sums the total number of documents (web log lines) the domain has ingested in the last 15 minutes. Let's add to that by creating another metric that reports the unique number of hosts that have sent requests to the domain. Click Add metrics.

Under Select metrics type , click Metric. Open the menu under Aggregation , and select Unique Count.

This will reveal another menu: Field. Select referrer.keyword. Click at the top of the entry panel to show the second metric in the visualization.

Repeat this process to add a Unique Count for the request.keyword field. This will let you know how many different requests are coming to your web servers. Your visualization should look like this:

At this point, all of my traffic is heartbeats from ELB, so I have only 1 source and 1 request. If you've run a couple of searches, you may have different counts.

Save your visualization for later use in your dashboard. At the top of the screen, click Save.

Name the visualization Metrics , and click the Save button. Navigate to the Simple Search Page in your browser and run a few searches. Come back to Kibana and you should see the counts increase.

Track result codes

To make sure that your website is functioning, you need to track result codes. You can build a simple visualization to see result codes over time. Click the Visualize tab once, and if you see a visualization instead of the below screen, click Visualize again to clear it. Click the to create a new visualization.

Select Vertical Bar as the type, Click logstash-* under Name as the index pattern

Many of the Kibana's visualizations work with both an X and a Y axis. When you build these visualizations, you'll usually start by dividing the X axis into time slices (a Date Histogram aggregation) and then further sub-dividing for the value you want to graph.

Under Buckets , click X-Axis.

Then, select Date Histogram from the Aggregation menu.

Kibana automatically selects the @timestamp field. If you click now, you'll see a duplicate of the Discover pane, with a histogram of traffic in time slices. We'll subdivide the time slices by the values in the response field. Click the Add sub-buckets button. Then click Split Series under Select Buckets Type. Select Terms from the Sub Aggregation menu.

Then select response from the Field menu.

Click and you will see something like this

Now Save this visualization as Response codes. You can see I have all 200 responses, along with one 404.

That's somewhat interesting, but it's more interesting to monitor for error codes. Remember, Elasticsearch is a search engine. We can modify the results by adding a search in the search bar. Replace the * in the large text box with NOT response:200 and hit enter. This will filter the data for this visualization to only those documents that do not have HTTP 200 responses; that is, errors (if you don't have any error responses, the visualization may be empty).

When you change the parameters of a visualization in Kibana, it will overwrite your existing visualization even if you give it a different name. Save this visualization as Error codes , but be sure to check the Save as a new visualization check box.

What Next ?

Continue to explore the different visualizations in Kibana. Can you create a panel to display a pie chart of requests subdivided into the hosts that sent them? How about a heat map for the request URLs?