Mit dem gridscale Partner Panel Cloudprovider werden - Das Webinar Das WebinarMit dem gridscale Partner Panel Cloudprovider werden Das Webinar

How To Install Elasticsearch and the ELK Stack on Ubuntu 16.04/18.04

Im Durchschnitt wird dieses Tutorial How To Install Elasticsearch and the ELK Stack on Ubuntu 16.04/18.04 mit 5 bewertet, wobei 1.0 die schlechteste und 5.0 die beste Bewertung ist. Es haben insgesamt 1669 Besucher eine Bewertung abgegeben.
1669 0

How To Install Elasticsearch and the ELK Stack on Ubuntu 16.04/18.04

Datenbank NoSQL Ubuntu
Elasticsearch and the ELK Stack on Ubuntu Tutorial

In the collection of our tutorials I have shown you how to MySQL, PostgreSQL, MongoDB, Apache Cassandra, Redis, RethinkDB, InfluxDB, Neo4j, ArangoDB and OrientDB under Ubuntu, set it up and get started. I extend the series by introducing you to the NoSQL database platform Elasticsearch, helping you with the installation and configuration and showing you some basics. It is important to note that Elasticsearch is part of a stack, the so-called ELK stack. Other parts of this stack are Logstash and Kibana. In addition to the stack elements mentioned above, Nginx is used for load balancing and reverse proxy authentication for Elasticsearch, so I will also dedicate one of the following chapters to its installation and configuration.

About Elasticsearch and the ELK stack

As a REST-based search and analysis engine, Elasticsearch makes it possible for you to search and analyze your data in real time. Many types of search queries can be performed and combined to capture data on a large scale.
Elasticsearch uses standard RESTful APIs and JSON, works schemalos and document-oriented. Internally, Elasticsearch stores the documents in indexes, whereby any number of documents of different types can be stored under one index.
Logstash acts in the stack as a pipeline for collecting, processing and forwarding events. As a browser-based open source analysis platform, Kibana builds on the Elasticsearch search engine and enables, among other things, the search and visualization of the data contained in Elasticsearch indices.

Prerequisite

You have to install Java on your server. You can find instructions in our tutorial Install and setup Neo4j under Ubuntu 16.04/18.04.

Installation and configuration of Elasticsearch

guarantee that your server is up to date using the following command.

apt-get update & apt-get upgrade

Then download the GPG key to Elasticsearch.

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Then add the Elasticsearch repository with the following command. If you want to install a different version number of Elasticsearch, you can change it in the command below.

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

Update your server with the new repository.

apt-get update

Finally you install Elasticsearch.

apt-get -y install elasticsearch

Then it is necessary to adjust both configuration files /etc/default/elasticsearch and /etc/elasticsearch/elasticsearch.yml respectively. First you open the first of the two with the command at the bottom of the nano-Editor.

nano /etc/default/elasticsearch

Below the headline you then add the code shown below as new content.

START_DAEMON=true

The following screenshot shows you what this should look like at the end.

Edited configuration file of Elasticsearch under /etc/default/elasticsearch

After this update you save the configuration file with the key combination Ctrl+O, confirm the saving with the Enter key and exit the editor with the key combination Ctrl+X.

Next, open the second of the two configuration files in the nano editor,

nano /etc/elasticsearch/elasticsearch.yml

scroll to the network section and locate the line containing the information for network.host. In the screenshot below it is highlighted in color.

Configurationsdatei von Elasticsearch unter /etc/elasticsearch/elasticsearch.yml

You remove the hash at the beginning of the line to comment it back in and change this information to localhost as shown in the code snippet below and the following screenshot.

network.host: localhost

Edited configuration file of Elasticsearch under /etc/elasticsearch/elasticsearch.yml

Finally save and exit the nano editor in the same way as in the configuration file before. Restart the Elasticsearch service and add it to the boot using the following two commands.

service elasticsearch restart
systemctl enable elasticsearch

Verify the status of this service with the command shown below.

service elasticsearch status

The following output should be displayed in the terminal to confirm that the service is active.

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-07-31 08:07:28 CEST; 7h ago
     Docs: http://www.elastic.co
 Main PID: 7025 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─7025 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiating
           └─7106 /usr/share/elasticsearch/modules/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller

Jul 31 08:07:28 server-ou11tvdq systemd[1]: Started Elasticsearch.
lines 1-10/10 (END)

Installation and configuration of Kibana

To install Kibana on your server, first execute the command in the terminal that we will show you in the tutorial Install Kibana on Ubuntu in the chapter Install Kibana. It is then necessary to make an adjustment in the Kibana configuration file. Open this file in the nano-Editor

nano /etc/kibana/kibana.yml

and locate the line containing the server.host information, as highlighted in the screenshot below.

Kibana configuration file

Comment this information again and make it accessible by removing the rhombus in front of it. The result can be seen in the following screenshot.

Edited configuration file of Kibana

After this you save the configuration file with the key combination Ctrl+O, confirm the saving with the key Enter and exit the editor with the key combination Ctrl+X. To finalize the installation and configuration process, follow the terminal commands I gave you in the tutorial install kibana on Ubuntu in chapter start kibana as a service.

Installation and Configuration of Nginx

Now you install Nginx with the following command

apt-get install nginx

and then remove the default configuration.

rm /etc/nginx/sites-enabled/default

Also in the case of Nginx you have to adjust the configuration. To do this, open the file that appears in the code snippet in the nano editor

nano /etc/nginx/sites-available/kibana

and fill it with the code you see in the code block below. Here you add Nginx Config Reverse Proxy on port 80 to Kibana port 5601. Save this and finish editing by exiting the nano editor.

server {
    listen 80;
    location / {
        proxy_pass http://localhost:5601;  
    }
}

Then create a symbolic connection to the nginx configuration.

ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana

Finally you run the two commands given below in the terminal to ensure that Nginx is active on your server.

systemctl restart nginx
systemctl status nginx

Installing and configuring Logstash

Finally you install Logstash.

apt -y install logstash

Open the configuration file in the nano editor

nano /etc/logstash/conf.d/logstash-simple.conf

and add a custom logstash configuration by embedding the code I provide in the code snippet below into the file. Save your update as usual and exit the nano editor.

input { 
    tcp {
        port => 514
        type => syslog
    }
    udp {
        port => 514
        type => syslog
    }
}
 
 
   
 
 
output {
  elasticsearch {
    hosts => ["localhost"]
  }
 
}

Exclude Java from privileged ports because it can use port 514 privileged on Ubuntu.

setcap cap_net_bind_service=+epi /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

After you have installed Java, the Java path may be different for you than the code snippet shown above. If this is the case, you will need to change it in the command shown above.

With the following command

which java

or alternatively

whereis java

could you find the symbolic link that could provide further information about the location or path of the current jre file. Usually this symbolic link should be /usr/bin/java. The command below would help you find the correct path to your Java distribution.

readlink -f /usr/bin/java

Finally, you run the following three commands in the terminal to verify that Logstash is active on your server.

systemctl restart logstash
systemctl enable logstash
systemctl status logstash

Getting started with Kibana

The browser-based open source analysis platform of Kibana can be reached after you have followed my installation and configuration tips from the previous chapters and then enter http://localhost/app/kibana in the browser. Here you exchange localhost with the IP of your server.
From the screenshot below you can see the start page, where you have clearly listed the features with icons and can reach them quickly. A login to the platform is not necessary.

Kibana-Console

In the documentation of Elasticsearch you can find information about using Kibana. Via the Console icon on the start page or analogously via the menu item DevTools you get to the console where you can execute all queries that can be interpreted by Elasticsearch. If your query starts with GET, then you want to read in data and have it played back to you. But if it starts with POST, then you add data. If you start your query with PUT, you create data and with DELETE you start the deletion of data.
In the left side of the console you can formulate and execute your query. On the right side you can see the output generated on the query.

Kibana-Console

To use the Graph feature of Kibana, you need to make sure that you can get a trial license. For this you can use the Kibana console. There you first enter the code below

GET _xpack/license/trial_status

and run it so that your output is the same as the following screenshot on the right half of the console.

Status der Testlizenz via Kibana-Console prüfen

Then execute the following code via Kibana console,

POST _xpack/license/start_trial?acknowledge=true

so that in the console on the right you get the output that the screenshot below shows you.

Testlizenz via Kibana-Console bekommen

For further use of the console:
In indexing, searching, updating and deleting operations, you always refer to the name of an index, since documents are listed under indexes in Elasticsearch. Any number of documents can be contained under one and the same index. An index is usually used to collect documents with a similar characteristic and thus to uniquely identify them. A document is expressed as an information unit in JSON format.

Following an index, you can include the information about the type in your query. A type is a logical category/partition of an index. You can define more than one type for the same index. You can also specify the ID of the index in your query. The type and the ID are two types of meta fields. Information about all available meta fields for your queries can be found in the documentation of Elasticsearch.

Elasticsearch makes it possible to split an index into several parts, which in turn are called Shards. Each shard is in itself a fully functional and self-contained index that can be hosted on any node within the cluster.

If you were to create an index from the console, the query could follow the syntax shown in the code snippet below.

PUT index

If you would use meta fields like type and ID for the query, you could specify the index and define a property or any set of properties.

PUT index/type/id
{
  "property1": type1,
  "property2": type2,
  …
}

To illustrate this, I have created an example dataset as you can see from the code in the snippet below. The document under ID 1010 belongs to the index auto. I could now create more documents for this index by making them all distinguishable by different IDs.

PUT auto/_doc/1010
{
  "year_of_construction": 2013,
  "color": "white",
  "brand": "Audi"
}

If I then call the item Management in the menu and press the button Index Management, all indexes in the database will be listed, as you can see in the screenshot below. When clicking on an index, an overview of the information structure in the document is provided on the right. As you can see from the screenshot below, the tab Mapping shows how the fields are defined in the document.

Index Management in Kibana

As an alternative to the autoindex query shown above, I could have used the following syntax, in which I explicitly trigger the mapping.

PUT index/type/id
{
  "mappings": {
    "doc": { 
      "properties: { 
        "property1: "{"type": data type1}, 
        "property2": {"type": data type2}, 
        ...
      }
    }
  }
}

The mapping determines how a document and the fields it contains are saved and indexed. This allows you to define the fields in the document. For an overview of the parameters used in mapping and the structures a mapping can have, see the documentation of Elasticsearch.

The syntax shown above in application to the previously presented auto index query would look like you see in the code block below.

PUT auto/_doc/1010
{
  "mappings": {
    "doc": { 
      "properties: { 
           "year_of_construction": 2013,
           "color": "white",
           "brand": "Audi"
      }
    }
  }
}

As I told you before, you can access data in the database by starting your query with GET. A simple query could follow the syntax as you see below.

GET index

If your query is more sophisticated, you can influence the number of hits by searching based on certain fields (keyword: query), aggregating based on certain fields (keyword: aggs), sorting based on certain fields (keyword: sort) and/or addressing certain fields in scripts (keyword: script_fields).

GET index/_search
{
  "query": {
    ...
  },
 "aggs": {
    ...
  },
  "sort": [
    ...
  ],
  "script_fields": {
    ...
  }
}

When deleting an index, use the syntax shown in the code snippet below.

DELETE index

Summary

We mastered the installation and configuration of the components of the ELK stack together. I showed you the basics of using the console in Kibana’s web interface and explained the syntax of the Elasticsearch queries. A detailed and extensive documentation provides you with Elasticsearch itself, so that you have an always available guide for the realization of your projects.

From our Open Source Database and Database Platform section also interesting and helpful:

In the collection of our tutorials I have shown you how to MySQL, PostgreSQL, MongoDB, Apache Cassandra, Redis, RethinkDB, InfluxDB, Neo4j, ArangoDB and OrientDB under Ubuntu, set it up and get started. I extend the series by introducing you to the NoSQL database platform Elasticsearch, helping you with the installation and configuration and showing you […]

Schade, dass dir der Artikel nicht gefallen hat.
Was sollten wir deiner Meinung nach besser machen?

Thank you for your feedback!
We will get back to you as soon as the article is finished.

Übrigens: kennst du schon unser Tutorial zum Thema Install Rancher v1.3 on Ubuntu 16.04?

×

Developer?

Get the latest gridscale developer tutorials here.
And don’t worry - we won’t spam you