October 21, 2021
How to install Elastic Stack on CentOS 8

How To Install Elastic Stack (Elasticsearch, Logstash, Filebeat and Kibana) On CentOS 8

On this article we will discuss how to install Elastic Stack (Elasticsearch, Logstash, Filebeat and Kibana) On CentOS 8. On this tutorial we are using Elasticsearch 7.8.1. Formerly the Elastic Stack was called as ELK Stack.

Introduction

Elasticsearch is an open-source search engine based on Lucene, developed in Java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web-interface (Kibana). The data is queried, retrieved and stored with a JSON document scheme. Elasticsearch is a scalable search engine that can be used to search for all kinds of text documents, including log files. Elasticsearch is the heart of the ‘Elastic Stack’ or ELK Stack.

Logstash is an open-source tool for managing events and logs. It provides real-time pipelining for data collections. Logstash will collect log data, convert the data into JSON documents, and store them in Elasticsearch.

Kibana is an open-source data visualization tool for Elasticsearch. Kibana provides a pretty dashboard web interface. It allows you to manage and visualize data from Elasticsearch. It’s not just beautiful, but also powerful.

In this tutorial, we will show you step-by-step installing and configuring the ‘Elastic Stack’ on the CentOS 8 server. We will install and setup the Elasticsearch, Logstash, and Kibana. And then set up the Beats ‘filebeat’ on clients Ubuntu and CentOS system.



Prerequisites

Before we are going to install Ealstic Stack, we have to prepare our environment for convenient installation process. Here are the prerequisite :

  • A system with CentOS 8 installed with the latest update
  • A user account with sudo or root privileges
  • Java version 8 or 11 installed on system (on our environment there is OpenJDK 11 installed)
  • Sufficient disk for Elastic Stack binnary files

The article will be divided into serveral step, namely :

  1. Install & Configure Elasticsearch
  2. Install & Configure Kibana
  3. Install & Configure Logstash
  4. Install & Configure Filebeat
  5. Testing

Install & Configure Elasticsearch

Elasticsearch provides a multi-tenant, distributed text search engine. Data is displayed, retrieved and stored in JSON format. Elasticsearch is a scalable search engine that can be used to search all types of text documents, including log files. Elasticsearch is the heart of the ‘Elastic Stack’ or ELK Stack.

The Elasticsearch installation has been explained on the other article. For detailed tutorial, please find the link https://otodiginet.com/software/how-to-install-elasticsearch-on-centos-8/.

Starting Elasticsearch services

[ramans@otodiginet ~]$ sudo systemctl restart elasticsearch
[ramans@otodiginet ~]$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-10-14 01:20:39 PDT; 9s ago
Docs: https://www.elastic.co
Main PID: 7937 (java)
Tasks: 98 (limit: 49614)
Memory: 1.3G
CGroup: /system.slice/elasticsearch.service
├─7937 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -D>
└─8170 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Oct 14 01:19:16 otodiginet systemd[1]: Starting Elasticsearch…
Oct 14 01:20:39 otodiginet systemd[1]: Started Elasticsearch.

Install & Configure Kibana

After Elasticsearch installation was completed done, then we’re going to install and configure the Kibana Dashboard on CentOS 8 server. Kibana is an open source data visualization tool for Elasticsearch. Kibana provides a beautiful dashboard web interface. We can use Kibana to manage and visualize data from Elasticsearch. Kibana is also not only beautiful, but also powerful.

1. Kibana installation on CentOS 8

Install the Kibana dashboard using the dnf command below.

[ramans@otodiginet bin]$ sudo dnf install kibana
install Kibana on CentOS 8
Kibana Installation

the output will be :

Downloading Packages:
kibana-7.8.1-x86_64.rpm 239 kB/s | 332 MB 23:44 A
Total 239 kB/s | 332 MB 23:44
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: kibana-7.8.1-1.x86_64 1/1
Installing : kibana-7.8.1-1.x86_64 1/1
Running scriptlet: kibana-7.8.1-1.x86_64 1/1
Verifying : kibana-7.8.1-1.x86_64 1/1
Installed products updated.
Installed:
kibana-7.8.1-1.x86_64

Complete!

2. Configuring Kibana

After Kibana installation was completed done, then we have to configure it by editing the Kibana configuration file which is located on /etc/kibana/kibana.yml file. In this tutorial article we will be using default value for all parameters, only uncomment the parameters that will be applied. On this tutorial we will let the configuration with the default values.

#Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
#Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
#The default is 'localhost', which usually means remote machines will not be able to connect.
#To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

/etc/kibana/kibana.yml
Kibana Configuration File



3. Starting Kibana Services

[ramans@otodiginet ~]$ sudo systemctl enable kibana
Synchronizing state of kibana.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable kibana
[ramans@otodiginet ~]$ sudo systemctl start kibana

Kibana services status :

[ramans@otodiginet ~]$ sudo systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-10-14 00:44:47 PDT; 32min ago
Main PID: 1120 (node)
Tasks: 11 (limit: 49614)
Memory: 529.3M
CGroup: /system.slice/kibana.service
└─1120 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli
Oct 14 01:17:09 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:09Z","tags":["warn>
Oct 14 01:17:09 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:09Z","tags":["warn>
Oct 14 01:17:10 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:10Z","tags":["warn>
Oct 14 01:17:10 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:10Z","tags":["warn>
Oct 14 01:17:12 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:12Z","tags":["warn>
Oct 14 01:17:12 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:12Z","tags":["warn>
Oct 14 01:17:15 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:15Z","tags":["warn>
Oct 14 01:17:15 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:15Z","tags":["warn>
Oct 14 01:17:17 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:17Z","tags":["warn>
Oct 14 01:17:17 otodiginet kibana[1120]: {"type":"log","@timestamp":"2020-10-14T08:17:17Z","tags":["warn>
Kibana services

Install and Configuring Logstash

Another component of Elastic Stack is Logstash. Logstash is an open source tool for managing activity and logs. Logstash provides real-time pipelining data collection.

1. Logstash Installation

[ramans@otodiginet ~]$ sudo dnf install logstash
[sudo] password for ramans:
Last metadata expiration check: 1:40:09 ago on Sat 08 Aug 2020 04:57:22 PM PDT.
Dependencies resolved.
Package Architecture Version Repository Size
Installing:
logstash noarch 1:7.8.1-1 elasticsearch-7.x 152 M
Transaction Summary
Install 1 Package
Total download size: 152 M
Installed size: 276 M
Logstash Installation

The otuput will be :

Successfully created system startup script for Logstash
Verifying : logstash-1:7.8.1-1.noarch 1/1
Installed products updated.
Installed:
logstash-1:7.8.1-1.noarch
Complete!

2. Starting Logstash Services

Once the installation was completed done, then we enable and start Logstash service.

[ramans@otodiginet ~]$ sudo systemctl enable logstash
[ramans@otodiginet ~]$ sudo systemctl start logstash

The Logstash custom configuration files will be located in the /etc/logstash/conf.d/ directory. On this tutorial we let the configuration as it was, no configuration was configured yet. We query the Logstash status.

[ramans@otodiginet ~]$ sudo systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-10-15 01:23:29 PDT; 1min 21s ago
Main PID: 4983 (java)
Tasks: 19 (limit: 49614)
Memory: 592.6M
CGroup: /system.slice/logstash.service
└─4983 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -X>
Oct 15 01:23:29 otodiginet systemd[1]: Started logstash.
Oct 15 01:23:29 otodiginet logstash[4983]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was>
Logstash Services status
Logstash Services Status



Install and Configuring Filebeat

Filebeat is needed for sending logs from each server to the logstash. We will install a filebeat and configure to ship logs from both servers to the Logstash on the elastic server.

[ramans@otodiginet ~]$ sudo yum install filebeat
Filebeat Installation
Filebeat Installation

The output will be as shown below :

Installing : filebeat-7.8.1-1.x86_64 1/1
Running scriptlet: filebeat-7.8.1-1.x86_64 1/1
Verifying : filebeat-7.8.1-1.x86_64 1/1
Installed products updated.
Installed:
filebeat-7.8.1-1.x86_64
Complete!
Filebeat 7.8.1 Installation

After filebeat installation, then we query its status by submitting command line : sudo systemctl status filebeat.

[ramans@otodiginet ~]$ sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-12-02 05:02:10 PST; 57min ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 1943 (filebeat)
Tasks: 16 (limit: 49614)
Memory: 67.5M
CGroup: /system.slice/filebeat.service
└─1943 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/fileb>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.756-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.758-0800 INFO tem>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.758-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.760-0800 INFO [in>
Dec 02 05:59:46 otodiginet filebeat[1943]: 2020-12-02T05:59:46.761-0800 INFO [pu>

For our testing we will add

[ramans@otodiginet ~]$ sudo filebeat modules enable system
[sudo] password for ramans:
Module system is already enabled
[ramans@otodiginet ~]$ sudo filebeat setup
Overwriting ILM policy is disabled. Set setup.ilm.overwrite:true for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ml.html
Loaded machine learning job configurations
Loaded Ingest pipelines
[ramans@otodiginet ~]$ sudo service filebeat start
Starting filebeat (via systemctl): [ OK ]

Testing Kibana Portal

We will test our Elastic stack by launching our Kibana dashboard in a browser by visiting the url : http://localhost:5601/app/kibana.

Kibana Dashboard|Elastic
Kibana 7.8.1 Main Dasboard
Elastic Stack Main Dashboard
Kibana 7.8.1 Main Dasboard
Flight data sample for Elastic Stack
Flight data sample for Elastic Stack

Conclusion

Share this article via :

Leave a Reply

Your email address will not be published. Required fields are marked *