by On this article we will discuss how to install Apache Kafka on Ubuntu 20.04 LTS and accessing it via Cluster Manager for Apache Kafka, CMAK.
Introduction
If we are going to build an application where the domain case is related to the event streaming, just try the Apache Kafka.Why ? Because Apache Kafka is event streaming platform which is designed to handle the high-performance data pipelines, streaming analytics applications.
As mentioned in the Apache Kafka official web, Event streaming is the digital equivalent of the human body’s central nervous system. It is the technological foundation for the ‘always-on’ world where businesses are increasingly software-defined and automated, and where the user of software is more software. Event streaming ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time.
Apache Kafka is an open-source distributed event streaming platform written in Scala and Java. It is used for high-performance data pipelines, streaming analytics, data integration, and mission-crititcal applications. Kafka was originally developed by LinkedIn and to be open sourced on early 2011 under Apache Software Foundation. On this article we will discuss how to install Apache Kafka on Ubuntu 20.04 LTS.
Apache Kafka Installation on Ubuntu 20.04 LTS
Before we are going to start the installation, we have to confirm if our system environment is supports for the Apache Kafka installation process. The installation process will consist of several stages, namely:
- Prerequisite
- Download And Extract Apache Kafka Binary Source
- Creating Kafka and Zookeeper Systemd Unit Files
- Installing and Configuring Cluster Manager for Apache Kafka (CMAK)
- Starting And Accessing Kafka services
- Creating Cluster and Topic (An Example)
- Command line
- CMAK Interface
Prerequisite
Before we are going to install Apache Kafka on Ubuntu 20.04 LTS, we have to prepare the environment first.
- Ubuntu 20.04 LTS System
- root or ordinary account with
sudoprivilege - Sufficient disk space
- Java installed (Oracle JDK or OpenJDK). We could verify the Java installed on our system by submitting command line
java --version, as mentioned below.
ramans@otodiginet:~$ java --version openjdk 11.0.8 2020-07-14 OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
If your system is still doesn’t have Java installed, just install Java first. We could found an articles related to Java installation on https://otodiginet.com/software/how-to-install-java-openjdk-11-on-centos-8/ for Open JDK installation or https://otodiginet.com/software/how-to-install-oracle-java-12-in-linux-ubuntu-18-04/ for Oracle Java installation.
Download And Extract Apache Kafka Binary File
After all requirement were fulfilled, then we download and extract binary file of Apach Kafka. For your reference, just check the latest version of Kafka on https://kafka.apache.org/downloads web page. The newest stable version of Apache Kafka when this article made is version 2.6.0 which was released on Auguts 03, 2020. We will download the source and extract it to the /usr/local/kafka-server/ directory.
1. Download Kafka source
ramans@otodiginet:~/Desktop$ cd ~ ramans@otodiginet:~$ wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz --2020-09-20 18:33:53-- https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz Resolving downloads.apache.org (downloads.apache.org)… 88.99.95.219, 2a01:4f8:10a:201a::2 Connecting to downloads.apache.org (downloads.apache.org)|88.99.95.219|:443… connected. HTTP request sent, awaiting response… 200 OK Length: 65537909 (63M) [application/x-gzip] Saving to: ‘kafka_2.13-2.6.0.tgz’ kafka_2.13-2.6.0.tgz 100%[=================================================>] 62.50M 369KB/s in 3m 48s 2020-09-20 18:37:45 (281 KB/s) - ‘kafka_2.13-2.6.0.tgz’ saved [65537909/65537909]
2. Create /usr/local/kafka-server/ directory and extract To It
The /usr/local/kafka-server/ will be used for storing the extracted Apace Kafka binary files.
ramans@otodiginet:~$ sudo mkdir /usr/local/kafka-server && cd /usr/local/kafka-server [sudo] password for ramans: ramans@otodiginet:/usr/local/kafka-server$ sudo tar -xvzf ~/kafka_2.13-2.6.0.tgz --strip 1 kafka_2.13-2.6.0/LICENSE kafka_2.13-2.6.0/NOTICE kafka_2.13-2.6.0/bin/ kafka_2.13-2.6.0/bin/kafka-delete-records.sh kafka_2.13-2.6.0/bin/trogdor.sh kafka_2.13-2.6.0/bin/kafka-preferred-replica-election.sh kafka_2.13-2.6.0/bin/connect-mirror-maker.sh
Creating Kafka and Zookeeper Systemd Unit Files
To enable Kafka and Zookeeper daemons are easy to manage, then we have to make system unit files for it. By creating Kafka and Zookeeper systemd unit file, we could adapt other services are started, stopped, and restarted which is beneficial and consistent. For this purpose we have to create two files: zookeeper.service and kafka.service.
1. /etc/systemd/system/zookeeper.service file
ramans@otodiginet:/usr/local/kafka-server$ sudo vi /etc/systemd/system/zookeeper.service [Unit] Description=Apache Zookeeper Server Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
2. /etc/systemd/system/kafka.service file
Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service After=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
3. Reload Services
After adding these two configuration files, then we should reload the systemd daemon to apply changes and then start the services. We can check the status, whether the services has been running properly. For this purpose we should submit command lines below :
ramans@otodiginet:/usr/local/kafka-server$ sudo systemctl daemon-reload ramans@otodiginet:/usr/local/kafka-server$ sudo systemctl enable --now zookeeper ramans@otodiginet:/usr/local/kafka-server$ sudo systemctl enable --now kafka ramans@otodiginet:/usr/local/kafka-server$ sudo systemctl status kafka zookeeper
Output should be like this :
● kafka.service - Apache Kafka Server Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2020-09-20 19:08:33 PDT; 10s ago Docs: http://kafka.apache.org/documentation.html Main PID: 14694 (java) Tasks: 68 (limit: 4624) Memory: 328.8M CGroup: /system.slice/kafka.service └─14694 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMilli> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,525] INFO [Transaction Marker Channel Man> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,643] INFO [ExpirationReaper-0-AlterAcls]:> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,726] INFO [/config/changes-event-process-> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,770] INFO [SocketServer brokerId=0] Start> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,799] INFO [SocketServer brokerId=0] Start> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,800] INFO [SocketServer brokerId=0] Start> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,827] INFO Kafka version: 2.6.0 (org.apach> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,828] INFO Kafka commitId: 62abe01bee03965> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,828] INFO Kafka startTimeMs: 160065411880> Sep 20 19:08:38 otodiginet kafka-server-start.sh[14694]: [2020-09-20 19:08:38,831] INFO [KafkaServer id=0] started (kaf> ● zookeeper.service - Apache Zookeeper Server Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2020-09-20 19:08:20 PDT; 23s ago Main PID: 14287 (java) Tasks: 37 (limit: 4624) Memory: 73.6M CGroup: /system.slice/zookeeper.service └─14287 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPer> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,737] INFO maxSessionTimeout set to 60> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,738] INFO Created server with tickTim> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,763] INFO Using org.apache.zookeeper.> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,772] INFO Configuring NIO connection > Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,787] INFO binding to port 0.0.0.0/0.0> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,828] INFO zookeeper.snapshotSizeFacto> Sep 20 19:08:22 otodiginet zookeeper-server-start.sh[14287]: [2020-09-20 19:08:22,834] INFO Snapshotting: 0x0 to /tmp/z>
Installing and Configuring Cluster Manager for Apache Kafka (CMAK)
Kafka need a tools for monitoring and managing its services. The developers has built a tools for its purpose called as CMAK (previously known as Kafka Manager) an opensource tool for managing Apache Kafka clusters which was developed by Yahoo. Here, we will use this tools too. So we will install it on our system.
We will clone the Cluster Manager for Apache Kafka (CMAK) files from github, by submitting command line below :
ramans@otodiginet:~$ cd ~ ramans@otodiginet:~$ git clone https://github.com/yahoo/CMAK.git
We will use the Zookeeper host for hosting the CMAK. For this purpose we will configure it on the CMAK configuration file which is located at ~/CMAK/conf/application.conf file. On this tutorial the Zookeeper host is the localhost.
ramans@otodiginet:~$ vi ~/CMAK/conf/application.conf
Settings prefixed with 'kafka-manager.' will be deprecated, use 'cmak.' instead.
https://github.com/yahoo/CMAK/issues/713
kafka-manager.zkhosts="kafka-manager-zookeeper:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="kafka-manager-zookeeper:2181"
cmak.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="localhost:2181"
After updating the file abvoe, then we will deploy the application by submitting command line below.
ramans@otodiginet:~$ cd ~/CMAK/ ramans@otodiginet:~/CMAK$ ./sbt clean dist
It will take time until the process is completed done. The process will be ended until we found the information below.
[info] Compilation completed in 18.304s. model contains 640 documentable templates [info] Main Scala API documentation successful. [success] All package validations passed [info] Your package is ready in /home/ramans/CMAK/target/universal/cmak-3.0.0.5.zip [success] Total time: 363 s (06:03), completed Sep 20, 2020, 8:25:09 PM
The CMAK package has been successfully created on /home/ramans/CMAK/target/universal/cmak-3.0.0.5.zip file. Unzip it, start and access the CMAK via web browser. Here is the command line for unziping the file.
ramans@otodiginet:~/CMAK$ cd ~/CMAK/target/universal ramans@otodiginet:~/CMAK/target/universal$ unzip cmak-3.0.0.5.zip
Starting And Accessing CMAK
1. Starting CMAK services
We will strating the CMAK daemon by submitting command line below.
ramans@otodiginet:~/CMAK/target/universal$ ramans@otodiginet:~/CMAK/target/universal/cmak-3.0.0.5bin/cmak
Until now, our Kafka installation has almost successfully done. We will access the Kafka service through dashboard (web based) which was provided by CMAK. For accessing it, we just hit the URL with our IP_Address or hostname that was configured on the configuration file above. The default port for this services is port 9000. On our tutorial here, we could hit the http://otodiginet:9000.
Creating Cluster and Topic (An Example)
So far, we have through so far. We will try to add new cluster and Topic for our new Kafka. On our scenario we will create a new cluster called as ‘Otodiginet_cluster1‘ and topic called as ‘Ramansah_Topic1‘.
Conclussion
On this article we have tried to install Apache Kafka on Ubuntu 20.04 LTS system. We have finally completed this simple tutorial. Have a nice day, stay at home and stay save.
