Setting Up and Running Apache Kafka on Windows – DZone Big Data

note : When you purchase through links on our locate, we may receive an affiliate deputation .

Introduction

In my last article, we covered setting up and using Hadoop. This article is all about configure and starting an Apache Kafka waiter on a Windows OS. This usher will besides provide instructions to set up Java and Apache ZooKeeper .
Apache Kafka is a fast and scalable message queue, adequate to of handling big loads in context of read and write, i.e. IO-related, stuff. You can find more about Kafka on hypertext transfer protocol : //kafka.apache.org/. Apache Kafka requires a running ZooKeeper example, which is used for dependable distribute coordination. Please find more about ZooKeeper on hypertext transfer protocol : //zookeeper.apache.org/ ..
You might besides consider these highly rated Apache Kafka courses :

  • Getting Started with Apache Kafka 5/5 stars from 785 reviews. 10 day free trial on Pluralsight
  • Apache Kafka for Beginners – Learn Kafka by Hands-On 4.3/5.0 stars from 751 reviews on Udemy.

You can get avail from this video for setting up Kafka on Windows .

Author’s GitHub:

I have created a bunch of Spark-Scala utilities at hypertext transfer protocol : //github.com/gopal-tiwari, might be helpful in some other cases .

Downloading the Required Files

  • Download Server JRE according to your OS and CPU architecture from http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
  • Download and install 7-zip from http://www.7-zip.org/download.html
  • Download and extract ZooKeeper using 7-zip from http://zookeeper.apache.org/releases.html
  • Download and extract Kafka using 7-zip from http://kafka.apache.org/downloads.html

For this tutorial, we are assuming that ZooKeeper and Kafka are unzipped in the C : drive, but you can unzip them in any location .
here, we are using full-fledged ZooKeeper and not the one packaged with Kafka because it will be a single-node ZooKeeper example. If you want, you can run Kafka with a packaged ZooKeeper located in a Kafka box inside the \kafka\bin\windows directory .

Installation

A. JDK Setup

1. Start the JRE initiation and hit the “ Change address folder ” checkbox, then click ‘Install. ‘
Image title
2. Change the initiation directory to any path without spaces in the folder diagnose. E.g. C : \Java\jre1.8.0_xx\. ( By default it will be C : \Program Files\Java\jre1.8.0_xx ), then snap ‘Next. ‘
3. nowadays open the system environment variables dialogue by opening Control Panel – > System – > Advanced system settings – > Environment Variables .
4. Hit the New User Variable button in the User variables section, then type JAVA_HOME in Variable name and give your jre path in the variable star value. It should look like the below prototype :
Image title ( Java way and translation may change according to the version of Kafka you are using )
5. now snap OK .
6. research for a Path variable in the “ System Variable ” segment in the “ Environment Variables ” dialogue box you just opened .
7. Edit the path and type “ ; % JAVA_HOME % \bin ” at the end of the textbook already written there, barely like the picture below :
Image title
8. To confirm the Java installation, good candid cmd and type “ java –version. ” You should be able to see the translation of Java you equitable installed .
Image title
If your command prompt reasonably looks like the image above, you are good to go. differently, you need to recheck whether your apparatus version matches the right OS architecture ( x86, x64 ), or if the environment variables path is adjust .

B. ZooKeeper Installation

1. Go to your ZooKeeper config directory. For me its C : \zookeeper-3.4.7\conf
2. Rename charge “ zoo_sample.cfg ” to “ zoo.cfg ”
3. overt zoo.cfg in any text editor program, like Notepad ; I prefer Notepad++ .
4. Find and edit dataDir=/tmp/zookeeper to :\zookeeper-3.4.7\data
5. Add an introduction in the System Environment Variables as we did for Java .
a. Add ZOOKEEPER_HOME = C:\zookeeper-3.4.7 to the System Variables .
b. Edit the System Variable named “ Path ” and add ;%ZOOKEEPER_HOME%\bin;
6. You can change the default Zookeeper port in zoo.cfg file ( Default port 2181 ) .
7. Run ZooKeeper by opening a fresh cmd and type zkserver .
8. You will see the command prompt with some details, like the picture below :

Image title
Congratulations, your ZooKeeper is up and running on port 2181 !

C. Setting Up Kafka

1. Go to your Kafka config directory. For me its C : \kafka_2.11-0.9.0.0\config
2. Edit the file “ server.properties. ”
3. Find and edit the cable log.dirs=/tmp/kafka-logs” to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs .
4. If your ZooKeeper is running on some early car or bunch you can edit “ zookeeper.connect:2181 ” to your customs IP and port. For this show, we are using the lapp machine so there ‘s no indigence to change. besides the Kafka port and broker.id are configurable in this file. Leave other settings as is .
5. Your Kafka will run on default port 9092 and connect to ZooKeeper ’ south default port, 2181 .

D. Running a Kafka Server

significant : Please ensure that your ZooKeeper case is up and running earlier starting a Kafka server .
1. Go to your Kafka installation directory : c : \kafka_2.11-0.9.0.0\
2. Open a command prompt here by pressing Shift + right click and choose the “ Open command window here ” option ) .
3. now type .\bin\windows\kafka-server-start.bat .\config\server.properties and press Enter .

.\bin\windows\kafka-server-start.bat .\config\server.properties

Image title
4. If everything went fine, your dominate prompt will look like this :
Image title
5. now your Kafka Server is improving and running, you can create topics to store messages. besides, we can produce or consume data from Java or Scala code or immediately from the command immediate .

E. Creating Topics

1. now create a topic with the name “ test ” and a replication factor of 1, as we have only one Kafka server running. If you have a bunch with more than one Kafka waiter melt, you can increase the replication-factor consequently, which will increase the data handiness and work like a fault-tolerant system .
2. Open a fresh command prompt in the location C : \kafka_2.11-0.9.0.0\bin\windows .
3. Type the following command and hit Enter :

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Image title

F. Creating a Producer and Consumer to Test Server

1. Open a newly dominate prompt in the localization C : \kafka_2.11-0.9.0.0\bin\windows
2. To start a producer type the follow instruction :

kafka-console-producer.bat --broker-list localhost:9092 --topic test

3. Again open a new command prompt in the same localization as C : \kafka_2.11-0.9.0.0\bin\windows
4. now start a consumer by typing the follow command :

Before kafka adaptation 2.0 ( < 2.0 ) :

kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

After kafka interpretation 2.0 ( > = 2.0 ) :

kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test

5. now you will have two command prompts, like the image below :
Image title
6. now type anything in the manufacturer control prompt and press Enter, and you should be able to see the message in the other consumer instruction prompt.

Image title
7. If you are able to push and see your messages on the consumer side, you are done with Kafka setup .

Some Other Useful Commands

  1. List Topics: kafka-topics.bat --list --zookeeper localhost:2181
  2. Describe Topic: kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]
  3. Read messages from the beginning
    1. Before version < 2.0: kafka-console-consumer.bat –zookeeper localhost:2181 –topic [Topic Name] –from-beginning
    2. After version > 2.0:  kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic [Topic Name] --from-beginn
  4. Delete Topic: kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [topic_to_delete] --zookeeper localhost:2181
source : https://thefartiste.com
Category : Tech

About admin

I am the owner of the website thefartiste.com, my purpose is to bring all the most useful information to users.

Check Also

articlewriting1

Manage participants in a zoom meeting webinar

Call the people who attend the meet as follows Alternate host host Who scheduled the …

Leave a Reply

Your email address will not be published.