Lets get into some demo, which will introduce us to lot of stuff. There will be a tendency to try to know everything but I suggest to hold on as in next blogs we will lot of stuff everything in detail.

Download and setup

Before installing/setting up kafka, please install java 8 on your dev machine. Java 8 from oracle or openJDK, any one is okay! Download the kafka binary tar from here. Scala is needed to run Kafka but since we are downloading tar (tgz), so we don’t need to worry about it as it is part of the package. Extract the tar to get the folder out of it. You can keep this extracted folder where ever you like, you may even chose to leave it in your downloads folder.

Navigate to config folder in your kafka installation directory and make three copies (namely server-1.props and so-on) of server.properties and in each of the files change the listeners property to a new port and also log.dirs property can point to a different folder, if folder dont exist kafka will create one.

//server-1.properties
listeners=PLAINTEXT://:9093
..
..
log.dirs=/tmp/kafka-logs-1

Each property file we create here will represent a broker instance. Lets start the cluster now., in following order zookeeper, server 1, 2, 3: Navigate to bin folder in installation directory then

//from inside bin folder -/Users/me/Downloads/kafka_2.12-2.5.0/bin
zookeeper-server-start ../config/zookeeper.properties
kafka-server-start ../config/server-1.properties
kafka-server-start ../config/server-2.properties
kafka-server-start ../config/server-3.properties

Take a look at logs of one of the brokers and see the details, all the configs that broker is running with are printed there. Now, we need to create a topic to which we can send messages:

//from inside bin folder
kafka-topics --create --topic mytopic --partitions 3 -replication-factor 3 --zookeeper localhost:2181

Don’t worry about command details we will see more details about them soon.

Once topic is create you can run following describe command to check details, as our topic is ready now

(base) svyas bin $ kafka-topics --describe --topic mytopic --zookeeper localhost:2181
Topic:mytopic	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: mytopic	Partition: 0	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: mytopic	Partition: 1	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: mytopic	Partition: 2	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1

We are starting a produce and a consumer now again from inside bin folder:

kafka-console-producer --broker-list localhost:9094 --topic mytopic

kafka-console-consumer --bootstrap-server localhost:9094 --topic mytopic --from-beginning

From producer command line type messages and see them appear in consumers terminal window

kafka-console-producer --broker-list localhost:9094 --topic mytopic
>Hello 1
>Hello 2
>Hello 3
>Hello 4
>Hello 5

kafka-console-consumer --bootstrap-server localhost:9094 --topic mytopic --from-beginning
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5