Lets get into some demo, which will introduce us to lot of stuff. There will be a tendency to try to know everything but I suggest to hold on as in next blogs we will lot of stuff everything in detail.
Download and setup
Before installing/setting up kafka, please install java 8 on your dev machine. Java 8 from oracle or openJDK, any one is okay! Download the kafka binary tar from here. Scala is needed to run Kafka but since we are downloading tar (tgz), so we don’t need to worry about it as it is part of the package. Extract the tar to get the folder out of it. You can keep this extracted folder where ever you like, you may even chose to leave it in your downloads folder.
Navigate to config folder in your kafka installation directory and make three copies (namely server-1.props and so-on) of server.properties and in each of the files change the listeners property to a new port and also log.dirs property can point to a different folder, if folder dont exist kafka will create one.
//server-1.properties listeners=PLAINTEXT://:9093 .. .. log.dirs=/tmp/kafka-logs-1
Each property file we create here will represent a broker instance. Lets start the cluster now., in following order zookeeper, server 1, 2, 3: Navigate to bin folder in installation directory then
//from inside bin folder -/Users/me/Downloads/kafka_2.12-2.5.0/bin zookeeper-server-start ../config/zookeeper.properties kafka-server-start ../config/server-1.properties kafka-server-start ../config/server-2.properties kafka-server-start ../config/server-3.properties
Take a look at logs of one of the brokers and see the details, all the configs that broker is running with are printed there. Now, we need to create a topic to which we can send messages:
//from inside bin folder kafka-topics --create --topic mytopic --partitions 3 -replication-factor 3 --zookeeper localhost:2181
Don’t worry about command details we will see more details about them soon.
Once topic is create you can run following describe command to check details, as our topic is ready now
(base) svyas bin $ kafka-topics --describe --topic mytopic --zookeeper localhost:2181 Topic:mytopic PartitionCount:3 ReplicationFactor:3 Configs: Topic: mytopic Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: mytopic Partition: 1 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3 Topic: mytopic Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
We are starting a produce and a consumer now again from inside bin folder:
kafka-console-producer --broker-list localhost:9094 --topic mytopic kafka-console-consumer --bootstrap-server localhost:9094 --topic mytopic --from-beginning
From producer command line type messages and see them appear in consumers terminal window
kafka-console-producer --broker-list localhost:9094 --topic mytopic >Hello 1 >Hello 2 >Hello 3 >Hello 4 >Hello 5
kafka-console-consumer --bootstrap-server localhost:9094 --topic mytopic --from-beginning Hello 1 Hello 2 Hello 3 Hello 4 Hello 5
So, this completes our quick setup and intro to kafka blog. We will dive deep into the basics and core of kafka soon. Also, we used lot of terms here which didn’t worry discussing, those too we will explore in next post.