Posts

Showing posts from July, 2018

Apache Kafka and flume installation guide (import data from Kafka to HDFS)

Image
This article contains a complete guide on how to install Apache Kafka, creating Kafka topics, publishing and subscribing Topic messages. In addition, it contains Apache Flume installation guide and how to import Kafka topic messages into HDFS using Apache Flume. 1. Environment ·           Hadoop  Version: 3.1.0 ·           Apache Kafka Version: 1.1.1 ·           Apache Flume Version: 1.8.0 ·           Operating System: Ubuntu 16.04 ·           Java Version: Java 8 2. Prerequisites 2.1. Install Java Apache Kafka requires Java. To ensure that Java is installed first update the Operating System then try to install it: sudo  apt-get update sudo  apt-get upgrade sudo  add-apt-repository -y ppa:webupd8team/java sudo  apt-get install oracle-java8-installer 2.2. Install Zookeeper Apache Kafka requires Zookeeper service to be installed, because it uses it to maintain its nodes heart beats, its configuration and to elect leaders. sudo  apt-get insta