Apache Flume 安装

Flume安装

上传安装包到数据源所在节点上
然后解压 tar -zxvf apache-flume-1.8.0-bin.tar.gz
然后进入flume的目录，修改conf下的flume-env.sh（把.sh.template复制为.sh），
在里面配置JAVA_HOME

tar -zxvf flume-ng-1.6.0-cdh5.14.0.tar.gz -C /export/servers/
cd  /export/servers/apache-flume-1.6.0-cdh5.14.0-bin/conf
cp  flume-env.sh.template flume-env.sh
vim flume-env.sh

export JAVA_HOME=/export/servers/jdk1.8.0_141

或者

export JAVA_HOME=${JAVA_HOME}

Flume接收telnet数据

测试一下程序环境是否正常
1、先在flume的conf目录下新建一个文件

vi   netcat-logger.conf

写入如下内容，保存。

# 定义这个agent中各组件的名字
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 描述和配置source组件：r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 192.168.100.201
a1.sources.r1.port = 44444

# 描述和配置sink组件：k1
a1.sinks.k1.type = logger

# 描述和配置channel组件，此处使用是内存缓存的方式
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 描述和配置source  channel   sink之间的连接关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2、启动agent去采集数据

bin/flume-ng agent -c conf -f conf/a1.conf -n a1  -Dflume.root.logger=INFO,console

-c conf 指定flume自身的配置文件所在目录
-f conf/netcat-logger.con 指定我们所描述的采集方案
-n a1 指定我们这个agent的名字

3、测试

先要往agent采集监听的端口上发送数据，让agent有数据可采。
随便在一个能跟agent节点联网的机器上（另一个节点上）：

telnet anget-hostname  port   （telnet localhost 44444）

如果没有安装talnet就安装它

yum install telnet-server -y
yum install telnet.*  -y

telnet命令还可做别的用途，比如确定远程服务的状态，比如确定远程服务器的某个端口是否能访问

2019-12-05 08:51:29,757 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31 32 0D                                        12. }
2019-12-05 08:51:30,016 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 33 31 32 0D                                     312. }
2019-12-05 08:51:30,197 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 33 0D                                           3. }
2019-12-05 08:51:31,245 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 31 32 33 0D                                     123. }
2019-12-05 08:51:46,249 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 61 68 61 68 61 68 61 0D                      hahahaha. }

然后hadoop01会打印出日志。说明Flume可用，安装成功