Flink on yarn以及实现jobManager 高可用(HA)

on yarn

https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/yarn_setup.html

flink on yarn两种方式

第一种方式：在yarn上启动一个长期运行的flink集群

可以把yarn和hdfs相关配置文件拷贝到flink配置目录下，或者直接指定yarn和hdfs配置文件对应的路径

export HADOOP_CONF_DIR=/root/flink-1.8.2/conf
cd flink-1.8.2/
./bin/yarn-session.sh -jm 1024m -tm 4096m -s 16

-jm：jobmanager的内存，-tm：每个taskmanager的内存，-s：the number of processing slots per Task Manager

日志如下

[root@master01 flink-1.8.2]# ./bin/yarn-session.sh -jm 1024m -tm 4096m -s 16
2019-12-10 10:05:40,010 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, master01.hadoop.xxx.cn
2019-12-10 10:05:40,012 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2019-12-10 10:05:40,012 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-12-10 10:05:40,012 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
2019-12-10 10:05:40,012 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-12-10 10:05:40,012 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2019-12-10 10:05:40,067 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-12-10 10:05:40,399 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to root (auth:SIMPLE)
2019-12-10 10:05:40,459 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at master01.hadoop.xxx.cn/xxx.xx.x.xxx:8032
2019-12-10 10:05:40,634 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=16}
2019-12-10 10:05:40,857 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-12-10 10:05:40,873 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/root/flink-1.8.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-12-10 10:05:42,434 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1570496850779_0463
2019-12-10 10:05:42,457 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1570496850779_0463
2019-12-10 10:05:42,457 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-12-10 10:05:42,458 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-12-10 10:05:46,234 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
2019-12-10 10:05:46,597 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
Flink JobManager is now running on worker03.hadoop.xxx.cn:38055 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://worker03.hadoop.xxx.cn:38055

查看web界面可以直接到yarn界面查看，也可以通过日志中给出的jobmanager界面查看

至此 flink on yarn的集群已经启动，下面提交任务测试，提交任务使用./bin/flink

cd flink-1.8.2/
./bin/flink run ./examples/batch/WordCount.jar

日志如下：

[root@master01 flink-1.8.2]# ./bin/flink run ./examples/batch/WordCount.jar
2019-12-10 11:01:43,553 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-12-10 11:01:43,553 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - Found Yarn properties file under /tmp/.yarn-properties-root.
2019-12-10 11:01:43,785 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 16
2019-12-10 11:01:43,785 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - YARN properties set default parallelism to 16
YARN properties set default parallelism to 16
2019-12-10 11:01:43,812 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at master01.hadoop.xxx.cn/xxx.xx.x.211:8032
2019-12-10 11:01:43,904 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-10 11:01:43,904 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-10 11:01:43,956 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Found application JobManager host name 'worker02.hadoop.xxx.cn' and port '39095' from supplied application id 'application_1570496850779_0467'
Starting execution of program
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
......

问题：在提交flink任务时候，flink是怎么找到对应的集群呢？

看日志高亮部分，查看/tmp/.yarn-properties-root文件内容

[root@master01 flink-1.8.2]# more /tmp/.yarn-properties-root
#Generated YARN properties file
#Tue Dec 10 10:40:29 CST 2019
parallelism=16
dynamicPropertiesString=
applicationID=application_1570496850779_0467

这个applicationID不就是我们提交到yarn上flink集群对应的id嘛。

到flink web ui查看任务记录：

此外，在启动on yarn flink集群时候可以使用-d or --detached实现类似后台运行的形式执行，此方式下，如果想停止集群，使用yarn application -kill <appId>

第二种方式：Run a single Flink job on YARN

上面第一种方式是在yarn上启动一个flink集群，然后提交任务时候向这个集群提交。此外，也可以在yarn上直接执行一个flink任务，有点类似spark-submit的感觉。

[root@master01 flink-1.8.2]# ./bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar

日志：

2019-12-10 11:44:56,912 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at master01.hadoop.xxx.cn/xxx.xx.x.xxx:8032
2019-12-10 11:44:57,004 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-10 11:44:57,004 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-12-10 11:44:57,101 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, numberTaskManagers=1, slotsPerTaskManager=1}
2019-12-10 11:44:57,379 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/root/flink-1.8.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-12-10 11:45:01,058 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1570496850779_0470
2019-12-10 11:45:01,093 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1570496850779_0470
2019-12-10 11:45:01,093 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-12-10 11:45:01,094 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-12-10 11:45:05,621 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
Starting execution of program
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
(a,5)
(action,1)
(after,1)
(against,1)
......

可以看到，第一件事是连接yarn的resourcemanager。

./bin/flink run 命令解析：

run [OPTIONS] <jar-file> <arguments>  
"run" 操作参数:  
-c,--class <classname>  如果没有在jar包中指定入口类，则需要在这里通过这个参数指定  
-m,--jobmanager <host:port>  指定需要连接的jobmanager(主节点)地址，使用这个参数可以指定一个不同于配置文件中的jobmanager  
-p,--parallelism <parallelism>   指定程序的并行度。可以覆盖配置文件中的默认值。
默认查找当前yarn集群中已有的yarn-session信息中的jobmanager【/tmp/.yarn-properties-root】：
./bin/flink run ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
连接指定host和port的jobmanager：
./bin/flink run -m hadoop100:1234 ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
启动一个新的yarn-session：
./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
注意：yarn session命令行的选项也可以使用./bin/flink 工具获得。它们都有一个y或者yarn的前缀
例如：./bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar

Flink on yarn的内部实现

既然是on yarn，那必然需要知道yarn以及hdfs的相关配置，获取相关配置流程如下：

1，先检查有没有设置 YARN_CONF_DIR, HADOOP_CONF_DIR or HADOOP_CONF_PATH环境变量，如果其中之一设置了的话，那就通过此方式读取环境信息。

2，如果第一部分没有设置任何内容，那么客户端会去找HADOOP_HOME环境变量，然后访问$HADOOP_HOME/etc/hadoop路径下的配置文件。

当flink在提交一个任务时，客户端首先会检查资源是否可用（内存和cpu），然后上传flink jar包到hdfs。

然后客户端申请container启动applicationMaster，被选中的nodeManager初始化container，比如下载相关文件，然后启动applicationMaster。

JobManager和AM在同一个container中运行。AM也就知道JobManager的地址。然后为taskManager生成一个新的Flink配置文件(以便它们可以连接到JobManager)。文件也被上传到HDFS。此外，AM container还提供Flink的web接口。（yarn分配的所有端口都是临时端口。并且允许用户并行执行多个Flink任务）

之后，AM开始为Flink的taskManager分配container，后者将从HDFS下载jar包和修改后的配置文件。即可接收job然后执行