标签：cluster 学会 Flink JobManager Application job 集群 官网
文章目录
前言
从2021年开始的2022年Flink官网翻译计划，希望能给你带来帮助！
部署（Deployment）
Flink is a versatile framework, supporting many different deployment scenarios in a mix and match fashion.
Flink是一个通用的框架，支持多种不同的部署方案。
Below, we briefly explain the building blocks of a Flink cluster, their purpose and available implementations. If you just want to start Flink locally, we recommend setting up a Standalone Cluster.
下面，我们简短的介绍下Flink集群的各个组件、它们的作用和可用的实现。如果你只是想在本地运行Flink，我们建议你搭建一个本地集群。
概览及相关架构（Overview and Reference Architecture）
The figure below shows the building blocks of every Flink cluster. There is always somewhere a client running. It takes the code of the Flink applications, transforms it into a JobGraph and submits it to the JobManager.
该部分重点介绍了Flink集群的必需组件，Client总是运行在集群中，它携带着Flink程序的源代码，在运行过程中，会将源代码转换成作业图（JobGraph），然后将作业图提交给JobManager。
The JobManager distributes the work onto the TaskManagers, where the actual operators (such as sources, transformations and sinks) are running.
JobManager将任务分发给TaskManager，TaskManager负责完成具体的Source、Transformation、Sink操作。
When deploying Flink, there are often multiple options available for each building block. We have listed them in the table below the figure.
部署Flink时，可以选择性的安装各个组件，在下表中，我们展示了它们的作用。
| Component | Purpose | Implementations |
|---|---|---|
| Flink Client | Compiles batch or streaming applications into a dataflow graph, which it then submits to the JobManager. | Command Line Interface REST Endpoint SQL Client Python REPL Scala REPL |
| Flink客户端 | 编译批处理或流式程序，生成数据流图，在提交任务时发送给JobManager | 命令行接口、REST API、SQL客户端、Python命令、Scala命令 |
| JobManager | JobManager is the name of the central work coordination component of Flink. It has implementations for different resource providers, which differ on high-availability, resource allocation behavior and supported job submission modes. JobManager modes for job submissions:Application Mode: runs the cluster exclusively for one application. The job’s main method (or client) gets executed on the JobManager. Calling execute/executeAsync multiple times in an application is supported.Per-Job Mode: runs the cluster exclusively for one job. The job’s main method (or client) runs only prior to the cluster creation.Session Mode: one JobManager instance manages multiple jobs sharing the same cluster of TaskManagers | Standalone (this is the barebone mode that requires just JVMs to be launched. Deployment with Docker, Docker Swarm / Compose, non-native Kubernetes and other models is possible through manual setup in this mode)KubernetesYARN |
| JobManager | JobManager是Flink 中的作业协调组件，它实现了不同与HA模式的资源提交方式、资源分配以及任务提交模式。任务提交模式：Application模式：对每一个应用运行一个专有集群，作业的main方法或client运行在JobManager上，支持在一个应用中多次调用execute或executeAsync命令；Per-Job模式：对每一个应用运行一个专有集群，作业的main方法或client只在集群创建之初；Session模式：一个JobManager实例管理多个作业，这些作业共享同一个Flink集群的TaskManager | |
| TaskManager | TaskManagers are the services actually performing the work of a Flink job. | |
| TaskManager | TaskManager是Flink作业的执行者 | |
| External Components (all optional) | 其他可选的组件 | |
| High Availability Service Provider | Flink’s JobManager can be run in high availability mode which allows Flink to recover from JobManager faults. In order to failover faster, multiple standby JobManagers can be started to act as backups. | ZookeeperKubernetes HA |
| HA高可用服务 | JobManager可以允许在高可用模式下，该模式使Flink在JobManager出现故障时可以快速恢复，即多个standby状态的JobManager可以从backup状态转至act状态 | |
| File Storage and Persistency | For checkpointing (recovery mechanism for streaming jobs) Flink relies on external file storage systems | See FileSystems page. |
| 文件存储和持久化 | Flink的checkpoint（存储streaming任务的状态）依赖额外的文件存储系统 | |
| Resource Provider | Flink can be deployed through different Resource Provider Frameworks, such as Kubernetes or YARN. | See JobManager implementations above. |
| 资源分配 | Flink可以结合第三方资源管理框架部署，如K8s、YARN等 | |
| Metrics Storage | Flink components report internal metrics and Flink jobs can report additional, job specific metrics as well. | See Metrics Reporter page. |
| 指标存储 | Flink组件可以汇报内部指标、额外指标以及作业指定指标 | |
| Application-level data sources and sinks | While application-level data sources and sinks are not technically part of the deployment of Flink cluster components, they should be considered when planning a new Flink production deployment. Colocating frequently used data with Flink can have significant performance benefits | For example:Apache KafkaAmazon S3ElasticSearchApache CassandraSee Connectors page. |
| 应用级数据源和目标 | 虽然应用级的数据源和目标不是Flink集群的内置部分，但在规划Flink的生成环境时应该考虑到它们，使用Flink来管理数据可以获得显著的性能提升 |
部署方式（Deployment Modes）
Flink can execute applications in one of three ways:
in Application Mode,
in a Per-Job Mode,
in Session Mode.
The above modes differ in:
the cluster lifecycle and resource isolation guarantees
whether the application’s main() method is executed on the client or on the cluster.
Flink可以使用如下三种方式运行应用：
- Application模式
- Per-Job模式
- Session模式
上述模式的不同点：
- 集群生命周期和资源隔离确认
- 应用的main方法运行的位置：client上还是集群的JobManager上
Application模式（Application Mode）
In all the other modes, the application’s main() method is executed on the client side. This process includes downloading the application’s dependencies locally, executing the main() to extract a representation of the application that Flink’s runtime can understand (i.e. the JobGraph) and ship the dependencies and the JobGraph(s) to the cluster. This makes the Client a heavy resource consumer as it may need substantial network bandwidth to download dependencies and ship binaries to the cluster, and CPU cycles to execute the main(). This problem can be more pronounced when the Client is shared across users.
在其他模式中，main方法运行在client上。这个进程会下载应用所需的本地依赖，执行main方法将应用转换成一个Flink可以理解的作业图（JobGraph），然后将依赖和作业图发送给集群。这导致Client消耗大量资源，因为它需要大量的网络带宽来下载依赖、发送二进制文件给集群以及需要CPU时间片来运行main方法。当这个Client被多个用户共享时，这个问题会更加明显。
Building on this observation, the Application Mode creates a cluster per submitted application, but this time, the main() method of the application is executed by the JobManager. Creating a cluster per application can be seen as creating a session cluster shared only among the jobs of a particular application, and torn down when the application finishes. With this architecture, the Application Mode provides the same resource isolation and load balancing guarantees as the Per-Job mode, but at the granularity of a whole application.
基于上述情况，Application模式为每个应用创建一个单独的集群，同时应用的main方法运行在JobManager上。创建一个应用专有集群即创建了一个会话集群，这个集群仅被这个应用的作业共享，并且在应用运行结束时被销毁。通过这种架构，Application模式提供了和Per-Job模式一样的资源隔离和负载均衡保证，当然仅限于应用内部。
The Application Mode builds on an assumption that the user jars are already available on the classpath (usrlib folder) of all Flink components that needs access to it (JobManager, TaskManager). In other words, your application comes bundled with the Flink distribution. This allows the application mode to speed up the deployment / recovery process, by not having to distribute the user jars to the Flink components via RPC as the other deployment modes do.
Application模式有个前提：用户的Jar包可以被所有需要访问它的Flink组件访问，如JobManager、TaskManager等。换句话说，你的应用需要和Flink捆绑在一起。这样可以加快部署、恢复的过程，而不像其他模式需要将Jar包通过RPC分发到Flink组件。
The application mode assumes that the user jars are bundled with the Flink distribution.
Executing the main() method on the cluster may have other implications for your code, such as any paths you register in your environment using the registerCachedFile() must be accessible by the JobManager of your application.
Application模式建立在用户Jar和Flink集群绑定在一起的基础上。
在集群中运行main方法可能对你的代码有影响，如使用registerCachedFile()在环境中注册的任何路径都必须可以被JobManager访问。
Compared to the Per-Job mode, the Application Mode allows the submission of applications consisting of multiple jobs. The order of job execution is not affected by the deployment mode but by the call used to launch the job. Using execute(), which is blocking, establishes an order and it will lead to the execution of the “next” job being postponed until “this” job finishes. Using executeAsync(), which is non-blocking, will lead to the “next” job starting before “this” job finishes.
相较于Per-Job模式，Application模式允许提交由多个作业组成的程序。作业执行的顺序不受部署模式的影响，而是由启动作业的调用决定。使用阻塞式的execute建立一个顺序，只有当前作业执行结束才会执行下一个作业；使用非阻塞的executeAsync，在当前作业结束前就会启动下一个作业。
The Application Mode allows for multi-execute() applications but High-Availability is not supported in these cases. High-Availability in Application Mode is only supported for single-execute() applications.
Application模式允许运行多个应用，但是HA不支持这些情况。HA下的Application模式只支持单次提交应用。
Additionally, when any of multiple running jobs in Application Mode (submitted for example using executeAsync()) gets cancelled, all jobs will be stopped and the JobManager will shut down. Regular job completions (by the sources shutting down) are supported.
此外，Application模式中，多个运行时作业（例如使用executeAsync提交）中的任意一个被取消时，所有的作业都会停止，JobManager将会关闭。正常完成的作业（使用源关闭）是支持的。
Per-Job Mode
Aiming at providing better resource isolation guarantees, the Per-Job mode uses the available resource provider framework (e.g. YARN, Kubernetes) to spin up a cluster for each submitted job. This cluster is available to that job only. When the job finishes, the cluster is torn down and any lingering resources (files, etc) are cleared up. This provides better resource isolation, as a misbehaving job can only bring down its own TaskManagers. In addition, it spreads the load of book-keeping across multiple JobManagers, as there is one per job. For these reasons, the Per-Job resource allocation model is the preferred mode by many production reasons.
为了提供更好的资源隔离策略，Per-Job模式使用第三方的资源分配框架（如YARN、K8s）来为每个提交的作业分配集群。这个集群只能被当前作业访问，当作业结束时，集群会被关闭并且所有资源会被清除。这提供了更好的资源隔离，因为一个出错的作业只能关闭它自己的TaskManager。此外，它将作业状态分发到多个JobManager，因为每个作业都有一个。基于这些原因，Per-Job资源隔离模式是生产环境的首选模式。
Session Mode
Session mode assumes an already running cluster and uses the resources of that cluster to execute any submitted application. Applications executed in the same (session) cluster use, and consequently compete for, the same resources. This has the advantage that you do not pay the resource overhead of spinning up a full cluster for every submitted job. But, if one of the jobs misbehaves or brings down a TaskManager, then all jobs running on that TaskManager will be affected by the failure. This, apart from a negative impact on the job that caused the failure, implies a potential massive recovery process with all the restarting jobs accessing the filesystem concurrently and making it unavailable to other services. Additionally, having a single cluster running multiple jobs implies more load for the JobManager, who is responsible for the book-keeping of all the jobs in the cluster.
Session模式假设存在一个运行中的集群，并且所有提交的应用都使用这个集群的资源。
应用执行在同一个集群中，共享集群资源。这样的好处是，你无需为每个提交的作业支付启动一个完整集群的开销。但是，如果一个故障作业出现或者关闭了某个TaskManager，那么运行在这个TaskManager上的所有作业都会收到影响。除了作业失败导致的影响外，所有同时重启的作业同时访问文件系统，会导致其对其他服务不可用。此外，单个集群运行多个作业会让JobManager的负载更大，因为JobManager管理当前作业的所有状态记录。
Summary
In Session Mode, the cluster lifecycle is independent of that of any job running on the cluster and the resources are shared across all jobs. The Per-Job mode pays the price of spinning up a cluster for every submitted job, but this comes with better isolation guarantees as the resources are not shared across jobs. In this case, the lifecycle of the cluster is bound to that of the job. Finally, the Application Mode creates a session cluster per application and executes the application’s main() method on the cluster.
在Session模式中，集群的生命周期取决于各个作业运行的状态，并且所有资源由各个作业共享。
Per-Job模式，虽然每个作业都有启动一个完整集群，这会消耗一定资源，但是换来了更好的资源隔离，各个作业无需共享资源，同时集群的生命周期与各自作业绑定。
Application模式为每个应用创建了一个会话集群，并在集群中运行main函数。
总结
以上就是Flink入门的第一步，有更多问题可以私信我一起交流学习！
标签：cluster,学会,Flink,JobManager,Application,job,集群,官网 来源： https://blog.csdn.net/qq_36369061/article/details/122176799
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。