-During our research process, we communicated with the main developers of both tools multiple times. After our repeated studies and assessments, we eventually decided to adopt StreamPark as our primary Flink development tool for now. +During our research process, we communicated with the main developers of both tools multiple times. After our repeated studies and assessments, we eventually decided to adopt Apache StreamPark as our primary Flink development tool for now. -
-After extended development and testing by our team, StreamPark currently boasts: +After extended development and testing by our team, Apache StreamPark currently boasts: * Comprehensive SQL validation capabilities * It has achieved automatic build/push for images @@ -143,21 +143,21 @@ This effectively addresses most of the challenges we currently face in developme -
-In its latest release, version 1.2.0, StreamPark provides robust support for both K8s-Native-Application and K8s-Session-Application modes. +In its latest release, version 1.2.0, Apache StreamPark provides robust support for both K8s-Native-Application and K8s-Session-Application modes. -
### K8s Native Application Mode -Within StreamPark, all we need to do is configure the relevant parameters, fill in the corresponding dependencies in the Maven POM, or upload the dependency jar files. Once we click on 'Apply', the specified dependencies will be generated. This implies that we can also compile all the UDFs we use into jar files, as well as various connector.jar files, and use them directly in SQL. As illustrated below: +Within Apache StreamPark, all we need to do is configure the relevant parameters, fill in the corresponding dependencies in the Maven POM, or upload the dependency jar files. Once we click on 'Apply', the specified dependencies will be generated. This implies that we can also compile all the UDFs we use into jar files, as well as various connector.jar files, and use them directly in SQL. As illustrated below:  @@ -169,7 +169,7 @@ We can also specify resources, designate dynamic parameters within Flink Run as  -After saving the program, when clicking to run, we can also specify a savepoint. Once the task is successfully submitted, StreamPark will, based on the FlinkPod's network Exposed Type (be it loadBalancer, NodePort, or ClusterIp), return the corresponding WebURL, seamlessly enabling a WebUI redirect. However, as of now, due to security considerations within our online private K8s cluster, there hasn't been a connection established between the Pod and client node network (and there's currently no plan for this). Hence, we only employ NodePort. If the number of future tasks increases significantly, and there's a need for ClusterIP, we might consider deploying StreamPark in K8s or further integrate it with Ingress. +After saving the program, when clicking to run, we can also specify a savepoint. Once the task is successfully submitted, Apache StreamPark will, based on the FlinkPod's network Exposed Type (be it loadBalancer, NodePort, or ClusterIp), return the corresponding WebURL, seamlessly enabling a WebUI redirect. However, as of now, due to security considerations within our online private K8s cluster, there hasn't been a connection established between the Pod and client node network (and there's currently no plan for this). Hence, we only employ NodePort. If the number of future tasks increases significantly, and there's a need for ClusterIP, we might consider deploying Apache StreamPark in K8s or further integrate it with Ingress.  @@ -185,7 +185,7 @@ Below is the specific submission process in the K8s Application mode: ### K8s Native Session Mode -StreamPark also offers robust support for the K8s Native-Session mode, which lays a solid technical foundation for our subsequent offline FlinkSQL development or for segmenting certain resources. +Apache StreamPark also offers robust support for the K8s Native-Session mode, which lays a solid technical foundation for our subsequent offline FlinkSQL development or for segmenting certain resources. To use the Native-Session mode, one must first use the Flink command to create a Flink cluster that operates within K8s. For instance: @@ -203,7 +203,7 @@ To use the Native-Session mode, one must first use the Flink command to create a  -As shown in the image above, we use that ClusterId as the Kubernetes ClusterId task parameter for StreamPark. Once the task is saved and submitted, it quickly transitions to a 'Running' state: +As shown in the image above, we use that ClusterId as the Kubernetes ClusterId task parameter for Apache StreamPark. Once the task is saved and submitted, it quickly transitions to a 'Running' state:  @@ -211,13 +211,13 @@ Following the application info's WebUI link:  -It becomes evident that StreamPark essentially uploads the jar package to the Flink cluster through REST API and then schedules the task for execution. +It becomes evident that Apache StreamPark essentially uploads the jar package to the Flink cluster through REST API and then schedules the task for execution.
### Custom Code Mode -To our delight, StreamPark also provides support for coding DataStream/FlinkSQL tasks. For special requirements, we can achieve our implementations in Java/Scala. You can compose tasks following the scaffold method recommended by StreamPark or write a standard Flink task. By adopting this approach, we can delegate code management to git, utilizing the platform for automated compilation, packaging, and deployment. Naturally, if functionality can be achieved via SQL, we would prefer not to customize DataStream, thereby minimizing unnecessary operational complexities. +To our delight, Apache StreamPark also provides support for coding DataStream/FlinkSQL tasks. For special requirements, we can achieve our implementations in Java/Scala. You can compose tasks following the scaffold method recommended by Apache StreamPark or write a standard Flink task. By adopting this approach, we can delegate code management to git, utilizing the platform for automated compilation, packaging, and deployment. Naturally, if functionality can be achieved via SQL, we would prefer not to customize DataStream, thereby minimizing unnecessary operational complexities.
@@ -225,26 +225,26 @@ To our delight, StreamPark also provides support for coding DataStream/FlinkSQL ## Suggestions for Improvement -StreamPark, similar to any other new tools, does have areas for further enhancement based on our current evaluations: +Apache StreamPark, similar to any other new tools, does have areas for further enhancement based on our current evaluations: * **Strengthening Resource Management**: Features like multi-file system jar resources and robust task versioning are still awaiting additions. * **Enriching Frontend Features**: For instance, once a task is added, functionalities like copying could be integrated. * **Visualization of Task Submission Logs**: The process of task submission involves loading class files, jar packaging, building and submitting images, and more. A failure at any of these stages could halt the task. However, error logs are not always clear, or due to some anomaly, the exceptions aren't thrown as expected, leaving users puzzled about rectifications. -It's a universal truth that innovations aren't perfect from the outset. Although minor issues exist and there are areas for improvement with StreamPark, its merits outweigh its limitations. As a result, we've chosen StreamPark as our Flink DevOps platform. We're also committed to collaborating with its main developers to refine StreamPark further. We wholeheartedly invite others to use it and contribute towards its advancement. +It's a universal truth that innovations aren't perfect from the outset. Although minor issues exist and there are areas for improvement with Apache StreamPark, its merits outweigh its limitations. As a result, we've chosen Apache StreamPark as our Flink DevOps platform. We're also committed to collaborating with its main developers to refine Apache StreamPark further. We wholeheartedly invite others to use it and contribute towards its advancement.
## Future Prospects * We'll keep our focus on Doris and plan to unify business data with log data in Doris, leveraging Flink to realize lakehouse capabilities. -* Our next step is to explore integrating StreamPark with DolphinScheduler 2.x. This would enhance DolphinScheduler's offline tasks, and gradually we aim to replace Spark with Flink for a unified batch-streaming solution. +* Our next step is to explore integrating Apache StreamPark with DolphinScheduler 2.x. This would enhance DolphinScheduler's offline tasks, and gradually we aim to replace Spark with Flink for a unified batch-streaming solution. * Drawing from our own experiments with S3, after building the fat-jar, we're considering bypassing image building. Instead, we'll mount PVC directly to the Flink Pod's directory using Pod Template, refining the code submission process even further. -* We plan to persistently implement StreamPark in our production environment. Collaborating with community developers, we aim to boost StreamPark's Flink stream development, deployment, and monitoring capabilities. Our collective vision is to evolve StreamPark into a holistic stream data DevOps platform. +* We plan to persistently implement Apache StreamPark in our production environment. Collaborating with community developers, we aim to boost Apache StreamPark's Flink stream development, deployment, and monitoring capabilities. Our collective vision is to evolve Apache StreamPark into a holistic stream data DevOps platform. Resources: -StreamPark GitHub: [https://github.com/apache/incubator-streampark](https://github.com/apache/incubator-streampark)
+Apache StreamPark GitHub: [https://github.com/apache/incubator-streampark](https://github.com/apache/incubator-streampark)
Doris GitHub: [https://github.com/apache/doris](https://github.com/apache/doris)  diff --git a/blog/2-streampark-usercase-chinaunion.md b/blog/2-streampark-usercase-chinaunion.md index 6f4ea6bfd..a7a2121b2 100644 --- a/blog/2-streampark-usercase-chinaunion.md +++ b/blog/2-streampark-usercase-chinaunion.md @@ -1,7 +1,7 @@ --- slug: streampark-usercase-chinaunion title: China Union's Flink Real-Time Computing Platform Ops Practice -tags: [StreamPark, Production Practice, FlinkSQL] +tags: [Apache StreamPark, Production Practice, FlinkSQL] ---  @@ -10,7 +10,7 @@ tags: [StreamPark, Production Practice, FlinkSQL] - Introduction to the Real-Time Computing Platform Background - Operational Challenges of Flink Real-Time Jobs -- Integrated Management Based on StreamPark +- Integrated Management Based on Apache StreamPark - Future Planning and Evolution @@ -68,28 +68,28 @@ In terms of job operation and maintenance dilemmas, firstly, the job deployment Due to various factors in the job operation and maintenance difficulties, business support challenges arise, such as a high rate of failures during launch, impact on data quality, lengthy launch times, high data latency, and issues with missed alarm handling, leading to complaints. In addition, the impact on our business is unclear, and once a problem arises, addressing the issue becomes the top priority. -## **基于 StreamPark 一体化管理** +## **基于 Apache StreamPark 一体化管理**  -In response to the two dilemmas mentioned above, we have resolved many issues through StreamPark's integrated management. First, let's take a look at the dual evolution of StreamPark, which includes Flink Job Management and Flink Job DevOps Platform. In terms of job management, StreamPark supports deploying Flink real-time jobs to different clusters, such as Flink's native Standalone mode, and the Session, Application, and PerJob modes of Flink on Yarn. In the latest version, it will support Kubernetes Native Session mode. The middle layer includes project management, job management, cluster management, team management, variable management, and alarm management. +In response to the two dilemmas mentioned above, we have resolved many issues through Apache StreamPark's integrated management. First, let's take a look at the dual evolution of Apache StreamPark, which includes Flink Job Management and Flink Job DevOps Platform. In terms of job management, Apache StreamPark supports deploying Flink real-time jobs to different clusters, such as Flink's native Standalone mode, and the Session, Application, and PerJob modes of Flink on Yarn. In the latest version, it will support Kubernetes Native Session mode. The middle layer includes project management, job management, cluster management, team management, variable management, and alarm management. - Project Management: When deploying a Flink program, you can fill in the git address in project management and select the branch you want to deploy. -- Job Management: You can specify the execution mode of the Flink job, such as which type of cluster you want to submit to. You can also configure some resources, such as the number of TaskManagers, the memory size of TaskManager/JobManager, parallelism, etc. Additionally, you can set up some fault tolerance measures; for instance, if a Flink job fails, StreamPark can support automatic restarts, and it also supports the input of some dynamic parameters. +- Job Management: You can specify the execution mode of the Flink job, such as which type of cluster you want to submit to. You can also configure some resources, such as the number of TaskManagers, the memory size of TaskManager/JobManager, parallelism, etc. Additionally, you can set up some fault tolerance measures; for instance, if a Flink job fails, Apache StreamPark can support automatic restarts, and it also supports the input of some dynamic parameters. - Cluster Management: You can add and manage big data clusters through the interface. - Team Management: In the actual production process of an enterprise, there are multiple teams, and these teams are isolated from each other. - Variable Management: You can maintain some variables in one place. For example, you can define Kafka's Broker address as a variable. When configuring Flink jobs or SQL, you can replace the Broker's IP with a variable. Moreover, if this Kafka needs to be decommissioned later, you can also use this variable to check which jobs are using this cluster, facilitating some subsequent processes. - Alarm Management: Supports multiple alarm modes, such as WeChat, DingTalk, SMS, and email. -StreamPark supports the submission of Flink SQL and Flink Jar, allows for resource configuration, and supports state tracking, indicating whether the state is running, failed, etc. Additionally, it provides a metrics dashboard and supports the viewing of various logs. +Apache StreamPark supports the submission of Flink SQL and Flink Jar, allows for resource configuration, and supports state tracking, indicating whether the state is running, failed, etc. Additionally, it provides a metrics dashboard and supports the viewing of various logs.  The Flink Job DevOps platform primarily consists of the following parts: -- Teams: StreamPark supports multiple teams, each with its team administrator who has all permissions. There are also team developers who only have a limited set of permissions. +- Teams: Apache StreamPark supports multiple teams, each with its team administrator who has all permissions. There are also team developers who only have a limited set of permissions. - Compilation and Packaging: When creating a Flink project, you can configure the git address, branch, and packaging commands in the project, and then compile and package with a single click of the build button. - Release and Deployment: During release and deployment, a Flink job is created. Within the Flink job, you can choose the execution mode, deployment cluster, resource settings, fault tolerance settings, and fill in variables. Finally, the Flink job can be started or stopped with a single click. - State Monitoring: After the Flink job is started, real-time tracking of its state begins, including Flink's running status, runtime duration, Checkpoint information, etc. There is also support for one-click redirection to Flink's Web UI. @@ -97,7 +97,7 @@ The Flink Job DevOps platform primarily consists of the following parts:  -Companies generally have multiple teams working on real-time jobs simultaneously. In our company, this includes a real-time data collection team, a data processing team, and a real-time marketing team. StreamPark supports resource isolation for multiple teams. +Companies generally have multiple teams working on real-time jobs simultaneously. In our company, this includes a real-time data collection team, a data processing team, and a real-time marketing team. Apache StreamPark supports resource isolation for multiple teams.  @@ -110,7 +110,7 @@ Management of the Flink job platform faces the following challenges: -Based on the challenges mentioned above, StreamPark has addressed the issues of unclear ownership and untraceable branches through project management. This is because when creating a project, you need to manually specify certain branches. Once the packaging is successful, these branches are recorded. Job management centralizes configurations, preventing scripts from being too dispersed. Additionally, there is strict control over the permissions for starting and stopping jobs, preventing an uncontrollable state due to script permissions. StreamPark interacts with clusters through interfaces to obtain job information, allowing for more precise job control. +Based on the challenges mentioned above, Apache StreamPark has addressed the issues of unclear ownership and untraceable branches through project management. This is because when creating a project, you need to manually specify certain branches. Once the packaging is successful, these branches are recorded. Job management centralizes configurations, preventing scripts from being too dispersed. Additionally, there is strict control over the permissions for starting and stopping jobs, preventing an uncontrollable state due to script permissions. Apache StreamPark interacts with clusters through interfaces to obtain job information, allowing for more precise job control. @@ -118,15 +118,15 @@ Referring to the image above, you can see at the bottom of the diagram that pack  -In the early stages, we needed to go through seven steps for deployment, including connecting to a VPN, logging in through 4A, executing compile scripts, executing start scripts, opening Yarn, searching for the job name, and entering the Flink UI. StreamPark supports one-click deployment for four of these steps, including one-click packaging, one-click release, one-click start, and one-click access to the Flink UI. +In the early stages, we needed to go through seven steps for deployment, including connecting to a VPN, logging in through 4A, executing compile scripts, executing start scripts, opening Yarn, searching for the job name, and entering the Flink UI. Apache StreamPark supports one-click deployment for four of these steps, including one-click packaging, one-click release, one-click start, and one-click access to the Flink UI.  -The image above illustrates the job submission process of our StreamPark platform. Firstly, StreamPark proceeds to release the job, during which some resources are uploaded. Following that, the job is submitted, accompanied by various configured parameters, and it is published to the cluster using the Flink Submit method via an API call. At this point, there are multiple Flink Submit instances corresponding to different execution modes, such as Yarn Session, Yarn Application, Kubernetes Session, Kubernetes Application, and so on; all of these are controlled here. After submitting the job, if it is a Flink on Yarn job, the platform will acquire the Application ID or Job ID of the Flink job. This ID is then stored in our database. Similarly, if the job is executed based on Kubernetes, a Job ID will be obtained. Subsequently, when tracking the job status, we primarily use these stored IDs to monitor the state of the job. +The image above illustrates the job submission process of our Apache StreamPark platform. Firstly, Apache StreamPark proceeds to release the job, during which some resources are uploaded. Following that, the job is submitted, accompanied by various configured parameters, and it is published to the cluster using the Flink Submit method via an API call. At this point, there are multiple Flink Submit instances corresponding to different execution modes, such as Yarn Session, Yarn Application, Kubernetes Session, Kubernetes Application, and so on; all of these are controlled here. After submitting the job, if it is a Flink on Yarn job, the platform will acquire the Application ID or Job ID of the Flink job. This ID is then stored in our database. Similarly, if the job is executed based on Kubernetes, a Job ID will be obtained. Subsequently, when tracking the job status, we primarily use these stored IDs to monitor the state of the job.  -As mentioned above, in the case of Flink on Yarn jobs, two IDs are acquired upon job submission: the Application ID and the Job ID. These IDs are used to retrieve the job status. However, when there is a large number of Flink jobs, certain issues may arise. StreamPark utilizes a status retriever that periodically sends requests to the ResourceManager every five seconds, using the Application ID or Job ID stored in our database. If there are a considerable number of jobs, during each polling cycle, the ResourceManager is responsible for calling the Job Manager's address to access its status. This can lead to significant pressure on the number of connections to the ResourceManager and an overall increase in the number of connections. +As mentioned above, in the case of Flink on Yarn jobs, two IDs are acquired upon job submission: the Application ID and the Job ID. These IDs are used to retrieve the job status. However, when there is a large number of Flink jobs, certain issues may arise. Apache StreamPark utilizes a status retriever that periodically sends requests to the ResourceManager every five seconds, using the Application ID or Job ID stored in our database. If there are a considerable number of jobs, during each polling cycle, the ResourceManager is responsible for calling the Job Manager's address to access its status. This can lead to significant pressure on the number of connections to the ResourceManager and an overall increase in the number of connections. @@ -134,38 +134,38 @@ In the diagram mentioned earlier, the connection count to the ResourceManager sh  -To address the issues mentioned above, we have made some optimizations in StreamPark. Firstly, after submitting a job, StreamPark saves the Application ID or Job ID, and it also retrieves and stores the direct access address of the Job Manager in the database. Therefore, instead of polling the ResourceManager for job status, it can directly call the addresses of individual Job Managers to obtain the real-time status. This significantly reduces the number of connections to the ResourceManager. As can be seen from the latter part of the diagram above, there are basically no significant spikes in connection counts, which substantially alleviates the pressure on the ResourceManager. Moreover, this ensures that as the number of Flink jobs continues to grow, the system will not encounter bottlenecks in status retrieval. +To address the issues mentioned above, we have made some optimizations in Apache StreamPark. Firstly, after submitting a job, Apache StreamPark saves the Application ID or Job ID, and it also retrieves and stores the direct access address of the Job Manager in the database. Therefore, instead of polling the ResourceManager for job status, it can directly call the addresses of individual Job Managers to obtain the real-time status. This significantly reduces the number of connections to the ResourceManager. As can be seen from the latter part of the diagram above, there are basically no significant spikes in connection counts, which substantially alleviates the pressure on the ResourceManager. Moreover, this ensures that as the number of Flink jobs continues to grow, the system will not encounter bottlenecks in status retrieval.  -Another issue that StreamPark resolves is safeguarding Flink's state recovery. In the past, when we used scripts for operations and maintenance, especially during business upgrades, it was necessary to recover from the latest checkpoint when starting Flink. However, developers often forgot to recover from the previous checkpoint, leading to significant data quality issues and complaints. StreamPark's process is designed to mitigate this issue. Upon the initial start of a Flink job, it polls every five seconds to retrieve checkpoint records, saving them in a database. When manually stopping a Flink job through StreamPark, users have the option to perform a savepoint. If this option is selected, the path of the savepoint is saved in the database. In addition, records of each checkpoint are also stored in the database. When restarting a Flink job, the system defaults to using the latest checkpoint or savepoint record. This effectively prevents issues associated with failing to recover from the previous checkpoint. It also avoids the resource wastage caused by having to rerun jobs with offset rollbacks to address problems, while ensuring consistency in data processing. +Another issue that Apache StreamPark resolves is safeguarding Flink's state recovery. In the past, when we used scripts for operations and maintenance, especially during business upgrades, it was necessary to recover from the latest checkpoint when starting Flink. However, developers often forgot to recover from the previous checkpoint, leading to significant data quality issues and complaints. Apache StreamPark's process is designed to mitigate this issue. Upon the initial start of a Flink job, it polls every five seconds to retrieve checkpoint records, saving them in a database. When manually stopping a Flink job through Apache StreamPark, users have the option to perform a savepoint. If this option is selected, the path of the savepoint is saved in the database. In addition, records of each checkpoint are also stored in the database. When restarting a Flink job, the system defaults to using the latest checkpoint or savepoint record. This effectively prevents issues associated with failing to recover from the previous checkpoint. It also avoids the resource wastage caused by having to rerun jobs with offset rollbacks to address problems, while ensuring consistency in data processing.  -StreamPark also addresses the challenges associated with referencing multiple components across various environments. In a corporate setting, there are typically multiple environments, such as development, testing, and production. Each environment generally includes multiple components, such as Kafka, HBase, Redis, etc. Additionally, within a single environment, there may be multiple instances of the same component. For example, in a real-time computing platform at China Union, when consuming data from an upstream Kafka cluster and writing the relevant data to a downstream Kafka cluster, two sets of Kafka are involved within the same environment. It can be challenging to determine the specific environment and component based solely on IP addresses. To address this, we define the IP addresses of all components as variables. For instance, the Kafka cluster variable, Kafka.cluster, exists in development, testing, and production environments, but it points to different Broker addresses in each. Thus, regardless of the environment in which a Flink job is configured, referencing this variable is sufficient. This approach significantly reduces the incidence of operational failures in production environments. +Apache StreamPark also addresses the challenges associated with referencing multiple components across various environments. In a corporate setting, there are typically multiple environments, such as development, testing, and production. Each environment generally includes multiple components, such as Kafka, HBase, Redis, etc. Additionally, within a single environment, there may be multiple instances of the same component. For example, in a real-time computing platform at China Union, when consuming data from an upstream Kafka cluster and writing the relevant data to a downstream Kafka cluster, two sets of Kafka are involved within the same environment. It can be challenging to determine the specific environment and component based solely on IP addresses. To address this, we define the IP addresses of all components as variables. For instance, the Kafka cluster variable, Kafka.cluster, exists in development, testing, and production environments, but it points to different Broker addresses in each. Thus, regardless of the environment in which a Flink job is configured, referencing this variable is sufficient. This approach significantly reduces the incidence of operational failures in production environments.  -StreamPark supports multiple execution modes for Flink, including three deployment modes based on Yarn: Application, Perjob, and Session. Additionally, it supports two deployment modes for Kubernetes: Application and Session, as well as some Remote modes. +Apache StreamPark supports multiple execution modes for Flink, including three deployment modes based on Yarn: Application, Perjob, and Session. Additionally, it supports two deployment modes for Kubernetes: Application and Session, as well as some Remote modes.  -StreamPark also supports multiple versions of Flink. For example, while our current version is 1.14.x, we would like to experiment with the new 1.16.x release. However, it’s not feasible to upgrade all existing jobs to 1.16.x. Instead, we can opt to upgrade only the new jobs to 1.16.x, allowing us to leverage the benefits of the new version while maintaining compatibility with the older version. +Apache StreamPark also supports multiple versions of Flink. For example, while our current version is 1.14.x, we would like to experiment with the new 1.16.x release. However, it’s not feasible to upgrade all existing jobs to 1.16.x. Instead, we can opt to upgrade only the new jobs to 1.16.x, allowing us to leverage the benefits of the new version while maintaining compatibility with the older version. ## **Future Planning and Evolution**  -In the future, we will increase our involvement in the development of StreamPark, and we have planned the following directions for enhancement: -- High Availability: StreamPark currently does not support high availability, and this aspect needs further strengthening. +In the future, we will increase our involvement in the development of Apache StreamPark, and we have planned the following directions for enhancement: +- High Availability: Apache StreamPark currently does not support high availability, and this aspect needs further strengthening. - State Management: In enterprise practices, each operator in a Flink job has a UID. If the Flink UID is not set, it could lead to situations where state recovery is not possible when upgrading the Flink job. This issue cannot be solved through the platform at the moment. Therefore, we plan to add this functionality to the platform. We will introduce a feature that checks whether the operator has a UID set when submitting a Flink Jar. If not, a reminder will be issued to avoid state recovery issues every time a Flink job is deployed. Previously, when facing such situations, we had to use the state processing API to deserialize from the original state, and then create a new state using the state processing API for the upgraded Flink to load. -- More Detailed Monitoring: Currently, StreamPark supports sending alerts when a Flink job fails. We hope to also send alerts when a Task fails, and need to know the reason for the failure. In addition, enhancements are needed in job backpressure monitoring alerts, Checkpoint timeout alerts, failure alerts, and performance metric collection. +- More Detailed Monitoring: Currently, Apache StreamPark supports sending alerts when a Flink job fails. We hope to also send alerts when a Task fails, and need to know the reason for the failure. In addition, enhancements are needed in job backpressure monitoring alerts, Checkpoint timeout alerts, failure alerts, and performance metric collection. - Stream-Batch Integration: Explore a platform that integrates both streaming and batch processing, combining the Flink stream-batch unified engine with data lake storage that supports stream-batch unification.  -The above diagram represents the Roadmap for StreamPark. -- Data Source: StreamPark will support rapid integration with more data sources, achieving one-click data onboarding. +The above diagram represents the Roadmap for Apache StreamPark. +- Data Source: Apache StreamPark will support rapid integration with more data sources, achieving one-click data onboarding. - Operation Center: Acquire more Flink Metrics to further enhance the capabilities in monitoring and operation. - K8S-operator: The existing Flink on K8S is somewhat cumbersome, having gone through the processes of packaging Jars, building images, and pushing images. There is a need for future improvements and optimization, and we are actively embracing the upstream K8S-operator integration. - Streaming Data Warehouse: Enhance support for Flink SQL job capabilities, simplify the submission of Flink SQL jobs, and plan to integrate with Flink SQL Gateway. Enhance capabilities in the SQL data warehouse domain, including metadata storage, unified table creation syntax validation, runtime testing, and interactive queries, while actively embracing Flink upstream to explore real-time data warehouses and streaming data warehouses. diff --git a/blog/3-streampark-usercase-bondex-paimon.md b/blog/3-streampark-usercase-bondex-paimon.md index d3b645350..2c8d8abd6 100644 --- a/blog/3-streampark-usercase-bondex-paimon.md +++ b/blog/3-streampark-usercase-bondex-paimon.md @@ -1,12 +1,12 @@ --- slug: streampark-usercase-bondex-with-paimon -title: Based on Apache Paimon + StreamPark's Streaming Data Warehouse Practice by Bondex -tags: [StreamPark, Production Practice, paimon, streaming-warehouse] +title: Based on Apache Paimon + Apache StreamPark's Streaming Data Warehouse Practice by Bondex +tags: [Apache StreamPark, Production Practice, paimon, streaming-warehouse] ---  -**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. +**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + Apache StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. - Introduction to Company Business - Pain Points and Selection in Big Data Technology @@ -102,11 +102,11 @@ Continuing the characteristics of the Kappa architecture with a single stream pr ## 03 Production Practices -This solution adopts Flink Application on K8s clusters, with Flink CDC for real-time ingestion of relational database data from business systems. Tasks for Flink + Paimon Streaming Data Warehouse are submitted through the StreamPark task platform, with the Trino engine ultimately used to access Finereport for service provision and developer queries. Paimon's underlying storage supports the S3 protocol, and as the company's big data services rely on Alibaba Cloud, Object Storage Service (OSS) is used as the data filesystem. +This solution adopts Flink Application on K8s clusters, with Flink CDC for real-time ingestion of relational database data from business systems. Tasks for Flink + Paimon Streaming Data Warehouse are submitted through the Apache StreamPark task platform, with the Trino engine ultimately used to access Finereport for service provision and developer queries. Paimon's underlying storage supports the S3 protocol, and as the company's big data services rely on Alibaba Cloud, Object Storage Service (OSS) is used as the data filesystem. -[StreamPark](https://github.com/apache/incubator-streampark) is a real-time computing platform that leverages the powerful capabilities of [Paimon](https://github.com/apache/incubator-paimon) to process real-time data streams. This platform offers the following key features: +[Apache StreamPark](https://github.com/apache/incubator-streampark) is a real-time computing platform that leverages the powerful capabilities of [Paimon](https://github.com/apache/incubator-paimon) to process real-time data streams. This platform offers the following key features: -**Real-time Data Processing: **StreamPark supports the submission of real-time data stream tasks, capable of real-time acquisition, transformation, filtering, and analysis of data. This is extremely important for applications that require rapid response to real-time data, such as real-time monitoring, real-time recommendations, and real-time risk control. +**Real-time Data Processing: **Apache StreamPark supports the submission of real-time data stream tasks, capable of real-time acquisition, transformation, filtering, and analysis of data. This is extremely important for applications that require rapid response to real-time data, such as real-time monitoring, real-time recommendations, and real-time risk control. **Scalability: **Capable of efficiently processing large-scale real-time data with good scalability. It can operate in a distributed computing environment, automatically handling parallelization, fault recovery, and load balancing to ensure efficient and reliable data processing. @@ -114,7 +114,7 @@ This solution adopts Flink Application on K8s clusters, with Flink CDC for real- **Ease of Use: **Provides a straightforward graphical interface and simplified API, enabling easy construction and deployment of data processing tasks without needing to delve into underlying technical details. -By submitting Paimon tasks on the StreamPark platform, we can establish a full-link real-time flowing, queryable, and layered reusable Pipline. +By submitting Paimon tasks on the Apache StreamPark platform, we can establish a full-link real-time flowing, queryable, and layered reusable Pipline.  @@ -127,7 +127,7 @@ The main components versions used are as follows: ### **Environment Setup** -Download flink-1.16.0-scala-2.12.tar.gz which can be obtained from the official Flink website. Download the corresponding version of the package to the StreamPark deployment server. +Download flink-1.16.0-scala-2.12.tar.gz which can be obtained from the official Flink website. Download the corresponding version of the package to the Apache StreamPark deployment server. ```shell # Unzip @@ -182,7 +182,7 @@ export PATH=$PATH:$FLINK_HOME/bin source /etc/profile ``` -In StreamPark, add Flink conf: +In Apache StreamPark, add Flink conf:  @@ -236,7 +236,7 @@ docker push registry-vpc.cn-zhangjiakou.aliyuncs.com/xxxxx/flink-table-store:v1. Next, prepare the Paimon jar package. You can download the corresponding version from the Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon). It's important to note that it should be consistent with the major version of Flink. -### **Managing Jobs with StreamPark** +### **Managing Jobs with Apache StreamPark** **Prerequisites:** @@ -247,7 +247,7 @@ Next, prepare the Paimon jar package. You can download the corresponding version **Kubernetes Client Connection Configuration:** -Copy the k8s master node's `~/.kube/config` configuration directly to the directory on the StreamPark server, then execute the following command on the StreamPark server to display the k8s cluster as running, which indicates successful permission and network verification. +Copy the k8s master node's `~/.kube/config` configuration directly to the directory on the Apache StreamPark server, then execute the following command on the Apache StreamPark server to display the k8s cluster as running, which indicates successful permission and network verification. ```shell kubectl cluster-info @@ -272,11 +272,11 @@ kubectl create secret docker-registry streamparksecret In this case, Alibaba Cloud's Container Registry Service (ACR) is used, but you can also substitute it with a self-hosted image service such as Harbor. -Create a namespace named StreamPark (set the security setting to private). +Create a namespace named Apache StreamPark (set the security setting to private).  -Configure the image repository in StreamPark; task build images will be pushed to the repository. +Configure the image repository in Apache StreamPark; task build images will be pushed to the repository.  @@ -935,4 +935,4 @@ In complex real-time tasks, resources can be increased by modifying dynamic para - Subsequently, we will integrate with Trino Catalog to access Doris, realizing a one-service solution for both offline and real-time data. - We will continue to advance the pace of building an integrated streaming and batch data warehouse within the group, adopting the architecture of Doris + Paimon. -Here, I would like to thank Teacher Zhixin and the StreamPark community for their strong support during the use of StreamPark + Paimon. The problems encountered in the learning process are promptly clarified and resolved. We will also actively participate in community exchanges and contributions in the future, enabling Paimon to provide more developers and enterprises with integrated stream and batch data lake solutions. +Here, I would like to thank Teacher Zhixin and the Apache StreamPark community for their strong support during the use of Apache StreamPark + Paimon. The problems encountered in the learning process are promptly clarified and resolved. We will also actively participate in community exchanges and contributions in the future, enabling Paimon to provide more developers and enterprises with integrated stream and batch data lake solutions. diff --git a/blog/4-streampark-usercase-shunwang.md b/blog/4-streampark-usercase-shunwang.md index 33d621618..fc4ef8ce6 100644 --- a/blog/4-streampark-usercase-shunwang.md +++ b/blog/4-streampark-usercase-shunwang.md @@ -1,17 +1,17 @@ --- slug: streampark-usercase-shunwang -title: StreamPark in the Large-Scale Production Practice at Shunwang Technology -tags: [StreamPark, Production Practice, FlinkSQL] +title: Apache StreamPark in the Large-Scale Production Practice at Shunwang Technology +tags: [Apache StreamPark, Production Practice, FlinkSQL] ---  -**Preface:** This article primarily discusses the challenges encountered by Shunwang Technology in using the Flink computation engine, and how StreamPark is leveraged as a real-time data platform to address these challenges, thus supporting the company's business on a large scale. +**Preface:** This article primarily discusses the challenges encountered by Shunwang Technology in using the Flink computation engine, and how Apache StreamPark is leveraged as a real-time data platform to address these challenges, thus supporting the company's business on a large scale. - Introduction to the company's business - Challenges encountered -- Why choose StreamPark +- Why choose Apache StreamPark - Implementation in practice - Benefits Brought - Future planning @@ -43,7 +43,7 @@ Flink, as one of the most popular real-time computing frameworks currently, boas Facing a series of pain points in the management and operation of Flink jobs, we have been looking for suitable solutions to lower the barrier to entry for our developers using Flink and improve work efficiency. -Before we encountered StreamPark, we researched some companies' Flink management solutions and found that they all developed and managed Flink jobs through self-developed real-time job platforms. Consequently, we decided to develop our own real-time computing management platform to meet the basic needs of our developers for Flink job management and operation. Our platform is called Streaming-Launcher, with the following main functions: +Before we encountered Apache StreamPark, we researched some companies' Flink management solutions and found that they all developed and managed Flink jobs through self-developed real-time job platforms. Consequently, we decided to develop our own real-time computing management platform to meet the basic needs of our developers for Flink job management and operation. Our platform is called Streaming-Launcher, with the following main functions:  @@ -71,19 +71,19 @@ To view logs for a job, developers must go through multiple steps, which to some  -## **Why Use StreamPark** +## **Why Use Apache StreamPark** Faced with the defects of our self-developed platform Streaming-Launcher, we have been considering how to further lower the barriers to using Flink and improve work efficiency. Considering the cost of human resources and time, we decided to seek help from the open-source community and look for an appropriate open-source project to manage and maintain our Flink tasks. -### 01 **StreamPark: A Powerful Tool for Solving Flink Issues** +### 01 **Apache StreamPark: A Powerful Tool for Solving Flink Issues** -Fortunately, in early June 2022, we stumbled upon StreamPark on GitHub and embarked on a preliminary exploration full of hope. We found that StreamPark's capabilities can be broadly divided into three areas: user permission management, job operation and maintenance management, and development scaffolding. +Fortunately, in early June 2022, we stumbled upon Apache StreamPark on GitHub and embarked on a preliminary exploration full of hope. We found that Apache StreamPark's capabilities can be broadly divided into three areas: user permission management, job operation and maintenance management, and development scaffolding. **User Permission Management** -In the StreamPark platform, to avoid the risk of users having too much authority and making unnecessary misoperations that affect job stability and the accuracy of environmental configurations, some user permission management functions are provided. This is very necessary for enterprise-level users. +In the Apache StreamPark platform, to avoid the risk of users having too much authority and making unnecessary misoperations that affect job stability and the accuracy of environmental configurations, some user permission management functions are provided. This is very necessary for enterprise-level users. @@ -93,13 +93,13 @@ In the StreamPark platform, to avoid the risk of users having too much authority **Job Operation and Maintenance Management** -**Our main focus during the research on StreamPark was its capability to manage the entire lifecycle of jobs:** from development and deployment to management and problem diagnosis. **Fortunately, StreamPark excels in this aspect, relieving developers from the pain points associated with Flink job management and operation.** Within StreamPark’s job operation and maintenance management, there are three main modules: basic job management functions, Jar job management, and FlinkSQL job management as shown below: +**Our main focus during the research on Apache StreamPark was its capability to manage the entire lifecycle of jobs:** from development and deployment to management and problem diagnosis. **Fortunately, Apache StreamPark excels in this aspect, relieving developers from the pain points associated with Flink job management and operation.** Within Apache StreamPark’s job operation and maintenance management, there are three main modules: basic job management functions, Jar job management, and FlinkSQL job management as shown below:  **Development Scaffolding** -Further research revealed that StreamPark is not just a platform but also includes a development scaffold for Flink jobs. It offers a better solution for code-written Flink jobs, standardizing program configuration, providing a simplified programming model, and a suite of Connectors that lower the barrier to entry for DataStream development. +Further research revealed that Apache StreamPark is not just a platform but also includes a development scaffold for Flink jobs. It offers a better solution for code-written Flink jobs, standardizing program configuration, providing a simplified programming model, and a suite of Connectors that lower the barrier to entry for DataStream development. @@ -109,9 +109,9 @@ Further research revealed that StreamPark is not just a platform but also includ -### 02 **How StreamPark Addresses Issues of the Self-Developed Platform** +### 02 **How Apache StreamPark Addresses Issues of the Self-Developed Platform** -We briefly introduced the core capabilities of StreamPark above. During the technology selection process at Shunwang Technology, we found that StreamPark not only includes the basic functions of our existing Streaming-Launcher but also offers a more complete set of solutions to address its many shortcomings. Here, we focus on the solutions provided by StreamPark for the deficiencies of our self-developed platform, Streaming-Launcher. +We briefly introduced the core capabilities of Apache StreamPark above. During the technology selection process at Shunwang Technology, we found that Apache StreamPark not only includes the basic functions of our existing Streaming-Launcher but also offers a more complete set of solutions to address its many shortcomings. Here, we focus on the solutions provided by Apache StreamPark for the deficiencies of our self-developed platform, Streaming-Launcher. @@ -121,21 +121,21 @@ We briefly introduced the core capabilities of StreamPark above. During the tech **One-Stop Flink Job Development Capability** -To lower the barriers to Flink job development and improve developers' work efficiency, **StreamPark provides features like FlinkSQL IDE, parameter management, task management, code management, one-click compilation, and one-click job deployment and undeployment**. Our research found that these integrated features of StreamPark could further enhance developers’ work efficiency. To some extent, developers no longer need to concern themselves with the difficulties of Flink job management and operation and can focus on developing the business logic. These features also solve the pain point of cumbersome SQL development processes in Streaming-Launcher. +To lower the barriers to Flink job development and improve developers' work efficiency, **Apache StreamPark provides features like FlinkSQL IDE, parameter management, task management, code management, one-click compilation, and one-click job deployment and undeployment**. Our research found that these integrated features of Apache StreamPark could further enhance developers’ work efficiency. To some extent, developers no longer need to concern themselves with the difficulties of Flink job management and operation and can focus on developing the business logic. These features also solve the pain point of cumbersome SQL development processes in Streaming-Launcher.  **Support for Multiple Deployment Modes** -The Streaming-Launcher was not flexible for developers since it only supported the Yarn Session mode. StreamPark provides a comprehensive solution for this aspect. **StreamPark fully supports all of Flink's deployment modes: Remote, Yarn Per-Job, Yarn Application, Yarn Session, K8s Session, and K8s Application,** allowing developers to freely choose the appropriate running mode for different business scenarios. +The Streaming-Launcher was not flexible for developers since it only supported the Yarn Session mode. Apache StreamPark provides a comprehensive solution for this aspect. **Apache StreamPark fully supports all of Flink's deployment modes: Remote, Yarn Per-Job, Yarn Application, Yarn Session, K8s Session, and K8s Application,** allowing developers to freely choose the appropriate running mode for different business scenarios. **Unified Job Management Center** -For developers, job running status is one of their primary concerns. In Streaming-Launcher, due to the lack of a unified job management interface, developers had to rely on alarm information and Yarn application status to judge the state of tasks, which was very unfriendly. StreamPark addresses this issue by providing a unified job management interface, allowing for a clear view of the running status of each task. +For developers, job running status is one of their primary concerns. In Streaming-Launcher, due to the lack of a unified job management interface, developers had to rely on alarm information and Yarn application status to judge the state of tasks, which was very unfriendly. Apache StreamPark addresses this issue by providing a unified job management interface, allowing for a clear view of the running status of each task.  -In the Streaming-Launcher, developers had to go through multiple steps to diagnose job issues and locate job runtime logs. StreamPark offers a one-click jump feature that allows quick access to job runtime logs. +In the Streaming-Launcher, developers had to go through multiple steps to diagnose job issues and locate job runtime logs. Apache StreamPark offers a one-click jump feature that allows quick access to job runtime logs.  @@ -143,17 +143,17 @@ In the Streaming-Launcher, developers had to go through multiple steps to diagno ## Practical Implementation -When introducing StreamPark to Shunwang Technology, due to the company's business characteristics and some customized requirements from the developers, we made some additions and optimizations to the functionalities of StreamPark. We have also summarized some problems encountered during the use and corresponding solutions. +When introducing Apache StreamPark to Shunwang Technology, due to the company's business characteristics and some customized requirements from the developers, we made some additions and optimizations to the functionalities of Apache StreamPark. We have also summarized some problems encountered during the use and corresponding solutions. ### 01 **Leveraging the Capabilities of Flink-SQL-Gateway** -At Shunwang Technology, we have developed the ODPS platform based on the Flink-SQL-Gateway to facilitate business developers in managing the metadata of Flink tables. Business developers perform DDL operations on Flink tables in ODPS, and then carry out analysis and query operations on the created Flink tables in StreamPark. Throughout the entire business development process, we have decoupled the creation and analysis of Flink tables, making the development process appear more straightforward. +At Shunwang Technology, we have developed the ODPS platform based on the Flink-SQL-Gateway to facilitate business developers in managing the metadata of Flink tables. Business developers perform DDL operations on Flink tables in ODPS, and then carry out analysis and query operations on the created Flink tables in Apache StreamPark. Throughout the entire business development process, we have decoupled the creation and analysis of Flink tables, making the development process appear more straightforward. -If developers wish to query real-time data in ODPS, we need to provide a Flink SQL runtime environment. We have used StreamPark to run a Yarn Session Flink environment to support ODPS in conducting real-time queries. +If developers wish to query real-time data in ODPS, we need to provide a Flink SQL runtime environment. We have used Apache StreamPark to run a Yarn Session Flink environment to support ODPS in conducting real-time queries.  -Currently, the StreamPark community is intergrating with Flink-SQL-Gateway to further lower the barriers to developing real-time jobs. +Currently, the Apache StreamPark community is intergrating with Flink-SQL-Gateway to further lower the barriers to developing real-time jobs. https://github.com/apache/streampark/issues/2274 @@ -171,7 +171,7 @@ In practice, we found it difficult for business developers to intuitively know h ### 03 **Enhancing Alert Capabilities** -Since each company's SMS alert platform is implemented differently, the StreamPark community has not abstracted a unified SMS alert function. Here, we have implemented the SMS alert function through the Webhook method. +Since each company's SMS alert platform is implemented differently, the Apache StreamPark community has not abstracted a unified SMS alert function. Here, we have implemented the SMS alert function through the Webhook method.  @@ -179,7 +179,7 @@ Since each company's SMS alert platform is implemented differently, the StreamPa ### 04 **Adding a Blocking Queue to Solve Throttling Issues** -In production practice, we found that when a large number of tasks fail at the same time, such as when a Yarn Session cluster goes down, platforms like Feishu/Lark and WeChat have throttling issues when multiple threads call the alert interface simultaneously. Consequently, only a portion of the alert messages will be sent by StreamPark due to the throttling issues of platforms like Feishu/Lark and WeChat, which can easily mislead developers in troubleshooting. At Shunwang Technology, we added a blocking queue and an alert thread to solve the throttling issue. +In production practice, we found that when a large number of tasks fail at the same time, such as when a Yarn Session cluster goes down, platforms like Feishu/Lark and WeChat have throttling issues when multiple threads call the alert interface simultaneously. Consequently, only a portion of the alert messages will be sent by Apache StreamPark due to the throttling issues of platforms like Feishu/Lark and WeChat, which can easily mislead developers in troubleshooting. At Shunwang Technology, we added a blocking queue and an alert thread to solve the throttling issue.  @@ -191,11 +191,11 @@ https://github.com/apache/streampark/issues/2142 ## Benefits Brought -We started exploring and using StreamX 1.2.3 (the predecessor of StreamPark) and after more than a year of running in, we found that StreamPark truly resolves many pain points in the development management and operation and maintenance of Flink jobs. +We started exploring and using StreamX 1.2.3 (the predecessor of Apache StreamPark) and after more than a year of running in, we found that Apache StreamPark truly resolves many pain points in the development management and operation and maintenance of Flink jobs. -The greatest benefit that StreamPark has brought to Shunwang Technology is the lowered threshold for using Flink and improved development efficiency. Previously, our business development colleagues had to use multiple tools such as vscode, GitLab, and a scheduling platform in the original Streaming-Launcher to complete a FlinkSQL job development. The process was tedious, going through multiple tools from development to compilation to release. StreamPark provides one-stop service, allowing job development, compilation, and release to be completed on StreamPark, simplifying the entire development process. +The greatest benefit that Apache StreamPark has brought to Shunwang Technology is the lowered threshold for using Flink and improved development efficiency. Previously, our business development colleagues had to use multiple tools such as vscode, GitLab, and a scheduling platform in the original Streaming-Launcher to complete a FlinkSQL job development. The process was tedious, going through multiple tools from development to compilation to release. Apache StreamPark provides one-stop service, allowing job development, compilation, and release to be completed on Apache StreamPark, simplifying the entire development process. -**At present, StreamPark has been massively deployed in the production environment at Shunwang Technology, with the number of FlinkSQL jobs managed by StreamPark increasing from the initial 500+ to nearly 700, while also managing more than 10 Yarn Session Clusters.** +**At present, Apache StreamPark has been massively deployed in the production environment at Shunwang Technology, with the number of FlinkSQL jobs managed by Apache StreamPark increasing from the initial 500+ to nearly 700, while also managing more than 10 Yarn Session Clusters.**  @@ -203,11 +203,11 @@ The greatest benefit that StreamPark has brought to Shunwang Technology is the l ## Future Plans -As one of the early users of StreamPark, Shunwang Technology has been communicating with the community for a year and participating in the polishing of StreamPark's stability. We have submitted the Bugs encountered in production operations and new Features to the community. In the future, we hope to manage the metadata information of Flink tables on StreamPark, and implement cross-data-source query analysis functions based on the Flink engine through multiple Catalogs. Currently, StreamPark is integrating with Flink-SQL-Gateway capabilities, which will greatly help in the management of table metadata and cross-data-source query functions in the future. +As one of the early users of Apache StreamPark, Shunwang Technology has been communicating with the community for a year and participating in the polishing of Apache StreamPark's stability. We have submitted the Bugs encountered in production operations and new Features to the community. In the future, we hope to manage the metadata information of Flink tables on Apache StreamPark, and implement cross-data-source query analysis functions based on the Flink engine through multiple Catalogs. Currently, Apache StreamPark is integrating with Flink-SQL-Gateway capabilities, which will greatly help in the management of table metadata and cross-data-source query functions in the future. -Since Shunwang Technology primarily runs jobs in Yarn Session mode, we hope that StreamPark can provide more support for Remote clusters, Yarn Session clusters, and K8s Session clusters, such as monitoring and alerts, and optimizing operational processes. +Since Shunwang Technology primarily runs jobs in Yarn Session mode, we hope that Apache StreamPark can provide more support for Remote clusters, Yarn Session clusters, and K8s Session clusters, such as monitoring and alerts, and optimizing operational processes. -Considering the future, as the business develops, we may use StreamPark to manage more Flink real-time jobs, and StreamPark in single-node mode may not be safe. Therefore, we are also looking forward to the High Availability (HA) of StreamPark. +Considering the future, as the business develops, we may use Apache StreamPark to manage more Flink real-time jobs, and Apache StreamPark in single-node mode may not be safe. Therefore, we are also looking forward to the High Availability (HA) of Apache StreamPark. -We will also participate in the construction of the capabilities of StreamPark is integrating with Flink-SQL-Gateway, enriching Flink Cluster functionality, and StreamPark HA. +We will also participate in the construction of the capabilities of Apache StreamPark is integrating with Flink-SQL-Gateway, enriching Flink Cluster functionality, and Apache StreamPark HA. diff --git a/blog/5-streampark-usercase-dustess.md b/blog/5-streampark-usercase-dustess.md index 3a1910e6e..92a63a929 100644 --- a/blog/5-streampark-usercase-dustess.md +++ b/blog/5-streampark-usercase-dustess.md @@ -1,10 +1,10 @@ --- slug: streampark-usercase-dustess -title: StreamPark's Best Practices at Dustess, Simplifying Complexity for the Ultimate Experience -tags: [StreamPark, Production Practice, FlinkSQL] +title: Apache StreamPark's Best Practices at Dustess, Simplifying Complexity for the Ultimate Experience +tags: [Apache StreamPark, Production Practice, FlinkSQL] --- -**Abstract:** This article originates from the production practices of StreamPark at Dustess Information, written by the senior data development engineer, Gump. The main content includes: +**Abstract:** This article originates from the production practices of Apache StreamPark at Dustess Information, written by the senior data development engineer, Gump. The main content includes: 1. Technology selection 2. Practical implementation @@ -35,7 +35,7 @@ Firstly, in terms of the computing engine: We chose Flink for the following reas - Flink supports both batch and stream processing. Although the company's current batch processing architecture is based on Hive, Spark, etc., using Flink for stream computing facilitates the subsequent construction of unified batch and stream processing and lake-house architecture. - The domestic ecosystem of Flink has become increasingly mature, and Flink is starting to break boundaries towards the development of stream-based data warehousing. -At the platform level, we comprehensively compared StreamPark, Apache Zeppelin, and flink-streaming-platform-web, also thoroughly read their source code and conducted an analysis of their advantages and disadvantages. We won’t elaborate on the latter two projects in this article, but those interested can search for them on GitHub. We ultimately chose StreamPark for the following reasons: +At the platform level, we comprehensively compared Apache StreamPark, Apache Zeppelin, and flink-streaming-platform-web, also thoroughly read their source code and conducted an analysis of their advantages and disadvantages. We won’t elaborate on the latter two projects in this article, but those interested can search for them on GitHub. We ultimately chose Apache StreamPark for the following reasons: ### **High Completion** @@ -51,11 +51,11 @@ When creating a task, you can **freely choose the Flink version**. The Flink bin #### **2. Supports Multiple Deployment Modes** -StreamPark supports **all the mainstream submission modes** for Flink, such as standalone, yarn-session, yarn application, yarn-perjob, kubernetes-session, kubernetes-application. Moreover, StreamPark does not simply piece together Flink run commands to submit tasks. Instead, it introduces the Flink Client source package and directly calls the Flink Client API for task submission. The advantages of this approach include modular code, readability, ease of extension, stability, and the ability to quickly adapt to upgrades of the Flink version. +Apache StreamPark supports **all the mainstream submission modes** for Flink, such as standalone, yarn-session, yarn application, yarn-perjob, kubernetes-session, kubernetes-application. Moreover, Apache StreamPark does not simply piece together Flink run commands to submit tasks. Instead, it introduces the Flink Client source package and directly calls the Flink Client API for task submission. The advantages of this approach include modular code, readability, ease of extension, stability, and the ability to quickly adapt to upgrades of the Flink version.  -Flink SQL can greatly improve development efficiency and the popularity of Flink. StreamPark’s support for **Flink SQL is very comprehensive**, with an excellent SQL editor, dependency management, multi-version task management, etc. The StreamPark official website states that it will introduce metadata management integration for Flink SQL in the future. Stay tuned. +Flink SQL can greatly improve development efficiency and the popularity of Flink. Apache StreamPark’s support for **Flink SQL is very comprehensive**, with an excellent SQL editor, dependency management, multi-version task management, etc. The Apache StreamPark official website states that it will introduce metadata management integration for Flink SQL in the future. Stay tuned.  @@ -69,33 +69,33 @@ Flink SQL can greatly improve development efficiency and the popularity of Flink Although Flink SQL is now powerful enough, using JVM languages like Java and Scala to develop Flink tasks can be more flexible, more customizable, and better for tuning and improving resource utilization. The biggest problem with submitting tasks via Jar packages, compared to SQL, is the management of the Jar uploads. Without excellent tooling products, this can significantly reduce development efficiency and increase maintenance costs. -Besides supporting Jar uploads, StreamPark also provides an **online update build** feature, which elegantly solves the above problems: +Besides supporting Jar uploads, Apache StreamPark also provides an **online update build** feature, which elegantly solves the above problems: -1. Create Project: Fill in the GitHub/Gitlab (supports enterprise private servers) address and username/password, and StreamPark can Pull and Build the project. +1. Create Project: Fill in the GitHub/Gitlab (supports enterprise private servers) address and username/password, and Apache StreamPark can Pull and Build the project. -2. When creating a StreamPark Custom-Code task, refer to the Project, specify the main class, and optionally automate Pull, Build, and bind the generated Jar when starting the task, which is very elegant! +2. When creating a Apache StreamPark Custom-Code task, refer to the Project, specify the main class, and optionally automate Pull, Build, and bind the generated Jar when starting the task, which is very elegant! -At the same time, the StreamPark community is also perfecting the entire task compilation and launch process. The future StreamPark will be even more refined and professional on this foundation. +At the same time, the Apache StreamPark community is also perfecting the entire task compilation and launch process. The future Apache StreamPark will be even more refined and professional on this foundation.  #### **5. Comprehensive Task Parameter Configuration** -For data development using Flink, the parameters submitted with Flink run are almost impossible to maintain. StreamPark has also **elegantly solved** this kind of problem, mainly because, as mentioned above, StreamPark directly calls the Flink Client API and has connected the entire process through the StreamPark product frontend. +For data development using Flink, the parameters submitted with Flink run are almost impossible to maintain. Apache StreamPark has also **elegantly solved** this kind of problem, mainly because, as mentioned above, Apache StreamPark directly calls the Flink Client API and has connected the entire process through the Apache StreamPark product frontend.  -As you can see, StreamPark's task parameter settings cover all the mainstream parameters, and every parameter has been thoughtfully provided with an introduction and an optimal recommendation based on best practices. This is also very beneficial for newcomers to Flink, helping them to avoid common pitfalls! +As you can see, Apache StreamPark's task parameter settings cover all the mainstream parameters, and every parameter has been thoughtfully provided with an introduction and an optimal recommendation based on best practices. This is also very beneficial for newcomers to Flink, helping them to avoid common pitfalls! #### **6. Excellent Configuration File Design** -In addition to the native parameters for Flink tasks, which are covered by the task parameters above, StreamPark also provides a powerful **Yaml configuration file** mode and **programming model**. +In addition to the native parameters for Flink tasks, which are covered by the task parameters above, Apache StreamPark also provides a powerful **Yaml configuration file** mode and **programming model**.  -1. For Flink SQL tasks, you can configure the parameters that StreamPark has already built-in, such as **CheckPoint, retry mechanism, State Backend, table planner, mode**, etc., directly using the task's Yaml configuration file. +1. For Flink SQL tasks, you can configure the parameters that Apache StreamPark has already built-in, such as **CheckPoint, retry mechanism, State Backend, table planner, mode**, etc., directly using the task's Yaml configuration file. -2. For Jar tasks, StreamPark offers a generic programming model that encapsulates the native Flink API. Combined with the wrapper package provided by StreamPark, it can very elegantly retrieve custom parameters from the configuration file. For more details, see the documentation: +2. For Jar tasks, Apache StreamPark offers a generic programming model that encapsulates the native Flink API. Combined with the wrapper package provided by Apache StreamPark, it can very elegantly retrieve custom parameters from the configuration file. For more details, see the documentation: Programming model: @@ -111,13 +111,13 @@ http://www.streamxhub.com/docs/development/config In addition: -StreamPark also **supports Apache Flink native tasks**. The parameter configuration can be statically maintained within the Java task internal code, covering a wide range of scenarios, such as seamless migration of existing Flink tasks, etc. +Apache StreamPark also **supports Apache Flink native tasks**. The parameter configuration can be statically maintained within the Java task internal code, covering a wide range of scenarios, such as seamless migration of existing Flink tasks, etc. #### **7. Checkpoint Management** -Regarding Flink's Checkpoint (Savepoint) mechanism, the greatest difficulty is maintenance. StreamPark has also elegantly solved this problem: +Regarding Flink's Checkpoint (Savepoint) mechanism, the greatest difficulty is maintenance. Apache StreamPark has also elegantly solved this problem: -- StreamPark will **automatically maintain** the task Checkpoint directory and versions in the system for easy retrieval. +- Apache StreamPark will **automatically maintain** the task Checkpoint directory and versions in the system for easy retrieval. - When users need to update and restart an application, they can choose whether to save a Savepoint. - When restarting a task, it is possible to choose to recover from a specified version of Checkpoint/Savepoint. @@ -129,42 +129,42 @@ As shown below, developers can very intuitively and conveniently upgrade or deal #### **8. Comprehensive Alerting Features** -For streaming computations, which are 7*24H resident tasks, monitoring and alerting are very important. StreamPark also has a **comprehensive solution** for these issues: +For streaming computations, which are 7*24H resident tasks, monitoring and alerting are very important. Apache StreamPark also has a **comprehensive solution** for these issues: - It comes with an email-based alerting method, which has zero development cost and can be used once configured. -- Thanks to the excellent modularity of the StreamPark source code, it's possible to enhance the code at the Task Track point and introduce the company's internal SDK for telephone, group, and other alerting methods, all with a very low development cost. +- Thanks to the excellent modularity of the Apache StreamPark source code, it's possible to enhance the code at the Task Track point and introduce the company's internal SDK for telephone, group, and other alerting methods, all with a very low development cost.  ### **Excellent Source Code** -Following the principle of technology selection, a new technology must be sufficiently understood in terms of underlying principles and architectural ideas before it is considered for production use. Before choosing StreamPark, its architecture and source code were subjected to in-depth study and reading. It was found that the underlying technologies used by StreamPark are very familiar to Chinese developers: MySQL, Spring Boot, Mybatis Plus, Vue, etc. The code style is unified and elegantly implemented with complete annotations. The modules are independently abstracted and reasonable, employing numerous design patterns, and the code quality is very high, making it highly suitable for troubleshooting and further development in the later stages. +Following the principle of technology selection, a new technology must be sufficiently understood in terms of underlying principles and architectural ideas before it is considered for production use. Before choosing Apache StreamPark, its architecture and source code were subjected to in-depth study and reading. It was found that the underlying technologies used by Apache StreamPark are very familiar to Chinese developers: MySQL, Spring Boot, Mybatis Plus, Vue, etc. The code style is unified and elegantly implemented with complete annotations. The modules are independently abstracted and reasonable, employing numerous design patterns, and the code quality is very high, making it highly suitable for troubleshooting and further development in the later stages.  -In November 2021, StreamPark was successfully selected by Open Source China as a GVP - Gitee "Most Valuable Open Source Project," which speaks volumes about its quality and potential. +In November 2021, Apache StreamPark was successfully selected by Open Source China as a GVP - Gitee "Most Valuable Open Source Project," which speaks volumes about its quality and potential.  ### **03 Active Community** -The community is currently very active. Since the end of November 2021, when StreamPark (based on 1.2.0-release) was implemented, StreamPark had just started to be recognized by everyone, and there were some minor bugs in the user experience (not affecting core functionality). At that time, in order to go live quickly, some features were disabled and some minor bugs were fixed. Just as we were preparing to contribute back to the community, we found that these had already been fixed, indicating that the community's iteration cycle is very fast. In the future, our company's team will also strive to stay in sync with the community, quickly implement new features, and improve data development efficiency while reducing maintenance costs. +The community is currently very active. Since the end of November 2021, when Apache StreamPark (based on 1.2.0-release) was implemented, Apache StreamPark had just started to be recognized by everyone, and there were some minor bugs in the user experience (not affecting core functionality). At that time, in order to go live quickly, some features were disabled and some minor bugs were fixed. Just as we were preparing to contribute back to the community, we found that these had already been fixed, indicating that the community's iteration cycle is very fast. In the future, our company's team will also strive to stay in sync with the community, quickly implement new features, and improve data development efficiency while reducing maintenance costs. ## **02 Implementation Practice** -StreamPark's environment setup is very straightforward, following the official website's building tutorial you can complete the setup within a few hours. It now supports a front-end and back-end separation packaging deployment model, which can meet the needs of more companies, and there has already been a Docker Build related PR, suggesting that StreamPark's compilation and deployment will become even more convenient and quick in the future. Related documentation is as follows: +Apache StreamPark's environment setup is very straightforward, following the official website's building tutorial you can complete the setup within a few hours. It now supports a front-end and back-end separation packaging deployment model, which can meet the needs of more companies, and there has already been a Docker Build related PR, suggesting that Apache StreamPark's compilation and deployment will become even more convenient and quick in the future. Related documentation is as follows: ``` http://www.streamxhub.com/docs/user-guide/deployment ``` -For rapid implementation and production use, we chose the reliable On Yarn resource management mode (even though StreamPark already supports K8S quite well), and there are already many companies that have deployed using StreamPark on K8S, which you can refer to: +For rapid implementation and production use, we chose the reliable On Yarn resource management mode (even though Apache StreamPark already supports K8S quite well), and there are already many companies that have deployed using Apache StreamPark on K8S, which you can refer to: ``` http://www.streamxhub.com/blog/flink-development-framework-streamx ``` -Integrating StreamPark with the Hadoop ecosystem can be said to be zero-cost (provided that Flink is integrated with the Hadoop ecosystem according to the Flink official website, and tasks can be launched via Flink scripts). +Integrating Apache StreamPark with the Hadoop ecosystem can be said to be zero-cost (provided that Flink is integrated with the Hadoop ecosystem according to the Flink official website, and tasks can be launched via Flink scripts). Currently, we are also conducting K8S testing and solution design, and will be migrating to K8S in the future. @@ -174,7 +174,7 @@ At present, our company's tasks based on Flink SQL are mainly for simple real-ti  -StreamPark has thoughtfully prepared a demo SQL task that can be run directly on a newly set up platform. This attention to detail demonstrates the community's commitment to user experience. Initially, our simple tasks were written and executed using Flink SQL, and StreamPark's support for Flink SQL is excellent, with a superior SQL editor and innovative POM and Jar package dependency management that can meet many SQL scenario needs. +Apache StreamPark has thoughtfully prepared a demo SQL task that can be run directly on a newly set up platform. This attention to detail demonstrates the community's commitment to user experience. Initially, our simple tasks were written and executed using Flink SQL, and Apache StreamPark's support for Flink SQL is excellent, with a superior SQL editor and innovative POM and Jar package dependency management that can meet many SQL scenario needs. Currently, we are researching and designing solutions related to metadata, permissions, UDFs, etc. @@ -182,11 +182,11 @@ Currently, we are researching and designing solutions related to metadata, permi Since most of the data development team members have a background in Java and Scala, we've implemented Jar-based builds for more flexible development, transparent tuning of Flink tasks, and to cover more scenarios. Our implementation was in two phases: -**First Phase:** StreamPark provides support for native Apache Flink projects. We configured our existing tasks' Git addresses in StreamPark, used Maven to package them as Jar files, and created StreamPark Apache Flink tasks for seamless migration. In this process, StreamPark was merely used as a platform tool for task submission and state maintenance, without leveraging the other features mentioned above. +**First Phase:** Apache StreamPark provides support for native Apache Flink projects. We configured our existing tasks' Git addresses in Apache StreamPark, used Maven to package them as Jar files, and created Apache StreamPark Apache Flink tasks for seamless migration. In this process, Apache StreamPark was merely used as a platform tool for task submission and state maintenance, without leveraging the other features mentioned above. -**Second Phase:** After migrating tasks to StreamPark in the first phase and having them run on the platform, the tasks' configurations, such as checkpoint, fault tolerance, and adjustments to business parameters within Flink tasks, required source code modifications, pushes, and builds. This was very inefficient and opaque. +**Second Phase:** After migrating tasks to Apache StreamPark in the first phase and having them run on the platform, the tasks' configurations, such as checkpoint, fault tolerance, and adjustments to business parameters within Flink tasks, required source code modifications, pushes, and builds. This was very inefficient and opaque. -Therefore, following StreamPark's QuickStart, we quickly integrated StreamPark's programming model, which is an encapsulation for StreamPark Flink tasks (for Apache Flink). +Therefore, following Apache StreamPark's QuickStart, we quickly integrated Apache StreamPark's programming model, which is an encapsulation for Apache StreamPark Flink tasks (for Apache Flink). Example： @@ -194,7 +194,7 @@ Example： StreamingContext = ParameterTool + StreamExecutionEnvironment ``` -- StreamingContext is the encapsulation object for StreamPark +- StreamingContext is the encapsulation object for Apache StreamPark - ParameterTool is the parameter object after parsing the configuration file ``` @@ -205,7 +205,7 @@ StreamingContext = ParameterTool + StreamExecutionEnvironment ## **03 Business Support & Capability Opening** -Currently, Dustess Info's real-time computing platform based on StreamPark has been online since the end of November last year and has launched 50+ Flink tasks, including 10+ Flink SQL tasks and 40+ Jar tasks. At present, it is mainly used internally by the data team, and the real-time computing platform will be opened up for use by business teams across the company shortly, which will significantly increase the number of tasks. +Currently, Dustess Info's real-time computing platform based on Apache StreamPark has been online since the end of November last year and has launched 50+ Flink tasks, including 10+ Flink SQL tasks and 40+ Jar tasks. At present, it is mainly used internally by the data team, and the real-time computing platform will be opened up for use by business teams across the company shortly, which will significantly increase the number of tasks.  @@ -213,17 +213,17 @@ Currently, Dustess Info's real-time computing platform based on StreamPark has b The real-time data warehouse mainly uses Jar tasks because the model is more generic. Using Jar tasks can generically handle a large number of data table synchronization and calculations, and even achieve configuration-based synchronization. Our real-time data warehouse mainly uses Apache Doris for storage, with Flink handling the cleaning and calculations (the goal being storage-computation separation). -Using StreamPark to integrate other components is also very straightforward, and we have also abstracted the configuration related to Apache Doris and Kafka into the configuration file, which greatly enhances our development efficiency and flexibility. +Using Apache StreamPark to integrate other components is also very straightforward, and we have also abstracted the configuration related to Apache Doris and Kafka into the configuration file, which greatly enhances our development efficiency and flexibility. ### **02 Capability Opening** -Other business teams outside the data team also have many stream processing scenarios. Hence, after secondary development of the real-time computing platform based on StreamPark, we opened up the following capabilities to all business teams in the company: +Other business teams outside the data team also have many stream processing scenarios. Hence, after secondary development of the real-time computing platform based on Apache StreamPark, we opened up the following capabilities to all business teams in the company: - Business capability opening: The upstream real-time data warehouse collects all business tables through log collection and writes them into Kafka. Business teams can base their business-related development on Kafka, or they can perform OLAP analysis through the real-time data warehouse (Apache Doris). - Computing capability opening: The server resources of the big data platform are made available for use by business teams. - Solution opening: The mature Connectors of the Flink ecosystem and support for Exactly Once semantics can reduce the development and maintenance costs related to stream processing for business teams. -Currently, StreamPark does not support multi-business group functions. The multi-business group function will be abstracted and contributed to the community. +Currently, Apache StreamPark does not support multi-business group functions. The multi-business group function will be abstracted and contributed to the community.  @@ -240,7 +240,7 @@ Currently, all our company's Flink tasks run on Yarn, which meets current needs, - **Separation of Storage and Computation**. Flink's computational resources and state storage are separated; computational resources can be mixed with other component resources, improving machine utilization. - **Elastic Scaling**. It is capable of elastic scaling, better saving manpower and material costs. -I am also currently organizing and implementing related technical architectures and solutions and have completed the technical verification of Flink on Kubernetes using StreamPark in an experimental environment. With the support of the StreamPark platform and the enthusiastic help of the community, I believe that production implementation is not far off. +I am also currently organizing and implementing related technical architectures and solutions and have completed the technical verification of Flink on Kubernetes using Apache StreamPark in an experimental environment. With the support of the Apache StreamPark platform and the enthusiastic help of the community, I believe that production implementation is not far off. ### **02 Stream-Batch Unification Construction** @@ -259,7 +259,7 @@ Regarding the unification of stream and batch, I am also currently researching a ## **05 Closing Words** -That's all for the sharing of StreamPark in the production practice at Dustess Info. Thank you all for reading this far. The original intention of writing this article was to bring a bit of StreamPark's production practice experience and reference to everyone, and together with the buddies in the StreamPark community, to jointly build StreamPark. In the future, I plan to participate and contribute more. A big thank you to the developers of StreamPark for providing such an excellent product; in many details, we can feel everyone's dedication. Although the current production version used by the company (1.2.0-release) still has some room for improvement in task group search, edit return jump page, and other interactive experiences, the merits outweigh the minor issues. I believe that StreamPark will get better and better, **and I also believe that StreamPark will promote the popularity of Apache Flink**. Finally, let's end with a phrase from the Apache Flink community: The future is real-time! +That's all for the sharing of Apache StreamPark in the production practice at Dustess Info. Thank you all for reading this far. The original intention of writing this article was to bring a bit of Apache StreamPark's production practice experience and reference to everyone, and together with the buddies in the Apache StreamPark community, to jointly build Apache StreamPark. In the future, I plan to participate and contribute more. A big thank you to the developers of Apache StreamPark for providing such an excellent product; in many details, we can feel everyone's dedication. Although the current production version used by the company (1.2.0-release) still has some room for improvement in task group search, edit return jump page, and other interactive experiences, the merits outweigh the minor issues. I believe that Apache StreamPark will get better and better, **and I also believe that Apache StreamPark will promote the popularity of Apache Flink**. Finally, let's end with a phrase from the Apache Flink community: The future is real-time! diff --git a/blog/6-streampark-usercase-joyme.md b/blog/6-streampark-usercase-joyme.md index 511527813..522886f9f 100644 --- a/blog/6-streampark-usercase-joyme.md +++ b/blog/6-streampark-usercase-joyme.md @@ -1,12 +1,12 @@ --- slug: streampark-usercase-joyme -title: StreamPark's Production Practice in Joyme -tags: [StreamPark, Production Practice, FlinkSQL] +title: Apache StreamPark's Production Practice in Joyme +tags: [Apache StreamPark, Production Practice, FlinkSQL] --- -**Abstract:** This article presents the production practices of StreamPark at Joyme, written by Qin Jiyong, a big data engineer at Joyme. The main contents include: +**Abstract:** This article presents the production practices of Apache StreamPark at Joyme, written by Qin Jiyong, a big data engineer at Joyme. The main contents include: -- Encountering StreamPark +- Encountering Apache StreamPark - Flink SQL job development - Custom code job development - Monitoring and alerting @@ -16,9 +16,9 @@ tags: [StreamPark, Production Practice, FlinkSQL] -## 1 Encountering StreamPark +## 1 Encountering Apache StreamPark -Encountering StreamPark was inevitable. Based on our existing real-time job development mode, we had to find an open-source platform to support our company's real-time business. Our current situation was as follows: +Encountering Apache StreamPark was inevitable. Based on our existing real-time job development mode, we had to find an open-source platform to support our company's real-time business. Our current situation was as follows: - We wrote jobs and packaged them to servers, then executed the Flink run command to submit them, which was a cumbersome and inefficient process. - Flink SQL was submitted through a self-developed old platform. The developers of the old platform had left, and no one maintained the subsequent code, even if someone did, they would have to face the problem of high maintenance costs. @@ -27,13 +27,13 @@ Encountering StreamPark was inevitable. Based on our existing real-time job deve For all these reasons, we needed an open-source platform to manage our real-time jobs, and we also needed to refactor to unify the development mode and language and centralize project management. -The first encounter with StreamPark basically confirmed our choice. We quickly deployed and installed according to the official documentation, performed some operations after setting up, and were greeted with a user-friendly interface. StreamPark's support for multiple versions of Flink, permission management, job monitoring, and other series of functions already met our needs well. Further understanding revealed that its community is also very active. We have witnessed the process of StreamPark's feature completion since version 1.1.0. The development team is very ambitious, and we believe they will continue to improve. +The first encounter with Apache StreamPark basically confirmed our choice. We quickly deployed and installed according to the official documentation, performed some operations after setting up, and were greeted with a user-friendly interface. Apache StreamPark's support for multiple versions of Flink, permission management, job monitoring, and other series of functions already met our needs well. Further understanding revealed that its community is also very active. We have witnessed the process of Apache StreamPark's feature completion since version 1.1.0. The development team is very ambitious, and we believe they will continue to improve. ## 2 Development of Flink SQL Jobs The Flink SQL development mode has brought great convenience. For some simple metric developments, it is possible to complete them with just a few SQL statements, without the need to write a single line of code. Flink SQL has facilitated the development work for many colleagues, especially since writing code can be somewhat difficult for those who work on warehouses. -To add a new task, you open the task addition interface of StreamPark, where the default Development Mode is Flink SQL mode. You can write the SQL logic directly in the Flink SQL section. +To add a new task, you open the task addition interface of Apache StreamPark, where the default Development Mode is Flink SQL mode. You can write the SQL logic directly in the Flink SQL section. For the Flink SQL part, you can progressively write the logic SQL following the Flink official website's documentation. Generally, for our company, it consists of three parts: the Source connection, intermediate logic processing, and finally the Sink. Essentially, the Source is consuming data from Kafka, the logic layer will involve MySQL for dimension table queries, and the Sink part is mostly Elasticsearch, Redis, or MySQL. @@ -76,7 +76,7 @@ SELECT Data.uid FROM source_table; ### **2. Add Dependency** -In terms of dependencies, it's an unique feature to Streampark. A complete Flink SQL job is innovatively split into two components within StreamPark: the SQL and the dependencies. The SQL part is easy to understand and requires no further explanation, but the dependencies are the various Connector JARs needed within the SQL, such as Kafka and MySQL Connectors. If these are used within the SQL, then these Connector dependencies must be introduced. In StreamPark, there are two ways to add dependencies: one is based on the standard Maven pom coordinates, and the other is by uploading the required Jars from a local source. These two methods can also be mixed and used as needed; simply apply, and these dependencies will be automatically loaded when the job is submitted. +In terms of dependencies, it's an unique feature to Streampark. A complete Flink SQL job is innovatively split into two components within Apache StreamPark: the SQL and the dependencies. The SQL part is easy to understand and requires no further explanation, but the dependencies are the various Connector JARs needed within the SQL, such as Kafka and MySQL Connectors. If these are used within the SQL, then these Connector dependencies must be introduced. In Apache StreamPark, there are two ways to add dependencies: one is based on the standard Maven pom coordinates, and the other is by uploading the required Jars from a local source. These two methods can also be mixed and used as needed; simply apply, and these dependencies will be automatically loaded when the job is submitted.  @@ -106,7 +106,7 @@ For streaming jobs, we use Flink Java for development, having refactored previou  -Once the configuration is completed, compile the corresponding project to finish the packaging phase. Thus, the Custom code jobs can also reference it. Compilation is required every time the code needs to go live; otherwise, only the last compiled code is available. Here's an issue: for security reasons, our company’s GitLab account passwords are regularly updated. This leads to a situation where the StreamPark projects have the old passwords configured, resulting in a failure when pulling projects from Git during compilation. To address this problem, we contacted the community and learned that the capability to modify projects has been added in the subsequent version 1.2.1. +Once the configuration is completed, compile the corresponding project to finish the packaging phase. Thus, the Custom code jobs can also reference it. Compilation is required every time the code needs to go live; otherwise, only the last compiled code is available. Here's an issue: for security reasons, our company’s GitLab account passwords are regularly updated. This leads to a situation where the Apache StreamPark projects have the old passwords configured, resulting in a failure when pulling projects from Git during compilation. To address this problem, we contacted the community and learned that the capability to modify projects has been added in the subsequent version 1.2.1.  @@ -120,7 +120,7 @@ As well as the task’s parallelism, monitoring method, etc., memory size should ## 4 Monitoring and Alerts -The monitoring in StreamPark requires configuration in the setting module to set up the basic information for sending emails. +The monitoring in Apache StreamPark requires configuration in the setting module to set up the basic information for sending emails.  @@ -132,7 +132,7 @@ When our jobs fail, we can receive alerts through email. These alerts are quite  -Regarding alerts, we have developed a scheduled task based on StreamPark's t_flink_app table. Why do this? Because most people might not check their emails promptly when it comes to email notifications. Therefore, we opted to monitor the status of each task and send corresponding monitoring information to our Lark (Feishu) alert group, enabling us to promptly identify and address issues with the tasks. It's a simple Python script, then configured with crontab to execute at scheduled times. +Regarding alerts, we have developed a scheduled task based on Apache StreamPark's t_flink_app table. Why do this? Because most people might not check their emails promptly when it comes to email notifications. Therefore, we opted to monitor the status of each task and send corresponding monitoring information to our Lark (Feishu) alert group, enabling us to promptly identify and address issues with the tasks. It's a simple Python script, then configured with crontab to execute at scheduled times. ## 5 Common Issues @@ -152,10 +152,10 @@ If the task has started but fails during the running phase, this situation might ## 6 Community Impression -Often when we discuss issues in the StreamPark user group, we get immediate responses from community members. Issues that cannot be resolved at the moment are generally fixed in the next version or the latest code branch. In the group, we also see many non-community members actively helping each other out. There are many big names from other communities as well, and many members actively join the community development work. The whole community feels very active to me! +Often when we discuss issues in the Apache StreamPark user group, we get immediate responses from community members. Issues that cannot be resolved at the moment are generally fixed in the next version or the latest code branch. In the group, we also see many non-community members actively helping each other out. There are many big names from other communities as well, and many members actively join the community development work. The whole community feels very active to me! ## 7 Conclusion -Currently, our company runs 60 real-time jobs online, with Flink SQL and custom code each making up about half. More real-time tasks will be put online subsequently. Many colleagues worry about the stability of StreamPark, but based on several months of production practice in our company, StreamPark is just a platform to help you develop, deploy, monitor, and manage jobs. Whether it is stable or not depends on the stability of our own Hadoop Yarn cluster (we use the onyan mode) and has little to do with StreamPark itself. It also depends on the robustness of the Flink SQL or code you write. These two aspects should be the primary concerns. Only when these two aspects are problem-free can the flexibility of StreamPark be fully utilized to improve job performance. To discuss the stability of StreamPark in isolation is somewhat extreme. +Currently, our company runs 60 real-time jobs online, with Flink SQL and custom code each making up about half. More real-time tasks will be put online subsequently. Many colleagues worry about the stability of Apache StreamPark, but based on several months of production practice in our company, Apache StreamPark is just a platform to help you develop, deploy, monitor, and manage jobs. Whether it is stable or not depends on the stability of our own Hadoop Yarn cluster (we use the onyan mode) and has little to do with Apache StreamPark itself. It also depends on the robustness of the Flink SQL or code you write. These two aspects should be the primary concerns. Only when these two aspects are problem-free can the flexibility of Apache StreamPark be fully utilized to improve job performance. To discuss the stability of Apache StreamPark in isolation is somewhat extreme. -That is all the content shared by StreamPark at Joyme. Thank you for reading this article. We are very grateful for such an excellent product provided by StreamPark, which is a true act of benefiting others. From version 1.0 to 1.2.1, the bugs encountered are promptly fixed, and every issue is taken seriously. We are still using the on yarn deployment mode. Restarting yarn will still cause jobs to be lost, but restarting yarn is not something we do every day. The community will also look to fix this problem as soon as possible. I believe that StreamPark will get better and better, with a promising future ahead. \ No newline at end of file +That is all the content shared by Apache StreamPark at Joyme. Thank you for reading this article. We are very grateful for such an excellent product provided by Apache StreamPark, which is a true act of benefiting others. From version 1.0 to 1.2.1, the bugs encountered are promptly fixed, and every issue is taken seriously. We are still using the on yarn deployment mode. Restarting yarn will still cause jobs to be lost, but restarting yarn is not something we do every day. The community will also look to fix this problem as soon as possible. I believe that Apache StreamPark will get better and better, with a promising future ahead. \ No newline at end of file diff --git a/blog/7-streampark-usercase-haibo.md b/blog/7-streampark-usercase-haibo.md index 5badbe01a..cde7deca5 100644 --- a/blog/7-streampark-usercase-haibo.md +++ b/blog/7-streampark-usercase-haibo.md @@ -1,13 +1,13 @@ --- slug: streampark-usercase-haibo title: An All-in-One Computation Tool in Haibo Tech's Production Practice and facilitation in Smart City Construction -tags: [StreamPark, Production Practice, FlinkSQL] +tags: [Apache StreamPark, Production Practice, FlinkSQL] --- -**Summary:** The author of this article, "StreamPark: An All-in-One Computation Tool in Haibo Tech's Production Practice and facilitation in Smart City Construction," is the Big Data Architect at Haibo Tech. The main topics covered include: +**Summary:** The author of this article, "Apache StreamPark: An All-in-One Computation Tool in Haibo Tech's Production Practice and facilitation in Smart City Construction," is the Big Data Architect at Haibo Tech. The main topics covered include: -1. Choosing StreamPark +1. Choosing Apache StreamPark 2. Getting Started Quickly 3. Application Scenarios 4. Feature Extensions @@ -18,11 +18,11 @@ Haibo Tech is an industry-leading company offering AI IoT products and solutions -## **01. Choosing StreamPark** +## **01. Choosing Apache StreamPark** Haibo Tech started using Flink SQL to aggregate and process various real-time IoT data since 2020. With the accelerated pace of smart city construction in various cities, the types and volume of IoT data to be aggregated are also increasing. This has resulted in an increasing number of Flink SQL tasks being maintained online, making a dedicated platform for managing numerous Flink SQL tasks an urgent need. -After evaluating Apache Zeppelin and StreamPark, we chose StreamPark as our real-time computing platform. Compared to Apache Zeppelin, StreamPark may not be as well-known. However, after experiencing the initial release of StreamPark and reading its design documentation, we recognized that its all-in-one design philosophy covers the entire lifecycle of Flink task development. This means that configuration, development, deployment, and operations can all be accomplished on a single platform. Our developers, operators, and testers can collaboratively work on StreamPark. The **low-code** + **all-in-one** design principles solidified our confidence in using StreamPark. +After evaluating Apache Zeppelin and Apache StreamPark, we chose Apache StreamPark as our real-time computing platform. Compared to Apache Zeppelin, Apache StreamPark may not be as well-known. However, after experiencing the initial release of Apache StreamPark and reading its design documentation, we recognized that its all-in-one design philosophy covers the entire lifecycle of Flink task development. This means that configuration, development, deployment, and operations can all be accomplished on a single platform. Our developers, operators, and testers can collaboratively work on Apache StreamPark. The **low-code** + **all-in-one** design principles solidified our confidence in using Apache StreamPark. // Video link (streampark official video) @@ -32,7 +32,7 @@ After evaluating Apache Zeppelin and StreamPark, we chose StreamPark as our real ### **1. Quick Start** -Using StreamPark to accomplish a real-time aggregation task is as simple as putting an elephant into a fridge, and it can be done in just three steps: +Using Apache StreamPark to accomplish a real-time aggregation task is as simple as putting an elephant into a fridge, and it can be done in just three steps: - Edit SQL @@ -50,11 +50,11 @@ With just the above three steps, you can complete the aggregation task from Mysq ### **2. Production Practice** -StreamPark is primarily used at Haibo for running real-time Flink SQL tasks: reading data from Kafka, processing it, and outputting to Clickhouse or Elasticsearch. +Apache StreamPark is primarily used at Haibo for running real-time Flink SQL tasks: reading data from Kafka, processing it, and outputting to Clickhouse or Elasticsearch. -Starting from October 2021, the company gradually migrated Flink SQL tasks to the StreamPark platform for centralized management. It supports the aggregation, computation, and alerting of our real-time IoT data. +Starting from October 2021, the company gradually migrated Flink SQL tasks to the Apache StreamPark platform for centralized management. It supports the aggregation, computation, and alerting of our real-time IoT data. -As of now, StreamPark has been deployed in various government and public security production environments, aggregating and processing real-time IoT data, as well as capturing data on people and vehicles. Below is a screenshot of the StreamPark platform deployed on a city's dedicated network: +As of now, Apache StreamPark has been deployed in various government and public security production environments, aggregating and processing real-time IoT data, as well as capturing data on people and vehicles. Below is a screenshot of the Apache StreamPark platform deployed on a city's dedicated network:  @@ -62,31 +62,31 @@ As of now, StreamPark has been deployed in various government and public securit #### **1. Real-time IoT Sensing Data Aggregation** -For aggregating real-time IoT sensing data, we directly use StreamPark to develop Flink SQL tasks. For methods not provided by Flink SQL, StreamPark also supports UDF-related functionalities. Users can upload UDF packages through StreamPark, and then call the relevant UDF in SQL to achieve more complex logical operations. +For aggregating real-time IoT sensing data, we directly use Apache StreamPark to develop Flink SQL tasks. For methods not provided by Flink SQL, Apache StreamPark also supports UDF-related functionalities. Users can upload UDF packages through Apache StreamPark, and then call the relevant UDF in SQL to achieve more complex logical operations. -The "SQL+UDF" approach meets most of our data aggregation scenarios. If business changes in the future, we only need to modify the SQL statement in StreamPark to complete business changes and deployment. +The "SQL+UDF" approach meets most of our data aggregation scenarios. If business changes in the future, we only need to modify the SQL statement in Apache StreamPark to complete business changes and deployment.  #### **2. Flink CDC Database Synchronization** -To achieve synchronization between various databases and data warehouses, we use StreamPark to develop Flink CDC SQL tasks. With the capabilities of Flink CDC, we've implemented data synchronization between Oracle and Oracle, as well as synchronization between Mysql/Postgresql and Clickhouse. +To achieve synchronization between various databases and data warehouses, we use Apache StreamPark to develop Flink CDC SQL tasks. With the capabilities of Flink CDC, we've implemented data synchronization between Oracle and Oracle, as well as synchronization between Mysql/Postgresql and Clickhouse.  **3. Data Analysis Model Management** -For tasks that can't use Flink SQL and need Flink code development, such as real-time control models and offline data analysis models, StreamPark offers a Custom code approach, allowing users to upload executable Flink Jar packages and run them. +For tasks that can't use Flink SQL and need Flink code development, such as real-time control models and offline data analysis models, Apache StreamPark offers a Custom code approach, allowing users to upload executable Flink Jar packages and run them. -Currently, we have uploaded over 20 analysis models, such as personnel and vehicles, to StreamPark, which manages and operates them. +Currently, we have uploaded over 20 analysis models, such as personnel and vehicles, to Apache StreamPark, which manages and operates them.  -**In Summary:** Whether it's Flink SQL tasks or Custom code tasks, StreamPark provides excellent support to meet various business scenarios. However, StreamPark lacks task scheduling capabilities. If you need to schedule tasks regularly, StreamPark currently cannot meet this need. Community members are actively developing scheduling-related modules, and the soon-to-be-released version 1.2.3 will support task scheduling capabilities, so stay tuned. +**In Summary:** Whether it's Flink SQL tasks or Custom code tasks, Apache StreamPark provides excellent support to meet various business scenarios. However, Apache StreamPark lacks task scheduling capabilities. If you need to schedule tasks regularly, Apache StreamPark currently cannot meet this need. Community members are actively developing scheduling-related modules, and the soon-to-be-released version 1.2.3 will support task scheduling capabilities, so stay tuned. ## **04. Feature Extension** -Datahub is a metadata management platform developed by Linkedin, offering data source management, data lineage, data quality checks, and more. Haibo Tech has developed an extension based on StreamPark and Datahub, implementing table-level/field-level lineage features. With the data lineage feature, users can check the field lineage relationship of Flink SQL and save the lineage relationship to the Linkedin/Datahub metadata management platform. +Datahub is a metadata management platform developed by Linkedin, offering data source management, data lineage, data quality checks, and more. Haibo Tech has developed an extension based on Apache StreamPark and Datahub, implementing table-level/field-level lineage features. With the data lineage feature, users can check the field lineage relationship of Flink SQL and save the lineage relationship to the Linkedin/Datahub metadata management platform. // Two video links (Data lineage feature developed based on streampark) @@ -94,10 +94,10 @@ Datahub is a metadata management platform developed by Linkedin, offering data s ## **05. Future Expectations** -Currently, the StreamPark community's Roadmap indicates that StreamPark 1.3.0 will usher in a brand new Workbench experience, a unified resource management center (unified management of JAR/UDF/Connectors), batch task scheduling, and more. These are also some of the brand-new features we are eagerly anticipating. +Currently, the Apache StreamPark community's Roadmap indicates that Apache StreamPark 1.3.0 will usher in a brand new Workbench experience, a unified resource management center (unified management of JAR/UDF/Connectors), batch task scheduling, and more. These are also some of the brand-new features we are eagerly anticipating. -The Workbench will use a new workbench-style SQL development style. By selecting a data source, SQL can be generated automatically, further enhancing Flink task development efficiency. The unified UDF resource center will solve the current problem where each task has to upload its dependency package. The batch task scheduling feature will address StreamPark's current inability to schedule tasks. +The Workbench will use a new workbench-style SQL development style. By selecting a data source, SQL can be generated automatically, further enhancing Flink task development efficiency. The unified UDF resource center will solve the current problem where each task has to upload its dependency package. The batch task scheduling feature will address Apache StreamPark's current inability to schedule tasks. -Below is a prototype designed by StreamPark developers, so please stay tuned. +Below is a prototype designed by Apache StreamPark developers, so please stay tuned.  diff --git a/blog/8-streampark-usercase-ziru.md b/blog/8-streampark-usercase-ziru.md index eda608f9f..1b8b2a918 100644 --- a/blog/8-streampark-usercase-ziru.md +++ b/blog/8-streampark-usercase-ziru.md @@ -1,16 +1,16 @@ --- slug: streampark-usercase-ziru title: Ziroom's Real-Time Computing Platform Practice Based on Apache StreamPark -tags: [StreamPark, Production Practice] +tags: [Apache StreamPark, Production Practice] ---  -**Introduction:** Ziroom, an O2O internet company focusing on providing rental housing products and services, has built an online, data-driven, and intelligent platform that covers the entire chain of urban living. Real-time computing has always played an important role in Ziroom. To date, Ziroom processes TB-level data daily. This article, brought by the real-time computing team from Ziroom, introduces the in-depth practice of Ziroom's real-time computing platform based on StreamPark. +**Introduction:** Ziroom, an O2O internet company focusing on providing rental housing products and services, has built an online, data-driven, and intelligent platform that covers the entire chain of urban living. Real-time computing has always played an important role in Ziroom. To date, Ziroom processes TB-level data daily. This article, brought by the real-time computing team from Ziroom, introduces the in-depth practice of Ziroom's real-time computing platform based on Apache StreamPark. - Challenges in real-time computing - The journey to the solution -- In-depth practice based on StreamPark +- In-depth practice based on Apache StreamPark - Summary of practical experience and examples - Benefits brought by the implementation - Future plans @@ -63,7 +63,7 @@ Therefore, there is a need to improve the efficiency of development and debuggin In the early stages of platform construction, we comprehensively surveyed almost all relevant projects in the industry, covering both commercial paid versions and open-source versions, starting from early 2022. After investigation and comparison, we found that these projects have their limitations to varying extents, and their usability and stability could not be effectively guaranteed. -Overall, StreamPark performed best in our evaluation. It was the only project without major flaws and with strong extensibility: supporting both SQL and JAR jobs, with the most complete and stable deployment mode for Flink jobs. Its unique architectural design not only avoids locking in specific Flink versions but also supports convenient version switching and parallel processing, effectively solving job dependency isolation and conflict issues. The job management & operations capabilities we focused on were also very complete, including monitoring, alerts, SQL validation, SQL version comparison, CI, etc. StreamPark's support for Flink on K8s was the most comprehensive among all the open-source projects we surveyed. However, StreamPark's K8s mode submission required local image building, leading to storage resource consumption. +Overall, Apache StreamPark performed best in our evaluation. It was the only project without major flaws and with strong extensibility: supporting both SQL and JAR jobs, with the most complete and stable deployment mode for Flink jobs. Its unique architectural design not only avoids locking in specific Flink versions but also supports convenient version switching and parallel processing, effectively solving job dependency isolation and conflict issues. The job management & operations capabilities we focused on were also very complete, including monitoring, alerts, SQL validation, SQL version comparison, CI, etc. Apache StreamPark's support for Flink on K8s was the most comprehensive among all the open-source projects we surveyed. However, Apache StreamPark's K8s mode submission required local image building, leading to storage resource consumption. In the latest 2.2 version, the community has already restructured this part. @@ -73,19 +73,19 @@ After analyzing the pros and cons of many open-source projects, we decided to pa 2. In the selection of open-source components, after comprehensive comparison and evaluation of various indicators, we finally chose what was then StreamX. Subsequent close communication with the community allowed us to deeply appreciate the serious and responsible attitude of the founders and the united and friendly atmosphere of the community. We also witnessed the project's inclusion in the Apache Incubator in September 2022, making us hopeful for its future. -3. On the basis of StreamPark, we aim to promote integration with the existing ecosystem of the company to better meet our business needs. +3. On the basis of Apache StreamPark, we aim to promote integration with the existing ecosystem of the company to better meet our business needs. -## **In-depth Practice Based on StreamPark** +## **In-depth Practice Based on Apache StreamPark** -Based on the above decisions, we initiated the evolution of the real-time computing platform, oriented by "pain point needs," and built a stable, efficient, and easy-to-maintain real-time computing platform based on StreamPark. Since the beginning of 2022, we have participated in the construction of the community while officially scheduling our internal platform construction. +Based on the above decisions, we initiated the evolution of the real-time computing platform, oriented by "pain point needs," and built a stable, efficient, and easy-to-maintain real-time computing platform based on Apache StreamPark. Since the beginning of 2022, we have participated in the construction of the community while officially scheduling our internal platform construction. -First, we further improved related functionalities on the basis of StreamPark: +First, we further improved related functionalities on the basis of Apache StreamPark:  ### **01 LDAP Login Support** -On the basis of StreamPark, we further improved related functionalities, including support for LDAP, so that in the future we can fully open up real-time capabilities, allowing analysts from the company's four business lines to use the platform, expected to reach about 170 people. With the increase in numbers, account management becomes increasingly important, especially in the case of personnel changes, account cancellation, and application become frequent and time-consuming operations. Therefore, integrating LDAP becomes particularly important. We communicated with the community in a timely manner and initiated a discussion, eventually contributing this Feature. Now, starting LDAP in StreamPark has become very simple, requiring just two steps: +On the basis of Apache StreamPark, we further improved related functionalities, including support for LDAP, so that in the future we can fully open up real-time capabilities, allowing analysts from the company's four business lines to use the platform, expected to reach about 170 people. With the increase in numbers, account management becomes increasingly important, especially in the case of personnel changes, account cancellation, and application become frequent and time-consuming operations. Therefore, integrating LDAP becomes particularly important. We communicated with the community in a timely manner and initiated a discussion, eventually contributing this Feature. Now, starting LDAP in Apache StreamPark has become very simple, requiring just two steps: #### step1: Fill in the corresponding LDAP @@ -116,7 +116,7 @@ On the login interface, click LDAP login method, then enter the corresponding ac ### **02 Automatic Ingress Generation for Job Submission** -Due to the company's network security policy, only port 80 is opened on the Kubernetes host machines by the operation team, making it impossible to directly access the job WebUI on Kubernetes via "domain + random port." To solve this problem, we needed to use Ingress to add a proxy layer to the access path, achieving the effect of access routing. In StreamPark version 2.0, we contributed the functionality related to Ingress [3]. We adopted a strategy pattern implementation, initially obtaining Kubernetes metadata information to identify its version and accordingly constructing respective objects, ensuring smooth use of the Ingress function across various Kubernetes environments. +Due to the company's network security policy, only port 80 is opened on the Kubernetes host machines by the operation team, making it impossible to directly access the job WebUI on Kubernetes via "domain + random port." To solve this problem, we needed to use Ingress to add a proxy layer to the access path, achieving the effect of access routing. In Apache StreamPark version 2.0, we contributed the functionality related to Ingress [3]. We adopted a strategy pattern implementation, initially obtaining Kubernetes metadata information to identify its version and accordingly constructing respective objects, ensuring smooth use of the Ingress function across various Kubernetes environments. The specific configuration steps are as follows: @@ -138,7 +138,7 @@ You will notice that the generated address consists of three parts: domain + job ### **03 Support for Viewing Job Deployment Logs** -In the process of continuous job deployment, we gradually realized that without logs, we cannot perform effective operations. Log retention, archiving, and viewing became an important part in our later problem-solving process. Therefore, in StreamPark version 2.0, we contributed the ability to archive startup logs in On Kubernetes mode and view them on the page [4]. Now, by clicking the log viewing button in the job list, it is very convenient to view the real-time logs of the job. +In the process of continuous job deployment, we gradually realized that without logs, we cannot perform effective operations. Log retention, archiving, and viewing became an important part in our later problem-solving process. Therefore, in Apache StreamPark version 2.0, we contributed the ability to archive startup logs in On Kubernetes mode and view them on the page [4]. Now, by clicking the log viewing button in the job list, it is very convenient to view the real-time logs of the job.  @@ -148,7 +148,7 @@ In actual use, as the number of jobs increased, the number of users rose, and mo To solve this problem, we proposed a demand in the community: we hoped that each job could directly jump to the corresponding monitoring chart and log archive page through a hyperlink, so that users could directly view the monitoring information and logs related to their jobs. This avoids tedious searches in complex system interfaces, thus improving the efficiency of troubleshooting. -We had a discussion in the community, and it was quickly responded to, as everyone thought it was a common need. Soon, a developer contributed a design and related PR, and the issue was quickly resolved. Now, to enable this feature in StreamPark has become very simple: +We had a discussion in the community, and it was quickly responded to, as everyone thought it was a common need. Soon, a developer contributed a design and related PR, and the issue was quickly resolved. Now, to enable this feature in Apache StreamPark has become very simple: #### step1: Create a badge label @@ -225,11 +225,11 @@ User B's actual execution SQL: SELECT name, Encryption_function(age), price, Sensitive_field_functions(phone) FROM user; ``` -### **06 Data Synchronization Platform Based on StreamPark** +### **06 Data Synchronization Platform Based on Apache StreamPark** -With the successful implementation of StreamPark's technical solutions in the company, we achieved deep support for Flink jobs, bringing a qualitative leap in data processing. This prompted us to completely revamp our past data synchronization logic, aiming to reduce operational costs through technical optimization and integration. Therefore, we gradually replaced historical Sqoop jobs, Canal jobs, and Hive JDBC Handler jobs with Flink CDC jobs, Flink stream, and batch jobs. In this process, we continued to optimize and strengthen StreamPark's interface capabilities, adding a status callback mechanism and achieving perfect integration with the DolphinScheduler [7] scheduling system, further enhancing our data processing capabilities. +With the successful implementation of Apache StreamPark's technical solutions in the company, we achieved deep support for Flink jobs, bringing a qualitative leap in data processing. This prompted us to completely revamp our past data synchronization logic, aiming to reduce operational costs through technical optimization and integration. Therefore, we gradually replaced historical Sqoop jobs, Canal jobs, and Hive JDBC Handler jobs with Flink CDC jobs, Flink stream, and batch jobs. In this process, we continued to optimize and strengthen Apache StreamPark's interface capabilities, adding a status callback mechanism and achieving perfect integration with the DolphinScheduler [7] scheduling system, further enhancing our data processing capabilities. -External system integration with StreamPark is simple, requiring only a few steps: +External system integration with Apache StreamPark is simple, requiring only a few steps: 1. First, create a token for API access: @@ -255,11 +255,11 @@ curl -X POST '/flink/app/start' \ ## **Summary of Practical Experience** -During our in-depth use of StreamPark, we summarized some common issues and explored solutions in the practice process, which we have compiled into examples for reference. +During our in-depth use of Apache StreamPark, we summarized some common issues and explored solutions in the practice process, which we have compiled into examples for reference. ### **01 Building Base Images** -To deploy a Flink job on Kubernetes using StreamPark, you first need to prepare a Base image built on Flink. Then, on the Kubernetes platform, the user-provided image is used to start the Flink job. If we continue to use the official "bare image," it is far from sufficient for actual development. Business logic developed by users often involves multiple upstream and downstream data sources, requiring related data source Connectors and dependencies like Hadoop. Therefore, these dependencies need to be included in the image. Below, I will introduce the specific operation steps. +To deploy a Flink job on Kubernetes using Apache StreamPark, you first need to prepare a Base image built on Flink. Then, on the Kubernetes platform, the user-provided image is used to start the Flink job. If we continue to use the official "bare image," it is far from sufficient for actual development. Business logic developed by users often involves multiple upstream and downstream data sources, requiring related data source Connectors and dependencies like Hadoop. Therefore, these dependencies need to be included in the image. Below, I will introduce the specific operation steps. #### step1: First, create a folder containing two folders and a Dockerfile file @@ -328,7 +328,7 @@ RUN unzip -d arthas-latest-bin arthas-packaging-latest-bin.zip ### **03 Resolution of Dependency Conflicts in Images** -In the process of using StreamPark, we often encounter dependency conflict exceptions like NoClassDefFoundError, ClassNotFoundException, and NoSuchMethodError in Flink jobs running on base images. The troubleshooting approach is to find the package path of the conflicting class indicated in the error. For example, if the error class is in org.apache.orc:orc-core, go to the corresponding module directory, run `mvn dependency::tree`, search for orc-core, see who brought in the dependency, and remove it using exclusion. Below, I will introduce in detail a method of custom packaging to resolve dependency conflicts, illustrated by a dependency conflict caused by the flink-shaded-hadoop-3-uber JAR package in a base image. +In the process of using Apache StreamPark, we often encounter dependency conflict exceptions like NoClassDefFoundError, ClassNotFoundException, and NoSuchMethodError in Flink jobs running on base images. The troubleshooting approach is to find the package path of the conflicting class indicated in the error. For example, if the error class is in org.apache.orc:orc-core, go to the corresponding module directory, run `mvn dependency::tree`, search for orc-core, see who brought in the dependency, and remove it using exclusion. Below, I will introduce in detail a method of custom packaging to resolve dependency conflicts, illustrated by a dependency conflict caused by the flink-shaded-hadoop-3-uber JAR package in a base image. #### step1: Clone the flink-shaded project locally👇 @@ -346,7 +346,7 @@ git clone https://github.com/apache/flink-shaded.git ### **04 Centralized Job Configuration Example** -One of the great conveniences of using StreamPark is centralized configuration management. You can configure all settings in the conf file in the Flink directory bound to the platform. +One of the great conveniences of using Apache StreamPark is centralized configuration management. You can configure all settings in the conf file in the Flink directory bound to the platform. ```shell cd /flink-1.14.5/conf @@ -363,9 +363,9 @@ Clicking Sync Conf will synchronize the global configuration file, and new jobs  -### **05 StreamPark DNS Resolution Configuration** +### **05 Apache StreamPark DNS Resolution Configuration** -A correct and reasonable DNS resolution configuration is very important when submitting FlinkSQL on the StreamPark platform. It mainly involves the following points: +A correct and reasonable DNS resolution configuration is very important when submitting FlinkSQL on the Apache StreamPark platform. It mainly involves the following points: 1. Flink jobs' Checkpoint writing to HDFS requires a snapshot write to an HDFS node obtained through ResourceManager. If there are expansions in the Hadoop cluster in the enterprise, and these new nodes are not covered by the DNS resolution service, this will directly lead to Checkpoint failure, affecting online stability. @@ -449,7 +449,7 @@ To achieve isolation between production and testing environments, we introduced export HADOOP_CONF_DIR=/home/streamx/conf ``` -This effectively cut off the default logic of Flink on K8s loading HDFS configuration. This operation ensures that A StreamPark only connects to A Hadoop environment, while B StreamPark connects to B Hadoop environment, thus achieving complete isolation between testing and production environments. +This effectively cut off the default logic of Flink on K8s loading HDFS configuration. This operation ensures that A Apache StreamPark only connects to A Hadoop environment, while B Apache StreamPark connects to B Hadoop environment, thus achieving complete isolation between testing and production environments. Specifically, after this command takes effect, we can ensure that Flink jobs submitted on port 10002 connect to the B Hadoop environment. Thus, the B Hadoop environment is isolated from the Hadoop environment used by Flink jobs submitted on port 10000 in the past, effectively preventing interference between different environments and ensuring system stability and reliability. @@ -540,16 +540,16 @@ netstat -tlnp | grep 10002 ## **Benefits Brought** -Our team has been using StreamX (the predecessor of StreamPark) and, after more than a year of practice and refinement, StreamPark has significantly improved our challenges in developing, managing, and operating Apache Flink jobs. As a one-stop service platform, StreamPark greatly simplifies the entire development process. Now, we can complete job development, compilation, and release directly on the StreamPark platform, not only lowering the management and deployment threshold of Flink but also significantly improving development efficiency. +Our team has been using StreamX (the predecessor of Apache StreamPark) and, after more than a year of practice and refinement, Apache StreamPark has significantly improved our challenges in developing, managing, and operating Apache Flink jobs. As a one-stop service platform, Apache StreamPark greatly simplifies the entire development process. Now, we can complete job development, compilation, and release directly on the Apache StreamPark platform, not only lowering the management and deployment threshold of Flink but also significantly improving development efficiency. -Since deploying StreamPark, we have been using the platform on a large scale in a production environment. From initially managing over 50 FlinkSQL jobs to nearly 500 jobs now, as shown in the diagram, StreamPark is divided into 7 teams, each with dozens of jobs. This transformation not only demonstrates StreamPark's scalability and efficiency but also fully proves its strong practical value in actual business. +Since deploying Apache StreamPark, we have been using the platform on a large scale in a production environment. From initially managing over 50 FlinkSQL jobs to nearly 500 jobs now, as shown in the diagram, Apache StreamPark is divided into 7 teams, each with dozens of jobs. This transformation not only demonstrates Apache StreamPark's scalability and efficiency but also fully proves its strong practical value in actual business.  ## **Future Expectations** -As one of the early users of StreamPark, we have maintained close communication with the community, participating in the stability improvement of StreamPark. We have submitted bugs encountered in production operation and new features to the community. In the future, we hope to manage the metadata information of Apache Paimon lake tables and the capability of auxiliary jobs for +As one of the early users of Apache StreamPark, we have maintained close communication with the community, participating in the stability improvement of Apache StreamPark. We have submitted bugs encountered in production operation and new features to the community. In the future, we hope to manage the metadata information of Apache Paimon lake tables and the capability of auxiliary jobs for -Paimon's Actions on StreamPark. Based on the Flink engine, by interfacing with the Catalog of lake tables and Action jobs, we aim to realize the management and optimization of lake table jobs in one integrated capability. Currently, StreamPark is working on integrating the capabilities with Paimon data, which will greatly assist in real-time data lake ingestion in the future. +Paimon's Actions on Apache StreamPark. Based on the Flink engine, by interfacing with the Catalog of lake tables and Action jobs, we aim to realize the management and optimization of lake table jobs in one integrated capability. Currently, Apache StreamPark is working on integrating the capabilities with Paimon data, which will greatly assist in real-time data lake ingestion in the future. -We are very grateful for the technical support that the StreamPark team has provided us all along. We wish Apache StreamPark continued success, more users, and its early graduation to become a top-level Apache project. +We are very grateful for the technical support that the Apache StreamPark team has provided us all along. We wish Apache StreamPark continued success, more users, and its early graduation to become a top-level Apache project. diff --git a/community/contribution_guide/become_committer.md b/community/contribution_guide/become_committer.md index 96c4b493d..c84cd7ff4 100644 --- a/community/contribution_guide/become_committer.md +++ b/community/contribution_guide/become_committer.md @@ -38,10 +38,10 @@ only by code. Apache StreamPark community strives to be meritocratic. Thus, once someone has contributed sufficiently to any area of CoPDoC they can be a -candidate for committer-ship and at last voted in as a StreamPark +candidate for committer-ship and at last voted in as a Apache StreamPark committer. Being an Apache StreamPark committer does not necessarily mean you must commit code with your commit privilege to the codebase; it -means you are committed to the StreamPark project and are productively +means you are committed to the Apache StreamPark project and are productively contributing to our community's success. ## Committer requirements: @@ -58,8 +58,8 @@ and fair. Committer candidates should have a decent amount of continuous engagements and contributions (fixing bugs, adding new features, writing documentation, maintaining issues boards, code review, or answering -community questions) to StreamPark either by contributing to the codebase -of the main website or StreamPark's GitHub repositories. +community questions) to Apache StreamPark either by contributing to the codebase +of the main website or Apache StreamPark's GitHub repositories. - +3 months with light activity and engagement. - +2 months of medium activity and engagement. diff --git a/community/contribution_guide/become_pmc_member.md b/community/contribution_guide/become_pmc_member.md index 725626622..d52abb902 100644 --- a/community/contribution_guide/become_pmc_member.md +++ b/community/contribution_guide/become_pmc_member.md @@ -38,10 +38,10 @@ only by code. Apache StreamPark community strives to be meritocratic. Thus, once someone has contributed sufficiently to any area of CoPDoC they can be a -candidate for PMC membership and at last voted in as a StreamPark +candidate for PMC membership and at last voted in as a Apache StreamPark PMC member. Being an Apache StreamPark PMC member does not necessarily mean you must commit code with your commit privilege to the codebase; it -means you are committed to the StreamPark project and are productively +means you are committed to the Apache StreamPark project and are productively contributing to our community's success. ## PMC member requirements: @@ -58,8 +58,8 @@ and fair. PMC member candidates should have a decent amount of continuous engagements and contributions (fixing bugs, adding new features, writing documentation, maintaining issues boards, code review, or answering -community questions) to StreamPark either by contributing to the codebase -of the main website or StreamPark's GitHub repositories. +community questions) to Apache StreamPark either by contributing to the codebase +of the main website or Apache StreamPark's GitHub repositories. - +5 months with light activity and engagement. - +4 months of medium activity and engagement. diff --git a/community/contribution_guide/mailing_lists.md b/community/contribution_guide/mailing_lists.md index 3f6affcf4..7f008966f 100644 --- a/community/contribution_guide/mailing_lists.md +++ b/community/contribution_guide/mailing_lists.md @@ -21,7 +21,7 @@ sidebar_position: 1 limitations under the License. --> -The StreamPark developer mailing list is the preferred means for all your questions when using StreamPark, which pushes your doubts out to the entire community. +The Apache StreamPark developer mailing list is the preferred means for all your questions when using Apache StreamPark, which pushes your doubts out to the entire community. This is the best way to keep up-to-date with the community. Before you post anything to the mailing lists, be sure that you already **subscribe** to them. @@ -31,12 +31,12 @@ The currently available lists are listed in the below table. ### Developer List -- Use this list for your StreamPark questions -- Used by StreamPark contributors to discuss development of StreamPark +- Use this list for your Apache StreamPark questions +- Used by Apache StreamPark contributors to discuss development of Apache StreamPark ### Commits List -- Notifications on changes to the StreamPark codebase +- Notifications on changes to the Apache StreamPark codebase | List Name | Address | Subscribe | Unsubscribe | Archive | |---------------------|------------------------------|-------------------------------------------------------------|-----------------------------------------------------------------|-----------------------------------------------------------------------------| diff --git a/community/contribution_guide/new_committer_process.md b/community/contribution_guide/new_committer_process.md index 4baadf7f6..6a1c3d995 100644 --- a/community/contribution_guide/new_committer_process.md +++ b/community/contribution_guide/new_committer_process.md @@ -71,7 +71,7 @@ Subject: [VOTE] New committer: ${NEW_COMMITTER_NAME} ``` ```text -Hi StreamPark PPMC, +Hi Apache StreamPark PPMC, This is a formal vote about inviting ${NEW_COMMITTER_NAME} as our new committer. @@ -92,7 +92,7 @@ Subject: [RESULT] [VOTE] New committer: ${NEW_COMMITTER_NAME} ``` ```text -Hi StreamPark PPMC, +Hi Apache StreamPark PPMC, The vote has now closed. The results are: @@ -110,13 +110,13 @@ The vote is ***successful/not successful*** ```text To: ${NEW_COMMITTER_EMAIL} Cc: private@streampark.apache.org -Subject: Invitation to become StreamPark committer: ${NEW_COMMITTER_NAME} +Subject: Invitation to become Apache StreamPark committer: ${NEW_COMMITTER_NAME} ``` ```text Hello ${NEW_COMMITTER_NAME}, -The StreamPark Project Management Committee (PMC) +The Apache StreamPark Project Management Committee (PMC) hereby offers you committer privileges to the project. These privileges are offered on the understanding that you'll use them reasonably and with common sense. @@ -165,7 +165,7 @@ establishing you as a committer. ```text To: ${NEW_COMMITTER_EMAIL} Cc: private@streampark.apache.org -Subject: Re: invitation to become StreamPark committer +Subject: Re: invitation to become Apache StreamPark committer ``` ```text diff --git a/community/contribution_guide/new_pmc_member_process.md b/community/contribution_guide/new_pmc_member_process.md index 9828d0f31..8da0d4cb2 100644 --- a/community/contribution_guide/new_pmc_member_process.md +++ b/community/contribution_guide/new_pmc_member_process.md @@ -69,7 +69,7 @@ Subject: [VOTE] New PMC member candidate: ${NEW_PMC_NAME} ``` ```text -Hi StreamPark PPMC, +Hi Apache StreamPark PPMC, This is a formal vote about inviting ${NEW_PMC_NAME} as our new PMC member. @@ -90,7 +90,7 @@ Subject: [RESULT] [VOTE] New PMC member: ${NEW_PMC_NAME} ``` ```text -Hi StreamPark PPMC, +Hi Apache StreamPark PPMC, The vote has now closed. The results are: @@ -108,11 +108,11 @@ The vote is ***successful/not successful*** ```text To: board@apache.org Cc: private@
@@ -208,7 +208,7 @@ public interface SQLResultFunction extends Serializable {
## JDBC Read Write
-In `StreamPark`, `JdbcSink` is used to write data, let's see how to write data with `JdbcSink`, the example is to read data from `kakfa` and write to `mysql`.
+In `Apache StreamPark`, `JdbcSink` is used to write data, let's see how to write data with `JdbcSink`, the example is to read data from `kakfa` and write to `mysql`.
@@ -229,7 +229,7 @@ jdbc:
password: 123456
```
:::danger Cautions
-The configuration under `jdbc` **semantic** is the semantics of writing, as described in [Jdbc Info Configuration](#jdbc-info-config), the configuration will only take effect on `JdbcSink`, `StreamPark` is based on two-phase commit to achieve **EXACTLY_ONCE** semantics,
+The configuration under `jdbc` **semantic** is the semantics of writing, as described in [Jdbc Info Configuration](#jdbc-info-config), the configuration will only take effect on `JdbcSink`, `Apache StreamPark` is based on two-phase commit to achieve **EXACTLY_ONCE** semantics,
This requires that the database being manipulated supports transactions(`mysql`, `oracle`, `MariaDB`, `MS SQL Server`), theoretically all databases that support standard Jdbc transactions can do EXACTLY_ONCE (exactly once) write
:::
diff --git a/docs/connector/3-clickhouse.md b/docs/connector/3-clickhouse.md
index dfaf1718c..e87608c72 100755
--- a/docs/connector/3-clickhouse.md
+++ b/docs/connector/3-clickhouse.md
@@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem';
[ClickHouse](https://clickhouse.com/) is a columnar database management system (DBMS) for online analytics (OLAP).
Currently, Flink does not officially provide a connector for writing to ClickHouse and reading from ClickHouse.
Based on the access form supported by [ClickHouse - HTTP client](https://clickhouse.com/docs/zh/interfaces/http/)
-and [JDBC driver](https://clickhouse.com/docs/zh/interfaces/jdbc), StreamPark encapsulates ClickHouseSink for writing data to ClickHouse in real-time.
+and [JDBC driver](https://clickhouse.com/docs/zh/interfaces/jdbc), Apache StreamPark encapsulates ClickHouseSink for writing data to ClickHouse in real-time.
`ClickHouse` writes do not support transactions, using JDBC write data to it could provide (AT_LEAST_ONCE) semanteme. Using the HTTP client to write asynchronously,
it will retry the asynchronous write multiple times. The failed data will be written to external components (Kafka, MySQL, HDFS, HBase),
@@ -65,10 +65,10 @@ public class ClickHouseUtil {
The method of splicing various parameters into the request url is cumbersome and hard-coded, which is very inflexible.
-### Write with StreamPark
+### Write with Apache StreamPark
-To access `ClickHouse` data with `StreamPark`, you only need to define the configuration file in the specified format and then write code.
-The configuration and code are as follows. The configuration of `ClickHose JDBC` in `StreamPark` is in the configuration list, and the sample running program is scala
+To access `ClickHouse` data with `Apache StreamPark`, you only need to define the configuration file in the specified format and then write code.
+The configuration and code are as follows. The configuration of `ClickHose JDBC` in `Apache StreamPark` is in the configuration list, and the sample running program is scala
#### configuration list
@@ -147,14 +147,14 @@ Clickhouse INSERT must insert data through the POST method. The general operatio
$ echo 'INSERT INTO t VALUES (1),(2),(3)' | curl 'http://localhost:8123/' --data-binary @-
```
-The operation of the above method is relatively simple. Sure java could also be used for writing. StreamPark adds many functions to the http post writing method,
+The operation of the above method is relatively simple. Sure java could also be used for writing. Apache StreamPark adds many functions to the http post writing method,
including encapsulation enhancement, adding cache, asynchronous writing, failure retry, and data backup after reaching the retry threshold，
To external components (kafka, mysql, hdfs, hbase), etc., the above functions only need to define the configuration file in the prescribed format,
and write the code.
### Write to ClickHouse
-The configuration of `ClickHose JDBC` in `StreamPark` is in the configuration list, and the sample running program is scala, as follows:
+The configuration of `ClickHose JDBC` in `Apache StreamPark` is in the configuration list, and the sample running program is scala, as follows:
asynchttpclient is used as an HTTP asynchronous client for writing. first, import the jar of asynchttpclient
```xml
diff --git a/docs/connector/4-doris.md b/docs/connector/4-doris.md
index cdedc7fc0..017990326 100644
--- a/docs/connector/4-doris.md
+++ b/docs/connector/4-doris.md
@@ -12,11 +12,11 @@ import TabItem from '@theme/TabItem';
[Apache Doris](https://doris.apache.org/) is a high-performance, and real-time analytical database,
which could support high-concurrent point query scenarios.
-StreamPark encapsulates DoirsSink for writing data to Doris in real-time, based on [Doris' stream loads](https://doris.apache.org/administrator-guide/load-data/stream-load-manual.html)
+Apache StreamPark encapsulates DoirsSink for writing data to Doris in real-time, based on [Doris' stream loads](https://doris.apache.org/administrator-guide/load-data/stream-load-manual.html)
-### Write with StreamPark
+### Write with Apache StreamPark
-Use `StreamPark` to write data to `Doris`. DorisSink only supports JSON format (single-layer) writing currently,
+Use `Apache StreamPark` to write data to `Doris`. DorisSink only supports JSON format (single-layer) writing currently,
such as: {"id":1,"name":"streampark"} The example of the running program is java, as follows:
#### configuration list
diff --git a/docs/connector/5-es.md b/docs/connector/5-es.md
index 7f1874f9b..1d8b1b14e 100755
--- a/docs/connector/5-es.md
+++ b/docs/connector/5-es.md
@@ -14,12 +14,12 @@ for Elasticsearch, which is used to write data to Elasticsearch, which can provi
ElasticsearchSink uses TransportClient (before 6.x) or RestHighLevelClient (starting with 6.x) to communicate with the
Elasticsearch cluster.
-`StreamPark` further encapsulates Flink-connector-elasticsearch6, shields development details, and simplifies write
+`Apache StreamPark` further encapsulates Flink-connector-elasticsearch6, shields development details, and simplifies write
operations for Elasticsearch6 and above.
:::tip hint
-Because there are conflicts between different versions of Flink Connector Elasticsearch, StreamPark temporarily only
+Because there are conflicts between different versions of Flink Connector Elasticsearch, Apache StreamPark temporarily only
supports write operations of Elasticsearch6 and above. If you wants to using Elasticsearch5, you need to exclude the
flink-connector-elasticsearch6 dependency and introduce the flink-connector-elasticsearch5 dependency to create
org.apache.flink.streaming.connectors.elasticsearch5.ElasticsearchSink instance writes data.
@@ -200,11 +200,11 @@ input.addSink(esSinkBuilder.build)
-The ElasticsearchSink created above is very inflexible to add parameters. `StreamPark` follows the concept of convention over configuration and automatic configuration.
-Users only need to configure es connection parameters and Flink operating parameters, and StreamPark will automatically assemble source and sink,
+The ElasticsearchSink created above is very inflexible to add parameters. `Apache StreamPark` follows the concept of convention over configuration and automatic configuration.
+Users only need to configure es connection parameters and Flink operating parameters, and Apache StreamPark will automatically assemble source and sink,
which greatly simplifies development logic and improves development efficiency and maintainability.
-## Using StreamPark writes to Elasticsearch
+## Using Apache StreamPark writes to Elasticsearch
Please ensure that operation requests are sent to the Elasticsearch cluster at least once after enabling Flink checkpointing in ESSink.
@@ -234,7 +234,7 @@ host: localhost:9200
### 2. 写入Elasticsearch
-Using StreamPark writes to Elasticsearch
+Using Apache StreamPark writes to Elasticsearch
@@ -289,7 +289,7 @@ object ConnectorApp extends FlinkStreaming {
-Flink ElasticsearchSinkFunction可以执行多种类型请求，如（DeleteRequest、 UpdateRequest、IndexRequest）,StreamPark也对以上功能进行了支持，对应方法如下：
+Flink ElasticsearchSinkFunction可以执行多种类型请求，如（DeleteRequest、 UpdateRequest、IndexRequest）,Apache StreamPark也对以上功能进行了支持，对应方法如下：
```scala
import org.apache.streampark.flink.core.scala.StreamingContext
@@ -376,8 +376,8 @@ See [Official Documentation](https://nightlies.apache.org/flink/flink-docs-relea
The BulkProcessor inside es can further configure its behavior of how to refresh the cache operation request,
see the [official documentation](https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/connectors/datastream/elasticsearch/#elasticsearch-sink) for details - **Configuring the Internal** Bulk Processor
-### StreamPark configuration
+### Apache StreamPark configuration
-All other configurations must comply with the StreamPark configuration.
+All other configurations must comply with the Apache StreamPark configuration.
For [specific configurable](/docs/development/conf) items and the role of each parameter,
please refer to the project configuration
diff --git a/docs/connector/6-hbase.md b/docs/connector/6-hbase.md
index eeecb3bf3..2ae38e749 100755
--- a/docs/connector/6-hbase.md
+++ b/docs/connector/6-hbase.md
@@ -11,15 +11,15 @@ import TabItem from '@theme/TabItem';
large-scale structured storage clusters can be built on cheap PC Servers. Unlike general relational databases,
HBase is a database suitable for unstructured data storage because HBase storage is based on a column rather than a row-based schema.
-Flink does not officially provide a connector for Hbase DataStream. StreamPark encapsulates HBaseSource and HBaseSink based on `Hbase-client`.
-It supports automatic connection creation based on configuration and simplifies development. StreamPark reading Hbase can record the latest status of the read data when the checkpoint is enabled,
+Flink does not officially provide a connector for Hbase DataStream. Apache StreamPark encapsulates HBaseSource and HBaseSink based on `Hbase-client`.
+It supports automatic connection creation based on configuration and simplifies development. Apache StreamPark reading Hbase can record the latest status of the read data when the checkpoint is enabled,
and the offset corresponding to the source can be restored through the data itself. Implement source-side AT_LEAST_ONCE.
HbaseSource implements Flink Async I/O to improve streaming throughput. The sink side supports AT_LEAST_ONCE by default.
EXACTLY_ONCE is supported when checkpointing is enabled.
:::tip hint
-StreamPark reading HBASE can record the latest state of the read data when checkpoint is enabled.
+Apache StreamPark reading HBASE can record the latest state of the read data when checkpoint is enabled.
Whether the previous state can be restored after the job is resumed depends entirely on whether the data itself has an offset identifier,
which needs to be manually specified in the code. The recovery logic needs to be specified in the func parameter of the getDataStream method of HBaseSource.
:::
@@ -239,11 +239,11 @@ class HBaseWriter extends RichSinkFunction {
-Reading and writing HBase in this way is cumbersome and inconvenient. `StreamPark` follows the concept of convention over configuration and automatic configuration.
-Users only need to configure Hbase connection parameters and Flink operating parameters. StreamPark will automatically assemble source and sink,
+Reading and writing HBase in this way is cumbersome and inconvenient. `Apache StreamPark` follows the concept of convention over configuration and automatic configuration.
+Users only need to configure Hbase connection parameters and Flink operating parameters. Apache StreamPark will automatically assemble source and sink,
which greatly simplifies development logic and improves development efficiency and maintainability.
-## write and read Hbase with StreamPark
+## write and read Hbase with Apache StreamPark
### 1. Configure policies and connection information
@@ -260,7 +260,7 @@ hbase:
### 2. Read and write HBase
-Writing to Hbase with StreamPark is very simple, the code is as follows:
+Writing to Hbase with Apache StreamPark is very simple, the code is as follows:
@@ -363,7 +363,7 @@ object HBaseSinkApp extends FlinkStreaming {
-When StreamPark writes to HBase, you need to create the method of HBaseQuery,
+When Apache StreamPark writes to HBase, you need to create the method of HBaseQuery,
specify the method to convert the query result into the required object, identify whether it is running,
and pass in the running parameters. details as follows
```scala
@@ -391,7 +391,7 @@ class HBaseSource(@(transient@param) val ctx: StreamingContext, property: Proper
}
```
-StreamPark HbaseSource implements flink Async I/O, which is used to improve the throughput of Streaming: first create a DataStream,
+Apache StreamPark HbaseSource implements flink Async I/O, which is used to improve the throughput of Streaming: first create a DataStream,
then create an HBaseRequest and call requestOrdered() or requestUnordered() to create an asynchronous stream, as follows:
```scala
class HBaseRequest[T: TypeInformation](@(transient@param) private val stream: DataStream[T], property: Properties = new Properties()) {
@@ -430,7 +430,7 @@ class HBaseRequest[T: TypeInformation](@(transient@param) private val stream: Da
}
```
-StreamPark supports two ways to write data: 1. addSink() 2. writeUsingOutputFormat Examples are as follows:
+Apache StreamPark supports two ways to write data: 1. addSink() 2. writeUsingOutputFormat Examples are as follows:
```scala
//1）Insert way 1
HBaseSink().sink[TestEntity](source, "order")
@@ -445,4 +445,4 @@ StreamPark supports two ways to write data: 1. addSink() 2. writeUsingOutputForm
## Other configuration
-All other configurations must comply with the StreamPark configuration. For specific configurable items and the role of each parameter, please refer to the 【project configuration](/docs/development/conf)
+All other configurations must comply with the Apache StreamPark configuration. For specific configurable items and the role of each parameter, please refer to the 【project configuration](/docs/development/conf)
diff --git a/docs/connector/7-http.md b/docs/connector/7-http.md
index e6fd6ed4f..2df14ea9a 100755
--- a/docs/connector/7-http.md
+++ b/docs/connector/7-http.md
@@ -9,7 +9,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
Some background services receive data through HTTP requests. In this scenario, Flink can write result data through HTTP
-requests. Currently, Flink officially does not provide a connector for writing data through HTTP requests. StreamPark
+requests. Currently, Flink officially does not provide a connector for writing data through HTTP requests. Apache StreamPark
encapsulates HttpSink to write data asynchronously in real-time based on asynchttpclient.
`HttpSink` writes do not support transactions, writing data to the target service provides AT_LEAST_ONCE semantics. Data
@@ -29,7 +29,7 @@ Asynchronous writing uses asynchttpclient as the client, you need to import the
```
-## Write with StreamPark
+## Write with Apache StreamPark
### http asynchronous write support type
@@ -141,5 +141,5 @@ After the asynchronous write data reaches the maximum retry value, the data will
:::
## Other configuration
-All other configurations must comply with the **StreamPark** configuration.
+All other configurations must comply with the **Apache StreamPark** configuration.
For specific configurable items and the role of each parameter, please refer [Project configuration](/docs/development/conf)
diff --git a/docs/connector/8-redis.md b/docs/connector/8-redis.md
index 94b2badea..141293a7c 100644
--- a/docs/connector/8-redis.md
+++ b/docs/connector/8-redis.md
@@ -9,10 +9,10 @@ import TabItem from '@theme/TabItem';
[Redis](http://www.redis.cn/) is an open source in-memory data structure storage system that can be used as a database, cache, and messaging middleware. It supports many types of data structures such as strings, hashes, lists, sets, ordered sets and range queries, bitmaps, hyperlogloglogs and geospatial index radius queries. Redis has built-in transactions and various levels of disk persistence, and provides high availability through Redis Sentinel and Cluster.
- Flink does not officially provide a connector for writing reids data.StreamPark is based on [Flink Connector Redis](https://bahir.apache.org/docs/flink/current/flink-streaming-redis/)
+ Flink does not officially provide a connector for writing reids data.Apache StreamPark is based on [Flink Connector Redis](https://bahir.apache.org/docs/flink/current/flink-streaming-redis/)
It encapsulates RedisSink, configures redis connection parameters, and automatically creates redis connections to simplify development. Currently, RedisSink supports the following connection methods: single-node mode, sentinel mode, and cluster mode because it does not support transactions.
-StreamPark uses Redis' **MULTI** command to open a transaction and the **EXEC** command to commit a transaction, see the link for details:
+Apache StreamPark uses Redis' **MULTI** command to open a transaction and the **EXEC** command to commit a transaction, see the link for details:
http://www.redis.cn/topics/transactions.html , using RedisSink supports AT_LEAST_ONCE (at least once) processing semantics by default. EXACTLY_ONCE semantics are supported with checkpoint enabled.
:::tip tip
@@ -21,7 +21,7 @@ EXACTLY_ONCE semantics will write to redis in batch when the flink job checkpoin
:::
## Redis Write Dependency
-Flink Connector Redis officially provides two kinds, the following two api are the same, StreamPark is using org.apache.bahir dependency.
+Flink Connector Redis officially provides two kinds, the following two api are the same, Apache StreamPark is using org.apache.bahir dependency.
```xml
org.apache.bahir
@@ -163,10 +163,10 @@ public class FlinkRedisSink {
}
```
-The above creation of FlinkJedisPoolConfig is tedious, and each operation of redis has to build RedisMapper, which is very insensitive. `StreamPark` uses a convention over configuration and automatic configuration. This only requires configuring redis
-StreamPark automatically assembles the source and sink parameters, which greatly simplifies the development logic and improves development efficiency and maintainability.
+The above creation of FlinkJedisPoolConfig is tedious, and each operation of redis has to build RedisMapper, which is very insensitive. `Apache StreamPark` uses a convention over configuration and automatic configuration. This only requires configuring redis
+Apache StreamPark automatically assembles the source and sink parameters, which greatly simplifies the development logic and improves development efficiency and maintainability.
-## StreamPark Writes to Redis
+## Apache StreamPark Writes to Redis
RedisSink defaults to AT_LEAST_ONCE (at least once) processing semantics, two-stage segment submission supports EXACTLY_ONCE semantics with checkpoint enabled, available connection types: single-node mode, sentinel mode.
@@ -216,7 +216,7 @@ redis.sink:
### 2. Write to Redis
-Writing to redis with StreamPark is very simple, the code is as follows:
+Writing to redis with Apache StreamPark is very simple, the code is as follows:
@@ -277,7 +277,7 @@ case class RedisMapper[T](cmd: RedisCommand, additionalKey: String, key: T => St
-As the code shows, StreamPark automatically loads the configuration to create a RedisSink, and the user completes the redis write operation by creating the required RedisMapper object, **additionalKey is the outermost key when hset is invalid for other write commands**.
+As the code shows, Apache StreamPark automatically loads the configuration to create a RedisSink, and the user completes the redis write operation by creating the required RedisMapper object, **additionalKey is the outermost key when hset is invalid for other write commands**.
RedisSink.sink() write the corresponding key corresponding to the data is required to specify the expiration time, if not specified default expiration time is java Integer.MAX_VALUE (67 years). As shown in the code.
```scala
@@ -357,11 +357,11 @@ public enum RedisCommand {
```
:::info Warning
-RedisSink currently supports single-node mode and sentinel mode connections. And its cluster mode does not support transactions, but StreamPark is currently for support. Please call the official Flink Connector Redis api if you have a usage scenario.
+RedisSink currently supports single-node mode and sentinel mode connections. And its cluster mode does not support transactions, but Apache StreamPark is currently for support. Please call the official Flink Connector Redis api if you have a usage scenario.
Checkpoint must be enabled under EXACTLY_ONCE semantics, otherwise the program will throw parameter exceptions.
EXACTLY_ONCE semantics checkpoint data sink cache inside the memory, you need to reasonably set the checkpoint interval according to the actual data, otherwise there is a risk of **oom**.
::: ## Other Configuration -All other configurations must adhere to the **StreamPark** configuration, please refer to [project configuration](/docs/development/conf) for specific configurable items and the role of each parameter. +All other configurations must adhere to the **Apache StreamPark** configuration, please refer to [project configuration](/docs/development/conf) for specific configurable items and the role of each parameter. diff --git a/docs/development/alert-conf.md b/docs/development/alert-conf.md index b0680357c..56d07842c 100644 --- a/docs/development/alert-conf.md +++ b/docs/development/alert-conf.md @@ -7,14 +7,14 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -`StreamPark` supports a variety of alerts, mainly as follows： +`Apache StreamPark` supports a variety of alerts, mainly as follows： * **E-mail**: Mail notification * **DingTalk**: DingTalk Custom Group Robot * **WeChat**: Enterprise WeChat Custom Group Robot * **Lark**: Feishu Custom Group Robot -StreamPark support any combination of alerts +Apache StreamPark support any combination of alerts :::info Future plan @@ -24,7 +24,7 @@ StreamPark support any combination of alerts ## Added alert configuration -`Click StreamPark` -> Setting on the left, then click `Alert Setting` to enter the alert configuration. +`Click Apache StreamPark` -> Setting on the left, then click `Alert Setting` to enter the alert configuration.  Click `Add New` to add alert configuration: diff --git a/docs/development/conf.md b/docs/development/conf.md index 3c8b1a303..f91eb81d0 100755 --- a/docs/development/conf.md +++ b/docs/development/conf.md @@ -16,7 +16,7 @@ ClientFailureRate, ClientTables } from '../components/TableData.jsx'; -Configuration is very important in `StreamPark`. +Configuration is very important in `Apache StreamPark`. ## Why do I need to configure @@ -100,9 +100,9 @@ A simpler method should be used, such as simplifying some environment initializa **Absolutely** -`StreamPark` proposes the concept of unified program configuration, which is generated by configuring a series of parameters from development to deployment in the `application.yml`according to a specific format a general configuration template, so that the initialization of the environment can be completed by transferring the configuration of the project to the program when the program is started. This is the concept of `configuration file`. +`Apache StreamPark` proposes the concept of unified program configuration, which is generated by configuring a series of parameters from development to deployment in the `application.yml`according to a specific format a general configuration template, so that the initialization of the environment can be completed by transferring the configuration of the project to the program when the program is started. This is the concept of `configuration file`. -`StreamPark` provides a higher level of abstraction for the `Flink SQL`, developers only need to define SQL to `sql.yaml`, when the program is started, the `sql.yaml` is transferred to the main program, and the SQL will be automatically loaded and executed. This is the concept of `sql file`. +`Apache StreamPark` provides a higher level of abstraction for the `Flink SQL`, developers only need to define SQL to `sql.yaml`, when the program is started, the `sql.yaml` is transferred to the main program, and the SQL will be automatically loaded and executed. This is the concept of `sql file`. ## Terms @@ -112,7 +112,7 @@ The SQL extracted in Flink SQL task is put into `sql.yaml`, this file with speci ## Configuration file -In StreamPark, the configuration file of `DataStream` job and `Flink Sql` are common. In other words, this configuration file can define the configurations of `DataStream` and `Flink Sql` (the configuration file in Flink SQL job is optional). The format of the configuration file must be `yaml` and must meet the requirements of yaml. +In Apache StreamPark, the configuration file of `DataStream` job and `Flink Sql` are common. In other words, this configuration file can define the configurations of `DataStream` and `Flink Sql` (the configuration file in Flink SQL job is optional). The format of the configuration file must be `yaml` and must meet the requirements of yaml. How to configure this configuration file and what to pay attention to. @@ -127,7 +127,7 @@ flink: jobmanager: property: #@see: https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html $internal.application.main: org.apache.streampark.flink.quickstart.QuickStartApp - pipeline.name: StreamPark QuickStart App + pipeline.name: Apache StreamPark QuickStart App yarn.application.queue: taskmanager.numberOfTaskSlots: 1 parallelism.default: 2 @@ -227,7 +227,7 @@ There are many basic parameters. The five most basic parameters are as follows. `$internal.application.main` and `pipeline.name` must be set. ::: -If you need to set more parameters, please refer to [`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html), These parameters must be placed under the property and the parameter names must be correct. StreamPark will automatically resolve these parameters and take effect. +If you need to set more parameters, please refer to [`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html), These parameters must be placed under the property and the parameter names must be correct. Apache StreamPark will automatically resolve these parameters and take effect. ##### Memory parameters @@ -399,7 +399,7 @@ sql: | :::danger attention -In the above content, | after SQL is required. In addition, | will retain the format of the whole section. StreamPark can directly define multiple SQLs at once. Each SQLs must be separated by semicolons, and each section of SQLs must follow the format and specification specified by Flink SQL. +In the above content, | after SQL is required. In addition, | will retain the format of the whole section. Apache StreamPark can directly define multiple SQLs at once. Each SQLs must be separated by semicolons, and each section of SQLs must follow the format and specification specified by Flink SQL. ::: ## Summary diff --git a/docs/development/model.md b/docs/development/model.md index 809e61d95..06a8130de 100644 --- a/docs/development/model.md +++ b/docs/development/model.md @@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem'; There are some rules and conventions to be followed in any framework. Only by following and mastering these rules can we use them more easily and achieve twice the result with half the effort.When we develop Flink job, we actually use the API provided by Flink to write an executable program (which must have a `main()` function) according to the development method required by Flink. We access various`Connector`in the program, and after a series of `operator`operations, we finally sink the data to the target storage through the `Connector` . -We call this method of step-by-step programming according to certain agreed rules the "programming paradigm". In this chapter, we will talk about the "programming paradigm" of StreamPark and the development considerations. +We call this method of step-by-step programming according to certain agreed rules the "programming paradigm". In this chapter, we will talk about the "programming paradigm" of Apache StreamPark and the development considerations. Let's start from these aspects @@ -25,11 +25,11 @@ Let's start from these aspects  ## Programming paradigm -`streampark-core` is positioned as a programming time framework, rapid development scaffolding, specifically created to simplify Flink development. Developers will use this module during the development phase. Let's take a look at what the programming paradigm of `DataStream` and `Flink Sql` with StreamPark looks like, and what the specifications and requirements are. +`streampark-core` is positioned as a programming time framework, rapid development scaffolding, specifically created to simplify Flink development. Developers will use this module during the development phase. Let's take a look at what the programming paradigm of `DataStream` and `Flink Sql` with Apache StreamPark looks like, and what the specifications and requirements are. ### DataStream -StreamPark provides both `scala` and `Java` APIs to develop `DataStream` programs, the specific code development is as follows. +Apache StreamPark provides both `scala` and `Java` APIs to develop `DataStream` programs, the specific code development is as follows.
@@ -77,7 +77,7 @@ To develop with the `scala` API, the program must inherit from `FlinkStreaming`.
Development with the `Java` API can not omit the `main()` method due to the limitations of the language itself, so it will be a standard `main()` function,. The user needs to create the `StreamingContext` manually. `StreamingContext` is a very important class, which will be introduced later.
:::tip tip
-The above lines of `scala` and `Java` code are the basic skeleton code necessary to develop `DataStream` with StreamPark. Developing a `DataStream` program with StreamPark. Starting from these lines of code, Java API development requires the developer to manually start the task `start`.
+The above lines of `scala` and `Java` code are the basic skeleton code necessary to develop `DataStream` with Apache StreamPark. Developing a `DataStream` program with Apache StreamPark. Starting from these lines of code, Java API development requires the developer to manually start the task `start`.
:::
### Flink Sql
@@ -86,11 +86,11 @@ The TableEnvironment is used to create the contextual execution environment for
The Flink community has been promoting the batch processing capability of DataStream and unifying the stream-batch integration, and in Flink 1.12, the stream-batch integration is truly unified, many historical APIs such as: DataSet API, BatchTableEnvironment API, etc. are deprecated and retired from the history stage. TableEnvironment** and **StreamTableEnvironment**.
- StreamPark provides a more convenient API for the development of **TableEnvironment** and **StreamTableEnvironment** environments.
+ Apache StreamPark provides a more convenient API for the development of **TableEnvironment** and **StreamTableEnvironment** environments.
#### TableEnvironment
-To develop Table & SQL jobs, TableEnvironment will be the recommended entry class for Flink, supporting both Java API and Scala API, the following code demonstrates how to develop a TableEnvironment type job in StreamPark
+To develop Table & SQL jobs, TableEnvironment will be the recommended entry class for Flink, supporting both Java API and Scala API, the following code demonstrates how to develop a TableEnvironment type job in Apache StreamPark
@@ -128,7 +128,7 @@ public class JavaTableApp {
:::tip tip
-The above lines of Scala and Java code are the essential skeleton code for developing a TableEnvironment with StreamPark.
+The above lines of Scala and Java code are the essential skeleton code for developing a TableEnvironment with Apache StreamPark.
Scala API must inherit FlinkTable, Java API development needs to manually construct TableContext, and the developer needs to manually start the task `start`.
:::
@@ -136,7 +136,7 @@ Scala API must inherit FlinkTable, Java API development needs to manually constr
#### StreamTableEnvironment
`StreamTableEnvironment` is used in stream computing scenarios, where the object of stream computing is a `DataStream`. Compared to `TableEnvironment`, `StreamTableEnvironment` provides an interface to convert between `DataStream` and `Table`. If your application is written using the `DataStream API` in addition to the `Table API` & `SQL`, you need to use the `StreamTableEnvironment`.
-The following code demonstrates how to develop a `StreamTableEnvironment` type job in StreamPark.
+The following code demonstrates how to develop a `StreamTableEnvironment` type job in Apache StreamPark.
@@ -181,12 +181,12 @@ public class JavaStreamTableApp {
:::tip tip
-The above lines of scala and Java code are the essential skeleton code for developing `StreamTableEnvironment` with StreamPark, and for developing `StreamTableEnvironment` programs with StreamPark. Starting from these lines of code, Java code needs to construct `StreamTableContext` manually, and `Java API` development requires the developer to start the task `start` manually.
+The above lines of scala and Java code are the essential skeleton code for developing `StreamTableEnvironment` with Apache StreamPark, and for developing `StreamTableEnvironment` programs with Apache StreamPark. Starting from these lines of code, Java code needs to construct `StreamTableContext` manually, and `Java API` development requires the developer to start the task `start` manually.
:::
## RunTime Context
-**RunTime Context** - **StreamingContext** , **TableContext** , **StreamTableContext** are three very important objects in StreamPark, next we look at the definition and role of these three **Context**.
+**RunTime Context** - **StreamingContext** , **TableContext** , **StreamTableContext** are three very important objects in Apache StreamPark, next we look at the definition and role of these three **Context**.
@@ -224,7 +224,7 @@ class StreamingContext(val parameter: ParameterTool, private val environment: St
This object is very important and will be used throughout the lifecycle of the task in the `DataStream` job. The `StreamingContext` itself inherits from the `StreamExecutionEnvironment`, and the configuration file is fully integrated into the `StreamingContext`, so that it is very easy to get various parameters from the `StreamingContext`.
:::
-In StreamPark, `StreamingContext` is also the entry class for the Java API to write `DataStream` jobs, one of the constructors of `StreamingContext` is specially built for the Java API, the constructor is defined as follows:
+In Apache StreamPark, `StreamingContext` is also the entry class for the Java API to write `DataStream` jobs, one of the constructors of `StreamingContext` is specially built for the Java API, the constructor is defined as follows:
```scala
/**
@@ -301,7 +301,7 @@ class TableContext(val parameter: ParameterTool,
}
```
-In StreamPark, `TableContext` is also the entry class for the Java API to write `Table Sql` jobs of type `TableEnvironment`. One of the constructor methods of `TableContext` is a constructor specifically built for the `Java API`, which is defined as follows:
+In Apache StreamPark, `TableContext` is also the entry class for the Java API to write `Table Sql` jobs of type `TableEnvironment`. One of the constructor methods of `TableContext` is a constructor specifically built for the `Java API`, which is defined as follows:
```scala
@@ -391,7 +391,7 @@ class StreamTableContext(val parameter: ParameterTool,
```
-In StreamPark, `StreamTableContext` is the entry class for the Java API to write `Table Sql` jobs of type `StreamTableEnvironment`. One of the constructors of `StreamTableContext` is a function built specifically for the Java API, which is defined as follows:
+In Apache StreamPark, `StreamTableContext` is the entry class for the Java API to write `Table Sql` jobs of type `StreamTableEnvironment`. One of the constructors of `StreamTableContext` is a function built specifically for the Java API, which is defined as follows:
```scala
@@ -526,7 +526,7 @@ The **destroy** stage is an optional stage that requires developer participation
## Catalog Structure
-The recommended project directory structure is as follows, please refer to the directory structure and configuration in [StreamPark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart)
+The recommended project directory structure is as follows, please refer to the directory structure and configuration in [Apache StreamPark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart)
``` tree
.
@@ -602,11 +602,11 @@ assembly.xml is the configuration file needed for the assembly packaging plugin,
## Packaged Deployment
-The recommended packaging mode in [streampark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart/tree/dev/quickstart-flink) is recommended. It runs `maven package` directly to generate a standard StreamPark recommended project package, after unpacking the directory structure is as follows.
+The recommended packaging mode in [streampark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart/tree/dev/quickstart-flink) is recommended. It runs `maven package` directly to generate a standard Apache StreamPark recommended project package, after unpacking the directory structure is as follows.
``` text
.
-StreamPark-flink-quickstart-1.0.0
+Apache StreamPark-flink-quickstart-1.0.0
├── bin
│ ├── startup.sh //Launch Script
│ ├── setclasspath.sh //Java environment variable-related scripts (used internally, not of concern to users)
@@ -616,7 +616,7 @@ StreamPark-flink-quickstart-1.0.0
│ ├── application.yaml //Project's configuration file
│ ├── sql.yaml // flink sql file
├── lib
-│ └── StreamPark-flink-quickstart-1.0.0.jar //The project's jar package
+│ └── Apache StreamPark-flink-quickstart-1.0.0.jar //The project's jar package
└── temp
```
diff --git a/docs/flink-k8s/1-deployment.md b/docs/flink-k8s/1-deployment.md
index 0701713a3..b2698eda3 100644
--- a/docs/flink-k8s/1-deployment.md
+++ b/docs/flink-k8s/1-deployment.md
@@ -5,22 +5,22 @@ sidebar_position: 1
---
-StreamPark Flink Kubernetes is based on [Flink Native Kubernetes](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes/) and support deployment modes as below：
+Apache StreamPark Flink Kubernetes is based on [Flink Native Kubernetes](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes/) and support deployment modes as below：
* Native-Kubernetes Application
* Native-Kubernetes Session
-At now, one StreamPark only supports one Kubernetes cluster.You can submit [Fearure Request Issue](https://github.com/apache/incubator-streampark/issues) , when multiple Kubernetes clusters are needed.
+At now, one Apache StreamPark only supports one Kubernetes cluster.You can submit [Fearure Request Issue](https://github.com/apache/incubator-streampark/issues) , when multiple Kubernetes clusters are needed.
## Environments requirement -Additional operating environment to run StreamPark Flink-K8s is as below: +Additional operating environment to run Apache StreamPark Flink-K8s is as below: * Kubernetes -* Maven（StreamPark runNode） -* Docker（StreamPark runNode） +* Maven（Apache StreamPark runNode） +* Docker（Apache StreamPark runNode） -StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed on node out of Kubernetes cluster when there are **smooth network** between the node and cluster. +Apache StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed on node out of Kubernetes cluster when there are **smooth network** between the node and cluster.
@@ -29,7 +29,7 @@ StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed ### configuration for connecting Kubernetes -StreamPark connects to Kubernetes cluster by default connection credentials `～/.kube/config `.User can copy `.kube/config` from Kubernetes node to StreamPark nodes,or download it from Kubernetes provided by cloud service providers.If considering Permission constraints, User also can +Apache StreamPark connects to Kubernetes cluster by default connection credentials `～/.kube/config `.User can copy `.kube/config` from Kubernetes node to Apache StreamPark nodes,or download it from Kubernetes provided by cloud service providers.If considering Permission constraints, User also can generate custom account`s configuration by themselves. ```shell @@ -51,12 +51,12 @@ kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit ### Configuration for remote Docker service -On Setting page of StreamPark, user can configure the connection information for Docker service of Kubernetes cluster. +On Setting page of Apache StreamPark, user can configure the connection information for Docker service of Kubernetes cluster.  -Building a Namespace named `streampark`(other name should be set at Setting page of StreamPark) at remote Docker.The namespace is push/pull space of StreamPark Flink image and Docker Register User should own `pull`/`push` permission of this namespace. +Building a Namespace named `streampark`(other name should be set at Setting page of Apache StreamPark) at remote Docker.The namespace is push/pull space of Apache StreamPark Flink image and Docker Register User should own `pull`/`push` permission of this namespace. ```shell @@ -81,9 +81,9 @@ parameter descriptions are as below： * **Flink Base Docker Image**： Base Flink Docker Image Tag can be obtained from [DockerHub - offical/flink](https://hub.docker.com/_/flink) .And user can also use private image when Docker Register Account owns `pull` permission of it. * **Rest-Service Exposed Type**：Description of candidate values for native Flink K8s configuration [kubernetes.rest-service.exposed.type](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/config/#kubernetes) ： - * `ClusterIP`：ip that StreamPark can access； - * `LoadBalancer`：resource of LoadBalancer should be allocated in advance， Flink Namespace own permission of automatic binding，and StreamPark can access LoadBalancer`s gateway； - * `NodePort`：StreamPark can access all K8s nodes； + * `ClusterIP`：ip that Apache StreamPark can access； + * `LoadBalancer`：resource of LoadBalancer should be allocated in advance， Flink Namespace own permission of automatic binding，and Apache StreamPark can access LoadBalancer`s gateway； + * `NodePort`：Apache StreamPark can access all K8s nodes； * **Kubernetes Pod Template**： It`s Flink custom configuration of pod-template.The container-name must be flink-main-container. If the k8s pod needs a secret key to pull the docker image, please fill in the information about the secret key in the pod template file.The example pod-template is as below： @@ -114,7 +114,7 @@ The additional configuration of Flink-Native-Kubernetes Session Job will be deci ## other configuration -StreamPark parameter related to Flink-K8s in `applicaton.yml` are as below.And in most condition, it is no need to change it. +Apache StreamPark parameter related to Flink-K8s in `applicaton.yml` are as below.And in most condition, it is no need to change it. | Configuration item | Description | Default value | |-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|---------------| diff --git a/docs/flink-k8s/2-k8s-pvc-integration.md b/docs/flink-k8s/2-k8s-pvc-integration.md index 3ed1d97f0..e2bb50cd3 100755 --- a/docs/flink-k8s/2-k8s-pvc-integration.md +++ b/docs/flink-k8s/2-k8s-pvc-integration.md @@ -8,7 +8,7 @@ sidebar_position: 2 The support for pvc resource(mount file resources such as checkpoint/savepoint/logs and so on) is based on pod-template at current version。 -Users do not have to concern the Native-Kubernetes Session.It will be processed when Session Cluster is constructed .Native-Kubernetes Application can be constructed by configuring on StreamPark webpage using `pod-template`、`jm-pod-template`、`tm-pod-template`. +Users do not have to concern the Native-Kubernetes Session.It will be processed when Session Cluster is constructed .Native-Kubernetes Application can be constructed by configuring on Apache StreamPark webpage using `pod-template`、`jm-pod-template`、`tm-pod-template`.
@@ -45,9 +45,9 @@ There are three ways to provide the dependency when using `rocksdb-backend`. 1. Flink Base Docker Image contains the dependency（user fix the dependency conflict by themself）; -2. Put the dependency `flink-statebackend-rocksdb_xx.jar` to the path `Workspace/jars` in StreamPark ; +2. Put the dependency `flink-statebackend-rocksdb_xx.jar` to the path `Workspace/jars` in Apache StreamPark ; -3. Add the rockdb-backend dependency to StreamPark Dependency(StreamPark will fix the conflict automatically) ： +3. Add the rockdb-backend dependency to Apache StreamPark Dependency(Apache StreamPark will fix the conflict automatically) ：  diff --git a/docs/flink-k8s/3-hadoop-resource-integration.md b/docs/flink-k8s/3-hadoop-resource-integration.md index c03274d18..90bd0b20d 100644 --- a/docs/flink-k8s/3-hadoop-resource-integration.md +++ b/docs/flink-k8s/3-hadoop-resource-integration.md @@ -6,7 +6,7 @@ sidebar_position: 3 ## Using Hadoop resource in Flink on K8s -Using Hadoop resources under the StreamPark Flink-K8s runtime, such as checkpoint mount HDFS, read and write Hive, etc. The general process is as follows: +Using Hadoop resources under the Apache StreamPark Flink-K8s runtime, such as checkpoint mount HDFS, read and write Hive, etc. The general process is as follows: #### 1、HDFS @@ -26,7 +26,7 @@ flink-json-1.14.5.jar log4j-1.2-api-2.17.1.jar log4j-slf4j-impl- This is to download the shaded jar and put it in the lib directory of flink. Take hadoop2 as an example, download `flink-shaded-hadoop-2-uber`：https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-9.0/flink-shaded-hadoop-2-uber-2.7.5-9.0.jar - In addition, you can configure the shade jar in a dependent manner in the `Dependency` in the StreamPark task configuration. the following configuration: + In addition, you can configure the shade jar in a dependent manner in the `Dependency` in the Apache StreamPark task configuration. the following configuration: ```xml
@@ -128,7 +128,7 @@ public static String getHadoopConfConfigMapName(String clusterId) {
c、`flink-sql-connector-hive`：https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6_2.12/1.14.5/flink-sql-connector-hive-2.3.6_2.12-1.14.5.jar
- Similarly, the above-mentioned hive-related jars can also be dependently configured in the `Dependency` in the task configuration of StreamPark in a dependent manner, which will not be repeated here.
+ Similarly, the above-mentioned hive-related jars can also be dependently configured in the `Dependency` in the task configuration of Apache StreamPark in a dependent manner, which will not be repeated here.
##### ii、Add hive configuration file (hive-site.xml)
diff --git a/docs/intro.md b/docs/intro.md
index 6ed75b7c5..c2140f1fe 100644
--- a/docs/intro.md
+++ b/docs/intro.md
@@ -1,6 +1,6 @@
---
id: 'intro'
-title: 'What is StreamPark'
+title: 'What is Apache StreamPark'
sidebar_position: 1
---
@@ -12,15 +12,15 @@ make stream processing easier!!!
## 🚀 What is Apache StreamPark™
-`Apache StreamPark` is an easy-to-use stream processing application development framework and one-stop stream processing operation platform, Aimed at ease building and managing streaming applications, StreamPark provides scaffolding for writing streaming process logics with Apache Flink and Apache Spark.
-StreamPark also provides a professional task management including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
+`Apache StreamPark` is an easy-to-use stream processing application development framework and one-stop stream processing operation platform, Aimed at ease building and managing streaming applications, Apache StreamPark provides scaffolding for writing streaming process logics with Apache Flink and Apache Spark.
+Apache StreamPark also provides a professional task management including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
## Why Apache StreamPark™
Apache Flink and Apache Spark are widely used as the next generation of big data streaming computing engines. Based on a bench of excellent experiences combined with best practices, we extracted the task deployment and runtime parameters into the configuration files. In this way, an easy-to-use RuntimeContext with out-of-the-box connectors would bring easier and more efficient task development experience. It reduces the learning cost and development barriers, hence developers can focus on the business logic.
-On the other hand, It can be challenge for enterprises to use Flink & Spark if there is no professional management platform for Flink & Spark tasks during the deployment phase. StreamPark provides such a professional task management platform, including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
+On the other hand, It can be challenge for enterprises to use Flink & Spark if there is no professional management platform for Flink & Spark tasks during the deployment phase. Apache StreamPark provides such a professional task management platform, including task development, scheduling, interactive query, deployment, operation, maintenance, etc.
## 🎉 Features
* Apache Flink & Apache Spark application development scaffold
@@ -32,19 +32,19 @@ On the other hand, It can be challenge for enterprises to use Flink & Spark if t
## 🏳🌈 Architecture of Apache StreamPark™
-The overall architecture of Apache StreamPark is shown in the following figure. Apache StreamPark consists of three parts, they are StreamPark-core and StreamPark-console.
+The overall architecture of Apache StreamPark is shown in the following figure. Apache StreamPark consists of three parts, they are Apache StreamPark-core and Apache StreamPark-console.
-
+
-### 1️⃣ StreamPark-core
+### 1️⃣ Apache StreamPark-core
-The positioning of `StreamPark-core` is a framework uesd while developing, it focuses on coding development, regulates configuration files, and develops in the convention over configuration guide.
-StreamPark-core provides a development-time RunTime Content and a series of out-of-the-box Connectors. Cumbersome operations are simplified by extending `DataStream-related` methods and integrating DataStream and `Flink sql` api .
+The positioning of `Apache StreamPark-core` is a framework uesd while developing, it focuses on coding development, regulates configuration files, and develops in the convention over configuration guide.
+Apache StreamPark-core provides a development-time RunTime Content and a series of out-of-the-box Connectors. Cumbersome operations are simplified by extending `DataStream-related` methods and integrating DataStream and `Flink sql` api .
development efficiency and development experience will be highly improved because users can focus on the business.
-### 2️⃣ StreamPark-console
+### 2️⃣ Apache StreamPark-console
-`StreamPark-console` is a comprehensive real-time `low code` data platform that can manage `Flink` tasks more convenient.
+`Apache StreamPark-console` is a comprehensive real-time `low code` data platform that can manage `Flink` tasks more convenient.
It integrates the experience of many best practices and integrates many functions such as project compilation, release,
parameter configuration, startup, `savepoint`, `flame graph`, `Flink SQL`, monitoring, etc.,
which greatly simplifies the daily operation of Flink tasks and maintenance. The ultimate goal is to create a one-stop big data platform,
diff --git a/docs/user-guide/1-deployment.md b/docs/user-guide/1-deployment.md
index 90b576664..4de20a9be 100755
--- a/docs/user-guide/1-deployment.md
+++ b/docs/user-guide/1-deployment.md
@@ -6,9 +6,9 @@ sidebar_position: 1
import { DeploymentEnvs } from '../components/TableData.jsx';
-The overall component stack structure of StreamPark is as follows. It consists of two major parts: streampark-core and streampark-console. streampark-console is a very important module, positioned as a **integrated real-time data platform**, ** streaming data warehouse Platform**, **Low Code**, **Flink & Spark task hosting platform**, can better manage Flink tasks, integrate project compilation, publishing, parameter configuration, startup, savepoint, flame graph ( flame graph ), Flink SQL, monitoring and many other functions are integrated into one, which greatly simplifies the daily operation and maintenance of Flink tasks and integrates many best practices. Its ultimate goal is to create a one-stop big data solution that integrates real-time data warehouses and batches
+The overall component stack structure of Apache StreamPark is as follows. It consists of two major parts: streampark-core and streampark-console. streampark-console is a very important module, positioned as a **integrated real-time data platform**, ** streaming data warehouse Platform**, **Low Code**, **Flink & Spark task hosting platform**, can better manage Flink tasks, integrate project compilation, publishing, parameter configuration, startup, savepoint, flame graph ( flame graph ), Flink SQL, monitoring and many other functions are integrated into one, which greatly simplifies the daily operation and maintenance of Flink tasks and integrates many best practices. Its ultimate goal is to create a one-stop big data solution that integrates real-time data warehouses and batches
-
+
streampark-console provides an out-of-the-box installation package. Before installation, there are some requirements for the environment. The specific requirements are as follows:
@@ -16,7 +16,7 @@ streampark-console provides an out-of-the-box installation package. Before insta
@@ -34,9 +34,9 @@ can view or operate the applications of the corresponding team. ## Role Management In order to facilitate application management and prevent misoperation, the team also needs to distinguish between -administrator and developer, so StreamPark introduces role management. +administrator and developer, so Apache StreamPark introduces role management. -Currently, StreamPark supports two roles: team admin and developer. The team admin has all the permissions in the team. +Currently, Apache StreamPark supports two roles: team admin and developer. The team admin has all the permissions in the team. Compared with the team admin, the developer has fewer permissions to delete applications and add USER to the team.
diff --git a/docs/user-guide/8-YarnQueueManagement.md b/docs/user-guide/8-YarnQueueManagement.md index 5dcf6bd51..e8d44a7f2 100644 --- a/docs/user-guide/8-YarnQueueManagement.md +++ b/docs/user-guide/8-YarnQueueManagement.md @@ -19,7 +19,7 @@ will be time-consuming and accompanied by poor user experience. If a task is submitted to an incorrect queue due to the error-input, it's likely to affect the stability of yarn applications on the queue and the abuse of the queue resource. -So StreamPark introduced the queue management feature to ensure that a set of added queues are shared within the same team, +So Apache StreamPark introduced the queue management feature to ensure that a set of added queues are shared within the same team, that is, ensure that queues resource is isolated within the scope of the team. It can generate the following benefits: - When deploying Flink `yarn-application` applications or Flink `yarn-session` clusters, it could set quickly and accurately yarn queue(`yarn.application.queue`) & labels(`yarn.application.node-label`). @@ -88,13 +88,13 @@ due to the relationship between the queues and the `yarn-session` mode flink clu
- Session cluster is shared by all teams. Why is it that when creating a `yarn-session` flink cluster, only the queues in the current team instead of all teams can be used in the queues candidate list ? -> Based on the mentioned above, StreamPark hopes that when creating a `yarn-session` flink cluster, +> Based on the mentioned above, Apache StreamPark hopes that when creating a `yarn-session` flink cluster, administrators can specify the queue belonged to current of the current team only, which could be better for administrators to perceive the impact of current operations on the current team. - Why not support the isolation for `flink yarn-session clusters / general clusters` on team wide ? - The impact range caused by changes in cluster visibility is larger than that caused by changes in queue visibility. - - StreamPark need to face greater difficulties in backward compatibility while also considering the user experience. + - Apache StreamPark need to face greater difficulties in backward compatibility while also considering the user experience. - At present, there is no exact research on the users group and applications scale deployed using `yarn-application` & `yarn-session` cluster modes in the community. Based on this fact, the community didn't provide greater feature support. diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md b/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md index 39e96328f..fa2f4fc54 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md @@ -1,7 +1,7 @@ --- slug: streampark-flink-on-k8s -title: StreamPark Flink on Kubernetes 实践 -tags: [StreamPark, 生产实践, FlinkSQL, Kubernetes] +title: Apache StreamPark Flink on Kubernetes 实践 +tags: [Apache StreamPark, 生产实践, FlinkSQL, Kubernetes] --- 雾芯科技创立于2018年1月。目前主营业务包括 RELX 悦刻品牌产品的研发、设计、制造及销售。凭借覆盖全产业链的核心技术与能力，RELX 悦刻致力于为用户提供兼具高品质和安全性的产品。 @@ -20,9 +20,9 @@ Native Kubernetes 和 Standalone Kubernetes 主要区别在于 Flink 与 Kuberne  -## **当 Flink On Kubernetes 遇见 StreamPark** +## **当 Flink On Kubernetes 遇见 Apache StreamPark** -Flink on Native Kubernetes 目前支持 Application 模式和 Session 模式，两者对比 Application 模式部署规避了 Session 模式的资源隔离问题、以及客户端资源消耗问题，因此**生产环境更推荐采用 Application Mode 部署 Flink 任务。**下面我们分别看看使用原始脚本的方式和使用 StreamPark 开发部署一个 Flink on Native Kubernetes 作业的流程。 +Flink on Native Kubernetes 目前支持 Application 模式和 Session 模式，两者对比 Application 模式部署规避了 Session 模式的资源隔离问题、以及客户端资源消耗问题，因此**生产环境更推荐采用 Application Mode 部署 Flink 任务。**下面我们分别看看使用原始脚本的方式和使用 Apache StreamPark 开发部署一个 Flink on Native Kubernetes 作业的流程。 ### ***使用脚本方式部署Kubernetes*** @@ -69,17 +69,17 @@ kubectl -n flink-cluster get svc 以上就是使用 Flink 提供的最原始的脚本方式把一个 Flink 任务部署到 Kubernetes 上的过程，只做到了最基本的任务提交，如果要达到生产使用级别，还有一系列的问题需要解决，如：方式过于原始无法适配大批量任务、无法记录任务checkpoint 和实时状态跟踪、任务运维和监控困难、无告警机制、 无法集中化管理等等。 -## **使用 StreamPark 部署 Flink on Kubernetes** +## **使用 Apache StreamPark 部署 Flink on Kubernetes** ------ 对于企业级生产环境使用 Flink on Kubernetes 会有着更高的要求, 一般会选择自建平台或者购买相关商业产品, 不论哪种方案在产品能力上满足: **大批量任务开发部署、状态跟踪、运维监控、失败告警、任务统一管理、高可用性** 等这些是普遍的诉求。 -针对以上问题我们调研了开源领域支持开发部署 Flink on Kubernetes 任务的开源项目，调研的过程中也不乏遇到了其他优秀的开源项目，在综合对比了多个开源项目后得出结论: **StreamPark 不论是完成度还是使用体验、稳定性等整体表现都非常出色，因此最终选择了 StreamPark 作为我们的一站式实时计算平台。**下面我们看看 StreamPark 是如何支持 Flink on Kubernetes : +针对以上问题我们调研了开源领域支持开发部署 Flink on Kubernetes 任务的开源项目，调研的过程中也不乏遇到了其他优秀的开源项目，在综合对比了多个开源项目后得出结论: **Apache StreamPark 不论是完成度还是使用体验、稳定性等整体表现都非常出色，因此最终选择了 Apache StreamPark 作为我们的一站式实时计算平台。**下面我们看看 Apache StreamPark 是如何支持 Flink on Kubernetes : ### **基础环境配置** -基础环境配置包括 Kubernetes 和 Docker 仓库信息以及 Flink 客户端的信息配置。对于 Kubernetes 基础环境最为简单的方式是直接拷贝 Kubernetes 节点的 .kube/config 到 StreamPark 节点用户目录，之后使用 kubectl 命令创建 Flink 专用的 Kubernetes Namespace 以及进行 RBAC 配置。 +基础环境配置包括 Kubernetes 和 Docker 仓库信息以及 Flink 客户端的信息配置。对于 Kubernetes 基础环境最为简单的方式是直接拷贝 Kubernetes 节点的 .kube/config 到 Apache StreamPark 节点用户目录，之后使用 kubectl 命令创建 Flink 专用的 Kubernetes Namespace 以及进行 RBAC 配置。 ```shell # 创建Flink作业使用的k8s namespace @@ -92,17 +92,17 @@ Docker 账户信息直接在 Docker Setting 界面配置即可：  -StreamPark 可适配多版本 Flink 作业开发，Flink 客户端直接在 StreamPark Setting 界面配置即可： +Apache StreamPark 可适配多版本 Flink 作业开发，Flink 客户端直接在 Apache StreamPark Setting 界面配置即可：  ### **作业开发** -StreamPark 做好基础环境配置之后只需要三步即可开发部署一个 Flink 作业: +Apache StreamPark 做好基础环境配置之后只需要三步即可开发部署一个 Flink 作业:  -StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink SQL 作业只需要输入SQL 和 依赖项即可, 该方式大大提升了开发体验,** **并且规避了依赖冲突等问题，**对此部分本文不重点介绍。 +Apache StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink SQL 作业只需要输入SQL 和 依赖项即可, 该方式大大提升了开发体验,** **并且规避了依赖冲突等问题，**对此部分本文不重点介绍。 这里需要选择部署模式为 kubernetes application, 并且需要在作业开发页面进行以下参数的配置：红框中参数为 Flink on Kubernetes 基础参数。 @@ -118,7 +118,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ### **作业上线** -作业开发完成之后是作业上线环节，在这一环节中 StreamPark 做了大量的工作，具体如下: +作业开发完成之后是作业上线环节，在这一环节中 Apache StreamPark 做了大量的工作，具体如下: - 准备环境 - 作业中的依赖下载 @@ -130,7 +130,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink  -在镜像构建和推送的时候我们可以看到 StreamPark 做的一系列工作: **读取配置、构建镜像、推送镜像到远程仓库...** 这里要给StreamPark 一个大大的赞！ +在镜像构建和推送的时候我们可以看到 Apache StreamPark 做的一系列工作: **读取配置、构建镜像、推送镜像到远程仓库...** 这里要给Apache StreamPark 一个大大的赞！  @@ -140,19 +140,19 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink  -整个过程仅需上述三步，即可完成在 StreamPark 上开发和部署一个Flink on Kubernetes 作业。而 StreamPark 对于 Flink on Kubernetes 的支持远远不止提交个任务这么简单。 +整个过程仅需上述三步，即可完成在 Apache StreamPark 上开发和部署一个Flink on Kubernetes 作业。而 Apache StreamPark 对于 Flink on Kubernetes 的支持远远不止提交个任务这么简单。 ### **作业管理** -**在作业提交之后，StreamPark 能实时获取到任务的最新 checkpoint 地址、任务的运行状态、集群实时的资源消耗信息，针对运行的任务可以非常方便的一键启停, 在停止作业时支持记录 savepoint 的位置, 以及再次启动时从 savepoint 恢复状态等功能，从而保证了生产环境的数据一致性，真正具备 Flink on Kubernetes 的 一站式开发、部署、运维监控的能力。** +**在作业提交之后，Apache StreamPark 能实时获取到任务的最新 checkpoint 地址、任务的运行状态、集群实时的资源消耗信息，针对运行的任务可以非常方便的一键启停, 在停止作业时支持记录 savepoint 的位置, 以及再次启动时从 savepoint 恢复状态等功能，从而保证了生产环境的数据一致性，真正具备 Flink on Kubernetes 的 一站式开发、部署、运维监控的能力。** -接下来我们来看看这一块的能力 StreamPark 是如何进行支持的: +接下来我们来看看这一块的能力 Apache StreamPark 是如何进行支持的: - **实时记录checkpoint** ------ -在作业提交之后，有时候需要更改作业逻辑但是要保证数据的一致性，那么就需要平台具有实时记录每一次 checkpoint 位置的能力, 以及停止时记录最后的 savepoint 位置的能力，StreamPark 在 Flink on Kubernetes 上很好的实现了该功能。默认会每隔5秒获取一次 checkpoint 信息记录到对应的表中，并且会按照 Flink 中保留 checkpoint 数量的策略，只保留 state.checkpoints.num-retained 个，超过的部分则删除。在任务停止时有勾选 savepoint 的选项，如勾选了savepoint 选项，在任务停止时会做 savepoint 操作，同样会记录 savepoint 具体位置到表中。 +在作业提交之后，有时候需要更改作业逻辑但是要保证数据的一致性，那么就需要平台具有实时记录每一次 checkpoint 位置的能力, 以及停止时记录最后的 savepoint 位置的能力，Apache StreamPark 在 Flink on Kubernetes 上很好的实现了该功能。默认会每隔5秒获取一次 checkpoint 信息记录到对应的表中，并且会按照 Flink 中保留 checkpoint 数量的策略，只保留 state.checkpoints.num-retained 个，超过的部分则删除。在任务停止时有勾选 savepoint 的选项，如勾选了savepoint 选项，在任务停止时会做 savepoint 操作，同样会记录 savepoint 具体位置到表中。 默认 savepoint 的根路径只需要在 Flink Home flink-conf.yaml 文件中配置即可自动识别，除了默认地址，在停止时也可以自定义指定 savepoint 的根路径。 @@ -164,7 +164,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ------ -对于生产环境的挑战，很重要的一点就是监控是否到位，Flink on Kubernetes 更是如此。这点很重要, 也是最基本需要具备的能力，StreamPark 可实时监控 Flink on Kubernetes 作业的运行状态并在平台上展示给用户，在页面上可以很方便的根据各种运行状态来检索任务。 +对于生产环境的挑战，很重要的一点就是监控是否到位，Flink on Kubernetes 更是如此。这点很重要, 也是最基本需要具备的能力，Apache StreamPark 可实时监控 Flink on Kubernetes 作业的运行状态并在平台上展示给用户，在页面上可以很方便的根据各种运行状态来检索任务。  @@ -172,21 +172,21 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ------ -除此之外 StreamPark 还具备完善的报警功能: 支持邮件、钉钉、微信和短信 等。这也是当初公司调研之后选择 StreamPark 作为 Flink on Kubernetes 一站式平台的重要原因。 +除此之外 Apache StreamPark 还具备完善的报警功能: 支持邮件、钉钉、微信和短信 等。这也是当初公司调研之后选择 Apache StreamPark 作为 Flink on Kubernetes 一站式平台的重要原因。  -通过以上我们看到 StreamPark 在支持 Flink on Kubernetes 开发部署过程中具备的能力, 包括：**作业的开发能力、部署能力、监控能力、运维能力、异常处理能力等，StreamPark 提供的是一套相对完整的解决方案。 且已经具备了一些 CICD/DevOps 的能力，整体的完成度还在持续提升。是在整个开源领域中对于 Flink on Kubernetes 一站式开发部署运维工作全链路都支持的产品，StreamPark 是值得被称赞的。** +通过以上我们看到 Apache StreamPark 在支持 Flink on Kubernetes 开发部署过程中具备的能力, 包括：**作业的开发能力、部署能力、监控能力、运维能力、异常处理能力等，Apache StreamPark 提供的是一套相对完整的解决方案。 且已经具备了一些 CICD/DevOps 的能力，整体的完成度还在持续提升。是在整个开源领域中对于 Flink on Kubernetes 一站式开发部署运维工作全链路都支持的产品，Apache StreamPark 是值得被称赞的。** -## **StreamPark 在雾芯科技的落地实践** +## **Apache StreamPark 在雾芯科技的落地实践** -StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作业和实时指标计算作业的开发部署，有 Jar 任务也有 Flink SQL 任务，全部使用 Native Kubernetes 部署；数据源有CDC、Kafka 等，Sink 端有 Maxcompute、kafka、Hive 等，以下是公司开发环境StreamPark 平台截图： +Apache StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作业和实时指标计算作业的开发部署，有 Jar 任务也有 Flink SQL 任务，全部使用 Native Kubernetes 部署；数据源有CDC、Kafka 等，Sink 端有 Maxcompute、kafka、Hive 等，以下是公司开发环境Apache StreamPark 平台截图：  ## 遇到的问题 -任何新技术都有探索与踩坑的过程，失败的经验是宝贵的，这里介绍下 StreamPark 在雾芯科技落地过程中踩的一些坑和经验，**这块的内容不仅仅关于 StreamPark 的部分, 相信会带给所有使用 Flink on Kubernetes 的小伙伴一些参考**。 +任何新技术都有探索与踩坑的过程，失败的经验是宝贵的，这里介绍下 Apache StreamPark 在雾芯科技落地过程中踩的一些坑和经验，**这块的内容不仅仅关于 Apache StreamPark 的部分, 相信会带给所有使用 Flink on Kubernetes 的小伙伴一些参考**。 ### **常见问题总结如下** @@ -196,7 +196,7 @@ StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作 - **scala 版本不一致** -由于 StreamPark 部署需要 Scala 环境，而且 Flink SQL 运行需要用到 StreamPark 提供的 Flink SQL Client，因此一定要保证 Flink 作业的 Scala 版本和 StreamPark 的 Scala 版本保持一致。 +由于 Apache StreamPark 部署需要 Scala 环境，而且 Flink SQL 运行需要用到 Apache StreamPark 提供的 Flink SQL Client，因此一定要保证 Flink 作业的 Scala 版本和 Apache StreamPark 的 Scala 版本保持一致。 - **注意类冲突** @@ -228,7 +228,7 @@ HDFS 阿里云 OSS/AWS S3 都可以进行 checkpoint 和 savepoint 存储，Flin - **任务每次重启都会导致多出一个 Job 实例** -在配置了基于 kubernetes 的HA的前提条件下，当需要停止 Flink 任务时，需要通过 StreamPark 的 cancel 来进行，不要直接通过 kubernetes 集群删除 Flink 任务的 Deployment。因为 Flink 的关闭有其自有的关闭流程，在删除 pod 同时 Configmap 中的相应配置文件也会被一并删除，而直接删除 pod 会导致 Configmap 的残留。当相同名称的任务重启时，会出现两个相同 Job 现象，因为在启动时，任务会加载之前残留的配置文件，尝试将已经关闭的任务恢复。 +在配置了基于 kubernetes 的HA的前提条件下，当需要停止 Flink 任务时，需要通过 Apache StreamPark 的 cancel 来进行，不要直接通过 kubernetes 集群删除 Flink 任务的 Deployment。因为 Flink 的关闭有其自有的关闭流程，在删除 pod 同时 Configmap 中的相应配置文件也会被一并删除，而直接删除 pod 会导致 Configmap 的残留。当相同名称的任务重启时，会出现两个相同 Job 现象，因为在启动时，任务会加载之前残留的配置文件，尝试将已经关闭的任务恢复。 - **kubernetes pod 域名访问怎么实现** @@ -300,14 +300,14 @@ push k8s-harbor.xxx.com/streamx/udf_flink_1.13.6-scala_2.11:latest ## **未来期待** -- **StreamPark 对于 Flink 作业 Metric 监控的支持** +- **Apache StreamPark 对于 Flink 作业 Metric 监控的支持** -StreamPark 如果可以对接 Flink Metric 数据而且可以在 StreamPark 平台上展示每时每刻 Flink 的实时消费数据情况就太棒了 +Apache StreamPark 如果可以对接 Flink Metric 数据而且可以在 Apache StreamPark 平台上展示每时每刻 Flink 的实时消费数据情况就太棒了 -- **StreamPark 对于Flink 作业日志持久化的支持** +- **Apache StreamPark 对于Flink 作业日志持久化的支持** -对于部署到 YARN 的 Flink 来说，如果 Flink 程序挂了，我们可以去 YARN 上看历史日志，但是对于 Kubernetes 来说，如果程序挂了，那么 Kubernetes 的 pod 就消失了，就没法查日志了。所以用户需要借助 Kubernetes 上的工具进行日志持久化，如果 StreamPark 支持 Kubernetes 日志持久化接口就更好了。 +对于部署到 YARN 的 Flink 来说，如果 Flink 程序挂了，我们可以去 YARN 上看历史日志，但是对于 Kubernetes 来说，如果程序挂了，那么 Kubernetes 的 pod 就消失了，就没法查日志了。所以用户需要借助 Kubernetes 上的工具进行日志持久化，如果 Apache StreamPark 支持 Kubernetes 日志持久化接口就更好了。 - **镜像过大的问题改进** -StreamPark 目前对于 Flink on Kubernetes 作业的镜像支持是将基础镜像和用户代码打成一个 Fat 镜像推送到 Docker 仓库，这种方式存在的问题就是镜像过大的时候耗时比较久，希望未来基础镜像可以复用不需要每次都与业务代码打到一起，这样可以极大地提升开发效率和节约成本。 +Apache StreamPark 目前对于 Flink on Kubernetes 作业的镜像支持是将基础镜像和用户代码打成一个 Fat 镜像推送到 Docker 仓库，这种方式存在的问题就是镜像过大的时候耗时比较久，希望未来基础镜像可以复用不需要每次都与业务代码打到一起，这样可以极大地提升开发效率和节约成本。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md b/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md index 3604cdf9f..381622535 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md @@ -1,7 +1,7 @@ --- slug: flink-development-framework-streampark -title: Flink 开发利器 StreamPark -tags: [StreamPark, DataStream, FlinkSQL] +title: Flink 开发利器 Apache StreamPark +tags: [Apache StreamPark, DataStream, FlinkSQL] ---
@@ -58,18 +58,18 @@ Flink 从 1.13 版本开始，就支持 Pod Template，我们可以在 Pod Templ
-## 引入 StreamPark +## 引入 Apache StreamPark 之前我们写 Flink SQL 基本上都是使用 Java 包装 SQL，打 jar 包，提交到 S3 平台上。通过命令行方式提交代码，但这种方式始终不友好，流程繁琐，开发和运维成本太大。我们希望能够进一步简化流程，将 Flink TableEnvironment 抽象出来，有平台负责初始化、打包运行 Flink 任务，实现 Flink 应用程序的构建、测试和部署自动化。 -这是个开源兴起的时代，我们自然而然的将目光投向开源领域中：在一众开源项目中，经过对比各个项目综合评估发现 Zeppelin 和 StreamPark 这两个项目对 Flink 的支持较为完善，都宣称支持 Flink on K8s ，最终进入到我们的目标选择范围中，以下是两者在 K8s 相关支持的简单比较（目前如果有更新，麻烦批评指正）。 +这是个开源兴起的时代，我们自然而然的将目光投向开源领域中：在一众开源项目中，经过对比各个项目综合评估发现 Zeppelin 和 Apache StreamPark 这两个项目对 Flink 的支持较为完善，都宣称支持 Flink on K8s ，最终进入到我们的目标选择范围中，以下是两者在 K8s 相关支持的简单比较（目前如果有更新，麻烦批评指正）。
+RedisSink currently supports single-node mode and sentinel mode connections. And its cluster mode does not support transactions, but Apache StreamPark is currently for support. Please call the official Flink Connector Redis api if you have a usage scenario.
Checkpoint must be enabled under EXACTLY_ONCE semantics, otherwise the program will throw parameter exceptions.
EXACTLY_ONCE semantics checkpoint data sink cache inside the memory, you need to reasonably set the checkpoint interval according to the actual data, otherwise there is a risk of **oom**.
::: ## Other Configuration -All other configurations must adhere to the **StreamPark** configuration, please refer to [project configuration](/docs/development/conf) for specific configurable items and the role of each parameter. +All other configurations must adhere to the **Apache StreamPark** configuration, please refer to [project configuration](/docs/development/conf) for specific configurable items and the role of each parameter. diff --git a/docs/development/alert-conf.md b/docs/development/alert-conf.md index b0680357c..56d07842c 100644 --- a/docs/development/alert-conf.md +++ b/docs/development/alert-conf.md @@ -7,14 +7,14 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -`StreamPark` supports a variety of alerts, mainly as follows： +`Apache StreamPark` supports a variety of alerts, mainly as follows： * **E-mail**: Mail notification * **DingTalk**: DingTalk Custom Group Robot * **WeChat**: Enterprise WeChat Custom Group Robot * **Lark**: Feishu Custom Group Robot -StreamPark support any combination of alerts +Apache StreamPark support any combination of alerts :::info Future plan @@ -24,7 +24,7 @@ StreamPark support any combination of alerts ## Added alert configuration -`Click StreamPark` -> Setting on the left, then click `Alert Setting` to enter the alert configuration. +`Click Apache StreamPark` -> Setting on the left, then click `Alert Setting` to enter the alert configuration.  Click `Add New` to add alert configuration: diff --git a/docs/development/conf.md b/docs/development/conf.md index 3c8b1a303..f91eb81d0 100755 --- a/docs/development/conf.md +++ b/docs/development/conf.md @@ -16,7 +16,7 @@ ClientFailureRate, ClientTables } from '../components/TableData.jsx'; -Configuration is very important in `StreamPark`. +Configuration is very important in `Apache StreamPark`. ## Why do I need to configure @@ -100,9 +100,9 @@ A simpler method should be used, such as simplifying some environment initializa **Absolutely** -`StreamPark` proposes the concept of unified program configuration, which is generated by configuring a series of parameters from development to deployment in the `application.yml`according to a specific format a general configuration template, so that the initialization of the environment can be completed by transferring the configuration of the project to the program when the program is started. This is the concept of `configuration file`. +`Apache StreamPark` proposes the concept of unified program configuration, which is generated by configuring a series of parameters from development to deployment in the `application.yml`according to a specific format a general configuration template, so that the initialization of the environment can be completed by transferring the configuration of the project to the program when the program is started. This is the concept of `configuration file`. -`StreamPark` provides a higher level of abstraction for the `Flink SQL`, developers only need to define SQL to `sql.yaml`, when the program is started, the `sql.yaml` is transferred to the main program, and the SQL will be automatically loaded and executed. This is the concept of `sql file`. +`Apache StreamPark` provides a higher level of abstraction for the `Flink SQL`, developers only need to define SQL to `sql.yaml`, when the program is started, the `sql.yaml` is transferred to the main program, and the SQL will be automatically loaded and executed. This is the concept of `sql file`. ## Terms @@ -112,7 +112,7 @@ The SQL extracted in Flink SQL task is put into `sql.yaml`, this file with speci ## Configuration file -In StreamPark, the configuration file of `DataStream` job and `Flink Sql` are common. In other words, this configuration file can define the configurations of `DataStream` and `Flink Sql` (the configuration file in Flink SQL job is optional). The format of the configuration file must be `yaml` and must meet the requirements of yaml. +In Apache StreamPark, the configuration file of `DataStream` job and `Flink Sql` are common. In other words, this configuration file can define the configurations of `DataStream` and `Flink Sql` (the configuration file in Flink SQL job is optional). The format of the configuration file must be `yaml` and must meet the requirements of yaml. How to configure this configuration file and what to pay attention to. @@ -127,7 +127,7 @@ flink: jobmanager: property: #@see: https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html $internal.application.main: org.apache.streampark.flink.quickstart.QuickStartApp - pipeline.name: StreamPark QuickStart App + pipeline.name: Apache StreamPark QuickStart App yarn.application.queue: taskmanager.numberOfTaskSlots: 1 parallelism.default: 2 @@ -227,7 +227,7 @@ There are many basic parameters. The five most basic parameters are as follows. `$internal.application.main` and `pipeline.name` must be set. ::: -If you need to set more parameters, please refer to [`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html), These parameters must be placed under the property and the parameter names must be correct. StreamPark will automatically resolve these parameters and take effect. +If you need to set more parameters, please refer to [`here`](https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html), These parameters must be placed under the property and the parameter names must be correct. Apache StreamPark will automatically resolve these parameters and take effect. ##### Memory parameters @@ -399,7 +399,7 @@ sql: | :::danger attention -In the above content, | after SQL is required. In addition, | will retain the format of the whole section. StreamPark can directly define multiple SQLs at once. Each SQLs must be separated by semicolons, and each section of SQLs must follow the format and specification specified by Flink SQL. +In the above content, | after SQL is required. In addition, | will retain the format of the whole section. Apache StreamPark can directly define multiple SQLs at once. Each SQLs must be separated by semicolons, and each section of SQLs must follow the format and specification specified by Flink SQL. ::: ## Summary diff --git a/docs/development/model.md b/docs/development/model.md index 809e61d95..06a8130de 100644 --- a/docs/development/model.md +++ b/docs/development/model.md @@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem'; There are some rules and conventions to be followed in any framework. Only by following and mastering these rules can we use them more easily and achieve twice the result with half the effort.When we develop Flink job, we actually use the API provided by Flink to write an executable program (which must have a `main()` function) according to the development method required by Flink. We access various`Connector`in the program, and after a series of `operator`operations, we finally sink the data to the target storage through the `Connector` . -We call this method of step-by-step programming according to certain agreed rules the "programming paradigm". In this chapter, we will talk about the "programming paradigm" of StreamPark and the development considerations. +We call this method of step-by-step programming according to certain agreed rules the "programming paradigm". In this chapter, we will talk about the "programming paradigm" of Apache StreamPark and the development considerations. Let's start from these aspects @@ -25,11 +25,11 @@ Let's start from these aspects  ## Programming paradigm -`streampark-core` is positioned as a programming time framework, rapid development scaffolding, specifically created to simplify Flink development. Developers will use this module during the development phase. Let's take a look at what the programming paradigm of `DataStream` and `Flink Sql` with StreamPark looks like, and what the specifications and requirements are. +`streampark-core` is positioned as a programming time framework, rapid development scaffolding, specifically created to simplify Flink development. Developers will use this module during the development phase. Let's take a look at what the programming paradigm of `DataStream` and `Flink Sql` with Apache StreamPark looks like, and what the specifications and requirements are. ### DataStream -StreamPark provides both `scala` and `Java` APIs to develop `DataStream` programs, the specific code development is as follows. +Apache StreamPark provides both `scala` and `Java` APIs to develop `DataStream` programs, the specific code development is as follows.
@@ -224,7 +224,7 @@ class StreamingContext(val parameter: ParameterTool, private val environment: St
This object is very important and will be used throughout the lifecycle of the task in the `DataStream` job. The `StreamingContext` itself inherits from the `StreamExecutionEnvironment`, and the configuration file is fully integrated into the `StreamingContext`, so that it is very easy to get various parameters from the `StreamingContext`.
:::
-In StreamPark, `StreamingContext` is also the entry class for the Java API to write `DataStream` jobs, one of the constructors of `StreamingContext` is specially built for the Java API, the constructor is defined as follows:
+In Apache StreamPark, `StreamingContext` is also the entry class for the Java API to write `DataStream` jobs, one of the constructors of `StreamingContext` is specially built for the Java API, the constructor is defined as follows:
```scala
/**
@@ -301,7 +301,7 @@ class TableContext(val parameter: ParameterTool,
}
```
-In StreamPark, `TableContext` is also the entry class for the Java API to write `Table Sql` jobs of type `TableEnvironment`. One of the constructor methods of `TableContext` is a constructor specifically built for the `Java API`, which is defined as follows:
+In Apache StreamPark, `TableContext` is also the entry class for the Java API to write `Table Sql` jobs of type `TableEnvironment`. One of the constructor methods of `TableContext` is a constructor specifically built for the `Java API`, which is defined as follows:
```scala
@@ -391,7 +391,7 @@ class StreamTableContext(val parameter: ParameterTool,
```
-In StreamPark, `StreamTableContext` is the entry class for the Java API to write `Table Sql` jobs of type `StreamTableEnvironment`. One of the constructors of `StreamTableContext` is a function built specifically for the Java API, which is defined as follows:
+In Apache StreamPark, `StreamTableContext` is the entry class for the Java API to write `Table Sql` jobs of type `StreamTableEnvironment`. One of the constructors of `StreamTableContext` is a function built specifically for the Java API, which is defined as follows:
```scala
@@ -526,7 +526,7 @@ The **destroy** stage is an optional stage that requires developer participation
## Catalog Structure
-The recommended project directory structure is as follows, please refer to the directory structure and configuration in [StreamPark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart)
+The recommended project directory structure is as follows, please refer to the directory structure and configuration in [Apache StreamPark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart)
``` tree
.
@@ -602,11 +602,11 @@ assembly.xml is the configuration file needed for the assembly packaging plugin,
## Packaged Deployment
-The recommended packaging mode in [streampark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart/tree/dev/quickstart-flink) is recommended. It runs `maven package` directly to generate a standard StreamPark recommended project package, after unpacking the directory structure is as follows.
+The recommended packaging mode in [streampark-flink-quickstart](https://github.com/apache/incubator-streampark-quickstart/tree/dev/quickstart-flink) is recommended. It runs `maven package` directly to generate a standard Apache StreamPark recommended project package, after unpacking the directory structure is as follows.
``` text
.
-StreamPark-flink-quickstart-1.0.0
+Apache StreamPark-flink-quickstart-1.0.0
├── bin
│ ├── startup.sh //Launch Script
│ ├── setclasspath.sh //Java environment variable-related scripts (used internally, not of concern to users)
@@ -616,7 +616,7 @@ StreamPark-flink-quickstart-1.0.0
│ ├── application.yaml //Project's configuration file
│ ├── sql.yaml // flink sql file
├── lib
-│ └── StreamPark-flink-quickstart-1.0.0.jar //The project's jar package
+│ └── Apache StreamPark-flink-quickstart-1.0.0.jar //The project's jar package
└── temp
```
diff --git a/docs/flink-k8s/1-deployment.md b/docs/flink-k8s/1-deployment.md
index 0701713a3..b2698eda3 100644
--- a/docs/flink-k8s/1-deployment.md
+++ b/docs/flink-k8s/1-deployment.md
@@ -5,22 +5,22 @@ sidebar_position: 1
---
-StreamPark Flink Kubernetes is based on [Flink Native Kubernetes](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes/) and support deployment modes as below：
+Apache StreamPark Flink Kubernetes is based on [Flink Native Kubernetes](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/resource-providers/native_kubernetes/) and support deployment modes as below：
* Native-Kubernetes Application
* Native-Kubernetes Session
-At now, one StreamPark only supports one Kubernetes cluster.You can submit [Fearure Request Issue](https://github.com/apache/incubator-streampark/issues) , when multiple Kubernetes clusters are needed.
+At now, one Apache StreamPark only supports one Kubernetes cluster.You can submit [Fearure Request Issue](https://github.com/apache/incubator-streampark/issues) , when multiple Kubernetes clusters are needed.
## Environments requirement -Additional operating environment to run StreamPark Flink-K8s is as below: +Additional operating environment to run Apache StreamPark Flink-K8s is as below: * Kubernetes -* Maven（StreamPark runNode） -* Docker（StreamPark runNode） +* Maven（Apache StreamPark runNode） +* Docker（Apache StreamPark runNode） -StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed on node out of Kubernetes cluster when there are **smooth network** between the node and cluster. +Apache StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed on node out of Kubernetes cluster when there are **smooth network** between the node and cluster.
@@ -29,7 +29,7 @@ StreamPark entity can be deployed on Kubernetes nodes, and can also be deployed ### configuration for connecting Kubernetes -StreamPark connects to Kubernetes cluster by default connection credentials `～/.kube/config `.User can copy `.kube/config` from Kubernetes node to StreamPark nodes,or download it from Kubernetes provided by cloud service providers.If considering Permission constraints, User also can +Apache StreamPark connects to Kubernetes cluster by default connection credentials `～/.kube/config `.User can copy `.kube/config` from Kubernetes node to Apache StreamPark nodes,or download it from Kubernetes provided by cloud service providers.If considering Permission constraints, User also can generate custom account`s configuration by themselves. ```shell @@ -51,12 +51,12 @@ kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit ### Configuration for remote Docker service -On Setting page of StreamPark, user can configure the connection information for Docker service of Kubernetes cluster. +On Setting page of Apache StreamPark, user can configure the connection information for Docker service of Kubernetes cluster.  -Building a Namespace named `streampark`(other name should be set at Setting page of StreamPark) at remote Docker.The namespace is push/pull space of StreamPark Flink image and Docker Register User should own `pull`/`push` permission of this namespace. +Building a Namespace named `streampark`(other name should be set at Setting page of Apache StreamPark) at remote Docker.The namespace is push/pull space of Apache StreamPark Flink image and Docker Register User should own `pull`/`push` permission of this namespace. ```shell @@ -81,9 +81,9 @@ parameter descriptions are as below： * **Flink Base Docker Image**： Base Flink Docker Image Tag can be obtained from [DockerHub - offical/flink](https://hub.docker.com/_/flink) .And user can also use private image when Docker Register Account owns `pull` permission of it. * **Rest-Service Exposed Type**：Description of candidate values for native Flink K8s configuration [kubernetes.rest-service.exposed.type](https://ci.apache.org/projects/flink/flink-docs-stable/docs/deployment/config/#kubernetes) ： - * `ClusterIP`：ip that StreamPark can access； - * `LoadBalancer`：resource of LoadBalancer should be allocated in advance， Flink Namespace own permission of automatic binding，and StreamPark can access LoadBalancer`s gateway； - * `NodePort`：StreamPark can access all K8s nodes； + * `ClusterIP`：ip that Apache StreamPark can access； + * `LoadBalancer`：resource of LoadBalancer should be allocated in advance， Flink Namespace own permission of automatic binding，and Apache StreamPark can access LoadBalancer`s gateway； + * `NodePort`：Apache StreamPark can access all K8s nodes； * **Kubernetes Pod Template**： It`s Flink custom configuration of pod-template.The container-name must be flink-main-container. If the k8s pod needs a secret key to pull the docker image, please fill in the information about the secret key in the pod template file.The example pod-template is as below： @@ -114,7 +114,7 @@ The additional configuration of Flink-Native-Kubernetes Session Job will be deci ## other configuration -StreamPark parameter related to Flink-K8s in `applicaton.yml` are as below.And in most condition, it is no need to change it. +Apache StreamPark parameter related to Flink-K8s in `applicaton.yml` are as below.And in most condition, it is no need to change it. | Configuration item | Description | Default value | |-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|---------------| diff --git a/docs/flink-k8s/2-k8s-pvc-integration.md b/docs/flink-k8s/2-k8s-pvc-integration.md index 3ed1d97f0..e2bb50cd3 100755 --- a/docs/flink-k8s/2-k8s-pvc-integration.md +++ b/docs/flink-k8s/2-k8s-pvc-integration.md @@ -8,7 +8,7 @@ sidebar_position: 2 The support for pvc resource(mount file resources such as checkpoint/savepoint/logs and so on) is based on pod-template at current version。 -Users do not have to concern the Native-Kubernetes Session.It will be processed when Session Cluster is constructed .Native-Kubernetes Application can be constructed by configuring on StreamPark webpage using `pod-template`、`jm-pod-template`、`tm-pod-template`. +Users do not have to concern the Native-Kubernetes Session.It will be processed when Session Cluster is constructed .Native-Kubernetes Application can be constructed by configuring on Apache StreamPark webpage using `pod-template`、`jm-pod-template`、`tm-pod-template`.
@@ -45,9 +45,9 @@ There are three ways to provide the dependency when using `rocksdb-backend`. 1. Flink Base Docker Image contains the dependency（user fix the dependency conflict by themself）; -2. Put the dependency `flink-statebackend-rocksdb_xx.jar` to the path `Workspace/jars` in StreamPark ; +2. Put the dependency `flink-statebackend-rocksdb_xx.jar` to the path `Workspace/jars` in Apache StreamPark ; -3. Add the rockdb-backend dependency to StreamPark Dependency(StreamPark will fix the conflict automatically) ： +3. Add the rockdb-backend dependency to Apache StreamPark Dependency(Apache StreamPark will fix the conflict automatically) ：  diff --git a/docs/flink-k8s/3-hadoop-resource-integration.md b/docs/flink-k8s/3-hadoop-resource-integration.md index c03274d18..90bd0b20d 100644 --- a/docs/flink-k8s/3-hadoop-resource-integration.md +++ b/docs/flink-k8s/3-hadoop-resource-integration.md @@ -6,7 +6,7 @@ sidebar_position: 3 ## Using Hadoop resource in Flink on K8s -Using Hadoop resources under the StreamPark Flink-K8s runtime, such as checkpoint mount HDFS, read and write Hive, etc. The general process is as follows: +Using Hadoop resources under the Apache StreamPark Flink-K8s runtime, such as checkpoint mount HDFS, read and write Hive, etc. The general process is as follows: #### 1、HDFS @@ -26,7 +26,7 @@ flink-json-1.14.5.jar log4j-1.2-api-2.17.1.jar log4j-slf4j-impl- This is to download the shaded jar and put it in the lib directory of flink. Take hadoop2 as an example, download `flink-shaded-hadoop-2-uber`：https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-9.0/flink-shaded-hadoop-2-uber-2.7.5-9.0.jar - In addition, you can configure the shade jar in a dependent manner in the `Dependency` in the StreamPark task configuration. the following configuration: + In addition, you can configure the shade jar in a dependent manner in the `Dependency` in the Apache StreamPark task configuration. the following configuration: ```xml
@@ -34,9 +34,9 @@ can view or operate the applications of the corresponding team. ## Role Management In order to facilitate application management and prevent misoperation, the team also needs to distinguish between -administrator and developer, so StreamPark introduces role management. +administrator and developer, so Apache StreamPark introduces role management. -Currently, StreamPark supports two roles: team admin and developer. The team admin has all the permissions in the team. +Currently, Apache StreamPark supports two roles: team admin and developer. The team admin has all the permissions in the team. Compared with the team admin, the developer has fewer permissions to delete applications and add USER to the team.
diff --git a/docs/user-guide/8-YarnQueueManagement.md b/docs/user-guide/8-YarnQueueManagement.md index 5dcf6bd51..e8d44a7f2 100644 --- a/docs/user-guide/8-YarnQueueManagement.md +++ b/docs/user-guide/8-YarnQueueManagement.md @@ -19,7 +19,7 @@ will be time-consuming and accompanied by poor user experience. If a task is submitted to an incorrect queue due to the error-input, it's likely to affect the stability of yarn applications on the queue and the abuse of the queue resource. -So StreamPark introduced the queue management feature to ensure that a set of added queues are shared within the same team, +So Apache StreamPark introduced the queue management feature to ensure that a set of added queues are shared within the same team, that is, ensure that queues resource is isolated within the scope of the team. It can generate the following benefits: - When deploying Flink `yarn-application` applications or Flink `yarn-session` clusters, it could set quickly and accurately yarn queue(`yarn.application.queue`) & labels(`yarn.application.node-label`). @@ -88,13 +88,13 @@ due to the relationship between the queues and the `yarn-session` mode flink clu
- Session cluster is shared by all teams. Why is it that when creating a `yarn-session` flink cluster, only the queues in the current team instead of all teams can be used in the queues candidate list ? -> Based on the mentioned above, StreamPark hopes that when creating a `yarn-session` flink cluster, +> Based on the mentioned above, Apache StreamPark hopes that when creating a `yarn-session` flink cluster, administrators can specify the queue belonged to current of the current team only, which could be better for administrators to perceive the impact of current operations on the current team. - Why not support the isolation for `flink yarn-session clusters / general clusters` on team wide ? - The impact range caused by changes in cluster visibility is larger than that caused by changes in queue visibility. - - StreamPark need to face greater difficulties in backward compatibility while also considering the user experience. + - Apache StreamPark need to face greater difficulties in backward compatibility while also considering the user experience. - At present, there is no exact research on the users group and applications scale deployed using `yarn-application` & `yarn-session` cluster modes in the community. Based on this fact, the community didn't provide greater feature support. diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md b/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md index 39e96328f..fa2f4fc54 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/0-streampark-flink-on-k8s.md @@ -1,7 +1,7 @@ --- slug: streampark-flink-on-k8s -title: StreamPark Flink on Kubernetes 实践 -tags: [StreamPark, 生产实践, FlinkSQL, Kubernetes] +title: Apache StreamPark Flink on Kubernetes 实践 +tags: [Apache StreamPark, 生产实践, FlinkSQL, Kubernetes] --- 雾芯科技创立于2018年1月。目前主营业务包括 RELX 悦刻品牌产品的研发、设计、制造及销售。凭借覆盖全产业链的核心技术与能力，RELX 悦刻致力于为用户提供兼具高品质和安全性的产品。 @@ -20,9 +20,9 @@ Native Kubernetes 和 Standalone Kubernetes 主要区别在于 Flink 与 Kuberne  -## **当 Flink On Kubernetes 遇见 StreamPark** +## **当 Flink On Kubernetes 遇见 Apache StreamPark** -Flink on Native Kubernetes 目前支持 Application 模式和 Session 模式，两者对比 Application 模式部署规避了 Session 模式的资源隔离问题、以及客户端资源消耗问题，因此**生产环境更推荐采用 Application Mode 部署 Flink 任务。**下面我们分别看看使用原始脚本的方式和使用 StreamPark 开发部署一个 Flink on Native Kubernetes 作业的流程。 +Flink on Native Kubernetes 目前支持 Application 模式和 Session 模式，两者对比 Application 模式部署规避了 Session 模式的资源隔离问题、以及客户端资源消耗问题，因此**生产环境更推荐采用 Application Mode 部署 Flink 任务。**下面我们分别看看使用原始脚本的方式和使用 Apache StreamPark 开发部署一个 Flink on Native Kubernetes 作业的流程。 ### ***使用脚本方式部署Kubernetes*** @@ -69,17 +69,17 @@ kubectl -n flink-cluster get svc 以上就是使用 Flink 提供的最原始的脚本方式把一个 Flink 任务部署到 Kubernetes 上的过程，只做到了最基本的任务提交，如果要达到生产使用级别，还有一系列的问题需要解决，如：方式过于原始无法适配大批量任务、无法记录任务checkpoint 和实时状态跟踪、任务运维和监控困难、无告警机制、 无法集中化管理等等。 -## **使用 StreamPark 部署 Flink on Kubernetes** +## **使用 Apache StreamPark 部署 Flink on Kubernetes** ------ 对于企业级生产环境使用 Flink on Kubernetes 会有着更高的要求, 一般会选择自建平台或者购买相关商业产品, 不论哪种方案在产品能力上满足: **大批量任务开发部署、状态跟踪、运维监控、失败告警、任务统一管理、高可用性** 等这些是普遍的诉求。 -针对以上问题我们调研了开源领域支持开发部署 Flink on Kubernetes 任务的开源项目，调研的过程中也不乏遇到了其他优秀的开源项目，在综合对比了多个开源项目后得出结论: **StreamPark 不论是完成度还是使用体验、稳定性等整体表现都非常出色，因此最终选择了 StreamPark 作为我们的一站式实时计算平台。**下面我们看看 StreamPark 是如何支持 Flink on Kubernetes : +针对以上问题我们调研了开源领域支持开发部署 Flink on Kubernetes 任务的开源项目，调研的过程中也不乏遇到了其他优秀的开源项目，在综合对比了多个开源项目后得出结论: **Apache StreamPark 不论是完成度还是使用体验、稳定性等整体表现都非常出色，因此最终选择了 Apache StreamPark 作为我们的一站式实时计算平台。**下面我们看看 Apache StreamPark 是如何支持 Flink on Kubernetes : ### **基础环境配置** -基础环境配置包括 Kubernetes 和 Docker 仓库信息以及 Flink 客户端的信息配置。对于 Kubernetes 基础环境最为简单的方式是直接拷贝 Kubernetes 节点的 .kube/config 到 StreamPark 节点用户目录，之后使用 kubectl 命令创建 Flink 专用的 Kubernetes Namespace 以及进行 RBAC 配置。 +基础环境配置包括 Kubernetes 和 Docker 仓库信息以及 Flink 客户端的信息配置。对于 Kubernetes 基础环境最为简单的方式是直接拷贝 Kubernetes 节点的 .kube/config 到 Apache StreamPark 节点用户目录，之后使用 kubectl 命令创建 Flink 专用的 Kubernetes Namespace 以及进行 RBAC 配置。 ```shell # 创建Flink作业使用的k8s namespace @@ -92,17 +92,17 @@ Docker 账户信息直接在 Docker Setting 界面配置即可：  -StreamPark 可适配多版本 Flink 作业开发，Flink 客户端直接在 StreamPark Setting 界面配置即可： +Apache StreamPark 可适配多版本 Flink 作业开发，Flink 客户端直接在 Apache StreamPark Setting 界面配置即可：  ### **作业开发** -StreamPark 做好基础环境配置之后只需要三步即可开发部署一个 Flink 作业: +Apache StreamPark 做好基础环境配置之后只需要三步即可开发部署一个 Flink 作业:  -StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink SQL 作业只需要输入SQL 和 依赖项即可, 该方式大大提升了开发体验,** **并且规避了依赖冲突等问题，**对此部分本文不重点介绍。 +Apache StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink SQL 作业只需要输入SQL 和 依赖项即可, 该方式大大提升了开发体验,** **并且规避了依赖冲突等问题，**对此部分本文不重点介绍。 这里需要选择部署模式为 kubernetes application, 并且需要在作业开发页面进行以下参数的配置：红框中参数为 Flink on Kubernetes 基础参数。 @@ -118,7 +118,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ### **作业上线** -作业开发完成之后是作业上线环节，在这一环节中 StreamPark 做了大量的工作，具体如下: +作业开发完成之后是作业上线环节，在这一环节中 Apache StreamPark 做了大量的工作，具体如下: - 准备环境 - 作业中的依赖下载 @@ -130,7 +130,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink  -在镜像构建和推送的时候我们可以看到 StreamPark 做的一系列工作: **读取配置、构建镜像、推送镜像到远程仓库...** 这里要给StreamPark 一个大大的赞！ +在镜像构建和推送的时候我们可以看到 Apache StreamPark 做的一系列工作: **读取配置、构建镜像、推送镜像到远程仓库...** 这里要给Apache StreamPark 一个大大的赞！  @@ -140,19 +140,19 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink  -整个过程仅需上述三步，即可完成在 StreamPark 上开发和部署一个Flink on Kubernetes 作业。而 StreamPark 对于 Flink on Kubernetes 的支持远远不止提交个任务这么简单。 +整个过程仅需上述三步，即可完成在 Apache StreamPark 上开发和部署一个Flink on Kubernetes 作业。而 Apache StreamPark 对于 Flink on Kubernetes 的支持远远不止提交个任务这么简单。 ### **作业管理** -**在作业提交之后，StreamPark 能实时获取到任务的最新 checkpoint 地址、任务的运行状态、集群实时的资源消耗信息，针对运行的任务可以非常方便的一键启停, 在停止作业时支持记录 savepoint 的位置, 以及再次启动时从 savepoint 恢复状态等功能，从而保证了生产环境的数据一致性，真正具备 Flink on Kubernetes 的 一站式开发、部署、运维监控的能力。** +**在作业提交之后，Apache StreamPark 能实时获取到任务的最新 checkpoint 地址、任务的运行状态、集群实时的资源消耗信息，针对运行的任务可以非常方便的一键启停, 在停止作业时支持记录 savepoint 的位置, 以及再次启动时从 savepoint 恢复状态等功能，从而保证了生产环境的数据一致性，真正具备 Flink on Kubernetes 的 一站式开发、部署、运维监控的能力。** -接下来我们来看看这一块的能力 StreamPark 是如何进行支持的: +接下来我们来看看这一块的能力 Apache StreamPark 是如何进行支持的: - **实时记录checkpoint** ------ -在作业提交之后，有时候需要更改作业逻辑但是要保证数据的一致性，那么就需要平台具有实时记录每一次 checkpoint 位置的能力, 以及停止时记录最后的 savepoint 位置的能力，StreamPark 在 Flink on Kubernetes 上很好的实现了该功能。默认会每隔5秒获取一次 checkpoint 信息记录到对应的表中，并且会按照 Flink 中保留 checkpoint 数量的策略，只保留 state.checkpoints.num-retained 个，超过的部分则删除。在任务停止时有勾选 savepoint 的选项，如勾选了savepoint 选项，在任务停止时会做 savepoint 操作，同样会记录 savepoint 具体位置到表中。 +在作业提交之后，有时候需要更改作业逻辑但是要保证数据的一致性，那么就需要平台具有实时记录每一次 checkpoint 位置的能力, 以及停止时记录最后的 savepoint 位置的能力，Apache StreamPark 在 Flink on Kubernetes 上很好的实现了该功能。默认会每隔5秒获取一次 checkpoint 信息记录到对应的表中，并且会按照 Flink 中保留 checkpoint 数量的策略，只保留 state.checkpoints.num-retained 个，超过的部分则删除。在任务停止时有勾选 savepoint 的选项，如勾选了savepoint 选项，在任务停止时会做 savepoint 操作，同样会记录 savepoint 具体位置到表中。 默认 savepoint 的根路径只需要在 Flink Home flink-conf.yaml 文件中配置即可自动识别，除了默认地址，在停止时也可以自定义指定 savepoint 的根路径。 @@ -164,7 +164,7 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ------ -对于生产环境的挑战，很重要的一点就是监控是否到位，Flink on Kubernetes 更是如此。这点很重要, 也是最基本需要具备的能力，StreamPark 可实时监控 Flink on Kubernetes 作业的运行状态并在平台上展示给用户，在页面上可以很方便的根据各种运行状态来检索任务。 +对于生产环境的挑战，很重要的一点就是监控是否到位，Flink on Kubernetes 更是如此。这点很重要, 也是最基本需要具备的能力，Apache StreamPark 可实时监控 Flink on Kubernetes 作业的运行状态并在平台上展示给用户，在页面上可以很方便的根据各种运行状态来检索任务。  @@ -172,21 +172,21 @@ StreamPark 既支持 Upload Jar 也支持直接编写 Flink SQL 作业, **Flink ------ -除此之外 StreamPark 还具备完善的报警功能: 支持邮件、钉钉、微信和短信 等。这也是当初公司调研之后选择 StreamPark 作为 Flink on Kubernetes 一站式平台的重要原因。 +除此之外 Apache StreamPark 还具备完善的报警功能: 支持邮件、钉钉、微信和短信 等。这也是当初公司调研之后选择 Apache StreamPark 作为 Flink on Kubernetes 一站式平台的重要原因。  -通过以上我们看到 StreamPark 在支持 Flink on Kubernetes 开发部署过程中具备的能力, 包括：**作业的开发能力、部署能力、监控能力、运维能力、异常处理能力等，StreamPark 提供的是一套相对完整的解决方案。 且已经具备了一些 CICD/DevOps 的能力，整体的完成度还在持续提升。是在整个开源领域中对于 Flink on Kubernetes 一站式开发部署运维工作全链路都支持的产品，StreamPark 是值得被称赞的。** +通过以上我们看到 Apache StreamPark 在支持 Flink on Kubernetes 开发部署过程中具备的能力, 包括：**作业的开发能力、部署能力、监控能力、运维能力、异常处理能力等，Apache StreamPark 提供的是一套相对完整的解决方案。 且已经具备了一些 CICD/DevOps 的能力，整体的完成度还在持续提升。是在整个开源领域中对于 Flink on Kubernetes 一站式开发部署运维工作全链路都支持的产品，Apache StreamPark 是值得被称赞的。** -## **StreamPark 在雾芯科技的落地实践** +## **Apache StreamPark 在雾芯科技的落地实践** -StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作业和实时指标计算作业的开发部署，有 Jar 任务也有 Flink SQL 任务，全部使用 Native Kubernetes 部署；数据源有CDC、Kafka 等，Sink 端有 Maxcompute、kafka、Hive 等，以下是公司开发环境StreamPark 平台截图： +Apache StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作业和实时指标计算作业的开发部署，有 Jar 任务也有 Flink SQL 任务，全部使用 Native Kubernetes 部署；数据源有CDC、Kafka 等，Sink 端有 Maxcompute、kafka、Hive 等，以下是公司开发环境Apache StreamPark 平台截图：  ## 遇到的问题 -任何新技术都有探索与踩坑的过程，失败的经验是宝贵的，这里介绍下 StreamPark 在雾芯科技落地过程中踩的一些坑和经验，**这块的内容不仅仅关于 StreamPark 的部分, 相信会带给所有使用 Flink on Kubernetes 的小伙伴一些参考**。 +任何新技术都有探索与踩坑的过程，失败的经验是宝贵的，这里介绍下 Apache StreamPark 在雾芯科技落地过程中踩的一些坑和经验，**这块的内容不仅仅关于 Apache StreamPark 的部分, 相信会带给所有使用 Flink on Kubernetes 的小伙伴一些参考**。 ### **常见问题总结如下** @@ -196,7 +196,7 @@ StreamPark 在雾芯科技落地较晚，目前主要用于实时数据集成作 - **scala 版本不一致** -由于 StreamPark 部署需要 Scala 环境，而且 Flink SQL 运行需要用到 StreamPark 提供的 Flink SQL Client，因此一定要保证 Flink 作业的 Scala 版本和 StreamPark 的 Scala 版本保持一致。 +由于 Apache StreamPark 部署需要 Scala 环境，而且 Flink SQL 运行需要用到 Apache StreamPark 提供的 Flink SQL Client，因此一定要保证 Flink 作业的 Scala 版本和 Apache StreamPark 的 Scala 版本保持一致。 - **注意类冲突** @@ -228,7 +228,7 @@ HDFS 阿里云 OSS/AWS S3 都可以进行 checkpoint 和 savepoint 存储，Flin - **任务每次重启都会导致多出一个 Job 实例** -在配置了基于 kubernetes 的HA的前提条件下，当需要停止 Flink 任务时，需要通过 StreamPark 的 cancel 来进行，不要直接通过 kubernetes 集群删除 Flink 任务的 Deployment。因为 Flink 的关闭有其自有的关闭流程，在删除 pod 同时 Configmap 中的相应配置文件也会被一并删除，而直接删除 pod 会导致 Configmap 的残留。当相同名称的任务重启时，会出现两个相同 Job 现象，因为在启动时，任务会加载之前残留的配置文件，尝试将已经关闭的任务恢复。 +在配置了基于 kubernetes 的HA的前提条件下，当需要停止 Flink 任务时，需要通过 Apache StreamPark 的 cancel 来进行，不要直接通过 kubernetes 集群删除 Flink 任务的 Deployment。因为 Flink 的关闭有其自有的关闭流程，在删除 pod 同时 Configmap 中的相应配置文件也会被一并删除，而直接删除 pod 会导致 Configmap 的残留。当相同名称的任务重启时，会出现两个相同 Job 现象，因为在启动时，任务会加载之前残留的配置文件，尝试将已经关闭的任务恢复。 - **kubernetes pod 域名访问怎么实现** @@ -300,14 +300,14 @@ push k8s-harbor.xxx.com/streamx/udf_flink_1.13.6-scala_2.11:latest ## **未来期待** -- **StreamPark 对于 Flink 作业 Metric 监控的支持** +- **Apache StreamPark 对于 Flink 作业 Metric 监控的支持** -StreamPark 如果可以对接 Flink Metric 数据而且可以在 StreamPark 平台上展示每时每刻 Flink 的实时消费数据情况就太棒了 +Apache StreamPark 如果可以对接 Flink Metric 数据而且可以在 Apache StreamPark 平台上展示每时每刻 Flink 的实时消费数据情况就太棒了 -- **StreamPark 对于Flink 作业日志持久化的支持** +- **Apache StreamPark 对于Flink 作业日志持久化的支持** -对于部署到 YARN 的 Flink 来说，如果 Flink 程序挂了，我们可以去 YARN 上看历史日志，但是对于 Kubernetes 来说，如果程序挂了，那么 Kubernetes 的 pod 就消失了，就没法查日志了。所以用户需要借助 Kubernetes 上的工具进行日志持久化，如果 StreamPark 支持 Kubernetes 日志持久化接口就更好了。 +对于部署到 YARN 的 Flink 来说，如果 Flink 程序挂了，我们可以去 YARN 上看历史日志，但是对于 Kubernetes 来说，如果程序挂了，那么 Kubernetes 的 pod 就消失了，就没法查日志了。所以用户需要借助 Kubernetes 上的工具进行日志持久化，如果 Apache StreamPark 支持 Kubernetes 日志持久化接口就更好了。 - **镜像过大的问题改进** -StreamPark 目前对于 Flink on Kubernetes 作业的镜像支持是将基础镜像和用户代码打成一个 Fat 镜像推送到 Docker 仓库，这种方式存在的问题就是镜像过大的时候耗时比较久，希望未来基础镜像可以复用不需要每次都与业务代码打到一起，这样可以极大地提升开发效率和节约成本。 +Apache StreamPark 目前对于 Flink on Kubernetes 作业的镜像支持是将基础镜像和用户代码打成一个 Fat 镜像推送到 Docker 仓库，这种方式存在的问题就是镜像过大的时候耗时比较久，希望未来基础镜像可以复用不需要每次都与业务代码打到一起，这样可以极大地提升开发效率和节约成本。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md b/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md index 3604cdf9f..381622535 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/1-flink-framework-streampark.md @@ -1,7 +1,7 @@ --- slug: flink-development-framework-streampark -title: Flink 开发利器 StreamPark -tags: [StreamPark, DataStream, FlinkSQL] +title: Flink 开发利器 Apache StreamPark +tags: [Apache StreamPark, DataStream, FlinkSQL] ---
@@ -58,18 +58,18 @@ Flink 从 1.13 版本开始，就支持 Pod Template，我们可以在 Pod Templ
-## 引入 StreamPark +## 引入 Apache StreamPark 之前我们写 Flink SQL 基本上都是使用 Java 包装 SQL，打 jar 包，提交到 S3 平台上。通过命令行方式提交代码，但这种方式始终不友好，流程繁琐，开发和运维成本太大。我们希望能够进一步简化流程，将 Flink TableEnvironment 抽象出来，有平台负责初始化、打包运行 Flink 任务，实现 Flink 应用程序的构建、测试和部署自动化。 -这是个开源兴起的时代，我们自然而然的将目光投向开源领域中：在一众开源项目中，经过对比各个项目综合评估发现 Zeppelin 和 StreamPark 这两个项目对 Flink 的支持较为完善，都宣称支持 Flink on K8s ，最终进入到我们的目标选择范围中，以下是两者在 K8s 相关支持的简单比较（目前如果有更新，麻烦批评指正）。 +这是个开源兴起的时代，我们自然而然的将目光投向开源领域中：在一众开源项目中，经过对比各个项目综合评估发现 Zeppelin 和 Apache StreamPark 这两个项目对 Flink 的支持较为完善，都宣称支持 Flink on K8s ，最终进入到我们的目标选择范围中，以下是两者在 K8s 相关支持的简单比较（目前如果有更新，麻烦批评指正）。
| 功能 | Zeppelin | -StreamPark | +Apache StreamPark |