Apache Flink’s batch is already capable of supporting mass production, and the value that streaming batch brings to data development is clear, as we’ve heard from many business teams. In many practices, the main entry point is a kind of business scenario with both stream and batch requirements. In Shopee, the Flink team at Shopee Data Infra is exploring the potential value of stream and batch integration. On the one hand, we deeply integrate Flink with offline data system, including account and authority system, metadata system, scheduling and blood relationship management, etc. On the other hand, efforts are made to enhance the production capacity of Flink batch processing, including Remote Shuffle Service, Graph Config, Resource Profile, etc. In this way, many data development students in a platform can use unified code to complete the development and operation of all data processing requirements. In this talk, we will introduce these practical practices in detail.

Speakers:

Guichao Xiao: Shopee, Senior Expert Engineer, Shopee Data Infra real-time team Manager, engaged in engine development and mass production practice in the field of real-time computing of big Data for a long time. I used to be a member of Ali JStorm and Blink team, responsible for the application production of Flink in several leading e-commerce companies.