There are more and more tool frameworks in the field of data governance, data scheduling and quality control. As an enterprise application, how to organically organize a variety of tools into a unified platform has always been the area that Shang Silicon Valley is committed to exploring. As a communication organization dedicated to Internet technology, Silicon Valley has cooperated with a number of Internet companies and technical teams, combined with the current popular big data open source framework, integrated and created an open source data governance An integrated platform for data process scheduling, data quality monitoring and early warning. The functions of the platform include metadata management, blood relationship management, task scheduling, data quality monitoring and early warning, and data governance based on scoring and assessment. In addition to offline batch processing, it also integrates parts of real-time streaming processing, such as real-time task publishing and real-time task monitoring and warning. Many open source frameworks have been introduced in the platform, including Apache Dolphin Scheduler, StreamX, Prometheus and others. As a technology disseminator, Silicon Valley works with multiple open source teams and will share the features and usage of each major framework in this presentation.

Speakers:

Bryan Zhang,: Silicon Valley IT education, Project director, Shang Silicon Valley presided over the development of data management platform projects, user portrait, data warehouse and real-time data warehouse projects. He has been a technical lecturer for many years. He also participated in the first-line projects of State Grid Corporation of China, Citic Bank and Zhongzhi Group.