I want to build a lineage system for a real-time data warehouse，how can I extract table and column lineage from flink sql?
how to extract column&table lineage from flink sql
542 views Asked by LiJianing At
1
There are 1 answers
Related Questions in APACHE-FLINK
- Fine grained resource mangement and heap memory in flink task slot
- Does parallel flink tasks affect each other if they are unioned at the end?
- I am facing issue with ParquetFileWriting n hdfs in flink where parquet file size is around 382 KB . I want the parquet file in MB
- Apache Flink (AWS) does not recognize saved temporary function
- Flink 1.19 error Cannot determine simple type name "com"
- Unsupported options found for 'hudi'
- Flink 1.18 register custom API endpoint handler
- Flink Stuck on Broadcast
- Blunder about RichCoFlatMapFunction in flink 1.17.2 according to the official leanring guide
- Is there a way to store & retrieve a window's state in flink
- puzzled with flink window state
- Flink 1.15.2 OOM issue due to RocksDB
- How to create custom metrics with labels (python SDK + Flink Runner)
- flink-rpc-akka-loader - Security Vulnerability Issues
- I am new to Apache Flink and getting error FileNotFoundError: [WinError 2] at in_streaming_mode() The system cannot find the file specified
Related Questions in FLINK-STREAMING
- Fine grained resource mangement and heap memory in flink task slot
- Flink 1.19 error Cannot determine simple type name "com"
- Getting FlinkRuntime Exception during oracle exactly once jdbc sink
- Is there a way to store & retrieve a window's state in flink
- puzzled with flink window state
- Flink 1.15.2 OOM issue due to RocksDB
- If I emit an event from an operator after holding it in state for certain duration will the downstream operator accept it if it is past the watermark?
- How to write to Kafka Topic(Or to a file) from a Flink Stream
- Flink marks source late arriving events
- Why is flink UI not showing the right numbers?
- Union of bounded and unbounded streams in flink
- gRPC Connection Cancelled with "Multiplexer Hanging Up" Error in PyFlink
- Delta Lake as ingress for Flink Stateful Functions
- implement custom partitioning with windowAll()
- implementation of RoundRobin partitioning in Apache Flink
Related Questions in FLINK-SQL
- Flink 1.19 error Cannot determine simple type name "com"
- Unsupported options found for 'hudi'
- Flink - How to perform an aggregate function on all records resulting from JOIN and EXPLODE operations on the original record in streaming mode?
- How to Choose the Right Flink Join Type for Real-Time Data Warehouse Processing
- Exponential moving average with Flink SQL
- Using Flink SQL with Table Aggregate Function (UDTAGG)
- Apache Flink is not printing any new streaming data
- Flink SQL with Iceberg snapshot stream doesn't react if table has upsert
- Flink SQL OPTIONS for 'streaming'='true' doesn't work with JOIN
- How to read null value from Kafka using flink sql
- Flink SQL multiple over aggregations from a single source
- Flink SQL REGEXP_REPLACE Does Not Support Capturing Groups
- flink sql 1.12 how to get the time when window starts, and use this time caculate?
- Flink SQL Kafka Source Skip Failed Messages
- Getting nested fields in a Flink API Table fromDatastream
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
In addition to the answer in the Flink User Mailing List, I'd like to point out that there is ongoing work in the community to integrate Flink with Apache Atlas. This could be a more viable option for data governance than building your own custom implementation.
You can get some pointers in the initial discussion.