Data Lineage in Unity Catalog is not shown in lineage tab in databricks

34 views Asked by At

I have been trying to get the data lineage to populate with the simplest of queries on a unity enabled catalog with a unity enabled cluster. I am essentially running the example provided with more data to see how it works with various aggregates downstream.

However when I go to the lineage tab on the tables (which I have all privileges for the catalog and every schema/table underneath it) they are blank. There is history being collected and data is showing up in them.

I am running a cluster with 13.3 LTS and the following runtime settings enabled

CREATE TABLE IF NOT EXISTS
  unity_metastore.lineage_test.menu (
    recipe_id INT,
    app string,
    main string,
    dessert string
  )
  USING DELTA LOCATION '[external location]';

INSERT INTO unity_metastore.lineage_test.menu
    (recipe_id, app, main, dessert)
VALUES
    (1,"Ceviche", "Tacos", "Flan"),
    (2,"Tomato Soup", "Souffle", "Creme Brulee"),
    (3,"Chips","Grilled Cheese","Cheesecake");

CREATE TABLE IF NOT EXISTS
  unity_metastore.lineage_test.dinner (recipe_id INT,recipe_description string)
  USING DELTA LOCATION '[external location]';
INSERT INTO unity_metastore.lineage_test.dinner (recipe_id,recipe_description)
SELECT
  recipe_id, concat(app," + ", main," + ",dessert)
FROM
  unity_metastore.lineage_test.menu;

I expect lineage data in diagram and lineage items in lineage tab page of databricks

0

There are 0 answers