MS Fabric and geospatial data

Question

Can Microsoft Fabric's Data Lakehouse handle geospatial data such as shp, geojson, and other formats?

I've tried getting the geopandas geodataframe into pyspark/parquet spark_df.write.parquet(destination_path) but I can't seem to get the data in.

So, is it possible, and are there any good tutorials out there specifically for geospatial data?

Answer 1

First read the geospatial data from the shp or geojson file and convert it to spark dataframe. To do this use below code block.

import geopandas as gpd
gdf = gpd.read_file('/dbfs/json/points.geojson')
gdf['geom'] = [geom.wkt for geom in gdf['geometry']]
s_df = spark.createDataFrame(gdf.drop("geometry", axis=1,inplace=False))
s_df.printSchema()
s_df.show()

Next, writing data to onelake. Before writing make sure you enable Azure Data Lake Storage credential passthrough under Advance option.

Now, go to your one lake house > Files

create new folder. I've created a folder named geodata.

open properties of that folder and copy ABFS path.

Then run below code to write it.

oneLake = "<your_abfs_link>/Files/geodata"
s_df.write.format("parquet").mode("overwrite").save(oneLake)

It is written to onelake.

Again using the same link you can read it.

To convert geom column back to type geometry you can use below code.

from shapely.wkt import loads

odf = spark.read.parquet(oneLake)
p_df = odf.toPandas()
p_df['geometry'] = [loads(i) for i in p_df['geom']]
p_df = p_df.drop("geom",axis=1)
print(p_df.dtypes)

p_df