Spark2.4 Unable to overwrite table from same table

Question

I am trying to insert data into a table using insert overwrite statement but I am getting below error.

org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also being read from.;

command is as below

spark.sql("INSERT OVERWRITE TABLE edm_hive SELECT run_number+1 from edm_hive")

I am trying to use temp table, store the results and then update in final table but that is also not working.

Also I am trying to insert record into table using some variable but that is also not working. e.g.

spark.sql("INSERT into TABLE Feed_metadata_s2 values ('LOGS','StartTimestamp',$StartTimestamp)")

Please suggest

Answer 1

This solution works well for me. I added that property in sparkSession.

Spark HiveContext : Insert Overwrite the same table it is read from

val spark = SparkSession.builder()
      .config("spark.sql.hive.convertMetastoreParquet","false")