I am unable to access the kedro spark session configuration from an ipython console.
# /conf/base/spark.yml
spark.driver.maxResultSize: 30g
spark.scheduler.mode: FAIR
spark.driver.memory: 15g
spark.executor.memory: 15g
spark.executor.cores: 4
# settings for the UI
spark.ui.port: 10
spark.ui.enabled: true
As you can see the spark.driver.maxResultSize is defined. after running this I get a NoSuchElementException. Any idea why?
kedro ipython
%reload_kedro
spark = catalog.datasets.mydataset__spark._get_spark()
spark.conf.get('spark.driver.maxResultSize')
# Py4JJavaError: An error occurred while calling o28.get.
# : java.util.NoSuchElementException: spark.driver.maxResultSize
also, spark.conf._jconf.getAll().toString() yields nothing related to maxResultSize
Out[23]: 'Map(spark.sql.warehouse.dir -> file:/home/spark-warehouse, spark.driver.host -> xxx.xxx, spark.serializer.objectStreamReset -> 100, spark.driver.port -> xxx, spark.rdd.compress -> True, spark.app.name -> pyspark-shell, spark.submit.pyFiles -> , spark.ui.showConsoleProgress -> true, spark.app.startTime -> 1705924636973, spark.executor.id -> driver, spark.submit.deployMode -> client, spark.master -> local[*], spark.app.id -> local-1705924638008)'