Override core-site.xml property value at spark-submit execution time
Is there a way to override core-site property value when executing spark submit?
I can use HADOOP_CONF_DIR variable in spark-env.sh to point to a new core-site file, but I only want to override few values to to use different storage accounts for each spark job.
Found answer to my own question.
hadoop-related configuration can be overridden by pre-fixing "spark.hadoop" to property key, then submit to sparkconf.
i.e. spark-submit --sparkconf spark.hadoop.io.file.buffer.size 12952
See source code here: https://github.com/apache/spark/commit/b6cf1348170951396a6a5d8a65fb670382304f5b