Override core-site.xml property value at spark-submit execution time

Is there a way to override core-site property value when executing spark submit?

I can use HADOOP_CONF_DIR variable in spark-env.sh to point to a new core-site file, but I only want to override few values to to use different storage accounts for each spark job.

Answers


Found answer to my own question.

hadoop-related configuration can be overridden by pre-fixing "spark.hadoop" to property key, then submit to sparkconf.

i.e. spark-submit --sparkconf spark.hadoop.io.file.buffer.size 12952

See source code here: https://github.com/apache/spark/commit/b6cf1348170951396a6a5d8a65fb670382304f5b


Need Your Help

Visual Studio, Team Explorer: Where are the work items

visual-studio azure-devops

With Visual Studio 2013 and 2015 we used to manage and query work items directly from Team Explorer.

Wait for shell command to complete

excel shell vba batch-file

I'm running a simple shell command in Excel VBA that runs a batch file in a specified directory like below: