spark-submit详细参数说明在spark命令⾏输⼊
英语巴士论坛./bin/spark-submit --help
可以看到spark-submit的所⽤参数如下：
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]
Usage: spark-submit --kill [submission ID] --master [spark://...]
Usage: spark-submit --status [submission ID] --master [spark://...]
Usage: spark-submit run-example [options] example-class [example args]
Options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn,
k8s://host:port, or local (Default: local[*]).
--deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or
on one of the worker machines inside the cluster ("cluster")
(Default: client).
--class CLASS_NAME Your application's main class (for Java / Scala apps).
亚洲eee图>rtf--name NAME A name of your application.
--jars JARS Comma-parated list of jars to include on the driver
and executor classpaths.
河北宣布开学时间--packages Comma-parated list of maven coordinates of jars to include
on the driver and executor classpaths. Will arch the local专业英语翻译器
maven repo, then maven central and any additional remote
repositories given by --repositories. The format for the
coordinates should be groupId:artifactId:version.
-
-exclude-packages Comma-parated list of groupId:artifactId, to exclude while
resolving the dependencies provided in --packages to avoid
dependency conflicts.
--repositories Comma-parated list of additional remote repositories to
arch for the maven coordinates given with --packages.
--py-files PY_FILES Comma-parated list of .zip, .egg, or .py files to place
on the PYTHONPATH for Python apps.
--files FILES Comma-parated list of files to be placed in the working
directory of each executor. File paths of the files
in executors can be accesd (fileName).
--conf PROP=VALUE Arbitrary Spark configuration property.
-
-properties-file FILE Path to a file from which to load extra properties. If not
specified, this will look for f.
西安哪里学托福最好--driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
--driver-java-options Extra Java options to pass to the driver.
--driver-library-path Extra library path entries to pass to the driver.
英语国际音标发音表--driver-class-path Extra class path entries to pass to the driver. Note that
jars added with --jars are automatically included in the
classpath.
--executor-memory MEM Memory per executor (e.g. 1000M, 2G) (Default: 1G).
--proxy-ur NAME Ur to impersonate when submitting the application.
This argument does not work with --principal / --keytab.
-
-help, -h Show this help message and exit.
--verbo, -v Print additional debug output.
--version, Print the version of current Spark.
Cluster deploy mode only:
mixer
--driver-cores NUM Number of cores ud by the driver, only in cluster mode
(Default: 1).
Spark standalone or Mesos with cluster deploy mode only:
--supervi If given, restarts the driver on failure.
--kill SUBMISSION_ID If given, kills the driver specified.
--status SUBMISSION_ID If given, requests the status of the driver specified.
参数名
参数说明--master
master 的地址，提交任务到哪⾥执⾏，例如 spark://host:port, yarn, local --deploy-mode
在本地 (client) 启动 driver 或在 cluster 上启动，默认是 client --class
应⽤程序的主类，仅针对 java 或 scala 应⽤--name
应⽤程序的名称--jars
⽤逗号分隔的本地 jar 包，设置后，这些 jar 将包含在 driver 和 executor 的 classpath 下--packages
包含在driver 和executor 的 classpath 中的 jar 的 maven 坐标--exclude-packages个人房屋租赁发票
为了避免冲突 ⽽指定不包含的 package --repositories 远程 repository
--conf PROP=VALUE 指定 spark 配置属性的值，
例如 -aJavaOptions="-XX:MaxPermSize=256m"--properties-file 加载的配置⽂件，默认为 f
deposit是什么意思--driver-memory Driver内存，默认 1G
--driver-java-options 传给 driver 的额外的 Java 选项
--driver-library-path 传给 driver 的额外的库路径
--driver-class-path 传给 driver 的额外的类路径
--driver-cores Driver 的核数，默认是1。在 yarn 或者 standalone 下使⽤
--executor-memory 每个 executor 的内存，默认是1G
--total-executor-cores 所有 executor 总共的核数。仅仅在 mesos 或者 standalone 下使⽤
--num-executors
启动的 executor 数量。默认为2。在 yarn 下使⽤ --status SUBMISSION_ID If given, requests the status of the driver specified. Spark standalone and Mesos only:
--total-executor-cores NUM Total cores for all executors.
Spark standalone and YARN only:
-
-executor-cores NUM Number of cores per executor. (Default: 1 in YARN mode, or all available cores on the worker in standalone mode)
YARN-only:
--queue QUEUE_NAME The YARN queue to submit to (Default: "default"). --num-executors NUM Number of executors to launch (Default: 2).
If dynamic allocation is enabled, the initial number of
executors will be at least NUM.
--archives ARCHIVES Comma parated list of archives to be extracted into the working directory of each executor.
--principal PRINCIPAL Principal to be ud to login to KDC, while running on cure HDFS.
--keytab KEYTAB The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to
the node running the Application Master via the Secure
Distributed Cache, for renewing the login tickets and the
delegation tokens periodically.
1.
2.
相关意义如下：
--num-executors 启动的 executor 数量。默认为2。在 yarn 下使⽤
--executor-core 每个 executor 的核数。在yarn或者standalone下使⽤
