Different modes of execution

The Spark application can run in different modes, which are categorized by where and how we want to configure the master and what the executor's resource requirements are.

The master can run on the same local machine, along with executors; it can also run over a specific machine with the provided host and port. If we configure YARN as a Spark resource manager, the master can be managed by YARN:

# Run application locally on 8 cores
./bin/spark-submit \
 --class org.apache.spark.examples.SparkPi \
 --master local[8] \
  /path/to/examples.jar \
  100

# Run on a Spark standalone cluster in client deploy mode
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://host-ip:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \ # can be client for client mode
  --executor-memory 20G \
  --num-executors 50 \
  /path/to/examples.jar \
  1000

Source: https://spark.apache.org/docs/latest/submitting-applications.html.

Spark UI: Spark provides a web interface for application execution, which is accessible by default at port 4040: http://localhost:4040/jobs/: