Spark Driver Web UI

This section will provide some important aspects of the Spark driver's UI. We will see the statistics of the jobs we executed using Spark shell on Spark UI.

As described in the Getting started with Apache Spark section, Spark driver's UI runs at http://localhost:4040/ (unless you make any changes to default settings).

When you start Spark shell, Spark driver's UI will look as follows:

SparkContext is an entry point to every Spark application. Every Spark job is launched with a SparkContext and can consist of only one SparkContext.

Spark shell, being a Spark application starts with SparkContext and every SparkContext launches its own web UI. The default port is 4040. Spark UI can be enabled/disabled or can be launched on a separate port using the following properties:

For example, Spark shell application with Spark UI running on 5050 port can be launched as:

spark-shell --confspark.ui.port=5050

If multiple Spark applications are launched in parallel on one system without providing any of the preceding conf parameters, then Spark UI for those applications will be launched on successive ports starting from 4040 (that is, 4040, 4041, and so on).

Spark UI consists of the following tabs: