Stages

The Stages tab lists and provides, current state of all the stages of every job executed in the SparkContext. It can be accessed at http://localhost:4040/stages/. The Stages tab of Spark UI is rendered using the org.apache.spark.ui.jobs.StagesTab class.

Like the Jobs tab, the Stages tab also consists of three sections:

• Active Stages: This section is for the stages of an active that are currently running
• Completed Stages: This section is for the stages of active/failed/completed jobs that successfully completed
• Failed Stages: This section is for the stages of active/failed/completed jobs that were failed

After executing all the jobs mentioned in the Spark REPL also known as CLI section, the Stages tab will look as follows:

If you click on any of the stages you can see various details of the stage, that is, the DAG of tasks executed, Event Timeline, and so on. Also, this page provides metrics of various details of executors and tasks executed in the stage: