What this book covers

Chapter 1, Getting Started with Spark SQL, gives you an overview of Spark SQL while getting you comfortable with the Spark environment through hands-on sessions.

Chapter 2, Using Spark SQL for Processing Structured and Semistructured Data, will help you use Spark to work with a relational database (MySQL), NoSQL database (MongoDB), semistructured data (JSON), and data storage formats commonly used in the Hadoop ecosystem (Avro and Parquet).

Chapter 3, Using Spark SQL for Data Exploration, demonstrates the use of Spark SQL to explore datasets, perform basic data quality checks, generate samples and pivot tables, and visualize data with Apache Zeppelin.

Chapter 4, Using Spark SQL for Data Munging, uses Spark SQL for performing some basic data munging/wrangling tasks. It also introduces you to a few techniques to handle missing data, bad data, duplicate records, and so on.

Chapter 5, Using Spark SQL in Streaming Applications, provides a few examples of using Spark SQL DataFrame/Dataset APIs to build streaming applications. Additionally, it also shows how to use Kafka in structured streaming applications.

Chapter 6, Using Spark SQL in Machine Learning Applications, focuses on using Spark SQL in machine learning applications. In this chapter, we will mainly explore the key concepts in feature engineering and implement machine learning pipelines.

Chapter 7, Using Spark SQL in Graph Applications, introduces you to GraphFrame applications. It provides examples of using Spark SQL DataFrame/Dataset APIs to build graph applications and apply the various graph algorithms into your graph applications.

Chapter 8, Using Spark SQL with SparkR, covers the SparkR architecture and SparkR DataFrames API. It provides code examples for using SparkR for Exploratory Data Analysis (EDA) and data munging tasks, data visualization, and machine learning.

Chapter 9, Developing Applications with Spark SQL, helps you build Spark applications using a mix of Spark modules. It presents examples of applications that combine Spark SQL with Spark Streaming, Spark Machine Learning, and so on.

Chapter 10, Using Spark SQL in Deep Learning Applications, introduces you to deep learning in Spark. It covers the basic concepts of a few popular deep learning models before you delve into working with BigDL and Spark.

Chapter 11, Tuning Spark SQL Components for Performance, presents you with the foundational concepts related to tuning a Spark application, including data serialization using encoders. It also covers the key aspects of the cost-based optimizer introduced in Spark 2.2 to optimize Spark SQL execution automatically.

Chapter 12, Spark SQL in Large-Scale Application Architectures, teaches you to identify the use cases where Spark SQL can be used in large-scale application architectures to implement typical functional and non-functional requirements.