What this book covers

Chapter 1, Hadoop and HDInsight in a Heartbeat, covers the business value and the reason behind the big data hype. It provides a primer on Apache Hadoop, core concepts with HDFS, YARN, and the Hadoop 2.x ecosystem. Next, it discusses the Microsoft HDInsight platform, its key benefits, and deployment options.

Chapter 2, Enterprise Data Lake using HDInsight, covers the main points of the current Enterprise Data Warehouse and provides a path for an enterprise Data Lake based on the Hadoop platform. Additionally, it explains a use case built on the Azure HDInsight service.

Chapter 3, HDInsight Service on Azure, walks you through the steps for provisioning Azure HDInsight. Next, it explains how to explore, monitor, and delete the cluster using the Azure management portal. Next, it provides tools for developers to verify the cluster using a sample program and develop it using HDInsight Emulator.

Chapter 4, Administering Your HDInsight Cluster, covers steps to administer the HDInsight cluster using remote desktop connection to the head node of the cluster. It includes management of Azure Blob storage and introduces you to the Azure scripting environment known as Azure PowerShell.

Chapter 5, Ingest and Organize Data Lake, introduces you to an end-to-end Data Lake solution with a near real life size project and then focuses on various options to ingest data to a HDInsight cluster, including HDFS commands, Azure PowerShell, CloudExplorer, and Sqoop. Next, it provides details on how to organize data using Apache HCatalog. This chapter uses a real life size sample airline project to explain the various concepts.

Chapter 6, Transform Data in the Data Lake, provides you with various options to transform data, including MapReduce, Hive, and Pig. Additionally, it discusses Oozie and Spark, which are also commonly used for transformation. Throughout the chapter, you will be guided with a detailed code for the sample airline project.

Chapter 7, Analyze and Report from Data Lake, provides you with details on how to access and analyze data from the sample airline project using Excel Hive ODBC driver, Excel Power Query, Powerpivot, and PowerMap. Additionally, it discusses RHadoop, Giraph, and Mahout as alternatives to analyze data in the cluster.

Chapter 8, HDInsight 3.1 New Features, provides you with new features that are added to the evolving HDInsight platform with sample use cases for HBase, Tez, and Storm.

Chapter 9, Strategy for a Successful Data Lake Implementation, covers the key challenges for building a production Data Lake and provides guidance on the success path for a sustainable Data Lake. This chapter provides recommendations on architecture, organization, and links to online resources.