Designing and implementing data storage and data exploration layer
In this module, you will be introduced to the Azure tools available for designing and implementing data storage and data exploration layers. You will gain insight into the three important data engineering tools: Azure Synapse Analytics, Azure Databricks, and Azure Data Lake. You will gain insight into the key concepts and strategies of data partitioning for files, focusing on both Azure Synapse Analytics and Azure Data Lake Storage Gen2. Additionally, you will learn to browse and search metadata in Microsoft Purview Data Catalog for data exploration. The module also delves into the process of creating and executing a compute solution in Azure, which will enable you to make informed decisions regarding data partitioning, enhancing data cataloging capabilities, and efficiently managing data storage and processing resources in Azure.
Developing data processing
In this module, you will gain an understanding of data ingestion and transformation techniques. You will also learn about Transact SQL for data transformation, Azure Synapse Pipelines for ETL, Apache Spark for processing, and exploratory data analytics. The module also discusses data loading with PolyBase, the creation of data pipelines, and integration with Jupyter and Python notebooks. Additionally, it delves into configuring data snapshots with Delta Lake and building stream processing solutions using Spark Structured Streaming. Finally, you will gain a comprehensive understanding of data processing options and techniques.
Securing, monitoring, and optimizing data storage and data processing
In this module, you will learn about the importance of implementing robust data security measures in Azure, encompassing data encryption, access control through Azure RBAC, and secure data handling. Additionally, you can learn the steps to monitor data storage, optimize query performance through indexers and caching, and troubleshoot Spark job and pipeline run failures effectively. This knowledge is vital for maintaining efficient data storage and retrieval, contributing to improved data processing and analytics. You will also gain insight into two critical areas of Azure data management: Data security and performance optimization. You will also learn some insights on implementing robust data security measures to protect sensitive information effectively.