Big Data Analytics with Hadoop and Apache Spark

When data outgrows a single spreadsheet or database, you need tools built to share the work across many machines.

4 daysIntermediate10 modules

Enrol or enquire →Ask on WhatsApp

Overview

About this course

When data outgrows a single spreadsheet or database, you need tools built to share the work across many machines. This four-day course introduces the Hadoop ecosystem and Apache Spark, and shows how they store and process large volumes of data reliably.

Through guided labs in Python and SQL you will clean, join and analyse big data sets, build repeatable pipelines and tune jobs so they run efficiently. You leave ready to contribute to real data engineering and analytics projects.

What you'll learn

Explain the core ideas behind distributed storage and parallel processing of large data sets
Describe the Hadoop ecosystem, including HDFS and YARN, and where each component fits
Load, transform and analyse data with Spark DataFrames and Spark SQL
Write Spark applications in Python using PySpark
Build batch data pipelines that read from and write to common file formats
Tune Spark jobs through partitioning, caching and sensible resource settings
Introduce streaming data processing with Spark Structured Streaming

Curriculum

10 modules

Volume, velocity and variety
Why single machines are not enough
Batch versus streaming workloads
Typical big data architectures

HDFS storage concepts
Blocks, replication and fault tolerance
Resource management with YARN
The MapReduce model
Related tools such as Hive

Spark architecture: driver, executors and cluster managers
Resilient Distributed Datasets
Transformations and actions
Lazy evaluation and the execution plan

Creating DataFrames from files and tables
Selecting, filtering and aggregating data
Joins and window functions
Handling missing and messy data

Querying data with SQL
Temporary views and the catalogue
User-defined functions
Combining SQL and DataFrame code

CSV, JSON, Parquet and ORC
Schemas and schema evolution
Partitioned data layouts
Reading from and writing to HDFS and object storage

Designing extract, transform and load steps
Data quality checks
Scheduling and orchestration concepts
Packaging and submitting Spark applications

Partitioning and shuffles
Caching and persistence
Broadcast joins
Reading the Spark UI

Structured Streaming concepts
Sources, sinks and triggers
Windowed aggregations
Handling late data

Ingesting a large data set
Cleaning and transforming at scale
Producing analytical results
Presenting findings

Who it's for

For data analysts, developers and aspiring data engineers who know Python and SQL and want to work with data at scale.

Before you start

Working knowledge of Python
Basic SQL and familiarity with relational data

Let's plan your next step together.

Tell us the course or certification you have in mind and we'll send the outline, upcoming dates, fees and private-team options.

Talk to an advisor →WhatsApp