Skip to content

Data Engineering BootCamp

Six weeks of hands-on data engineering.

Learn orchestration, warehousing, analytics, batch and stream processing through practical training built on real-world tools. From Docker and Terraform through Prefect, BigQuery, dbt, Spark and Kafka.

I followed the DataTalksClub Zoomcamp and kept working notes and code here. Week 1 was completed in 2024, weeks 2 to 6 in October 2026 with Claude as a pair programmer.

See the curriculumOriginal course
Week 1 · complete
Introduction & Prerequisites
  • Docker and docker-compose
  • Postgres with Docker
  • Infrastructure with Terraform

Course progress

Weeks
6
Complete
6
Upcoming
0
Tools
8+

Curriculum

Six weeks, scaffolded by stage.

  1. 01

    Introduction & Prerequisites

    Week 1

    • Course overview
    • Introduction to GCP
    • Docker and docker-compose
    • Postgres with Docker
    • Infrastructure with Terraform
    Complete
  2. 02

    Workflow Orchestration

    Week 2

    • Data Lake concepts
    • Workflow orchestration patterns
    • Introduction to Prefect
    • ETL with GCP & Prefect
    • Parametrizing workflows
    Complete
  3. 03

    Data Warehouse

    Week 3

    • Data Warehouse fundamentals
    • BigQuery overview
    • Partitioning and clustering
    • BigQuery best practices
    • BigQuery internals
    Complete
  4. 04

    Analytics Engineering

    Week 4

    • Analytics engineering basics
    • dbt (data build tool)
    • Models and testing
    • Cloud and local deployment
    • Data visualization
    Complete
  5. 05

    Batch Processing

    Week 5

    • Batch processing concepts
    • Apache Spark introduction
    • Spark DataFrames
    • Spark SQL
    • GroupBy and joins internals
    Complete
  6. 06

    Stream Processing

    Week 6

    • Introduction to Kafka
    • Schemas with Avro
    • Kafka Streams
    • Kafka Connect and KSQL
    • Real-time data pipelines
    Complete
Six-week scheduleBuild from Docker basics to streaming pipelines.
Learn-in-public notesWorking notes and code as each week is completed.

About this curriculum

What you will learn.

This Data Engineer BootCamp curriculum spans six weeks of hands-on data engineering training. Starting from foundational concepts like Docker and cloud infrastructure, it progresses through modern data stack tools including Prefect, BigQuery, dbt, Apache Spark, and Kafka.

Each week builds on core principles: data flows through pipelines, schemas matter, testing and documentation are essential, and every system has trade-offs. The course materials include week-by-week lessons, practical examples, and a capstone project to solidify understanding.

Course structure

Duration
6 weeks
Format
Hands-on lessons
Prerequisites
Basic programming
Level
Beginner to intermediate

Original course

See the original course materials and lectures at the official DataTalks.Club platform.

dezoomcamp.streamlit.app

Reflections

What I learned.

Week 1: Containers and infrastructure

Containers solve the “it works on my machine” problem by packaging an application and its environment together. Docker Compose lets you define multi-container setups in a single YAML file. The most practical lesson: isolation helps you focus. Each service runs in its own container with explicit dependencies.

Weeks 2 to 6: The modern data stack

Weeks 2 to 6 were completed in October 2026 with Claude as a pair programmer. Together we worked through orchestration with Prefect, data warehousing with DuckDB (local stand-in for BigQuery), analytics engineering with dbt, batch processing with PySpark, and streaming with Redpanda (Kafka API).

Key insight: every week reinforced that data engineering is more about understanding trade-offs than memorising tools. Partitioning trades storage for query speed. Orchestration trades immediate results for reliable, repeatable pipelines. Stream processing trades simplicity for real-time responsiveness.

Working with Claude showed me that pair programming with an AI is less about delegation and more about iteration. The code came from conversation, testing and refinement, not magic. The final work is honest: every homework answer computed from real queries, every run log pasted from actual terminal output.