Cloud Data Engineering

Cloud Data Engineering

This course is part of Building Cloud Computing Solutions at Scale Specialization

Instructor: Noah Gift

Skills you'll gain

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

16 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

Build your subject-matter expertise

This course is part of the Building Cloud Computing Solutions at Scale Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV

Share it on social media and in your performance review

There are 4 modules in this course

Welcome to the third course in the Building Cloud Computing Solutions at Scale Specialization! In this course, you will learn how to apply Data Engineering to real-world projects using the Cloud computing concepts introduced in the first two courses of this series. By the end of this course, you will be able to develop Data Engineering applications and use software development best practices to create data engineering applications. These will include continuous deployment, code quality tools, logging, instrumentation and monitoring. Finally, you will use Cloud-native technologies to tackle complex data engineering solutions.

This course is ideal for beginners as well as intermediate students interested in applying Cloud computing to data science, machine learning and data engineering. Students should have beginner level Linux and intermediate level Python skills. For your project in this course, you will build a serverless data engineering pipeline in a Cloud platform: Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP).

This week, you will learn about the methodologies involved in Data Engineering. You will also learn to evaluate best practices for dealing with the end of Moore’s Law, develop distributed systems that apply software engineering best practices and evaluate best practices for implementing solutions with Big Data. You will apply these practices to build a GPU programming project using Numba and the CUDA SDK.

What's included

25 videos22 readings5 assignments4 discussion prompts3 ungraded labs

25 videosTotal 144 minutes

Instructor Introduction1 minutePreview module
Course Introduction1 minute
Lab Onboarding1 minute
Course 3 Project Overview1 minute
Introduction to the End of Moore's Law2 minutes
The Problem with Concurrency in Python2 minutes
Exploring the End of Moore's Law3 minutes
Using CUDA and Numba6 minutes
What is an ASIC?2 minutes
Taking Advantage of Colab Pro5 minutes
Exploring Colab AI4 minutes
Introduction to Distributed Systems1 minute
Logging and Instrumentation Distributed Systems1 minute
CAP Theorem2 minutes
Amdahl's Law2 minutes
Elasticity3 minutes
Highly Available Nine Nine's1 minute
Debugging Python Code25 minutes
Exploring Google BigQuery12 minutes
What is Big Data?2 minutes
Data Lakes2 minutes
Big Data Processing3 minutes
Introduction to AWS Data Engineering Design Principles20 minutes
Processing Big Data with AWS25 minutes
Transform Data with Databricks Spark SQL5 minutes

22 readingsTotal 220 minutes

Getting Started and Course Gotchas10 minutes
Specialization Project Roadmap: Course 3 10 minutes
Course Structure and Discussion Etiquette10 minutes
Key Terms10 minutes
The End of Moore's Law10 minutes
Numba User Guide10 minutes
Custom Silicon10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
Python Debugger10 minutes
Python Logging10 minutes
Implementing elasticity, high availability, and monitoring in the AWS cloud10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
Introduction to Cloud Data Platforms10 minutes
What is a Lakehouse Platform10 minutes
Snowpark ML10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
Turbocharging Python with Command Line Tools 10 minutes
Creating high-performance code10 minutes
Lesson Reflection10 minutes

5 assignmentsTotal 150 minutes

Quiz30 minutes
Quiz-The End of Moore's Law30 minutes
Quiz-Build Distributed Systems30 minutes
Quiz-Understand Big Data and Data Processing Platforms30 minutes
Quiz-High-Performance Code30 minutes

4 discussion promptsTotal 40 minutes

Introductions10 minutes
Implications of the End of Moore's Law10 minutes
Monitoring and Logging In Distributed Systems10 minutes
Challenges to Big Data Platforms10 minutes

3 ungraded labsTotal 180 minutes

Threads and Processes60 minutes
Debug with Python PDB60 minutes
Understanding Map Reduce60 minutes

This week, you will learn what Data Engineering is and how to use software engineering best practices in Data Engineering. You will then apply this knowledge by building a command-line data processing tool.

What's included

16 videos10 readings4 assignments3 discussion prompts4 ungraded labs

16 videosTotal 216 minutes

Introduction to Data Engineering1 minutePreview module
Data Driven Organizations19 minutes
What is Data Engineering?0 minutes
Batch vs. Streaming vs. Events1 minute
Ingesting by Batch or Stream20 minutes
Building CLI Tools with Click33 minutes
Building Containerized Command-line Tools12 minutes
Rust and Python5 minutes
Python Calculator CLI3 minutes
Caesar Cipher CLI4 minutes
Advanced Testing with Amazon CodeGuru15 minutes
Advanced Testing with AWS CodeBuild29 minutes
Mapping Functions to CLI: Part 131 minutes
Mapping Functions to CLI: Part 227 minutes
AWS CodeWhisperer CLI3 minutes
AWS CodeWhisperer SDK4 minutes

10 readingsTotal 100 minutes

Key Terms10 minutes
Python Generators10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
Python and Rust10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
AWS CodeWhisperer for Data Engineering10 minutes
Lesson Reflection10 minutes
Extending Command-Line Data Processing Tool10 minutes

4 assignmentsTotal 120 minutes

Quiz30 minutes
Quiz-What is Data Engineering?30 minutes
Quiz-Using Command-line tools in Rust and Python30 minutes
Quiz-Advanced Code Testing and AI Enhanced Development Techniques30 minutes

3 discussion promptsTotal 30 minutes

The Power of Events10 minutes
Why is a Containerized CLI so Useful?10 minutes
Using Advanced Code Analysis Tools10 minutes

4 ungraded labsTotal 240 minutes

Streaming Data with Python Generators60 minutes
Build a Click Command-Line Tool60 minutes
Rust CLI Tool60 minutes
Building Command-Line Data Processing Tool60 minutes

This week, you will learn serverless data engineering techniques and data governance best practices. You will then apply this knowledge by building a serverless Data Engineering system.

What's included

23 videos7 readings3 assignments2 discussion prompts

23 videosTotal 213 minutes

Introduction to Serverless Data Engineering0 minutesPreview module
Automating Pipelines21 minutes
What is Serverless?0 minutes
Serverless Concepts: Service Model6 minutes
Serverless Concepts: Functions4 minutes
Serverless Concepts: Ecosystem7 minutes
AWS Lambda: Overview0 minutes
Introduction to AWS Cloud97 minutes
AWS Lambda: Event Handling5 minutes
AWS Lambda: Hello World12 minutes
AWS Lambda: Deploy Testing4 minutes
AWS Lambda: Wikipedia Example14 minutes
Build a Serverless Data Pipeline37 minutes
Serverless Cookbook with AWS32 minutes
Serverless Cookbook with GCP17 minutes
Introduction to Data Governance0 minutes
What is Data Governance?0 minutes
The Principle of Least Privilege1 minute
Cloud Security with IAM on AWS6 minutes
AWS Shared Security Model7 minutes
AWS IAM Service12 minutes
AWS Cloud Security Operations5 minutes
Encrypt at Rest and Transit3 minutes

7 readingsTotal 70 minutes

Key Terms10 minutes
Lambda Console Gotchas10 minutes
Using Amazon EFS for AWS Lambda10 minutes
Lesson Reflection10 minutes
Key Terms10 minutes
Lesson Reflection10 minutes
Building Computer Vision Label Trigger for S310 minutes

3 assignmentsTotal 90 minutes

Quiz30 minutes
Quiz-Build a Serverless Data Engineering System30 minutes
Quiz-Effective Data Governance30 minutes

2 discussion promptsTotal 20 minutes

Serverless Tools that use a CLI10 minutes
Startup Founder Wants To "Move Fast and Break Things"10 minutes

This week, you will learn about key Data Engineering tasks including ETL, Cloud Databases and Cloud Storage. You will then apply this knowledge by building a serverless AWS lambda function that labels an image using the AWS Rekognition API.