In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipelines as code tools. You’ll also explore AWS and open source tools for monitoring your data systems and data quality.
Source Systems, Data Ingestion, and Pipelines
Source Systems, Data Ingestion, and Pipelines
This course is part of DeepLearning.AI Data Engineering Professional Certificate
Instructor: Joe Reis
Top Instructor
Sponsored by InternMart, Inc
5,478 already enrolled
Gain insight into a topic and learn the fundamentals.
4.7
(62 reviews)
Intermediate level
Recommended experience
34 hours to complete
3 weeks at 11 hours a week
Flexible schedule
Learn at your own pace
What you'll learn
Gather stakeholder needs and translate them into system requirements.
Implement a batch and a streaming ingestion process on AWS to ingest data from various source systems.
Integrate aspects of security, data management, DataOps and orchestration into the data systems you build.
Details to know
Shareable certificate
Add to your LinkedIn profile
Assessments
4 assignments
Taught in English
Recently updated!
September 2024
See how employees at top companies are mastering in-demand skills
Build your Cloud Computing expertise
This course is part of the DeepLearning.AI Data Engineering Professional Certificate
When you enroll in this course, you'll also be enrolled in this Professional Certificate.
- Learn new concepts from industry experts
- Gain a foundational understanding of a subject or tool
- Develop job-relevant skills with hands-on projects
- Earn a shareable career certificate from DeepLearning.AI
Earn a career certificate
Add this credential to your LinkedIn profile, resume, or CV
Share it on social media and in your performance review
There are 4 modules in this course
In lesson 1, you will explore source systems data engineers commonly interact with. Then in lesson 2, you will learn how to connect to various source systems and troubleshoot common connectivity issues.
What's included
21 videos10 readings1 assignment1 programming assignment3 ungraded labs
This week you will dive deep into the batch and streaming ingestion patterns. You will identify use cases and considerations for each, and then build a batch and a streaming ingestion pipeline. When looking at batch ingestion, you will compare and contrast the ETL and ELT paradigms. You will also explore various AWS services for batch and streaming ingestion.
What's included
11 videos6 readings1 assignment1 programming assignment1 ungraded lab
In the first lesson, you will explore DataOps automation practices, including applying CI/CD to both data and code, and using infrastructure as code tools like Terraform to automate the provisioning and management of your resources. Then in lesson 2, you will explore DataOps observability and monitoring practices, including using tools like Great Expectation to monitor data quality, and using Amazon CloudWatch to monitor your infrastructure.
What's included
17 videos5 readings1 assignment1 programming assignment2 ungraded labs
This week, you will learn all about orchestrating your data pipeline tasks. You'll identify the various orchestration tools, but will focus on Airflow -- one of the most popular and widely used tools in the field today. You'll explore the core components of Airflow, the Airflow UI, and how to create and manage DAGs using various Airflow features.
What's included
11 videos5 readings1 assignment1 programming assignment2 ungraded labs
Instructor
Instructor ratings
4.9 (20 ratings)
Why people choose Coursera for their career
Felipe M.
Learner since 2018
"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."
Jennifer J.
Learner since 2020
"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."
Larry W.
Learner since 2021
"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."
Chaitanya A.
"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."
Learner reviews
4.7
62 reviews
- 5 stars
88.88%
- 4 stars
3.17%
- 3 stars
0%
- 2 stars
4.76%
- 1 star
3.17%
Showing 3 of 62
TS
5
Reviewed on Nov 23, 2024
Really valuable, and I got an idea of data-related concepts and infrastructure management.
IH
5
Reviewed on Nov 15, 2024
Excellent course, with up to date technology, interesting labs and challenging quizzes. Highly recommended.
RA
5
Reviewed on Nov 19, 2024
All concepts related to these topics are explained clearly.
Recommended if you're interested in Information Technology
Google Cloud
Google Cloud
Amazon Web Services
Open new doors with Coursera Plus
Unlimited access to 10,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription
Advance your career with an online degree
Earn a degree from world-class universities - 100% online
Join over 3,400 global companies that choose Coursera for Business
Upskill your employees to excel in the digital economy