In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipelines as code tools. You’ll also explore AWS and open source tools for monitoring your data systems and data quality.
Source Systems, Data Ingestion, and Pipelines
Source Systems, Data Ingestion, and Pipelines
This course is part of DeepLearning.AI Data Engineering Professional Certificate
Instructor: Joe Reis
Top Instructor
4,844 already enrolled
Gain insight into a topic and learn the fundamentals.
4.7
(50 reviews)
Intermediate level
Recommended experience
34 hours to complete
3 weeks at 11 hours a week
Flexible schedule
Learn at your own pace
What you'll learn
Gather stakeholder needs and translate them into system requirements.
Implement a batch and a streaming ingestion process on AWS to ingest data from various source systems.
Integrate aspects of security, data management, DataOps and orchestration into the data systems you build.
Skills you'll gain
Details to know
Shareable certificate
Add to your LinkedIn profile
Recently updated!
September 2024
Assessments
4 assignments
Taught in English
See how employees at top companies are mastering in-demand skills
Build your Cloud Computing expertise
This course is part of the DeepLearning.AI Data Engineering Professional Certificate
When you enroll in this course, you'll also be enrolled in this Professional Certificate.
- Learn new concepts from industry experts
- Gain a foundational understanding of a subject or tool
- Develop job-relevant skills with hands-on projects
- Earn a shareable career certificate from DeepLearning.AI
Earn a career certificate
Add this credential to your LinkedIn profile, resume, or CV
Share it on social media and in your performance review
There are 4 modules in this course
In lesson 1, you will explore source systems data engineers commonly interact with. Then in lesson 2, you will learn how to connect to various source systems and troubleshoot common connectivity issues.
What's included
21 videos8 readings1 assignment3 app items1 ungraded lab
This week you will dive deep into the batch and streaming ingestion patterns. You will identify use cases and considerations for each, and then build a batch and a streaming ingestion pipeline. When looking at batch ingestion, you will compare and contrast the ETL and ELT paradigms. You will also explore various AWS services for batch and streaming ingestion.
What's included
11 videos7 readings1 assignment2 app items
In the first lesson, you will explore DataOps automation practices, including applying CI/CD to both data and code, and using infrastructure as code tools like Terraform to automate the provisioning and management of your resources. Then in lesson 2, you will explore DataOps observability and monitoring practices, including using tools like Great Expectation to monitor data quality, and using Amazon CloudWatch to monitor your infrastructure.
What's included
17 videos4 readings1 assignment3 app items
This week, you will learn all about orchestrating your data pipeline tasks. You'll identify the various orchestration tools, but will focus on Airflow -- one of the most popular and widely used tools in the field today. You'll explore the core components of Airflow, the Airflow UI, and how to create and manage DAGs using various Airflow features.
What's included
11 videos5 readings1 assignment3 app items
Instructor
Instructor ratings
4.9 (18 ratings)
Recommended if you're interested in Cloud Computing
Google Cloud
Amazon Web Services
Amazon Web Services
University of California San Diego
Why people choose Coursera for their career
Felipe M.
Learner since 2018
"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."
Jennifer J.
Learner since 2020
"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."
Larry W.
Learner since 2021
"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."
Chaitanya A.
"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."
Learner reviews
4.7
50 reviews
- 5 stars
86.53%
- 4 stars
3.84%
- 3 stars
0%
- 2 stars
5.76%
- 1 star
3.84%
Showing 3 of 50
IH
5
Reviewed on Nov 15, 2024
Excellent course, with up to date technology, interesting labs and challenging quizzes. Highly recommended.
TS
5
Reviewed on Nov 23, 2024
Really valuable, and I got an idea of data-related concepts and infrastructure management.
RA
5
Reviewed on Nov 19, 2024
All concepts related to these topics are explained clearly.
New to Cloud Computing? Start here.
Open new doors with Coursera Plus
Unlimited access to 10,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription
Advance your career with an online degree
Earn a degree from world-class universities - 100% online
Join over 3,400 global companies that choose Coursera for Business
Upskill your employees to excel in the digital economy
Frequently asked questions
Access to lectures and assignments depends on your type of enrollment. If you take a course in audit mode, you will be able to see most course materials for free. To access graded assignments and to earn a Certificate, you will need to purchase the Certificate experience, during or after your audit. If you don't see the audit option:
The course may not offer an audit option. You can try a Free Trial instead, or apply for Financial Aid.
The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile. If you only want to read and view the course content, you can audit the course for free.
If you subscribed, you get a 7-day free trial during which you can cancel at no penalty. After that, we don’t give refunds, but you can cancel your subscription at any time. See our full refund policy.