Master AWS Lambda Functions for Data Engineers using Python from Udemy

What's inside

Learning objectives

Setup required tools on windows to develop the code for etl data pipelines using python and aws services
Setup project or development environment to develop applications using python and aws services
Getting started with aws by creating account in aws and also configure aws cli as well as review data sets used for the project
Develop core logic to ingest data from source to aws s3 using python boto3
Getting started with aws lambda functions using python 3 run-time environment
Refactor the application, build zip file to deploy as aws lambda function
Create aws lambda function using zip file and validate

Troubleshoot issues related to aws lambda functions using aws cloudwatch
Build custom docker image for the application and push to aws ecr
Create aws lambda function using the custom docker image in aws ecr
Develop applications using aws lambda functions by adding python modules as layers
Show more
Show less

Setup required tools on windows to develop the code for etl data pipelines using python and aws services
Setup project or development environment to develop applications using python and aws services
Getting started with aws by creating account in aws and also configure aws cli as well as review data sets used for the project
Develop core logic to ingest data from source to aws s3 using python boto3
Getting started with aws lambda functions using python 3 run-time environment
Refactor the application, build zip file to deploy as aws lambda function
Create aws lambda function using zip file and validate
Troubleshoot issues related to aws lambda functions using aws cloudwatch
Build custom docker image for the application and push to aws ecr
Create aws lambda function using the custom docker image in aws ecr
Develop applications using aws lambda functions by adding python modules as layers
Show more
Show less

Syllabus

Introduction to Mastering AWS Lambda Functions for Data Engineers

Resources used for Mastering AWS Lambda Functions for Data Engineers

Getting Started on Windows with Required Tools

Introduction to Getting Started on Windows with Required Tools

Overview of Powershell on Windows 10 or Windows 11

Setup Ubuntu VM on Windows 10 or 11 using wsl

Setup Docker Desktop on Windows

Validate Docker on Windows using Command Line leveraging Power Shell

Review Docker Desktop Resource Configurations

Install Visual Studio Code on Windows

Install Remote Development Extension Kit for Visual Studio Code

Install Python 3.9 and Distutils on Windows using wsl Ubuntu

Review Tools Installed for Application Development using Python and AWS Service

Learn how to setup development environment to build data pipelines using AWS services such as boto3, lambda functions, ECR, etc.

Setup Project Folder using Visual Studio Code

Ensure Python 3.9 for the Project

Create Python Virtual Environment using Python 3

Install Required Dependencies for the Project using AWS Services

Ensure AWS CLI to interact with AWS Services using AWS CLI Commands

Recommendation to use Personal AWS Account for the course

Getting Started with AWS and Review Data Sets

Setup and Login into AWS Account

Setup AWS IAM User with Administrator Permissions

Configure and Validate AWS CLI

Configure AWS CLI with custom profile as default

Recap of Date Arithmetic using Python

Validate Python boto3 to interact with AWS Services

Setup and Validate Jupyter based Interactive Environment

Review GHActivity Data Details

Download GHActivity Data using requests

Review GHActivity Data using Pandas

Core Logic to Ingest Data from Web Service to AWS s3

Managing s3 using Python boto3

Overview of AWS Dynamodb

Create DynamoDB Table for Job Details

Create DynamoDB Table for Job Run Details

Get First Run Details to Copy GHActivity Data to AWS s3

Get Incremental Load Logic for next file

Understand AWS s3 concepts such as buckets and objects

Copying or Uploading Files to AWS s3 as objects using Python boto3

Writing Python Objects or Data as AWS s3 Objects using boto3

Save GHActivity Data to AWS s3

Convert Date Time to Integer Unix Epoch using Python

Save Job Run Details to DynamoDB Table

Validate Data Copied to AWS s3 and job run details

Run and Validate End to End Process

Getting Started with AWS Lambda Functions

Introduction to Getting Started with AWS Lambda Functions

Overview of AWS Lambda and Getting Started using Python 3

Passing Arguments to AWS Lambda and Processing using Python

Using Custom Handlers for AWS Lambda Functions using Python 3

Using AWS Services such as s3 in AWS Lambda Functions

Recap of handling permissions using AWS IAM Roles and User Groups

Develop AWS Lambda Function to list objects from AWS S3 Bucket

Passing Environment Variables to AWS Lambda Functions

Customizing Resources such as memory used for AWS Lambda Function

Setup Local Development Environment for AWS Lambda Functions

Develop logic for AWS Lambda Function using external packages

Build Zip file to deploy as AWS Lambda Function

Deploy Application with External Dependencies as AWS Lambda Function

Understand Problem Statement for Python Application for AWS

Setup Python Project for AWS Lambda using Visual Studio Code

Core Logic to upload files to AWS S3 using Python boto3

Develop Python Application to upload files to AWS s3 using Python boto3

Build Zip File for Python Application to deploy as AWS Lambda Function

Deploy Python Application as AWS Lambda Function using Zip File

Conclusion and request for rating and feedback

Build and Deploy AWS Lambda Function using Zip File

Introduction to Build and Deploy AWS Lambda Function using Zip File

Update Application Code with Core logic for Ingestion

Overview of Validating User Defined Functions using Python CLI

Validate Application using Core Logic to ingest data

Add Lambda Handler to ingest data to AWS s3

Upload Python Application Zip File to s3 and deploy as AWS Lambda Function

Set Custom Handler and required Environment Variables for AWS Lambda Function

Granting Permissions on AWS s3 and Dynamodb to AWS Lambda Function via Role

Change Memory and Timeout for AWS Lambda Function and Test

Recap and Overview of Monitoring Lambda Functions using Cloudwatch

Limitations of Deploying AWS Lambda Function using Zip file

Automate Build of AWS Lambda Function using Shell Scripts

Build and Deploy AWS Lambda Function using Custom Docker Image

Introduction to Build and Deploy AWS Lambda Function using Custom Docker Imag

Create Dockerfile for Custom Docker Image for AWS Lambda Function

Create Custom Docker Image for AWS Lambda Function using Python 3 Run-time

Validate Custom Docker Image by creating Docker Container

Run the application using Python CLI in the Docker Container

Run the Docker Container with the Credentials and Environment Variables

Validate AWS Lambda Function Locally using Docker and Curl

Create AWS ECR Repository for Custom Docker Image

Push Custom Docker Image for AWS Lambda Function to AWS ECR

Create AWS Lambda Function using Custom Docker Image in AWS ECR

Run and Validate AWS Lambda Function created using Custom Docker Image

Create Shell Script to Build and Push Docker Image to AWS ECR

Add Command to build script to reconfigure AWS Lambda Function to latest docker

Overview of AWS s3 Event Notifications with Lambda or s3 Triggers on Lambda

Introduction to AWS s3 Event Notifications with Lambda or s3 Triggers on Lamb

Switching to Different Profile for the demos on AWS s3 Event Notifications

Good to know

Know what's good

, what to watch for

, and possible dealbreakers

Develops skills and knowledge in Python boto3, AWS Lambda, Docker, EC2 in a real-world context

Covers AWS Lambda Functions by implementing an end-to-end pipeline and using AWS services like AWS EventsBridge and Step Functions

Introduces learners to Python 3.9 and teaches the use of distutils for package management

Provides personalized instruction with instructors like Durga Viswanatha Raju Gadiraju, Naga Bhuwaneshwar, and Kavitha Penmetsa

Requires learners to have basic familiarity with Python and AWS services, which may limit accessibility for beginners

Activities

Be better prepared before your course. Deepen your understanding during and after it. Supplement your coursework and achieve mastery of the topics covered in Master AWS Lambda Functions for Data Engineers using Python with these activities:

Explore AWS Lambda Tutorials

Show steps

Gain a foundational understanding of AWS Lambda and its capabilities to prepare for this course's focus on Lambda functions.

Browse courses on AWS Lambda

Show steps

Follow tutorials on AWS Lambda's architecture and features.
Build a simple Lambda function using AWS's online tutorials.

Review AWS Fundamentals

Show steps

Review the fundamentals of AWS to ensure you have a solid understanding before diving into Lambda Functions.

Show steps

Go through the AWS documentation on core services such as S3, EC2, and DynamoDB.
Complete the AWS Cloud Practitioner Essentials course on Coursera.

Review Python Data Science

Show steps

Solidify your understanding of Python Data Science concepts and tools to prepare for this course's material.

Browse courses on Python

Show steps

Review basic Python syntax and data types.
Go through a tutorial on Pandas for data manipulation.
Complete practice problems on data analysis and visualization using Python.

Nine other activities

Expand to see all activities and additional details

Show all 12 activities

Solve Python Coding Problems

Show steps

Enhance your problem-solving skills and refine your Python coding abilities to prepare for this course's programming assignments.

Browse courses on Problem Solving

Show steps

Solve Python coding problems on platforms like LeetCode or HackerRank.
Participate in coding challenges or hackathons.

Organize and Review Course Materials

Show steps

Review the course materials such as notes, assignments, and quizzes to solidify your understanding.

Show steps

Gather all course materials.
Organize the materials by topic or module.

Build a Simple AWS Lambda Function with Python

Show steps

Follow a guided tutorial to build a basic Lambda function in Python, which will reinforce the concepts covered in the course.

Show steps

Follow the tutorial on the AWS Lambda Developer Guide.
Deploy your Lambda function and test it with sample data.

Collaborate on a Lambda Function Project

Show steps

Enhance your understanding of Lambda functions and data pipelines by working collaboratively with peers on a project.

Browse courses on AWS Lambda

Show steps

Form a small group with other students.
Choose a project idea that involves building a data pipeline using Lambda functions.
Collaborate on designing, developing, and deploying your project.

Practice Troubleshooting Lambda Functions

Show steps

Engage in practice drills to troubleshoot common issues that may arise with Lambda Functions, enhancing your problem-solving skills.

Show steps

Review the AWS documentation on troubleshooting Lambda Functions.
Complete the AWS Lambda Troubleshooting Challenge on HackerRank.

Develop a Personal Project Using AWS

Show steps

Demonstrate your understanding of AWS services and data pipelines by building a practical project that utilizes Lambda functions.

Browse courses on AWS

Show steps

Identify a real-world problem that can be solved using AWS services.
Design and develop a solution using Lambda functions, S3, and other relevant AWS services.
Deploy and test your project on AWS.

Develop a Data Pipeline Using AWS Lambda Functions

Show steps

Create a data pipeline that utilizes AWS Lambda Functions to process and store data, putting your newfound knowledge into practice.

Show steps

Design the data pipeline architecture.
Implement the data pipeline using AWS Lambda Functions and other AWS services.
Test and deploy the data pipeline.

Participate in the AWS Lambda Hackathon

Show steps

Put your skills to the test by participating in the AWS Lambda Hackathon, offering an opportunity to collaborate and showcase your abilities.

Show steps

Form a team and register for the hackathon.
Develop an innovative solution using AWS Lambda Functions.
Submit your project and compete for prizes.

Mentor Junior Data Engineers

Show steps

Share your knowledge and skills by mentoring junior data engineers, reinforcing your understanding and fostering a supportive learning community.

Show steps

Volunteer as a mentor at a local university or coding bootcamp.
Provide guidance and support to junior data engineers on AWS Lambda Functions and data pipelines.

Career center

Learners who complete Master AWS Lambda Functions for Data Engineers using Python will develop knowledge and skills that may be useful to these careers:

Data Engineer

Data Engineers serve as the backbone of organizations that rely heavily on data to make informed decisions. Their primary responsibility is to design, build, and maintain data pipelines to extract insights and knowledge from large volumes of raw data. As a Data Engineer, you will be responsible for ensuring that data is reliable, accessible, and secure while using various tools like AWS Lambda Functions, Python, and other key AWS Services. This course will equip you with the essential knowledge and skills to succeed as a Data Engineer.

See salaries and explore the career path for Data Engineer

Cloud Engineer

As a Cloud Engineer, you will specialize in designing, implementing, and managing cloud computing infrastructure and services. This involves working with various cloud platforms such as AWS and leveraging services like Lambda Functions to build scalable and reliable applications. The course will provide you with a solid foundation in AWS Lambda Functions, allowing you to effectively develop and deploy applications in the cloud.

See salaries and explore the career path for Cloud Engineer

Full-Stack Developer

As a Full Stack Developer, you possess a comprehensive understanding of both front-end and back-end development. You are responsible for the entire software development lifecycle, from designing and implementing the user interface to managing the database and server-side logic. This course will provide you with the necessary skills to work with AWS Lambda Functions from both the front-end and back-end, enabling you to build robust and efficient full-stack applications.

See salaries and explore the career path for Full-Stack Developer

DevOps Engineer

DevOps Engineers combine software development (Dev) and information technology operations (Ops) practices to improve the speed, quality, and reliability of software delivery. This course will teach you how to use AWS Lambda Functions in your DevOps processes, helping you create a more efficient and collaborative environment for software development and deployment.

See salaries and explore the career path for DevOps Engineer

AWS Solutions Architect

AWS Solutions Architects design and implement scalable, highly available, and cost-effective solutions on the AWS platform. This course will provide you with a deep understanding of AWS Lambda Functions, enabling you to design and implement solutions that meet specific business requirements while adhering to best practices and security guidelines.

See salaries and explore the career path for AWS Solutions Architect

Data Analyst

Data Analysts are responsible for extracting insights and knowledge from data to inform decision-making. This course will teach you how to use AWS Lambda Functions to process and analyze data, enabling you to derive valuable insights that drive business strategies.

See salaries and explore the career path for Data Analyst

Database Administrator

Database Administrators are responsible for managing and maintaining databases. This course will provide you with a solid understanding of how to use AWS Lambda Functions to interact with databases, perform data operations, and ensure data integrity.

See salaries and explore the career path for Database Administrator

Software Architect

Software Architects design and develop the overall architecture of software systems. This course will give you a comprehensive understanding of how to use AWS Lambda Functions in your software designs, enabling you to create scalable, maintainable, and secure software systems.

See salaries and explore the career path for Software Architect

Cloud Architect

Cloud Architects design and develop cloud computing solutions. They work closely with cloud providers such as AWS to design and implement cloud-based infrastructures and applications. This course will provide you with a solid foundation in AWS Lambda Functions, enabling you to effectively design and deploy cloud-based solutions.

See salaries and explore the career path for Cloud Architect

Computer Systems Analyst

Computer Systems Analysts design, implement, and maintain computer systems. They work with stakeholders to gather requirements, develop solutions, and ensure that systems meet business needs. This course will provide you with a good understanding of how to use AWS Lambda Functions in your systems analysis and design work.

See salaries and explore the career path for Computer Systems Analyst

Business Analyst

Business Analysts work with stakeholders to understand their business needs and develop solutions that meet those needs. This course will teach you how to use AWS Lambda Functions to automate and streamline business processes, enabling you to deliver value to organizations.

See salaries and explore the career path for Business Analyst

Project Manager

Project Managers plan, execute, and close projects. They are responsible for ensuring that projects are delivered on time, within budget, and to the required quality standards. This course will provide you with an understanding of how to use AWS Lambda Functions in your project management processes, helping you manage projects more effectively.

See salaries and explore the career path for Project Manager

Quality Assurance Analyst

Quality Assurance Analysts test and evaluate software to ensure that it meets quality standards. This course will teach you how to use AWS Lambda Functions to automate and streamline your testing processes.

See salaries and explore the career path for Quality Assurance Analyst

Network Engineer

Network Engineers design, implement, and maintain computer networks. They work with a variety of technologies, including routers, switches, firewalls, and cables. This course may provide you with some insights into how to use AWS Lambda Functions in your network engineering work.

See salaries and explore the career path for Network Engineer

Information Security Analyst

Information Security Analysts plan and implement security measures to protect an organization's information systems. This course provides an overview of AWS Lambda Functions and how they can be used to enhance information security.

See salaries and explore the career path for Information Security Analyst